Keep index association when sorting assigned labels