directly adding labels to std::set<int> labelsToRemove