Change an unordered_map to an unordered_set for a small performance gain.