Change used_labels to a Python set