direct: fix cleanup of the label in set_label