Fix crash when removing labels with sub-labels