Fix error trying to toggle colored tag with no tag

(This should be fixed so that the setting is always removed, though.)