Ensure only categories with no tags can be removed