Factored remove_group and remove_context out of clear_empty_states