Only remove 'Ungrouped' group if empty.