Make sure ungrouped aggregates get deleted in the grouping manager.