[Refactoring] Cleanup code around aggregates

This is a separated pull request from #3384 due to reduce reviewing difficulty.