Fix performance regression in variance aggregation