Only allocate a new destination row for an aggregations when the old
row has actually been used, this seems to save around 5-10% time on a
simple group by query