elasticsearch/elasticsearch — pull request #6128 by markharwood (2014-05-12 13:28:03+00:00)
Title: Deferred aggregation

New BucketCollector classes to aid the recording and subsequent playback of "collect" streams in aggs to reduce combinatorial explosions of the type outlined in https://github.com/elasticsearch/dev/issues/103.
A "DeferredAggregator” is provided here to aid testing of performance/RAM use - it can sit between parent and child aggs to buffer collects. The only parent agg that knows how to use this currently is the LongTermsAggregator which will call "DeferredAggregator.prepareBuckets()" to build only the top child buckets it creates.
The initial commit is not intended as final but shared here in order to start trials on different datasets and promote discussion. While "DeferredAggregator" is useful for testing it may not remain in future as we reconsider exactly how the cached "collect streams" in RecordingBucketCollector can be invoked and used by aggs.