epfldata/squall — pull request #2 by avitorovic (2012-04-24 10:43:12+00:00)
Title: Custom acking mechanism

The list of the changes is as follows:
1) Added MultipleAggregateOperator, which collocates multiple aggregations within a single class .
2) SQLPlugin: Nephele superkey/key for isHashedBy method (collocating join and aggregation on the same level).
3) Load balancing achieved using a BalancedStreamGrouping (an implementation of CustomStreamGrouping). This class is instantiated with the full list of hashes which might occur - if the list is not specified, filed stream grouping is used (as before). Applicable when the last level has a small number of distinct hashes. In Hyracks we achieved 23% overall time speedup for 5G datasize, also applicable for TPCH 4, 5, 7 and 12.
4) There are two way we ensure all the tuples fully processed, so we can obtain the final result, the full execution time. If DIP_NUM_ACKERS is set to a non-zero value, each tuple is acked, which provide a user with possibility to measure latencies in UI. If the parameter is set to 0, each DataSourceComponents will send the ack message only at the very end. This incurs in average 2x speedup, but you loose the information about latencies from the UI. It also might lead to overflow in buffers which stores tuple from the time they arrive until the time they are processed. This is implemented by using a special message as the last one from a DataSourceComponent, which flushes all the previous communication. For all other tuples, BalancedStreamGrouping is used.
5) Documentation and (config files for plugin) are updated as well.