apache/spark — pull request #3247 by chenghao-intel (2014-11-13 14:29:59+00:00)
Title: [SPARK-4233] [SQL] WIP:Simplify the UDAF API (Interface)

Simplify the UDAF API is the first step of optimization for Aggregation (see https://issues.apache.org/jira/browse/SPARK-4366).

Currently UDAF cannot scale up when data volume grows, particularly for the `distinct` aggregation expressions. This PR doesn't aim for fixing the `distinct` performance, but facilitate

* More straightforward API for UDAF implementation
Developers will not write the distinct expression any more, like `DistinctAverage` is not necessary, since `Average` is provided, the framework will handle the `distinct` internally
* Schema-ed Aggregation Buffer
Aggregation Buffer is stored as `MutableRow` with schema, which means the UDAF developers will benefit from the Catalyst Expression framework in UDAF development. And shuffling the aggregation buffers cross the machine boundary is transparently from the UDAF developers, too, that provide us a chance to switch the aggregation algorithm (sort-based / hash-based)

It's a WIP PR, hopefully we could discuss the UDAF interface design in details, and a sample implementation for `max`, and `avg` is also provided.

Once we got generally agreement for the new UDAF design, I will continue to finish this PR:
* Integrated with Hive UDAF (Generic UDAF)
* Re-implemented the existed UDAF (e.g. `max`, `first`, `last` etc.)
* Code style issues.