mateiz/spark — pull request #1 by jerryshao (2014-06-09 10:20:29+00:00)
Title: Initial proposal of moving map-side combiner and reduce-side combiner to ShuffleManager's reader and writer

Hi Matei,

This is the WIP proposal of moving map-side combiner and reduce-side combiner to ShuffleManager's reader and writer. I have some concerns about implementations, so I posted here to see if you can give me some advises.

1. Since `Aggregator` use `Iterator[_ <: Product[K, V]]` as input, but for shuffle writer, it only accept one-by-one record, if we want to move aggregator into shuffle writer, we should locally create an adapter to match these two interfaces, or can we simply modified shuffle writer's interface to accept iterator?
2. We may lose type information about RDD's elements when executing `runTask()` in `ShuffleMapTask`. We may only get type information Product2[Any, Any] for each record, I'm not sure is it Ok when running aggregator and key comparator in shuffle writer?
3. Here in my implementation, I add one more combiner type parameter `C` in `ShuffledRDD` for `Aggregator` and `ShuffleDependency`, I think this is the only few entry points for creating `ShuffleDependency`. So I modified `ShuffleRDD`, since In Spark there are so many modules directly use `ShuffledRDD`, not only in core, so change this will greatly impact the whole code path. I'm not sure is there any better solution available?

Sorry for my rudeness and immature thoughts. Any comments would be greatly appreciated, thanks a lot.