mesos/spark — pull request #275 by JoshRosen (2012-10-14 00:14:28+00:00)
Title: Refactoring of shuffling-related classes

These commits refactor interfaces and classes related to shuffling.

The main change is to remove map-side combining and aggregation from the shuffling interfaces. Map-side combining can be performed by calling `mapPartitions()` on an RDD, then passing that transformed RDD to `ShuffledRDD`. The reduce-side combining of the combiners can be performed in a second `mapPartitions()` call.

In this branch,

- `ShuffleMapTask` only performs shuffling, not map-side combining.
- All of the specialized `ShuffledRDD` subclasses were removed and replaced with a single `ShuffledRDD` class, which is only responsible for fetching a shuffled RDD.
- `ShuffleFetcher` returns an iterator over its fetched partitions, rather than accepting a function to apply to those partitions.
- The map-side combiner tests were removed, since map-side combining no longer uses a special mechanism.
- A `preservesPartitioning` option was added to `mapPartitions()` to allow users to specify that a transformation preserves the RDD's partitioning; this is used when implementing map-side combiners and reduce-side aggregation.
- RDD methods were modified to use these new interfaces. For some methods, like `sort()`, this had the effect of grouping the method's code into a single file rather than splitting it between the method and a special RDD subclass.

These changes haven't had a noticeable performance impact in the tests that I've run, but I'm still in the process of gathering benchmarks. I'd appreciate any feedback on these changes, particularly the changes to `cogroup()` and `CoGroupedRDD`.