jerryshao/apache-spark — pull request #2 by jerryshao (2014-11-03 08:58:14+00:00)
Title: SortShuffleReader code refactor and improvement

Hi @sryza , these days I looked through the Hadoop 2.4 MapReduce Merger related code and made some changes:

1. Extract and make the template for MergeThread.
2. Create two merge threads( in-memory merge thread and on-disk merge thread) for background merging. The merging strategy is the same as your PR, also the same as MR code.
3. For only 1 in-memory block to be merged, directly write to file without serde-compression code which did previously.
4. Fix `keyComparator` Integer overflow problem.
5. Add a `MixedShuffleReader` for both `HashShuffleReader` and `SortShuffleReader`. Only for shuffle with `keyOrdering` defined will go `SortShuffleReader` code path currently. (to be defined)

Would you mind taking a look at it and give me some comments. Thanks a lot.