apache/spark — pull request #2963 by koertkuipers (2014-10-27 20:41:30+00:00)
Title: add foldLeftByKey to PairRDDFunctions for reduce algorithms that by key ...

...need to process values in a particular order

see:
https://issues.apache.org/jira/browse/SPARK-3655

this is the second of 2 competing pullreqs that try to address this issue. this one does so without making changes to core spark sorting routines. it is based on this suggestion by patrick wendell:
1. Map your RDD[(K, V)] to an RDD[((K, V), null)]
2. Write a custom partitioner that partitions based only on the K component of the key.
3. Call repartitionAndSortWithinPartition with your custom partitioner
4. Map the RDD back into RDD[(K, V)]

the downsides of this approach are that
1) a little more data goes through the shuffle (one extra object per row), i am not sure if this matters at all
2) the sorting by value is not generalized

the upside is that it's a much simpler and more self-contained change than the other pullreq.