mesos/spark — pull request #525 by stephenh (2013-03-14 17:17:58+00:00)
Title: Add PairRDDFunctions.subtractByKey.

This is currently a Request For Comment, but after having RDD.subtract, I found myself also wanting subtractByKey, where I could do a set difference based on a key, but also have a value to keep around extra data for use later in the report.

So, this changed SubtractedRDD to always use tupled RDDs, so that PairRDDFunctions.subtractByKey can use it directly.

And then RDD.subtract does the usual thing of promoting an RDD[T] into a fake-tupled RDD[T, Null], running subtractByKeys, and then de-tupling back to the T type.

I believe this makes sense and is simpler, e.g. some of the "p2" partitioner weirdness went away. (Although not entirely if the user as a partitioned RDD[K, V] and then wants to do a regular subtract (not subtractByKeys) because we have to adapt the existing partitioner to handle the fake-tupled values.)