apache/incubator-spark — pull request #635 by coderxiang (2014-02-23 05:30:17+00:00)
Title: allCollect functions for RDD

Two methods (`allCollect`, `allCollectBroadcast`) are added to `RDD[T]`, which output a new `RDD[Array[T]]` instance with each partition containing all of the records of the original RDD stored in a single `Array[T]` instance (the same as RDD.collect). This functionality can be useful in machine learning tasks that require sharing updated parameters across partitions.

Method `allCollect` creates a new `AllCollectedRDD` while method `allCollectBroadcast` applies broadcasting. Both of them need collecting the data and therefore should deliver similar performance.