mesos/spark — pull request #638 by lyogavin (2013-06-02 23:41:25+00:00)
Title: Enhance pipe to support more features we can do in hadoop streaming

In Yahoo we want to migrate some of our hadoop based pipelines to Spark. Our pipelines are mostly implemented in hadoop streaming. We hope we can reuse most of the hadoop streaming code so that we can reduce the effort of the migration, and maybe only focus on rewriting the perf bottleneck. In our prototyping we saw that reusing existing code we can still get significant perf improvement. This should be also helpful for other users if they are currently using hadoop and want to change to Spark.

Though Spark has rdd.pipe() to do some similar work as hadoop streaming, we found some use cases as following still couldn't be supported.

1. the reducer side hadoop streaming can't be implemented with rdd.pipe(), as it just simply cast the content in rdd to string.
2. In some cases we use distributed cache to distribute some of the shared data to all tasks.

In this Pull Request, we enhanced the pipeRdd and rdd.pipe() to support above 2 features. For 1st, we allow user of pipe() to provide a function to specify how to transform the content of RDD and do the outputing. For 2nd, we allow user to provide a Seq[String] which will be piped to the shell process. User can use broadcast values to implement the similar feature as distributed cache in hadoop streaming.