Enhance pipe to support more features we can do in hadoop streaming

In Yahoo we want to migrate some of our hadoop based pipelines to Spark. Our pipelines are mostly implemented in hadoop streaming. We hope we can reuse most of the hadoop streaming code so that we can reduce the effort of the migration, and maybe only focus on rewriting the perf bottleneck. In our prototyping we saw that reusing existing code we can still get significant perf improvement. This should be also helpful for other users if they are currently using hadoop and want to change to Spark.

Though Spark has rdd.pipe() to do some similar work as hadoop streaming, we found some use cases as following still couldn't be supported.

1. We have some problem with reuse the reducer side hadoop streaming code with pipe(). The problem we had was, in our use case, after keyBy() or groupBy() the size of the data for some key can be very big. If we resolve this problem using map() then pipe(), we need to construct a big string to concat all the strings in the Seq[String] of the RDD of one key, this would consume a quite big chunk of memory in some cases, and seems like a not very scalable and performant solution. Especially Spark is a so memory intensive system, optimizing memory usage is very important for us.
2. Sometimes in hadoop streaming we need to use distributed cache(-cacheFile, -cacheArchive) to distribute the shared data to all tasktrackers. We need some similar mechanism in spark to do the same thing.

In this Pull Request, we enhanced the pipeRdd and rdd.pipe() to support above 2 features. For 1st, we allow user of pipe() to provide a function to specify how to transform the content of RDD and do the outputing. For 2nd, we allow user to provide a Seq[String] which will be piped to the shell process. User can use broadcast values to implement the similar feature as distributed cache in hadoop streaming.