apache/incubator-spark — pull request #301 by tdas (2013-12-27 02:32:00+00:00)
Title: Request For Comments: Refactoring streaming project to separate out non-essential  functionality from the core ones

This is my proposal of separating out non-essential functionality like twitter, kafka, etc from the core streaming functionality. I have done it only with twitter as an illustrative example. Please ignore formatting, extra line spaces, etc.

The proposal is as follows.

1. All the functionality that require non-essential libraries are to be put in `SPARK_HOME/external/` as follows.
`external/twitter/`
`external/kafka/`
`etc.`

2. Each of them will be published as `spark-twitter`, `spark-kafka`, etc.

3. To use the extra functionality in a program, one will have to import the relevant package. For twitter, one will have to `import org.apache.spark.streaming.twitter._ ` as shown in the Twitter-based examples below. [Scala package object](http://www.scala-lang.org/docu/files/packageobjects/packageobjects.html) is used to automatically introduce implicit conversion from StreamingContext to StreamingContextWithTwitter just by import the package (see below). For Java, we will still have to use `new JavaStreamingContextWithTwitter(javaStreamingContext)`.

Some questions to ponder on:
1. Choice of directory name and location?
2. Since the twitter functionality is tied to Spark Streaming, would the maven name of `spark-streaming-twitter` make more sense than `spark-twitter`?
3. Should these projects be part of the JAR generate by `assembly/assembly`?
@mateiz @rxin @pwendell