apache/incubator-spark — pull request #313 by tdas (2013-12-31 09:03:11+00:00)
Title: Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc.

At a high level, these are the following changes.

1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules.

2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For the scala API, the developer will be able to call `StreamingContext.twitterStream(...)` using implicit conversions. This has been implemented using class `TwitterFunctions` that takes a `StreamingContext` as parameter (similar to `PairRDDFunctions` from `RDD`). For the java API, the developer will have to explicitly construct `JavaStreamingContextWithTwitter` from `JavaStreamingContext` (similar to `JavaPairRDD` from `JavaRDD`). Note that `JavaStreamingContextWithTwitter` has all the functionality of a `JavaStreamingContext`.

3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information).

4. In some files, imports have been rearrange to conform to the Spark coding guidelines.