apache/spark — pull request #3858 by industrial-sloth (2014-12-31 17:38:35+00:00)
Title: [SPARK-5037] dynamically loaded DStreams implementation and example

This PR adds a new reflection-based method of creating input DStreams to the scala StreamingContext, and wires it through to the python streaming API.

Trying to create DStream instances directly by reflection runs into trouble with unwanted stuff getting dragged into closures, so I worked around this by defining a new abstract serializable `ReflectedDStreamFactory` class. The idea is that one subclasses this with a concrete implementation that directly instantiates the desired InputDStream; then the StreamingContext uses reflection to dynamically load this new Factory implementation. This PR also has an example showing how this works with the existing ZeroMQ example code in both the scala and python streaming APIs.

Parameters are passed into the input DStream indirectly by first putting them into the factory constructor, then requiring the factory implementation to pass them on into the DStream instance. At the moment these parameters are limited to String type, which I think should cover the majority of use cases, but I'd think it should be possible to generalize this further.

Am throwing this out there for comment; suggestions and alternative approaches more than welcome.