apache/incubator-streams — pull request #131 by steveblackmon (2014-11-16 22:59:38+00:00)
Title: Streams 68,218

This is a big PR, because STREAMS-147/STREAMS-218 is tackling a big problem: the proliferation of classes and code snippets that perform simple or advanced conversion between typical document types String, ObjectNode, Activity, and pojos generated from schemas matching provider data schemas.

The goal is to have platform-level utilities and processors that perform these functions with a 'plug-in' architecture where individual modules can aid those processors in conversion. Thus the responsibility of the module with regard to conversion would be only to define schemas, classify documents that align with those schemas, and provide classes the generic processors can use to convert specific classified documents.

A major benefit of this approach are that the code footprint of each module can be smaller - because generic type converters are available and easy to use, components can set preconditions about what type of documents they will accept. Stream pipelines will be able to use a less diverse set of processors to accomplish the same work, so we can focus on making these utility components solid and all modules and streams will benefit.

Additionally use cases involving in-bound queues of many types of documents from multiple providers are now simple to implement, whereas without generic converters those use cases would have been hard and ugly.

Look forward to your comments and seeing this approach or some variation become the standard. I'm open to changes to the new interfaces or to the converter resolution/instantiation lifecycle. Also please note that these generic classes currently do not support a parallelism hint > 1, due to object serialization errors. Still looking into this and will add tests to ensure they can be parallelized.