apache/incubator-streams — pull request #104 by steveblackmon (2014-10-14 21:34:30+00:00)
Title: Streams 168

There's quite a bit going on in this PR, so here's a summary:

To date, modules in streams-contrib have implemented interfaces from streams-core. Code re-use has been intra-contrib-module, or occasionally cross-contrib-module, with a few helper classes in streams-pojo playing a supporting role.

This PR introduces a new top-level sub-module, streams-components, where I've placed streams-http. streams-http contains a provider and a processor, both of which can be employed in a stream to collect json documents from remote REST endpoints. The config properties of streams-http were chosen to align with camel's http support module, anticipating compatibility will be desirable at some point in the future.

In addition, the PR contains new processors (in streams-provider-twitter and streams-processor-peoplepattern) which employ streams-http to collect data from specific data supplier endpoints, with a high degree of code reuse from streams-http. Essentially they just override a few methods but do not need to introduce their configuration or alter any aspect of the REST call other than parameters and headers. I've built code that uses the provider as well.

The JSON responses from the REST calls in these reference processors are then appended to specific extensions fields (activity.extensions.X, activity.actor.extensions.X, activity.object.extensions.X, etc.. for arbitrary X, sited on the activity or below (on any sub-node whose field extends ActivityObject.) Utility classes supporting extensions are under streams-pojo-extensions. Modules that were importing these classes from streams-pojo would need to update to import streams-pojo-extensions as well.

Philosophically, future modules containing providers, processors, and persisters which are both useful as standalone components in streams, and which are designed to simplify/harden channel/API/database-specific components that extend them, would be placed under streams-components. This gives us a dedicated sub-partitioned module for valuable utility features (outside of core), and will drastically curtail tendancies for one contrib module to explicitly require another.

Lastly, I've built streams-example modules utilizing each of these processors, which are pushed to w2ogroup/streams-examples under the STREAMS-168 branch.

I look forward to your feedback on this proposed reorg.