apache/incubator-streams — pull request #141 by steveblackmon (2014-11-21 21:19:39+00:00)
Title: Streams 222

I implemented this requirement with a familiar technology well-suited for API applications - creating streams-runtime-dropwizard. Also refactored DatasiftPushProvider to run using this provider, eliminating the need for any overrides or other hacks in the implementation.

The runtime contains three endpoints: /streams/webhooks/json, /streams/webhooks/json_new_line, and /streams/webhooks/json_meta which with pass any json documents posted to them to a StreamProvider named GenericWebhookResouce.

Additionally any other StreamProviders with @Path annotation will be added as jersey resources, and registered as perpetual streams which pass documents to queues named for their respective classes. They can pick whatever path they want to bind. Using the yaml file used at dropwizard start-up, any DW/Jersey parameter including port can be set, and additional configuration for the stream can be supplied. DW and typesafe don't play nice together, so this is the recommended way of supplying configuration to this runtime.

https://github.com/w2ogroup/streams-examples/tree/STREAMS-222 modules dropwizard-webhook-console and dropwizard-webhook-elasticsearch are working examples of how to use this runtime.

I think the ability to build and run streams which expose webhooks is a critical feature gap this PR addresses, in a relatively flexible manner. I look forward to your feedback.