apache/incubator-streams — pull request #16 by smashew (2014-05-17 04:57:45+00:00)
Title: Overhaul of LocalStreamsBuilder

Commit Fixes:
https://issues.apache.org/jira/browse/STREAMS-81
https://issues.apache.org/jira/browse/STREAMS-80
https://issues.apache.org/jira/browse/STREAMS-82
https://issues.apache.org/jira/browse/STREAMS-83
https://issues.apache.org/jira/browse/STREAMS-84
https://issues.apache.org/jira/browse/STREAMS-85

Very large overhaul, pardon the large commit. The only major addition to this is to allow for the 'concept' of a known finite stream. There are many streams that are listened to and will be opened for a long time, however, there are other streams, that have a known termination point.

IE: Reading 1,000,000 records from an index in ElasticSearch to HDFS

There is a flag, now in StreamsResultSet, that when set to true, changes the 'queue' from being read as a Collection of items to an actual shared queue. This has many benefits, mainly, the queue is respected as a queue instead of being treated as a batch. In paradigms where items can be read in a stream, via a queue, this is an excellent choice.

There were many bugs that prevented the execution of this software from being run inside of a currently running JVM as it would cause memory and thread leaks. This has been updated and repaired in this version.

+++ tests +++
The tests that were in LocalStreamsBuilder have been restored and updated to be accurate (many of the tests were not accurate). This helps to ensure backward compatibility with the previous implementation. In addition, many (20) tests were added for edge cases and various amounts of behavior that could disrupt the execution of a stream. (RuntimeExceptions, excessive delays in processing, etc...) All tests have been written and are currently passing.

Thank you for your time!
Smashew