DataTorrent/Malhar — pull request #1028 by ilooner (2014-08-26 16:56:43+00:00)
Title: Added throughput based partitioning for reading files from directory inp...

* Fixed retrieving operator contexts.
* Repartitioning is now triggered when an operator has no files to process, and the minimum repartition interval has elapsed since the last repartition.
* Throw runtimeexception when an IOException is encountered and the maxRetryCount <= 0.
* removed FSInputOperator
* removed redundant logic in AbstractThroughputHashFSDirectoryInputOperator for definePartitions.
AbstractFSDirectoryInputOperator

Fixed timing issue with scanning that causes Unit Test to fail
Added files to failure list for retry if an IOException is thrown
Fixed bug where if two currentFiles got mapped to the same partition during repartition one of them would be dropped.
Fixed bug with directory scanner where each partition would accept a file if no regex was specified.
AbstractThroughputHashFSDirectoryInputOperator
Added unit test for making sure that repartitioning while a file is read is handled properly.
Added idempotent flag which allows the user to make the operator idempotent

Added AbstractThroughputHashFSDirectoryInputOperator. Does dynamic partitioning based on the number of files left in a directory.

Tested as thoroughly as I could, but ran into SPOI-3053 and SPOI-3054.