DataTorrent/Malhar — pull request #1013 by ilooner (2014-08-22 22:42:09+00:00)
Title: Mlhr 1339

AbstractFSDirectoryInputOperator

* Fixed timing issue with scanning that causes Unit Test to fail
* Added files to failure list for retry if an IOException is thrown
* Fixed bug where if two currentFiles got mapped to the same partition during repartition one of them would be dropped.
* Fixed bug with directory scanner where each partition would accept a file if no regex was specified.
* Scanning happens periodically to keep pending file lists up to date.

AbstractThroughputHashFSDirectoryInputOperator

* Added this class. Does dynamic partitioning based on the number of files left in a directory.

Added FSInputOperator to Malhar and added unit tests

Tested as thoroughly as I could, but ran into SPOI-3053 and SPOI-3054.