DataTorrent/Malhar — pull request #1017 by ilooner (2014-08-23 21:09:50+00:00)
Title: Mlhr 1339

AbstractFSDirectoryInputOperator

Fixed timing issue with scanning that causes Unit Test to fail
Added files to failure list for retry if an IOException is thrown
Fixed bug where if two currentFiles got mapped to the same partition during repartition one of them would be dropped.
Fixed bug with directory scanner where each partition would accept a file if no regex was specified.
AbstractThroughputHashFSDirectoryInputOperator
Added unit test for making sure that repartitioning while a file is read is handled properly.
Added idempotent flag which allows the user to make the operator idempotent

Added this class. Does dynamic partitioning based on the number of files left in a directory.
Added FSInputOperator to Malhar and added unit tests

Tested as thoroughly as I could, but ran into SPOI-3053 and SPOI-3054.