DataTorrent/Malhar — pull request #1018 by ilooner (2014-08-24 00:55:39+00:00)
Title: Mlhr 1339

* Cleaned up code, removed trailing whitespaces
* Removed unnecessary directory scanner in FSInputOperator
* Removed emitTuplesHelper, added comments about idempotent behavior.

AbstractFSDirectoryInputOperator

Fixed timing issue with scanning that causes Unit Test to fail
Added files to failure list for retry if an IOException is thrown
Fixed bug where if two currentFiles got mapped to the same partition during repartition one of them would be dropped.
Fixed bug with directory scanner where each partition would accept a file if no regex was specified.
AbstractThroughputHashFSDirectoryInputOperator
Added unit test for making sure that repartitioning while a file is read is handled properly.
Added idempotent flag which allows the user to make the operator idempotent

Added AbstractThroughputHashFSDirectoryInputOperator. Does dynamic partitioning based on the number of files left in a directory.
Added FSInputOperator to Malhar and added unit tests

Tested as thoroughly as I could, but ran into SPOI-3053 and SPOI-3054.