[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times

Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.

@pwendell Please take look. I still have not run long-running integration tests, so I cannot say for sure whether this has indeed solved the issue. You could do a first pass on this in the meantime. 
Just curious:  why does this test need to use wallclock time instead of the same sort of `ManualClock` + `Thread.sleep()` pattern that occurs elsewhere in the code?  I'm in the middle of a hunt to remove Thread.sleep() calls, so this caught my eye.