[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times

Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.

@pwendell Please take look. I still have not run long-running integration tests, so I cannot say for sure whether this has indeed solved the issue. You could do a first pass on this in the meantime. 
should this be called `MINIMUM_REMEMBER_DURATION`? There is also a variable called `durationToRemember` that might be different than this. I found it a bit confusing to have something called `durationToRemember` and another thing called `REMEMBER_DURATION` but they can have different values.