[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times

Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.
The does not convey much other than that it is a subclass of `PathFilter`. For instance I didn't realize off the bat that this also excluded files in `allFoundFiles` (when I was reading the `findNewFiles` function), I assumed from the constructor that it just filtered files older than that time.

Could this be named something like `ExcludeVisitedFilePathFilter`. Also, could it take `allFoundFiles` and `fileModTimes` as constructor arguments? This would localize the state used a bit more, right now it refers to these fields of the outer class directly and from the call site it's not really clear what logic it does.