[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times

Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.
This goes right into discussing how to work around file visibility issues but it doesn't discuss anything about the basic operation of the class first. It might be good to have something first that says:

```
At each batch interval, new files are detected and included in that batch. In this case "new" means files that became visible to readers during that time period. Some extra care is needed to deal with the fact that files may become visible after they are created.
```