[SPARK-4518][SPARK-4519][Streaming] Refactored file stream to prevent files from being processed multiple times

Because of a corner case, a file already selected for batch t can get considered again for batch t+2. This refactoring fixes it by remembering all the files selected in the last 1 minute, so that this corner case does not arise. Also uses spark context's hadoop configuration to access the file system API for listing directories.
In the best case I think it would be its own class and all dependencies would come through the constructor. If this is too difficult, however, you can just leave it as is. It's probably better to avoid major surgery for this bug fix.