Okay I looked a bit here, it looks like the read/writer semantics in HDFS are that a file can appear to readers but look empty if they are still being written. This is a problem because the entire model here assumes that file being visible is the same as it being fully readable by Spark Streaming. 

That said, I think that moves in HDFS are atomic (please verify this). The better workflow here is that the write creates a file and writes all the data and flushes it - then it calls `move` and renames the file to `file_COMPLETED` and the path filter only accepts these completed files.

This would avoid having the `slackTime` work around which again, I don't think will work in the general case.

We should make it clear then that whatever spooling directory is used for HDFS files for Spark Streaming ingest needs to be populated using atomic `move` operations. This should be documented if it's an assumption we make when consuming files.

https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-3/data-flow