Did you ever figure out why this was? Is the issue that the file wasn't yet `close()`d. Also, does `listStatus` only return files that are closed? If not, it seems like we have the more general issue that a user writing or copying a large file into HDFS might end up being in the middle of writing the file when we start to consume it. Does HDFS expose a way to make sure a file is closed?