fixed implementation and refactored the whole thing around HDFS Java API limitations