looks like this duplicates this functionality:

https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/filecache/DistributedCacheFile.scala#L51