ambiata/ivory — pull request #71 by raronson (2014-07-01 05:36:00+00:00)
Title: Remove duplicate partitions from FactsetGlob

`HdfsStore.list` calls `Hdfs.globFilesRecursively` - https://github.com/ambiata/ivory/blob/master/ivory-alien-hdfs/src/main/scala/com/ambiata/ivory/alien/hdfs/HdfsStore.scala#L29. This means a list of paths to each file under each partition will be returned, causing duplicate partitions to be returned by this code.

This code is used in snapshot, but the only impact will be possible slowness if there are multiple part files in partitions.

Note we did hit this in the old snapshot, but that was using the old ugly `PartitionExpantion` where this is using the new `FactsetGlob`