Fix a bug where records could be double-processed

If the download cache dir was used as the local data dir, then any files found
both remotely and locally would be processed twice. We now dedupe remote files
before partitioning.