openplanets/hawarp — pull request #7 by blekinge (2014-07-25 13:09:58+00:00)
Title: Numerous bugs in the code I supplied you

Hi

After testing, including on real data, I have now finished the real deduplicator. As it stands, whenever it encounters a file with -metadata- in the name, it migrates it twice. The first migration is a regular warc migration. The second is a deduplication migration. For examle, the file 99-metadata-1.arc will become 99-metadata-1.warc and 99-duplications-1.warc.