Added deduplication

Deduplication handled as explained in http://openplanetsfoundation.org/blogs/2014-03-24-arc-warc-migration-how-deal-de-duplicated-records