Landsbokasafn/deduplicator — pull request #3 by kris-sigur (2014-10-09 12:56:59+00:00)
Title: Redone for Heritrix 3.3.0

Index now uses Lucene 4.7.
Can now index based on WARCs.
Index must now contain info on original URL and time of capture.
Heritrix modules use new revisit profile feature in Heritrix 3.3.0 to write [proper WARC revisit records](https://iipc.github.io/warc-specifications/specifications/warc-deduplication/recording-arbitrary-duplicates.html).
Refactored into sub-projects. This makes creating indexes possible without having Heritrix around.
Build now creates a ready-to-go Heritrix build with the DeDuplicator baked in.
This is almost a complete rewrite.