internetarchive/heritrix3 — pull request #24 by nlevitt (2014-01-07 22:44:43+00:00)
Title: for url-agnostic dedup, follow "Proposal for Standardizing the Recording...

... of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0