for url-agnostic dedup, follow "Proposal for Standardizing the Recording of Arbitrary Duplicates in WARC Files" https://docs.google.com/document/d/1QyQBA7Ykgxie75V8Jziz_O7hbhwf7PF6_u9O6w6zgp0
Merge pull request #24 from nlevitt/url-agnostic-standard

for url-agnostic dedup, follow "Proposal for Standardizing the Recording...