Working on cleaning up WARC writer code.
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
Continuing to work on cleaning up WARC writer code.
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers.
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* ANVLRecord.java
    remove unneeded(?) serialVersionUID
Beginning work on deduplication by content digest (irrespective of url)
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
Load, store content digest history
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()
Write revisit records for URI-agnostic content digest duplicates.
* WARCConstants.java
    new header keys "WARC-Refers-To" and "X-WARC-Reference-Location"
* CrawlURI.java
    getContentDigestHistory() - return HashMap instead of Map so clone() can be called on the return value
* WARCWriterProcessor.java
    writeHttpRecords(), writeRevisitUriAgnosticDigest() - write revisit record for URI-agnostic content digest duplicate if enabled
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - fix construction of X-WARC-Reference-Location json value
* WARCConstants.java
    remove rfc6648-deprecated "X-" prefix to "WARC-Reference-Location", and document the field
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - add some comments
* WARCWriterProcessor.java
    writeRevisitUriAgnosticDigest() - write several separate warc header fields for each value instead of one json blob
* WARCConstants.java
    new header field names
Rename ContentDigestHistory -> BdbContentDigestHistory, since it's bdb-specific
New parent class AbstractContentDigestHistory for implementations to extend
* AbstractContentDigestHistory.java
    abstract methods load(), store(); non-abstract persistKeyFor(); javadocs
* BdbContentDigestHistory.java
    extend AbstractContentDigestHistory
* ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    use AbstractContentDigestHistory
Some tweaks on maintaining content digest history
* CrawlURI.java
    utility method hasContentDigestHistory()
* BdbContentDigestHistory.java
    load() - make sure to call curi.getContentDigestHistory() in all cases so the value is initialized and WARCWriterProcessor knows it should put the info in there
* WARCWriterProcessor.java
    updateMetadataAfterWrite() - update curi.getContentDigestHistory() only if curi.hasContentDigestHistory() for efficiency, like old uri-based fetch history; update the count after writing a revisit record
* WARCWriterProcessor.java
    writeHttpRecords() - avoid inadvertently creating content digest history map
* BdbContentDigestHistory.java
    store() - avoid clobbering good content digest history with empty one
* BdbContentDigestHistory.java
    class javadoc
Fix build
* WARCConstants.java, WARCWriter.java
    remove imports of WARCWriterProcessor from the modules/ area (was only used in javadoc)
Fix unit test
* WARCWriterTest.java
    writeRandomHTTPRecord() - fix method, had transposed some lines of code
Unit test for content digest history
* ContentDigestHistoryTest.java
    test basics
* BdbContentDigestHistory.java
    code formatting tweak