Real fix for HER-2016 FetchHTTP tries every url twice without credentials before sending credentials: Remember WWW-Authenticate headers, aka auth challenges, with CrawlURI, and after successful auth, with the CrawlServer. It's necessary to do this to know what Authorization header to send without getting another 401 first.
* CoreAttributeConstants.java
    new constant A_HTTP_AUTH_CHALLENGES key for CrawlURI.getData() for http auth challenges
* CrawlURI.java
    add A_HTTP_AUTH_CHALLENGES to keys to remember across URI processing; convenience getter and setter for auth challenges
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
    add httpAuthChallenges argument to populate()
* HttpAuthenticationCredential.java
    populate() - process httpAuthChallenges to set up auth state of http client to send the right auth header
* CrawlServer.java
    cache httpAuthChallenges
* FetchHTTP.java
    getAuthScheme() - cache auth challenges with CrawlURI
    configureHttp() - do not set authentication preemptive - this is handled by the configuration done in HttpAuthenticationCredential.populate()
    promoteCredentials() - cache auth challenges with the CrawlServer
* Credential.java, HtmlFormCredential.java, HttpAuthenticationCredential.java
    bump serialVersionUID because of changed method signature on populate()
Make speculative link extractor ignore urls with two consecutive slashes (not preceded by a colon as in http://). We've seen the double slashes lead to crawler traps.
* UriUtils.java
    isLikelyFalsePositive() - consider likely false positive if it contains '//' (but not '://')
* UriUtilsTest.java
    testDoubleSlashFalsePositives()
Fix for HER-2019 fetchHTTP httpBindAddress setting doesn't take effect
* FetchHTTP.java
    use keyed properties for httpBindAddress setting, as expected in configureBindAddress(CrawlURI, HostConfiguration)
Fix for HER-2017 XML representation for /engine/job/<jobName>/beans uses root node of type "script" instead of "beans"
* BeanBrowseResource.java
	replaced "script" string in root node with "beans"
Merge pull request #16 from adam-miller/HER-2017

Fix for HER-2017 XML representation of beans uses root node of type "script"
Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    Added encoding to URL generation in makePresentableMapFor().
    Added check to prevent URL generation from overwriting the job name portion of the path in addPresentableNestedNames()
* AbstractPersistProcessor.java
    shouldStore() - only save persist record if http, https or ftp
Fix HER-2021 usedBaseForVia annotation repeated many times
* CrawlURI.java
    getAnnotations() - use LinkedHashSet data structure for new annotations object to prevent duplicates
* Link.java
    addRelativeToVia() - only log use of base for via the first time, before the annotation has been added
Merge pull request #17 from adam-miller/HER-2018

Fix for HER-2018: XML representation for /engine/job/<jobName>/beans returns incorrect url for named beans
* JobRelatedResource.java
    fix code formatting from 01be6d4b
Make unit tests pass
* ContentExtractorTestBase.java
    assertNoSideEffects() - check curi.getAnnotations.isEmpty() instead of equality with Collections.EMPTY_LIST
Unsupported auth scheme causing recoverableProblem with SEVERE level log line and stack trace
* FetchHTTP.java
    getAuthScheme() - handle IllegalStateException thrown by AuthPolicy.getAuthScheme()
Merge branch 'master' of github.com:internetarchive/heritrix3
Working on cleaning up WARC writer code.
* WARCRecordInfo.java
    new class to hold warc record info to write, so we don't have to pass around all these long lists of variables
* WARCConstants.java
    new enum WARCRecordType replaces bunch of string constants
* WARCWriter.java, WARCWriterProcessor.java, WARCWriterTest.java
    use WARCRecordInfo, WARCRecordType
Continuing to work on cleaning up WARC writer code.
* WARCRecordInfo.java
    get rid of constructor with arguments, add setUrl() and setType()
* WARCWriter.java
    get rid of writeRecord() taking a million arguments
* Arc2Warc.java, WARCWriter.java, WARCWriterTest.java, WARCWriterProcessor.java, WARCWriterProcessorTest.java
    construct WARCRecordInfo object and call WARCWriter.writeRecord(recordInfo) instead of million argument method
Continuing to work on cleaning up WARC writer code: simply creation of WARC record headers.
* ANVLRecord.java
    extend LinkedList instead of ArrayList (it was inane to guess the number of records everywhere, when it was never much more than ~10)
* ANVLRecords.java
    remove (seemed to be an incomplete duplicate of ANVLRecord)
* WARCRecordInfo.java
    addExtraHeader() - convenience method
* Arc2Warc.java, WARCWriter.java, WARCWriter.java, WARCWriterProcessor.java
    use WARCRecordInfo.addExtraHeader() where appropriate, and use no-argument ANVLRecord constructor instead of deprecated ANVLRecord(int)
* ANVLRecord.java
    remove unneeded(?) serialVersionUID
Beginning work on deduplication by content digest (irrespective of url)
* ContentDigestHistory.java, ContentDigestHistoryLoader.java, ContentDigestHistoryStorer.java
    initial checkin
Load, store content digest history
* WARCRecordInfo.java
    new fields warcFilename, warcFileOffset
* WARCWriter.java
    temporarily remember warc records written
* CrawlURI.java
    convenience method getContentDigestHistory()
* ContentDigestHistory.java
    start up properly; load into and store from curi.getContentDigestHistory()
* RecrawlAttributeConstants.java
    bunch of hash keys for curi.getContentDigestHistory()
* WARCWriterProcessor.java
    save info in curi.getContentDigestHistory()