[HER-1976] H3: query-portion ignored when checking against robots.txt rules (using only path)
* RobotsPolicy.java: use UURI.getPathQuery -- like H1 does
[HER-1981] H3: overlay setting of alternate queue-precedence (via
BaseQueuePrecedencePolicy.basePrecedence) doesn't stick
(CrawlURI) don't null overlayNames in processingCleanup
(WorkQueueFrontier) move queue precedence recalc to only in handleQueue,
not all (post-wake) reenqueues
[HER-1976] query-portion ignored when checking against robots.txt rules
(using only path)
(RobotsPolicyTest) added for case that failed pre-fix
(RobotsTxtTest) added case similar to problem (but was already OK at
RobotsTxt level)
(RobotsPolicy, *RobotsPolicy) rename method to reflect new behavior
Fix for HER-1983 mangled dns records in w/arcs
* FetchDNS.java
    recordDNS() - use local variable for dummy read buffer, instead of member variable that was shared among threads, since RecordingInputStream.read() used the passed buffer as a work area
add processErrorOutlinks setting to CandidatesProcessor
default false is old behavior, skip candidates-handling of outlinks from response codes <200 and >=400
if true, these outlinks will be treated the same as others (get scope-tested and enqueued)
HER-1984 save script state - implement by adding a map to the application context for arbitrary use, and make sure the app context is available in all scripting environments
* PathSharingContext.java
    new member variable ConcurrentHashMap data and accessor getData()
* ScriptedProcessor.java, ScriptedDecideRule.java
    make appCtx available to scripts; also remove unused member sharedMap
* ActionDirectory.java
    formatting fix
Merge branch 'master' of github.com:internetarchive/heritrix3
* profile-crawler-beans.cxml
    commented-out default value for new CandidatesProcessor settings processErrorOutlinks
Sort script engines alphabetically so that they appear consistently.
Note that the name of the engine may be different than the name of the language
it parses. A good example is the Rhino script engine that parses javascript.
Fix for HER-1985 H3: SurtPrefixDecideRule forgets learned/seed-derived/seed-directive SURT prefixes in checkpoint-resume
* SurtPrefixedDecideRule.java
    make class implement Checkpointable, save surt prefixes to json on checkpoint and load them on recover
* profile-crawler-beans.cxml
    make main SurtPrefixedDecideRule a top-level bean so that it can be checkpointed
* Checkpoint.java, CheckpointService.java
    add some FINE level logging during checkpointing and recovery
Merge branch 'master' of https://github.com/travisfw/heritrix3
Fix problems handling crawl job paths with characters that require url-encoding, since we sometimes refer to them with file: urls
* PathSharingContext.java, CrawlJob.java
    use java.net.URI to convert paths to and from file: urls
Merge branch 'master' of github.com:internetarchive/heritrix3
* StatisticsTracker.java
    writeReportFile() - reduce log level from warning to info when reusing existing report (normal after crawl is finished)
BeanBrowseResource: fixed malformed HTML.
Second round of HTML cleanup:
- added missing <html> and <body>
- fixed non-working CSS link, added CSS link to all pages
- replaced layout by <br> and &nbsp; with structural tags with CSS
- removed superfluous output, notably in BeanBrowseResource
- <code>-ify variable names in ScriptResource
- added special handling of NaN in doubleToString()
* AbstractPersistProcessor.java, PersistProcessor.java, PersistStoreProcessor.java
    refactor some common persist processor code into new class AbstractPersistProcessor
restored deleted space after header "Crawl Log"
HER-1993: In JobResource, write error message instead of throwing RuntimeException when it failed to read 
job log or crawl log.
Merge branch 'master' of github.com:internetarchive/heritrix3 into html-cleanup