Reset Recorder state when uri processing is finished (had been seeing cases where a url's entry in crawl.log would sometimes have nonzero size value if the url was erroring out and RecordingOutputStream.open() was never called)
rename BdbUriUniqFilter.forgetSchemeHost() to forgetAllSchemeAuthorityMatching(), improve performance
further improve performance of BdbUriUniqFilter.forgetAllSchemeAuthorityMatching() by skipping conversion of key data to long and comparing bytes directly
HER-1895 support HTTP Refresh header
make special "host" "whois:" appear correctly in hosts report
some progress on forgetting old checkpoints when checkpointing anew
fix setForgetAllButLatest
BdbModule#doCheckpoint - support forgetting all but latest checkpoint
handle case where there is no whois server for a domain
handle case where there is no whois server for a domain
Merge branch 'master' into always-resumable
update public suffixes list
Merge branch 'master' into always-resumable
forgetting old checkpoints - roll up previous checkpointed CrawlerJournals (e.g. frontier.recover.gz) into new checkpoint
prevent redundant checkpoints when crawl is stopping
adjust PublicSuffixesTest for updated public suffixes list
Merge branch 'master' into always-resumable
don't log warning in normal case
Merge branch 'master' into always-resumable
CrawlController.reserveMemory - use byte[] instead of char[] for clarity of size