internetarchive/heritrix3 — pull request #36 by nlevitt (2014-01-29 00:22:46+00:00)
Title: Avoid the exception below by handling bad charsets in FetchHTTP. Restore...

... annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets.

SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
at java.nio.charset.Charset.forName(Charset.java:543)
at org.apache.http.entity.ContentType.(ContentType.java:111)
at org.apache.http.entity.ContentType.create(ContentType.java:210)
at org.apache.http.entity.ContentType.get(ContentType.java:258)
at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
at org.archive.modules.Processor.process(Processor.java:142)
at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)