BdbCookieStoreTest.java partly done
fix BdbCookieStore so that cookies get replaced correctly! and test it
restore useful comment
Merge pull request #30 from nlevitt/fix-bdb-cookie-store

Fix bdb cookie store (cookie is piling up due to lack of proper equality method in Cookie)
make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'?
Merge pull request #31 from nlevitt/proxy-port-configurable

make http proxy port configurable in cxml, avoiding this: org.springfram...
Switch to BasicHttpClientConnectionManager instead of
PoolingHttpClientConnectionManager, and make the connection manager a member
variable of HttpRequest. We had been seeing a strange bug where once in
every few thousand urls or so, the connection manager would get gc'd
while the http request was being sent, and RecordingOutputStream.flush()
would get called, as in the stack trace below.  The ToeThread was using
it at the same time, so if conditions were right, the recorded stream
would get mangled. This commit makes the connection manager a member of
FetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to
BasicHttpClientConnectionManager, since we're not doing any pooling.

  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133
  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166
  org.archive.io.RecordingOutputStream.flush() line: 469
  org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128
  org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314
  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520
  org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69
  org.apache.http.impl.conn.CPoolEntry.close() line: 95
  org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168
  java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]
Merge pull request #32 from nlevitt/hc43

Switch to BasicHttpClientConnectionManager instead of PoolingHttpClientConnectionManager
First pass at a processor to publish crawluris to AMQP channels
Use toString method instead
Pull out most umbra refs, only do http/https
Better Log messages
Merge pull request #33 from eldondev/master

First pass at a processor to publish crawluris to AMQP channels
fix NullPointerException in case of 401 with no auth challenge (includes unit test)
Merge pull request #34 from nlevitt/401-no-challenge

fix NullPointerException in case of 401 with no auth challenge (includes...
whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test)
Merge pull request #35 from nlevitt/urls-with-spaces

whoops! send escaped path+query on http request line; had been sending r...
Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation "unsatisfiableCharsetInHeader:...". Includes unit tests for good and bad charsets.

SEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in
java.nio.charset.UnsupportedCharsetException: binary
        at java.nio.charset.Charset.forName(Charset.java:543)
        at org.apache.http.entity.ContentType.<init>(ContentType.java:111)
        at org.apache.http.entity.ContentType.create(ContentType.java:210)
        at org.apache.http.entity.ContentType.get(ContentType.java:258)
        at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)
        at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)
        at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)
        at org.archive.modules.Processor.innerProcessResult(Processor.java:175)
        at org.archive.modules.Processor.process(Processor.java:142)
        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)
also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException
Merge pull request #36 from nlevitt/bad-charset

Avoid the exception below by handling bad charsets in FetchHTTP. Restore...