GitHub PushEvent at 2014-05-14 09:39:12
Repository: heritrix3
Actor: kris-sigur
Payload: {"shas":[["9a7e868022cb0b81ef3ff83fe1f80a7d2a2a6f0f","ed548340c7da3122212dc83b144b809582ac1662@archive.org","BdbCookieStoreTest.java partly done","Noah Levitt",false],["cba663a107bb219108c7a8da32da49662f5969e8","ed548340c7da3122212dc83b144b809582ac1662@archive.org","fix BdbCookieStore so that cookies get replaced correctly! and test it","Noah Levitt",false],["3ba476c39b23c90eb01d9f1af3261410445a8f87","ed548340c7da3122212dc83b144b809582ac1662@archive.org","restore useful comment","Noah Levitt",false],["fafcc1e28bdd1948f2bd6404c3db81fa6449b144","13ed0b3d6e4ca8f2a816e8ce2b41aa9f86f10512@gmail.com","Merge pull request #30 from nlevitt/fix-bdb-cookie-store\n\nFix bdb cookie store (cookie is piling up due to lack of proper equality method in Cookie)","Kenji Nagahashi",false],["c5c878822756ec7e14c5aec19c92fbc21f87f669","ed548340c7da3122212dc83b144b809582ac1662@archive.org","make http proxy port configurable in cxml, avoiding this: org.springframework.beans.NotWritablePropertyException: Invalid property 'httpProxyPort' of bean class [org.archive.modules.fetcher.FetchHTTP]: Bean property 'httpProxyPort' is not writable or has an invalid setter method. Did you mean 'httpProxyHost'?","Noah Levitt",false],["b1b2f2bc42a779a5ec1b98cbff78bd7e1e3aad3d","13ed0b3d6e4ca8f2a816e8ce2b41aa9f86f10512@gmail.com","Merge pull request #31 from nlevitt/proxy-port-configurable\n\nmake http proxy port configurable in cxml, avoiding this: org.springfram...","Kenji Nagahashi",false],["6b72ac9705ba73825871f4aa8003ce58a218329c","ed548340c7da3122212dc83b144b809582ac1662@archive.org","Switch to BasicHttpClientConnectionManager instead of\nPoolingHttpClientConnectionManager, and make the connection manager a member\nvariable of HttpRequest. We had been seeing a strange bug where once in\nevery few thousand urls or so, the connection manager would get gc'd\nwhile the http request was being sent, and RecordingOutputStream.flush()\nwould get called, as in the stack trace below.  The ToeThread was using\nit at the same time, so if conditions were right, the recorded stream\nwould get mangled. This commit makes the connection manager a member of\nFetchHTTPRequest so it doesn't get gc'd prematurely. Also switches to\nBasicHttpClientConnectionManager, since we're not doing any pooling.\n\n  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.dumpBuffer(boolean) line: 133\n  it.unimi.dsi.fastutil.io.FastBufferedOutputStream.flush() line: 166\n  org.archive.io.RecordingOutputStream.flush() line: 469\n  org.apache.http.impl.io.SessionOutputBufferImpl.flushStream() line: 128\n  org.apache.http.impl.io.SessionOutputBufferImpl.flush() line: 143\n  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection(org.apache.http.impl.BHttpConnectionBase).close() line: 314\n  org.archive.modules.fetcher.FetchHTTPRequest$RecordingHttpClientConnection.close() line: 520\n  org.apache.http.impl.conn.CPoolEntry.closeConnection() line: 69\n  org.apache.http.impl.conn.CPoolEntry.close() line: 95\n  org.apache.http.impl.conn.CPool(org.apache.http.pool.AbstractConnPool<T,C,E>).shutdown() line: 130\n  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.shutdown() line: 347\n  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.finalize() line: 168\n  java.lang.ref.Finalizer.invokeFinalizeMethod(java.lang.Object) line: not available [native method]","Noah Levitt",false],["7353f62d84efd3ac8dba11adfe391841ddf83137","13ed0b3d6e4ca8f2a816e8ce2b41aa9f86f10512@gmail.com","Merge pull request #32 from nlevitt/hc43\n\nSwitch to BasicHttpClientConnectionManager instead of PoolingHttpClientConnectionManager","Kenji Nagahashi",false],["d3e96b27700f146ef3c2599f911b3708451d5ab9","d6fb27dc330de5e957b2817091d91fd66296a676@eldondev.com","First pass at a processor to publish crawluris to AMQP channels","Eldon",false],["d837536dc67388b531475d4baf56d8c7da3d74e9","d6fb27dc330de5e957b2817091d91fd66296a676@eldondev.com","Use toString method instead","Eldon",false],["ab3aadf1999572ed5a69cc337e70dd1c4e9075e0","d6fb27dc330de5e957b2817091d91fd66296a676@eldondev.com","Pull out most umbra refs, only do http/https","Eldon",false],["b33d3faa756dfac94d0a5e1df4feeb358c94e72b","d6fb27dc330de5e957b2817091d91fd66296a676@eldondev.com","Better Log messages","Eldon",false],["b1a28358d7697b0a3e9fbe6ced0bbed96faa0a5f","ed548340c7da3122212dc83b144b809582ac1662@archive.org","Merge pull request #33 from eldondev/master\n\nFirst pass at a processor to publish crawluris to AMQP channels","Noah Levitt",false],["26a5dfebdc2569a1aab783428bf55bc396436116","ed548340c7da3122212dc83b144b809582ac1662@archive.org","fix NullPointerException in case of 401 with no auth challenge (includes unit test)","Noah Levitt",false],["21f04cdf79e2d319f781c353f1036df797cd73e2","e8e9f8f9e9922e2348cfccc3b14c6a9ef6c5c7bd@gmail.com","Merge pull request #34 from nlevitt/401-no-challenge\n\nfix NullPointerException in case of 401 with no auth challenge (includes...","Adam Miller",false],["b803ea0ec03b4e9d2b6a7bdaead8ccfc4ae760f9","ed548340c7da3122212dc83b144b809582ac1662@archive.org","whoops! send escaped path+query on http request line; had been sending raw spaces (includes unit test)","Noah Levitt",false],["693323258308c6dc1119573bb2c658b0d311e189","f0d7f9ab6791524662542a0331cf51f2915d6aab@yahoo.com","Merge pull request #35 from nlevitt/urls-with-spaces\n\nwhoops! send escaped path+query on http request line; had been sending r...","vonrosen",false],["161b3599797f3b87c5d58ad637979ebd019ea679","ed548340c7da3122212dc83b144b809582ac1662@archive.org","Avoid the exception below by handling bad charsets in FetchHTTP. Restore annotation \"unsatisfiableCharsetInHeader:...\". Includes unit tests for good and bad charsets.\n\nSEVERE thread-5973 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.nio.charset.UnsupportedCharsetException: binary occured when trying to process 'http://example.com/' at step ABOUT_TO_BEGIN_PROCESSOR in\njava.nio.charset.UnsupportedCharsetException: binary\n        at java.nio.charset.Charset.forName(Charset.java:543)\n        at org.apache.http.entity.ContentType.<init>(ContentType.java:111)\n        at org.apache.http.entity.ContentType.create(ContentType.java:210)\n        at org.apache.http.entity.ContentType.get(ContentType.java:258)\n        at org.apache.http.entity.ContentType.getOrDefault(ContentType.java:277)\n        at org.archive.modules.fetcher.FetchHTTP.setCharacterEncoding(FetchHTTP.java:580)\n        at org.archive.modules.fetcher.FetchHTTP.innerProcess(FetchHTTP.java:693)\n        at org.archive.modules.Processor.innerProcessResult(Processor.java:175)\n        at org.archive.modules.Processor.process(Processor.java:142)\n        at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)\n        at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)","Noah Levitt",false],["31aea014d9bb2ed406bc74507775b04b35e89d1f","ed548340c7da3122212dc83b144b809582ac1662@archive.org","also handle case where Charset.forName() throws IllegalCharsetNameException, as opposed to UnsupportedCharsetException","Noah Levitt",false],["10ec391da1144d1ae4063ff566b917fc03330048","64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@eldondev.com","Merge pull request #36 from nlevitt/bad-charset\n\nAvoid the exception below by handling bad charsets in FetchHTTP. Restore...","Eldon",false]],"size":145,"ref":"refs/heads/master","head":"3e213c82bc8a510f8355c11dcce373d7b9f907a4"}