internetarchive/heritrix3 — pull request #90 by kngenie (2014-08-26 18:18:59+00:00)
Title: Move Wayback-dedup module to heritrix-contrib

this patch adds `WbmPersistLoadProcessor`, a module for fetching recrawl info from Wayback CDX server.

I rewrote it for `org.apache.httpcomponent`/`httpclient` 4.3.3 (required by heritrix-common) and junit 3. During rewrite, I found heritrix-contrib's hadoop dependency is pulling in older version of httpclient and JUnit 4. So this patch also adds dependency exclusions for `jets3t` and `junit` to `hbase-client` to address this problem. (Note `heritrix-common` also depends on `jets3t` 0.5.0, which does not depend on `httpclient`).