archive-commons: Support for WaybackURLKeyMaker to create SURT and URL form canonicalized keys.
Support for -cdxURL operation in ResourceExtractor to create URL cdxs
(-cdx create surt form cdxs as before)
Merge github.com:internetarchive/archive-commons
Allow both 2-value and 1-value tuples.  If 2-value, assume they are already in (key,value) form.  If 1-value, split on first space into (key,value).
ia-tools: ZipNum Optimization:
Added ZipNulAllOutputFormat and ZipNumAllShardRecordWriter

Using these classes, the ZipNum output will contain part-NNNNN.ALL.summary files
which can then be cat-ed together to form the ALL.summary
This removes the need for running the summary-generator phase, as the shard
files are exactly the same format as the final summary.

Switched pig ZipNumStorage to use these output classes
(old java classes are unchanged, but should probably be deprecated eventually)
archive-commons: add exclusions to hadoop-core artifact to prevent pulling in (potentially old)
servlet and jsp versions automatically
configure <distributionManagement> section in root pom so that mvn deploy can work on build box, and make child projects point to parent so they inherit that config
Moving some stuff from heritrix, so it doesn't have to be a dependency of archive-commons, so that heritrix can have archive-commons as a dependency without creating a circular dependency
More on refactoring of stuff from heritrix
Fix bugs in refactoring of UURI->UsableURI
Bump ia-tools archive-commons dependency version
bump commons-io dependency
gzip.zipnum: Adding new streaming ZipNum block loader, which uses the Stream classes
to load zipnum blocks as a single stream per part file. Using SortedTextFile
for bin search of summary.
cdx: Added basic cdx loader support either via ZipNum block or
a plain cdx using SortedTextFile
GeneralURIStreamFactory: create Stream or SeekableLineReader based on general
uri. Returns appropriate class based on prefix, supporting http://, hdfs:// and (no prefix) local fs.
archive-commons: Adding CDXSearchResult for cdx queries which contains iterator and other state info (truncated or not, for now)
zipnum: added old-style location file support to new cluster implementation, optional loc file with multiple input paths may be specified
as constructor param
binsearch buffer sizes standardized to 8192
simplify ZipNumStreamingBlockIterator, using Peekable iterator and block to manager the buffered reader
archive-commons: Fully streaming blocks on the fly, instead of adding to array first!
SortedTextFile support for loading from beginning
CloseableIterator implements Closeable
CDX: Simplify ZipNum reading! Using a single ZipNumStreamingLoader with input stream to load entire part!
Cleaned up iterators!
commons: ZipNum streaming loader: use GZIPMemberInput stream and set higher buffer
fix badly encoded test string
cleanup ZipNumCluster, remove block PrefixStringIterator from iterator chain, getCdxLineIterator
returns an unbounded iterator, its up to consuming app to terminate iteration and close iterator
anyway