Double-check the encoding returned by chardet before using it.
For redirect links in CDX files, use unicode replacement character
instead of ignoring unicode characters that we can't decode.
Handle unknown charsets by falling back to utf-8.
Handle unicode issue with redirect urls by forcing both the
base url and redirect location to unicode before joining, using
the supplied charset and replacement character if necessary.
Add formfeed_in_url.arc.gz to test_small_warcs.py.
We no longer compute the redirect column, and write '-' instead,
per discussion with Aaron B, Ilya, and Kenji.
Add test for transposed ARC header.
Add test case for an ARC file missing a filedesc: header
Add test for handling spaces in the url field of an ARC header.
Handle ARC files with urls that contain the \r character.
Handle revisit records without a WARC-Payload-Digest header.
Add test for v1 ARC file with a v2 header.
Handle bad urls with colons in them.
Add arc files needed for test_small_warcs.py
Handle bad urls with colons in them.
Fix broken quoting.
No longer fatal if an exception is thrown while making a surt.
Catch all exceptions, and just return the original url in the surt field.
Handle 18 digit dates in ARC files.
Handle arc files with a hex string instead of a date in the header.
Escape null characters in the original url, since they may cause
problems with downstream C code that processes CDX files.