don't compute sha256 block digest for each warc record written; new constant PAYLOAD_DIGEST='WARC-Payload-Digest'
support passing in warc record content as a file
utility methods for generating uuids suitable for use as warc record ids
some constant strings for revisit records
streaming support when reading warcs partially working
some refactoring to support streamed reading, including using gzip.GzipFile instead of zlib; fixes (unexplained) issues with last commit
finally figured out a way to hook in to gzip.GzipFile and reliably get the raw offset of the start of the current member, i think
use bytes_to_eor to cut off reading of record at end of record
safe read and readline for reading content, will not read past the end of the payload, leaving the trailing \r\n\r\n, assuming self.bytes_to_eor is set
use streaming api, including safe reading, and use httplib to parse http record, since it supports streaming
don't need write access to read warcs
streaming support for arcs too, and some tweaks
account for different record trailing \n for arcs, and the fact that arc header length field includes the trailing newline
restore seek() support to GzipRecordStream
handle invalid record terminators, stuff other than "\r\n\r\n" for warcs or "\n" for arcs, specifically any number of \r and \n
fix line missed in last commit
tests for reading compressed and uncompressed arcs and warcs with different record terminators
use nose to run all tests with setup.py
improve documentation, mostly on ArchiveRecord.content_file; add unit tests for warc writing
turns out we don't need nose, 'unittest2.collector' works