internetarchive/warctools — pull request #6 by nlevitt (2013-11-05 22:27:43+00:00)
Title: streaming record content when reading, and some other tweaks

Hey tef, I added support for streaming record content when reading arcs or warcs. It should be backward compatible with the existing api. The new @property ArchiveRecord.content() is how it maintains backward compatibility, and is also an example of how to use the streaming api:

@property
def content(self):
if self._content is None:
content_type = self.get_header(self.CONTENT_TYPE)
try:
content = self.content_file.read()
self._content = (content_type, content)
finally:
self.content_file = None

return self._content

WarcParser.parse (or ArcParser.parse) reads and parses each record up to the start of the content. Then it sets record.content_file to the record stream, and makes a note of the content-length so it knows where to stop reading the current record. When asked to retrieve the next record, RecordStream streams through any remaining bytes of the current record.

(If you start reading from record.content_file and then ask for record.content, the result will be messed up. It's the callers responsibility not to do that. There's no enforcement at the moment but maybe there could be.)

The changes have only been moderately tested at this point. All the command-line tools seem to work for me. I changed one of them, warcpayload.py, to use the streaming api. It's very likely that malformed records, especially ones with incorrent content-length header, will break in a different way than in the existing code. I haven't tested those kinds of cases yet.

There's a handful of other changes in here, including some string constants for revisit records, and adjusting the default mode when reading so that write perms are not required.