Package: warcio
BackgroundThis library provides a fast, standalone way to read and writeWARC
Formatcommonly used in
web archives. Supports Python 2.7+ and Python 3.4+ (usingsix, the only external dependency)warcio supports reading and writing of WARC files compliant with both theWARC 1.0andWARC 1.1ISO standards.Install with:pip install warcioThis library is a spin-off of the WARC reading and writing component of
thepywbhigh-fidelity replay
library, a key component ofWebrecorderThe library is designed for fast, low-level access to web archival
content, oriented around a stream of WARC records rather than files.Reading WARC RecordsA key feature of the library is to be able to iterate over a stream of
WARC records using theArchiveIterator.It includes the following features:Reading a WARC 1.0, WARC 1.1 or ARC streamOn the fly ARC to WARC record conversionDecompressing and de-chunking HTTP payload content stored in WARC/ARC files.For example, the following prints the the url for each WARCresponserecord:fromwarcio.archiveiteratorimportArchiveIteratorwithopen('path/to/file','rb')asstream:forrecordinArchiveIterator(stream):ifrecord.rec_type=='response':print(record.rec_headers.get_header('WARC-Target-URI'))The stream object could be a file on disk or a remote network stream.
TheArchiveIteratorreads the WARC content in a single pass. Therecordis represented by anArcWarcRecordobject which contains
the format (ARC or WARC), record type, the record headers, http headers
(if any), and raw stream for reading the payload.classArcWarcRecord(object):def__init__(self,*args):(self.format,self.rec_type,self.rec_headers,self.raw_stream,self.http_headers,self.content_type,self.length)=argsReading WARC ContentTheraw_streamcan be used to read the rest of the payload directly.
A specialArcWarcRecord.content_stream()function provides a stream that
automatically decompresses and de-chunks the HTTP payload, if it is
compressed and/or transfer-encoding chunked.ARC FilesThe library provides support for reading (but not writing ARC) files.
The ARC format is legacy but is important to support in a consistent
matter. TheArchiveIteratorcan equally iterate over ARC and WARC
files to emitArcWarcRecordobjects. The specialarc2warcoption
converts ARC records to WARCs on the fly, allowing for them to be
accessed using the same API.(SpecialWARCIteratorandARCIteratorsubclasses ofArchiveIteratorare also available to read only WARC or only ARC files).WARC and ARC StreamingFor example, here is a snippet for reading an ARC and a WARC using the
same API.The example streams a WARC and ARC file over HTTP usingrequests, printing thewarcinforecord (or ARC header) and any response records (or all ARC
records) that contain HTML:importrequestsfromwarcio.archiveiteratorimportArchiveIteratordefprint_records(url):resp=requests.get(url,stream=True)forrecordinArchiveIterator(resp.raw,arc2warc=True):ifrecord.rec_type=='warcinfo':print(record.raw_stream.read())elifrecord.rec_type=='response':ifrecord.http_headers.get_header('Content-Type')=='text/html':print(record.rec_headers.get_header('WARC-Target-URI'))print(record.content_stream().read())print('')# WARCprint_records('https://archive.org/download/ExampleArcAndWarcFiles/IAH-20080430204825-00000-blackbook.warc.gz')# ARC with arc2warcprint_records('https://archive.org/download/ExampleArcAndWarcFiles/IAH-20080430204825-00000-blackbook.arc.gz')Writing WARC RecordsStarting with 1.6, warcio introduces a way to capture HTTP/S traffic directly
to a WARC file, by monkey-patching Python’shttp.clientlibrary.This approach works well with the popularrequestslibrary often used to fetch
HTTP/S content. Note thatrequestsmust be imported after thecapture_httpmodule.Quick Start to Writing a WARCFetching the urlhttps://example.com/while capturing the response and request
into a gzip compressed WARC file namedexample.warc.gzcan be done with the following four lines:fromwarcio.capture_httpimportcapture_httpimportrequests# requests must be imported after capture_httpwithcapture_http('example.warc.gz'):requests.get('https://example.com/')The WARCexample.warc.gzwill contain two records (the response is written first, then the request).To write to a default in-memory buffer (BufferWARCWriter), don’t specify a filename, usingwith capture_http() as writer:.Additional requests in thecapture_httpcontext and will be appended to the WARC as expected.TheWARC-IP-Addressheader will also be added for each record if the IP address is available.The following example (similar to aunit test from the test suite) demonstrates the resulting records created withcapture_http:withcapture_http()aswriter:requests.get('http://example.com/')requests.get('https://google.com/')expected=[('http://example.com/','response',True),('http://example.com/','request',True),('https://google.com/','response',True),('https://google.com/','request',True),('https://www.google.com/','response',True),('https://www.google.com/','request',True)]actual=[(record.rec_headers['WARC-Target-URI'],record.rec_type,'WARC-IP-Address'inrecord.rec_headers)forrecordinArchiveIterator(writer.get_stream())]assertactual==expectedCustomizing WARC WritingThe library provides a simple and extensible interface for writing
standards-compliant WARC files.The library comes with a basicWARCWriterclass for writing to a
single WARC file andBufferWARCWriterfor writing to an in-memory
buffer. TheBaseWARCWritercan be extended to support more complex
operations.(There is no support for writing legacy ARC files)For more flexibility, such as to use a customWARCWriterclass,
the above example can be written as:fromwarcio.capture_httpimportcapture_httpfromwarcioimportWARCWriterimportrequests# requests *must* be imported after capture_httpwithopen('example.warc.gz','wb')asfh:warc_writer=WARCWriter(fh)withcapture_http(warc_writer):requests.get('https://example.com/')WARC/1.1 SupportBy default, warcio creates WARC 1.0 records for maximum compatibility with existing tools.
To create WARC/1.1 records, simply specify the warc version as follows:withcapture_http('example.warc.gz',warc_version='1.1'):...WARCWriter(fh,warc_version='1.1)...When using WARC 1.1, the main difference is that theWARC-Datetimestamp header
will be written with microsecond precision, while WARC 1.0 only supports second precision.WARC 1.0:WARC/1.0
...
WARC-Date: 2018-12-26T10:11:12ZWARC 1.1:WARC/1.1
...
WARC-Date: 2018-12-26T10:11:12.456789ZFiltering HTTP CaptureWhen capturing via HTTP, it is possible to provide a custom filter function,
which can be used to determine if a particular request and response records
should be written to the WARC file or skipped.The filter function is called with the request and response record
before they are written, and can be used to substitute a different record (for example, a revisit
instead of a response), or to skip writing altogether by returning nothing, as shown below:deffilter_records(request,response,request_recorder):# return None, None to indicate records should be skippedifresponse.http_headers.get_statuscode()!='200':returnNone,None# the response record can be replaced with a revisit recordelifcheck_for_dedup():response=create_revisit_record(...)returnrequest,responsewithcapture_http('example.warc.gz',filter_records):requests.get('https://example.com/')Please refer totest/test_capture_http.pyfor additional examples
of capturingrequeststraffic to WARC.Manual/Advanced WARC WritingBefore 1.6, this was the primary method for fetching a url and then
writing to a WARC. This process is a bit more verbose,
but provides for full control of WARC creation and avoid monkey-patching.The following example loadshttp://example.com/, creates a WARC
response record, and writes it, gzip compressed, toexample.warc.gzThe block and payload digests are computed automatically.fromwarcio.warcwriterimportWARCWriterfromwarcio.statusandheadersimportStatusAndHeadersimportrequestswithopen('example.warc.gz','wb')asoutput:writer=WARCWriter(output,gzip=True)resp=requests.get('http://example.com/',headers={'Accept-Encoding':'identity'},stream=True)# get raw headers from urllib3headers_list=resp.raw.headers.items()http_headers=StatusAndHeaders('200 OK',headers_list,protocol='HTTP/1.0')record=writer.create_warc_record('http://example.com/','response',payload=resp.raw,http_headers=http_headers)writer.write_record(record)The library also includes additional semantics for:CreatingwarcinfoandrevisitrecordsWritingresponseandrequestrecords togetherWriting custom WARC recordsReading a full WARC record from a streamPlease refer towarcwriter.pyandtest/test_writer.pyfor additional examples.WARCIO CLI: Indexing and RecompressionThe library currently ships with a few simple command line tools.IndexThewarcio indexcmd will print a simple index of the records in the
warc file as newline delimited JSON lines (NDJSON).WARC header fields to include in the index can be specified via the-fflag, and are included in the JSON block (in order, for
convenience).warcio index ./test/data/example-iana.org-chunked.warc -f warc-type,warc-target-uri,content-length
{"warc-type": "warcinfo", "content-length": "137"}
{"warc-type": "response", "warc-target-uri": "http://www.iana.org/", "content-length": "7566"}
{"warc-type": "request", "warc-target-uri": "http://www.iana.org/", "content-length": "76"}HTTP header fields can be included by prefixing them with the prefixhttp:. The special fieldoffsetrefers to the record offset within
the warc file.warcio index ./test/data/example-iana.org-chunked.warc -f offset,content-type,http:content-type,warc-target-uri
{"offset": "0", "content-type": "application/warc-fields"}
{"offset": "405", "content-type": "application/http;msgtype=response", "http:content-type": "text/html; charset=UTF-8", "warc-target-uri": "http://www.iana.org/"}
{"offset": "8379", "content-type": "application/http;msgtype=request", "warc-target-uri": "http://www.iana.org/"}(Note: this library does not produce CDX or CDXJ format indexes often
associated with web archives. To create these indexes, please see thecdxj-indexertool which extends warcio indexing to provide this functionality)CheckThewarcio checkcommand will check the payload and block digests
of WARC records, if possible. An exit value of 1 indicates a failure.warcio check-vwill print verbose output for each record in the
WARC file.RecompressTherecompresscommand allows for re-compressing or normalizing WARC
(or ARC) files to a record-compressed, gzipped WARC file.Each WARC record is compressed individually and concatenated. This is
the ‘canonical’ WARC storage format used byWebrecorderand other
web archiving institutions, and usually stored with a.warc.gzextension.It can be used to: - Compress an uncompressed WARC - Convert any ARC
file to a compressed WARC - Fix an improperly compressed WARC file (eg.
a WARC compressed entirely instead of by record)warcio recompress ./input.arc.gz ./output.warc.gzExtractTheextractcommand provides a way to extract either the WARC and HTTP headers and/or payload of a WARC record
to stdout. Given a WARC filename and an offset,extractwill print the (decompressed) record at that offset
in the file to stdoutSpecifying –payload or –headers will output only the payload or only the WARC + HTTP headers (if any), respectively.warcio extract [--payload | --headers] filename offsetLicensewarciois licensed under the Apache 2.0 License and is part of the
Webrecorder project.SeeNOTICEandLICENSEfor details.