Package: warcprox
Warcprox is an HTTP proxy designed for web archiving applications. When used in
parallel withbrozzlerit
supports a comprehensive, modern, and distributed archival web capture system.
Warcprox stores its traffic to disk in theWeb ARChive (WARC) file format,
which may then be accessed with web archival replay software likeOpenWaybackandpywb. It captures encrypted HTTPS traffic by
using the “man-in-the-middle” technique (see theMan-in-the-middlesection
for more info).Warcprox was originally based onpymiproxyby Nadeem Douba.ContentsGetting startedMan-in-the-middleAPIDeduplicationStatisticsPluginsArchitectureLicenseGetting startedWarcprox runs on python 3.4+.To install latest release run:# apt-get install libffi-dev libssl-dev
pip install warcproxYou can also install the latest bleeding edge code:pip install git+https://github.com/internetarchive/warcprox.gitTo start warcprox run:warcproxTrywarcprox--helpfor documentation on command line options.Man-in-the-middleNormally, HTTP proxies can’t read encrypted HTTPS traffic. The browser uses the
HTTPCONNECTmethod to establish a tunnel through the proxy, and the proxy
merely routes raw bytes between the client and server. Since the bytes are
encrypted, the proxy can’t make sense of the information that it proxies. This
nonsensical encrypted data is not typically useful for web archiving purposes.In order to capture HTTPS traffic, warcprox acts as a “man-in-the-middle”
(MITM). When it receives aCONNECTdirective from a client, it generates a
public key certificate for the requested site, presents to the client, and
proceeds to establish an encrypted connection with the client. It then makes a
separate, normal HTTPS connection to the remote site. It decrypts, archives,
and re-encrypts traffic in both directions.Configuring a warcprox instance as a browser’s HTTP proxy will result in
security certificate warnings because none of the certificates will be signed
by trusted authorities. However, there is nothing malicious about warcprox
functions. To use warcprox effectively, the client needs to disable certificate
verification or add the CA certificate generated by warcprox as a trusted
authority. When using the latter, remember to undo this change when finished
using warcprox.APIThe warcprox API may be used to retrieve information from and interact with a
running warcprox instance, including:Retrieving status information via/statusURLWriting WARC records viaWARCPROX_WRITE_RECORDHTTP methodControlling warcprox settings via theWarcprox-MetaHTTP headerFor warcprox API documentation, see:api.rst.DeduplicationWarcprox avoids archiving redundant content by “deduplicating” it. The process
for deduplication works similarly to deduplication byHeritrixand other web archiving tools:While fetching URL, calculate payload content digest (typically SHA1
checksum value)Look up digest in deduplication database (warcprox currently supportssqliteby default,rethinkdbwith two different schemas, andtrough)If found, write warcrevisitrecord referencing the url and capture time
of the previous captureIf not found,Writeresponserecord with full payloadStore new entry in deduplication database (can be disabled, seeWarcprox-Meta HTTP request header)The deduplication database is partitioned into different “buckets”. URLs are
deduplicated only against other captures in the same bucket. If specified, thededup-bucketsfield of theWarcprox-Meta HTTP request headerdetermines the bucket(s). Otherwise,
the default bucket is used.Deduplication can be disabled entirely by starting warcprox with the argument--dedup-db-file=/dev/null.StatisticsWarcprox stores some crawl statistics to sqlite or rethinkdb. These are
consulted for enforcinglimitsandsoft-limits(seeWarcprox-Meta
fields), and can also be consulted by other
processes outside of warcprox, such as for crawl job reporting.Statistics are grouped by “bucket”. Every capture is counted as part of the__all__bucket. Other buckets can be specified in theWarcprox-Metarequest header. The fallback bucket in case none is specified is called__unspecified__.Within each bucket are three sub-buckets:new- tallies captures for which a complete record (usually aresponserecord) was written to a WARC filerevisit- tallies captures for which arevisitrecord was written to
a WARC filetotal- includes all URLs processed, even those not written to a WARC
file, and so may be greater than the sum of new and revisit recordsWithin each of these sub-buckets, warcprox generates two kinds of statistics:urls- simple count of URLswire_bytes- sum of bytes received over the wire from the remote server
for each URL, including HTTP headersFor historical reasons, the default sqlite store keeps statistics as JSON blobs:sqlite> select * from buckets_of_stats;
bucket           stats
---------------  ---------------------------------------------------------------------------------------------
__unspecified__  {"bucket":"__unspecified__","total":{"urls":37,"wire_bytes":1502781},"new":{"urls":15,"wire_bytes":1179906},"revisit":{"urls":22,"wire_bytes":322875}}
__all__          {"bucket":"__all__","total":{"urls":37,"wire_bytes":1502781},"new":{"urls":15,"wire_bytes":1179906},"revisit":{"urls":22,"wire_bytes":322875}}PluginsWarcprox supports a limited notion of plugins by way of the--plugincommand line argument. Plugin classes are loaded from the regular python module
search path. They are instantiated with one argument that contains the values
of all command line arguments,warcprox.Options. Legacy plugins with
constructors that take no arguments are also supported. Plugins should either
have a methodnotify(self, recorded_url, records)or should subclasswarcprox.BasePostfetchProcessor. More than one plugin can be configured by
specifying--pluginmultiples times.See a minimal examplehere.ArchitectureWarcprox is multithreaded. It has pool of http proxy threads (100 by default).
When handling a request, a proxy thread records data from the remote server to
an in-memory buffer that spills over to disk if necessary (after 512k by
default), while it streams the data to the proxy client. Once the HTTP
transaction is complete, it puts the recorded URL in a thread-safe queue, to be
picked up by the first processor in the postfetch chain.The postfetch chain normally includes processors for loading deduplication
information, writing records to the WARC, saving deduplication information, and
updating statistics. The exact set of processors in the chain depends on
command line arguments; for example, plugins specified with--pluginare
processors in the postfetch chain. Each postfetch processor has its own thread
or threads. Thus the processors are able to run in parallel, independent of one
another. This design also enables them to process URLs in batch. For example,
the statistics processor gathers statistics for up to 10 seconds or 500 URLs,
whichever comes first, then updates the statistics database with just a few
queries.LicenseWarcprox is a derivative work of pymiproxy, which is GPL. Thus warcprox is also
GPL.Copyright (C) 2012 Cygnos CorporationCopyright (C) 2013-2018 Internet ArchiveThis program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.