Package: warcdb
WarcDB: Web crawl data as SQLite databases.WarcDBis a anSQLite-based file format that makes web crawl data easier to share and query.It is based on the standardizedWeb ARChive format,
used by web archives, and defined inISO 28500:2017.Usagepipinstallwarcdb# Load the `archive.warcdb` file with data.warcdbimportarchive.warcdb./tests/google.warc./tests/frontpages.warc.gz"https://tselai.com/data/google.warc"warcdbenable-fts./archive.warcdbresponsepayload# Search for records that mention "stocks" in their response bodywarcdbsearch./archive.warcdbresponse"stocks"-c"WARC-Record-ID"As you can see you can use any mix of local/remote and raw/compressed archives.For example to get a part of theCommon Crawl January 2022 Crawl Archivein a streaming fashion:warcdbimportarchive.warcdb"https://data.commoncrawl.org/crawl-data/CC-MAIN-2022-05/segments/1642320306346.64/warc/CC-MAIN-20220128212503-20220129002503-00719.warc.gzYou can also import WARC files contained inWACZfiles, that are created by tools likeArchiveWeb.Page,Browsertrix-Crawler, andScoop.warcdbimportarchive.warcdbtests/scoop.waczHow It WorksIndividual.warcfiles are read and parsed and their data is inserted into an SQLite database with the relational schema seen below.SchemaIf there is a new major or minor version of warcdb you may need to migrate existing databases to use the new database schema (if there have been any changes). To do this you first upgrade warcdb, and then import into the database, which will make sure all migrations have been run. If you want to migrate the database explicitly you can:warcdbmigratearchive.warcdbIf there are no migrations to run themigratecommand will do nothing.Here's the relational schema of the.warcdbfile.MotivationFrom theWARCformal specification:The WARC (Web ARChive) file format offers a convention for concatenating multiple resource records (data objects),
each consisting of a set of simple text headers and an arbitrary data block into one long file.Many organizations such as Commoncrawl, WebRecorder, Archive.org and libraries around the world, use thewarcformat
to archive and store web data.The full datasets of these services range in the few pebibytes(PiB),
making them impractical to query using non-distributed systems.This project aims to makesubsetssuch data easier to access and query using SQL.Currently, this is implemented on top of SQLite and is a wrapper around the
excellentSQLite-Utilsutility."wrapper"means that all
existingsqlite-utilsCLI commandscan be called as expected likesqlite-utils<command>archive.warcdb`orwarcdb<command>example.warcdbExamplesPopulate withwgetwget--warc-filetselai"https://tselai.com"warcdbimportarchive.warcdbtselai.warc.gzGet all response headerssqlite3archive.warcdb<<SQLselect  json_extract(h.value, '$.header') as header,json_extract(h.value, '$.value') as valuefrom response,json_each(http_headers) hSQLGet Cookie Headers for requests and responsessqlite3archive.warcdb<<SQLselect json_extract(h.value, '$.header') as header, json_extract(h.value, '$.value') as valuefrom response,json_each(http_headers) hwhere json_extract(h.value, '$.header') like '%Cookie%'unionselect json_extract(h.value, '$.header') as header, json_extract(h.value, '$.value') as valuefrom request,json_each(http_headers) hwhere json_extract(h.value, '$.header') like '%Cookie%'SQLDevelopYou can use poetry to install dependencies and run the tests:$ git clone https://github.com/Florents-Tselai/WarcDB.git
$ cd WarcDB
$ poetry install
$ poetry run pytestThen when you are ready to publish to PyPI:$ poetry publish --buildResources on WARCThe stack: An introduction to the WARC file