Package: archive-query-log
📜 The Archive Query LogMining Millions of Search Result Pages of Hundreds of Search Engines from 25 Years of Web Archives.Start now by runningyour custom analysis/experiment, scrapingyour own query log,
or just look atour example files.ContentsIntegrationsCrawlingDevelopmentContributeAbstractIntegrationsRunning Experiments on the AQLThe data in the Archive Query Log is highly sensitive (still, you
canre-crawl everything from the Wayback Machine). For that reason, we ensure that custom experiments or
analyses can not leak sensitive data (pleaseget in touchif you have questions) by
usingTIRAas a platform for custom analyses/experiments. In TIRA, you submit a Docker image that
implements your experiment. Your software is then executed in sandboxed mode (without internet connection) to ensure
that your software does not leak sensitive information. After your software execution finished, administrators will
review your submission and unblind it so that you can access the outputs.Please refer to ourdedicated TIRA tutorialas starting point for your experiments.CrawlingFor running the CLI and crawl a query log on your own machine,
please refer to theinstructions for single-machine deployments.
If instead you want to scale up and run the crawling pipelines on a cluster,
please refer to theinstructions for cluster deployments.Single-Machine (PyPi/Docker)To run the Archive Query Log CLI on your machine,
you can either use ourPyPi packageor theDocker image.
(If you absolutely need to, you can also install thePython CLIor the Docker image from source.)Installation (PyPi)First you need to installPython 3.10andpipx(this allows you to install the AQL CLI in a virtual environment).
Then, you can install the Archive Query Log CLI by running:pipxinstallarchive-query-logNow you can run the Archive Query Log CLI by running:aql--helpInstallation (Python from source)First installPython 3.10, and clone this repository.
From inside the repository directory, create a virtual environment and activate it:python3.10-mvenvvenv/sourcevenv/bin/activateInstall the Archive Query Log by running:pipinstall-e.Now you can run the Archive Query Log CLI by running:aql--helpInstallation (Docker)You only need to installDocker.Note:The commands below use the syntax of thePyPi installation.
To run the same commands with the Docker installation, replaceaqlwithdocker run -it -v "$(pwd)"/config.override.yml:/workspace/config.override.yml ghcr.io/webis-de/archive-query-log,
for example:dockerrun-it-v"$(pwd)"/config.override.yml:/workspace/config.override.ymlghcr.io/webis-de/archive-query-log--helpInstallation (Docker from source)First installDocker, and clone this repository.
From inside the repository directory, build the Docker image like this:dockerbuild-taql.Note:The commands below use the syntax of thePyPi installation.
To run the same commands with the Docker installation, replaceaqlwithdocker run -it -v "$(pwd)"/config.override.yml:/workspace/config.override.yml aql,
for example:dockerrun-it-v"$(pwd)"/config.override.yml:/workspace/config.override.ymlaql--helpConfigurationCrawling the Archive Query Log requires access to an Elasticsearch cluster and some S3 block storage.
To configure access to the Elasticsearch cluster and S3,
add aconfig.override.ymlfile in the current directory, with the following contents.
Replace the placeholders with your actual credentials:es:host:"<HOST>"port:9200username:"<USERNAME>"password:"<PASSWORD>"s3:endpoint_url:"<URL>"bucket_name:archive-query-logaccess_key:"<KEY>"secret_key:"<KEY>"Toy Example: Crawl ChatNoir SERPs from the Wayback MachineThe crawling pipeline of the Archive Query Log can best be understood by looking at a small toy example.
Here, we want to crawl and parse SERPs of theChatNoir search enginefrom theWayback Machine.Add an archive serviceAdd new web archive services (e.g., theWayback Machine) to the AQL by running:aqlarchivesaddWe maintain a list of compatible web archivesbelow.Compatible archivesThe web archives below are known to be compatible with the Archive Query Log crawler and can be used to mine SERPs.NameCDX API URLMemento API URLWayback Machinehttps://web.archive.org/cdx/search/cdxhttps://web.archive.org/web/Add a search providerAdd new search providers (e.g.,Google) to the AQL by running:aqlprovidersaddA search provider can be any website that offers some kind of search functionality.
Ideally, you should also look at common prefixes of the URLs of the search results pages (e.g.,/searchfor Google).
Narrowing down URL prefixes helps to not crawl too many captures that do not contain search results.Refer to theimport instructions belowto import providers from the AQL-22 YAML file format.Build source pairsOnce you have added at least onearchiveand onesearch provider,
we want to crawl archived captures of SERPs for each search provider and for each archive service,
that is, we compute the cross product of archives and the search providers' domains
and URL prefixes (roughly: archive×provider).
Start building source pairs (i.e., archive–provider pairs) by running:aqlsourcesbuildRunning the command again after adding more archives or providers automatically creates the missing source pairs.Fetch capturesFor eachsource pair, we now fetch captures from the archive service, that correspond
to the provider's domain and URL prefix given in the source pair.aqlcapturesfetchAgain, running the command again after adding more source pairs automatically fetches the missing captures.Parse URLsNot every capture necessarily points to a search engine result page (SERP).
But usually, SERPs contain the user query in the URL, so we can filter out non-SERP captures by parsing the URLs.aqlserpsparseurl-queryParsing the query from the capture URL will add SERPs to a new, more focused index that only contains SERPs.
From the SERPs, we can also parse the page number and offset of the SERP, if available.aqlserpsparseurl-page
aqlserpsparseurl-offsetAll the above commands can be run in parallel, and they can be run multiple times to update the SERP index.
Already parsed SERPs will be skipped.Download WARCsUp to this point, we have only fetched the metadata of the captures, most prominently the URL.
However, the snippets of the SERPs are not contained in the metadata, but only on the web page.
So we need to download the actual web pages from the archive service.aqlserpsdownloadwarcThis command will download the contents of each SERP to a WARC file that is stored in the configured S3 bucket.
A pointer to the WARC file is stored in the SERP index so that we can quickly access a specific SERPs contents later.Parsing WARCsImport from AQL-22We support automatically importing providers and parsers from the AQL-22 YAML-file format
(seedata/selected-services.yaml).
To import the services and parsers from the AQL-22 YAML file, run:aqlprovidersimport
aqlparsersurl-queryimport
aqlparsersurl-pageimport
aqlparsersurl-offsetimportWe also support importing a previous crawl of captures from the AQL-22 file system backend:aqlcapturesimportaql-22Cluster (Helm/Kubernetes)Running the Archive Query Log on a cluster is recommended for large-scale crawls.
We provide a Helm chart that automatically starts crawling and parsing jobs for you
and stores the results in an Elasticsearch cluster.InstallationJust installHelmand configurekubectlfor your cluster.ConfigurationCrawling the Archive Query Log requires access to an Elasticsearch cluster and some S3 block storage.
Configure the Elasticsearch and S3 credentials in avalues.override.yamlfile like this:elasticsearch:host:"<HOST>"port:9200username:"<USERNAME>"password:"<PASSWORD>"s3:endpoint_url:"<URL>"bucket_name:archive-query-logaccess_key:"<KEY>"secret_key:"<KEY>"DeploymentLet's deploy the Helm chart on the cluster (we're testing first with--dry-runto see if everything works):helmupgrade--install--valueshelm/archive-query-log/values.override.yaml--dry-runarchive-query-loghelm/archive-query-logIf everything worked and the output looks good, you can remove the--dry-runflag to actually deploy the chart.UninstallIf you no longer need the chart, you can uninstall it:helmuninstallarchive-query-logCitationIf you use the Archive Query Log dataset or the crawling code in your research, please cite the following paper
describing the AQL and its use-cases:Jan Heinrich Reimer, Sebastian Schmidt, Maik Fröbe, Lukas Gienapp, Harrisen Scells, Benno Stein, Matthias Hagen, and
Martin Potthast.The Archive Query Log: Mining Millions of Search Result Pages of Hundreds of Search Engines
from 25 Years of Web Archives.In Hsin-Hsi Chen et al.,
editors,46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023),
pages 2848–2860, July 2023. ACM.You can use the following BibTeX entry for citation:@InProceedings{reimer:2023,author={{Jan Heinrich} Reimer and Sebastian Schmidt and Maik Fr{\"o}be and Lukas Gienapp and Harrisen Scells and Benno Stein and Matthias Hagen and Martin Potthast},booktitle={46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023)},doi={10.1145/3539618.3591890},editor={Hsin{-}Hsi Chen and Wei{-}Jou (Edward) Duh and Hen{-}Hsen Huang and Makoto P. Kato and Josiane Mothe and Barbara Poblete},ids={potthast:2023u},isbn={9781450394086},month=jul,numpages=13,pages={2848--2860},publisher={ACM},site={Taipei, Taiwan},title={{The Archive Query Log: Mining Millions of Search Result Pages of Hundreds of Search Engines from 25 Years of Web Archives}},url={https://dl.acm.org/doi/10.1145/3539618.3591890},year=2023}DevelopmentRefer to the localPython installationinstructions to set up the development
environment and install the dependencies.Then, also install the test dependencies:pipinstall-e.[tests]After having implemented a new feature, you should the check code format, inspect common LINT errors,
and run all unit tests with the following commands:flake8archive_query_log# Code formatpylintarchive_query_log# LINT errorsmypyarchive_query_log# Static typingbandit-cpyproject.toml-rarchive_query_log# Securitypytestarchive_query_log# Unit testsAdd new tests for parsersAt the moment, our workflow for adding new tests for parsers goes like this:Select the number of tests to run per service and the number of services.Auto-generate unit tests and download WARCs
withgenerate_tests.pyRun the tests.Failing tests will open a diff editor with the approval and a web browser tab with the Wayback URL.Use the web browser dev tools to find the query input field and search result CSS paths.Close diffs and tabs and re-run tests.ContributeIf you've found an important search provider to be missing from this query log,
please suggest it by creating anissue.
We also very gratefully acceptpull requestsfor adding search providers or new parser configurations!If you're unsure about anything, post anissue,
or contact us:heinrich.reimer@uni-jena.des.schmidt@uni-leipzig.demaik.froebe@uni-jena.delukas.gienapp@uni-leipzig.deharry.scells@uni-leipzig.debenno.stein@uni-weimar.dematthias.hagen@uni-jena.demartin.potthast@uni-leipzig.deWe're happy to help!LicenseThis repository is released under theMIT license.
Files in thedata/directory are exempt from this license.
If you use the AQL in your research, we'd be glad if you'dcite us.AbstractThe Archive Query Log (AQL) is a previously unused, comprehensive query log collected at the Internet Archive over the
last 25 years. Its first version includes 356 million queries, 166 million search result pages, and 1.7 billion search
results across 550 search providers. Although many query logs have been studied in the literature, the search providers
that own them generally do not publish their logs to protect user privacy and vital business data. Of the few query logs
publicly available, none combines size, scope, and diversity. The AQL is the first to do so, enabling research on new
retrieval models and (diachronic) search engine analyses. Provided in a privacy-preserving manner, it promotes open
research as well as more transparency and accountability in the search industry.