Working with WARCS: New Tools for Harvesting,Accessing, and Researching Web Archives
Jefferson Bailey

Maria Praetzellis

Vinay Goel

Internet Archive

Internet Archive

Internet Archive

300 Funston Avenue

300 Funston Avenue

300 Funston Avenue

San Francisco, CA, CA 94118, USA

San Francisco, CA, CA 94118, USA

San Francisco, CA, CA 94118, USA

jefferson@archive.org

maria@archive.org

vinay@archive.org

ABSTRACTThe process of collecting, managing, and serving archives of web
published materials poses many challenges. Web archives tend tobe quite voluminous, data-wise and number-of-items-wise, witheven highly curated collections being many terabytes in size andtens, if not hundreds, of millions of URLs. Web archives also
have longitudinal complexity, bearing a temporal aspect similar,
but different, to serial publications, with frequent changes incontent (and often state of existence) even at the same URL, gobs
of metadata both content-based and transactional that has researchor administrative value, and have overall characteristics that makethem highly suitable for data mining and computational analysis.
However, web archiving as a practice is still mostly fractionallystaffed, with few institutions able to contribute engineeringresources to innovation and research and development.
The Internet Archive has been pioneering web archiving
technologies for 20 years. A number of recent initiatives havesought to reinvigorate innovation around new tools designed toaddress improving content capture, reconceptualizing access anddiscovery, and facilitating the increasingly popular scholarly
approaches of computational analysis of large born-digitalcollections. This workshop will provide hands-on demonstrationsof, and, when possible, training in, new tools for capturing,
analyzing, facilitating use of web archive collections. This
includes training in browser based capture methods, curatorial
tools, data derivation and extraction methods for research use,demonstration of APIs and their uses for studying archived web
data, new search and aggregation tools, and frameworks for web
archive processing.
Workshop Scope & Intended Content
Attendees will use a range a tools as part of the workshop,
including the ability to run scripts and tools on web archive datathat they collector prior to, or bring with them to, the workshop.
In general the workshop will cover these areas of practice througha mix of hands-on work, demonstrations, and discussion:••

Participants will use and discuss new API based tools
for enhanced access to, and research use of, web archive
data. In-production and test APIs will be used for datatransport, content analysis, metadata aggregation, andother collection management and discovery activities.
Participants will be trained in how to generate datasetsfrom web archive collections, including gaining
familiarity with the suite of derivative formats, indexing
tools, and how to facilitate and support researchersinterested in using archived web data.

•Participants will be trained in the latest open-source
crawling technologies including browser based captureand API harvesting. This includes both Brozzler, a
distributed web crawler, and new other tools designed toharvest social media data. Participants will be given theopportunity to compare crawling mechanisms by using
a variety crawling methods, and discuss how they canincorporate these tools in their own institutions.

•Demonstration of recent innovations in providing accessto large web archive corpora, including user testing andtechnical review of new search technologies forquerying large-scale web archives, including a review ofmethods, technologies, interfaces, and other cuttingedge approach to dealing with web archive content
discovery.

•Review new dataset formats and tools designed toenable easy data processing, extraction and derivation of
web archive data for computational analysis. This
includes the development of a community space to sharemodules and recipes for analyzing data sets that can bereused among scholars as well as the exploration ofderivative data formats and interfaces like Python-base
Jupyter Notebooks for pre-scripted data analysis.

Requirements & Target Audience
The workshop requires a meeting room with wireless internet
access and a projector with screen. Participants will bring theirown laptop computers and there should be sufficient power
outlets. The workshop hosts will coordinate preliminary activities
over email and provide some basic technical support beforehand
for workshop attendees.
The target audience for this workshop includes professionals
working in digital library services that are collection, managing,or providing access to web archives, scholars using archived web
data in their work, or digital library and preservation professionals
working to provide computational access to digital collections.
Some knowledge of or experience with web archives is helpful
but not required and some familiarity or comfort with working at
the command line interface is helpful.

Keywords
Web archiving; digital collections; data mining; research