From WARC to Graph: Link Extraction for Web Archive Analytics
Creators: name: Schlesinger, Claus-Michael
name: Hein, Pascal
Publisher: name: Zenodo
Year: 2022
Subjects: web archiving
web archive analytics
graph data
network
hyperfiction
Susanne Berkenheger
digital literature
electronic literature

description: Presentation held on November 25, 2022, at the Conference DH_Budapest_2022 &amp; DARIAH Days, Eötvös-Lorand-University, Budapest. This presentation is located in the field of web archive analytics. It describes a case study of network analytics and network visualization of the hyperfiction work "Zeit für die Bombe" by Susanne Berkenheger, details the function of the software package Warc2graph that was developed for this task, and presents, as a conclusion and outlook, how transforming WARC files into a graph structure might advance the construction of analytics pipelines in web archive analytics. Book of Abstracts (see pp. 79-80): https://elte-dh.hu/wp-content/uploads/2022/11/dh_absztrakt.pdf Full Abstract as published in the book of abstracts: In hypertext fiction, the text, before reading, exists only in a state of potential. The text of a reading will only be realized through a series of decisions by the reader. For example, in Susanne Berkenheger’s Zeit für die Bombe (Time for the Bomb) (1997), the text is split into several pages. A page can contain hyperlinks and other reference types that are used to link to other pages or embed content like images into the site. Zeit für die Bombe also makes use of automatic forwards placed in the HTML header. Pages with one or multiple clickable links require the reader to take action in order to go to the next page. As each link leads to a different page, clicking a link means that the text will be constructed page by page based on the reader’s decision about which link to follow. Pages can be modeled as nodes of a network, hyperlinks, automatic forwards and other types of references as connections between nodes. Pages with multiple references are thus represented as nodes with an outgoing degree greater than 1. The pages and hyperlinks will thus form a specific network that represents the condition of all possible texts and that represents the site as a network of its constituting elements. As hypertext fiction is, in the space-time continuum of the world wide web, an old genre, chances are that works don’t exist on the live web any more, but only in web archives. Some web archiving software packages write hyperlinks found on a page into the web archive metadata, others do not. In Berkenheger’s Zeit für die Bombe, some pages use HTML header functions to implement an automatic forwarding after a specified amount of time. Also, links generated dynamically through javascript code can be difficult to extract for crawlers. While approaches working with large web archives prioritize scalability and focus on WARC metadata (Eldakar and Alsabbagh 2020, Lin et al. 2017), our approach prioritizes completeness of extracted reference information and applies several extraction methods including analysis of WARC payloads. In our presentation we describe the data model developed for deep extraction of reference data in web archives and our software implementation . Departing from our initial usecase - analyzing link structure in net literature - we describe how the link extraction procedure implies a transformation of WARC objects into graph data. We show that this transformation can be understood as an early step in web archive analytics pipelines not only for net literature, but more broadly for historical research which might be interested in the structure of single objects or small to medium web archive corpora. Bibliography Berkenheger, Susanne. 1997. “Zeit Für Die Bombe.” Hyperfiction. 1997. http://www.berkenheger.<br> netzliteratur.net/ouargla/wargla/zeit.htm.<br> Eldakar, Youssef, and Lana Alsabbagh. 2020. “LinkGate: Let’s Build a Scalable Visualization Tool<br> for Web Archive Research.” April 23, 2020. https://netpreserveblog.wordpress.com/2020/04/23/<br> linkgate-update/.<br> Hein, Pascal, Mona Ulrich, Claus-Michael Schlesinger, and Andre Blessing. 2022. “Warc2graph.”<br> https://github.com/dla-marbach/warc2graph.<br> Lin, Jimmy, Ian Milligan, Jeremy Wiebe, and Alice Zhou. 2017. “Warcbase: Scalable Analytics<br> Infrastructure for Exploring Web Archives.” J. Comput. Cult. Herit. 10 (4): 22:1-22:30. https://doi.<br> org/10.1145/3097570. Changes: For reasons of coherence, on slides no. 20 and 21, the two images showing raw WARC data and the image source information were exchanged for images derived from Berkenheger: "Zeit für die Bombe" (1997/2022) after the presentation was held at the conference. The images shown at the conference were derived from the work "Die Aaleskorte der Ölig".