# Narrative-based Understanding of Society

Luc Steels, Frank van Harmelen, John Bateman, Inès Blin, Alessandra Fornetti, Carlo Romano Marcello Alessandro Santagiustina, Ilda Mannino, Thomas Mildner, Mihai Pomarlan, Robert Porzel, et al.

## ► To cite this version:

Luc Steels, Frank van Harmelen (Dir.). Narrative-based Understanding of Society. 2025, 10.5281/zen-odo.15093258. hal-05031951

HAL Id: hal-05031951

https://hal.science/hal-05031951v1

Submitted on 24 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

Meaning and Understanding in Human-centric AI

# Narrative-based Understanding of Society

**Editors:**

Luc Steels (VIU)

Frank van Harmelen (VUA)

**Contributors:**

John Bateman (UHB)

Inès Blin (CSL, VUA)

Alessandra Fornetti (VIU)

Carlo Santagiustina (VIU)

Ilda Mannino (VIU)

Thomas Mildner (UHV)

Mihai Pomarlan (UHB)

Robert Porzel(UHB)

Folco Soffietti (VIU)

Laura Spillner (UHB)

Luc Steels (VIU)

Lise Stork (VUA)

Annette Ten Teije (VUA)

Ilaria Tiddi (VUA)

Remi van Trijp (CSL)

Richard Zijdeman (IIS)
Edited by Luc Steels (Venice International University) and Frank van Harmelen (Free University Amsterdam)

Cover Graphic Design - studio +fortuna, Trieste

Published by Venice International University, Isola di San Servolo 30133 Venice, Italy

Open Access: https://zenodo.org/communities/muhai/

Citation:

Steels, L. and Van Harmelen, F. (eds.) (2025) Narrative-based understanding of Society. Venice International University, Venice.

DOI: 10.5281/zenodo.6602456

Volume published in the framework of the MUHAI project, which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 951846. It reflects only the authors' view and the Commission is not responsible for any use that may be made of the information it contains.

This publication is released under the Creative Commons Attribution 4.0 International license, providing Open Access through zenodo.org. Copyrights for all articles and figures are retained by their authors or copyright holders. You can freely share, adapt and draw on this work as long as you give credit, as per the terms of the license. If you reproduce or draw on material from this publication, we'd be grateful if you could give credit and link back. The editors have used their best endeavours to ensure that the URLs for external websites referred to in this book are correct and active at the time of going to press. However, the editors have no responsibility for the websites and can make no guarantee that a site will remain live or that the content is or will remain appropriate. Every effort has been made to trace all copyright holders, but if any have been inadvertently overlooked the editors will be pleased to include any necessary credits in any subsequent reissue or edition.

<figure><img src="image_3.png" /></figure>

This project has received funding from the European Pathfinder Project under Grant Agreement N° 951846
# Executive Summary

The MUHAI project studies how it is possible to build AI systems that rest on meaning and understanding. The key hypothesis of the project is that it requires going beyond the application of patterns acquired by statistical induction (for example using deep learning methods) towards the use of symbolic models, inference processes, and functional language processing using form-meaning pairs to produce or interpret language.

This book is the third open access volume in a series reporting results of the project. The first volume https://zenodo.org/record/6666820 surveyed how the notion of understanding is being discussed and treated in other human-centred research fields, more specifically in social brain science, social psychology, linguistics, semiotics, economics, social history and medicine. The main conclusion was that understanding is the process of constructing a coherent narrative about a situation that integrates and is compatible with a multitude of knowledge sources. The knowledge sources range from visual and motor experiences to language processing, semantic memory, and episodic memory.

The second volume https://zenodo.org/records/14199174 explored these ideas with a number of concrete case studies that are all about understanding the world. The case studies come from the domain of cooking: the preparation and execution of recipes and the invention of recipes, particularly by constructing variations of existing ones. The different papers push the state of the art in technologies needed to operationalize understanding: computational linguistics, semantic web technology, knowledge-based robotics and generative AI. There are also contributions on data structures and strategies for integrating these components in the construction of integrative narrative networks.

The present volume contains further case studies that now focus on understanding society.

* *Part I. Tools for narrative understanding* introduces some of the tools that will be prominent in this volume. The first paper entitled *Structured representations for narratives* surveys structured representations for representing and processing narratives and choosing a knowledge representation paradigm that best fits with a specific class of narratives. The second one, entitle *Chronographer: Even-centric knowledge graph construction via informed graph traversal* introduces the ChronoGrapher, an event-centric knowledge graph construction tool based on informed graph traversal. It is designed specifically for handling temporal narratives, central to historical narratives, and for extracting them automatically from text.

* *Part II. Understanding society through demography* introduces a social science workbench with various tools for understanding society through social demography research.

* *Part III. Understanding society through social networks* applies semantic techniques to social networks to clarify opinion on important societal issues such as conflict or inequality. There are again three contributions: (i) A framework for tracking how social media reflect conflicts. (ii) A knowledge graph for representing and tracking discourse on inequality. And (iii) mechanisms for obtaining narratives on inequality based on an analysis of social media.

* *Part IV. Understanding society through Art* applies semantic techniques to the analysis of art works in order to see which societal structures and attitudes they reflect. There are two contributions: (i) A paper introducing Integrative Narrative Networks and ways in which
semantic web technologies can be used to derive them. The mechanisms are illustrated with a Renaissance painting by Lotto. (ii) A paper applying the same techniques but now for a contemporary painting by Luc Tuymans.

* *Part V. Generating Social Narratives* discusses ways in which knowledge graphs can be 'narrativized', in other words turned into narratives. There are two contributions: (i) A paper demonstrating how stories can be generated from knowledge graphs, and (ii) a paper demonstrating that there are unexpected gender stereotypes in stories generated by large language models, providing an argument for using semantic techniques to counter such biases.

* *Part VI. Conclusions* introduces first an analysis of AI in the public debate, and how AI researchers reflect on this debate, then there are some general conclusions on what has been learned within the MUHAI project and what are plausible steps forward towards human-centered trustworthy AI.
# Contents

<table><tr><td>I</td><td>Tools for Narrative Understanding</td><td>3</td></tr><tr><td>1</td><td>Structured Representations for Narratives</td><td>4</td></tr><tr><td>2</td><td>ChronoGrapher: Event-centric Knowledge Graph Construction via Informed Graph Traversal</td><td>27</td></tr><tr><td>II</td><td>Understanding Society through Social Demography</td><td>56</td></tr><tr><td>3</td><td>Semantic Technologies for Social Demography Research</td><td>57</td></tr><tr><td>III</td><td>Understanding Society through Social Media</td><td>80</td></tr><tr><td>4</td><td>Towards Conflictual Narrative Mechanics</td><td>81</td></tr><tr><td>5</td><td>OKG: A Knowledge Graph for Fine-grained Understanding of Social Media Discourse on Inequality</td><td>93</td></tr><tr><td>6</td><td>Mapping and Exploring the Dynamics of Inequality Narratives Through Social Media</td><td>111</td></tr><tr><td>IV</td><td>Understanding Society through Art</td><td>133</td></tr><tr><td>7</td><td>Integrative Narrative Networks for Art Interpretation</td><td>134</td></tr><tr><td>8</td><td>Understanding Artistic Creativity through the Lens of AI</td><td>152</td></tr><tr><td>V</td><td>Generating Social Narratives</td><td>165</td></tr><tr><td>9</td><td>Unexpected gender stereotypes in AI-generated stories§</td><td>166</td></tr><tr><td>10</td><td>Narrativizing Knowledge Graphs</td><td>178</td></tr><tr><td>VI</td><td>Looking Outward and forward</td><td>192</td></tr><tr><td>11</td><td>AI in the public debate: the perspective of AI researchers</td><td>193</td></tr><tr><td>12</td><td>MUHAI: The Path Forward</td><td>204</td></tr></table>
# Part I

## Tools for Narrative Understanding
# Chapter 1

# Structured Representations for
Narratives

Inès Blin*†, Annette ten Teije†, Frank van Harmelen†, Ilaria Tiddi†

*Sony Computer Science Laboratories - Paris

†Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

## Abstract

Narratives are essential to the human sense-making process, and have sparked interest across multidisciplinary fields including Computer Science. However, efforts to unify terminology and identify core narrative requirements are limited, complicating the choice of a suitable representation. We first map identified narrative requirements to appropriate knowledge representations (1). We identify and organise narrative requirements from interviews we conducted (1a), and map these requirements to their most suited knowledge representation forms (1b). We then conduct a systematic survey to select candidate event and narrative-centric ontologies (2). We analyse how effectively these ontologies represent the main narrative requirements (2a), and rank them based on a five-star rating system (2b). We lastly provide recommendations on how to choose the best ontology for a narrative for a specific use case (3).

**Keywords** Narratives, Ontologies, KGs, Event-centric KGs.

## 1.1 Introduction

Narratives are fundamental for human understanding [13], offering invaluable insights into intricate events. They provide coherence to otherwise fragmented and unrelated data, building up coherent explanations from past data. Systems capable of constructing such narratives would be beneficial in developing truly “human-centric” systems. Such systems have already proved valuable in helping building daily narratives, e.g. in the legal [15] and historical [55] domains.

Throughout the literature, the term “narrative” is widely used but with varying interpretations and expectations, drawing from fields such as linguistics, philosophy, narratology, and artificial intelligence [16]. Narrative construction has gained prominence within Computer Science, as evidenced by workshops like the Computational Models for Narratives Workshop Series (2009-2016, [56]), the Narrative Understanding Workshop Series<fnref n="1" /> (2019-2023), the Text2Story<fnref n="2" /> (2018-

<footnote n="1">¹https://sites.google.com/umass.edu/wnu2023/home</footnote>

<footnote n="2">²https://text2story24.inesctec.pt</footnote>
2024), or the SEMMES<fnref n="3" /> (2023-2024) workshops. Despite the importance of narratives, few efforts have been made to identify core elements common to each work such as events and agents, and to unify existing representations. A recurring simplification defines a narrative as a sequence of events.

The primary objective of this work is to identify the most suitable method, given a set of requirements, for representing a narrative in the form of a knowledge graph (KG) where an ontology is populated with data, that we call *narrative graphs*. The benefit of representing narratives in a structured format like KGs lies in their incorporation of a semantic layer, which allows for finer-grained representations. Designed to align with how humans naturally build and understand narratives [13], narrative graphs aim to facilitate understanding through improved querying and reasoning. KGs have been used as background knowledge to support narrative construction [23]. Debates persist at the syntactic level about optimal KG data models [83], encompassing options like simple RDF, Named Graphs⁴ or RDF-star⁵. At the semantic level, the increasing number of narrative resources (ontologies, KGs, etc) make it challenging to choose a suitable narrative representation for a given set of requirements or a use case.

In this paper, we propose to unify the terminology across the literature and to provide guidelines on which ontology model best to use for a specific use case. We divide our problem into three research questions: **RQ1**. What are the common requirements related to narratives? **RQ2**. Can existing ontologies represent these narrative requirements? **RQ3**. How to best choose a structured representation for a narrative? To answer **RQ1**, we conduct interviews to identify narrative requirements, and we organise these requirements into themes. We also map these requirements to their most suited knowledge representation forms. To answer **RQ2**, we conduct a systematic review on event-centric and narrative-centric ontologies, and evaluate the ontologies based on how effectively they can fullfill narrative requirements and on their vocabulary usage. To answer **RQ3**, we lastly provide recommendations on how to best identify a structured representation for a narrative, and apply them to the interviews use cases.

## 1.2 Narrative Requirements

We elicited narrative requirements from interviews to answer **RQ1**, and we organise these requirements into thematics. We also map these requirements to their most suited knowledge representation (KR) forms.

### 1.2.1 Identification of Narrative Requirements

We conducted 6 narrative interviews with 9 participants in the MUHAI⁶ EU research con-
sortium. MUHAI aims to advance AI systems by integrating narratives to enhance meaning
and understanding. The project has two main use cases: pragmatic everyday knowledge (e.g.
households tasks such as cooking) and societal knowledge (historical episodes, socio-historical
knowledge, and contemporary events). Since all participants were in MUHAI, this ensured
consistency and relevance to the use cases, yielding focused insights; however, this approach
possibly limits the generalisability of the findings to other domains, which could be explored
in future work. The open unstructured interviews took 20 minutes on average. The starting
point of the interviews were the following questions:

1. What elements are needed to represent your (domain-specific) narratives? Why and what

<footnote n="3">³https://anr-kflow.github.io/semmes/
⁴https://www.w3.org/2009/07/NamedGraph.html
⁵https://w3c.github.io/rdf-star/cg-spec/editors_draft.html
⁶Meaning and Understanding in Human-Centric AI: https://muhai.org</footnote>
<table><caption>Table 1.1: Interview Descriptions. #: number of people interviewed.</caption><thead><tr><td>Id #</td><td>Domain</td><td>Input Data</td><td>Narrative Example</td><td>Application scenario</td></tr></thead><tbody><tr><td>I1 1</td><td>History</td><td>Historical events</td><td>Main happenings during the French Revolution [7]</td><td>Query-answering about historical events</td></tr><tr><td>I2 1</td><td>Biomedical Science</td><td>Scientific Claims</td><td>PICOs (Population, Intervention, Comparison and Outcomes) [81]</td><td>Reasoning and decision support in clinical research</td></tr><tr><td>I3 2</td><td>Social Media Analysis</td><td>User-generated content</td><td>Claims by users on Twitter regarding the Russo-Ukrainian conflict [26]</td><td>Analyzing and deriving insights from user-generated content and trends</td></tr><tr><td>I4 2</td><td>Robotic Kitchen</td><td>Recipes</td><td>Robot execution of a recipe [23]</td><td>Robotic cooking task execution based on recipe understanding</td></tr><tr><td>I5 2</td><td>Arts &amp; Literature</td><td>Text (linguistics, semantics) &amp; Images (paintings)</td><td>Questions arising when reading a text/painting, that form a narrative network together with the answers to these questions [80]</td><td>Question-answering to enable interpretive analysis from artistic or literary works</td></tr><tr><td>I6 1</td><td>Cognitive Science</td><td>Mental Models</td><td>Importance of values in our beliefs [89]</td><td>Exploring and modeling belief systems and cognitive processes</td></tr></tbody></table>

do you need to represent with these?<fnref n="7" />

2. What kind of data structure is required to represent your narratives?

3. For which tasks do you use your narratives? E.g. querying, visualisation, etc.

4. Can you provide one example of a narrative in your use case?

Most participants had a computer science background. If not, we re-formulated in non-technical terms. Table 1.1 provides a description of the interviews, with the domains, all related to MUHAI, the type of input data used to build the narrative, one example of a narrative and a practical application scenario for each narrative graph to be built. In the history and in the cognitive science domains (I1, I6), the main elements to be represented can be seen as extended version of the five Ws: Who, What, Where, When and Why. For the biomedical science (I2), the knowledge to be represented is much more structured and concise. The main difference for the social media analysis (I3) is to enable the representation of conflicting viewpoints, that is important when analysing discourse in social media. Arts & Literature (I5) additionally require context and background knowledge to enhance the comprehension as the text or the image is read. Lastly, the Robotic Kitchen (I4) involving a cooking robot in a virtual simulation needs to plan its actions to execute a recipe. In this context, each action is seen as an episode in a narrative. We detail hereafter the 6 main requirements we gathered and organized from these interviews.

<footnote n="7">⁷From a KR perspective, expressions that do not very easily fit into the regular <s,p,o> representation of RDF, like higher-order or n-ary relations.</footnote>
* **Agents<fnref n="8" />** These are animate entities like humans or animals.

- *Types of agents* distinguish between humans, animals, animate objects, etc. Occasionally, people made the distinction between an active agent, who acts with a goal in mind, and a passive agent, who is more a spectator in a scene.

- *Properties* describe agent attributes, such as physical characteristics.

- *Roles* describe roles of agents in the events.

* **Objects.** These are physical or abstract objects that do not fall under the category of agents.

- *Types of objects* distinguish between different types of objects.

- *Properties* describe objects attributes, such as physical characteristics.

- *Roles* describe roles of objects in events.

* **Locations.** Places or settings in which events happens.

- *Relations between locations* link locations between them.

* **Provenance.** This is a description of the origin of the data or statements.

- *Narrative text* includes any original text narrative content.

* **Events and Actions.** Throughout the interviews, events appeared as the core element in the narrative. Similarly to agent, there is a distinction between an action and an event. An action happens because of the execution of a function by an agent, whereas an event simply happens. An action can be associated with pre-conditions and post-conditions, that enable the happening of the action, or result from the action.

- *Types* distinguish between different types of events and actions.

- *Properties* describe generic attributes, such as timestamps and locations.

- *Relations* describe relation between events, including:

* Mereological. Relations between parts and the wholes they form.

* Temporal. Temporal relationships between events, such as the ones from Allen's Algebra [4].

* Causal. Causal relationships between events.

* **Narrative Dynamics.** The higher-level elements that bind, influence, and drive the progression of the narrative. They play a critical role in shaping the overall story, providing the framework within which the narrative elements interact and evolve. They provide a type of narrative by bridging together elements from the above categories.

- *Script* is a pattern of sequences of events that are related, like the restaurant scene [75]. This includes archetypes, plans, frames and processes.

- *Storyline* is a coherent sequence of events, it is an instantiation of a script. This includes scenes, acts, episodes, sequences of events and situations.

- *State* is the particular condition that someone or something is at a specific time or during a specific time interval.

<footnote n="8">⁸In these requirements, we differentiate agents from objects, with agents not necessarily participating in events directly, but their participation would be modeled through relationships.</footnote>
- *Goal* is a state that is desired by an agent.

- *Outcome/Effect* is a state that results from a causal relation, representing the new condition after an event.

- *Perspectives* are different interpretations of the same event.

- *Conditions* include conditions under which events or actions can happens. This includes pre-conditions, results or post-conditions, complete-condition, coherence-condition, and enablement.

### 1.2.2 Narrative Requirements Representation

In Section 1.2.1, we identified requirements for including semantic concepts in the representations of narratives across a broad range of domains. In this section, we map these representations to a variety of syntactic constructs for KGs. We first describe each KR formalism, ordered by representation complexity. The complexity level is based on the introduction of new abstraction levels, the ability to handle more entities, or the incorporation of more sophisticated reasoning.

- **basic**: Only binary relations without variables;

- **n-ary**: Relations involving more than two entities;

- **variables**: Placeholders that can be replaced with actual values or entities;

- **higher-order**: Representations that involve functions or predicates applied to other functions or predicates;

- **temporal relation**: Relationships that involve time aspects;

- **epistemic**: Relates to knowledge and beliefs about knowledge;

- **causal**: Cause-and-effect relationships between entities or events;

- **rules**: Logic-based statements that define conditions and implications.

The left part of Table 1.2 summarises the mapping of narrative requirements to the interviews. The most frequent requirements are events and actions, agents, location, and time. The frequency of (physical) objects is low, except in specific use cases such as Robotic Kitchen and Arts & Literature. The right side of Table 1.2 maps narrative requirements to their most suited KG representation forms. Objects, locations, and agents require only basic or n-ary representations. Provenance requires higher-order representations, while most events are simple except for temporal and causal relationships. Narrative dynamics demand the most complex representations, involving variables, higher-order, epistemic, or rules. Objects, locations, and agents are therefore the easiest to represent, while events and narrative dynamics require more advanced KR formalisms.

Lastly, Table 1.3 provides a mapping of KR formalisms (cf. Table 1.2) to various syntactic constructs (KG, query, rules). The left part of Table 1.3 describes the different existing types of KGs. The right part of Table 1.3 shows which KR formalisms can be used to represent each construct. Since some KGs are additive, e.g. blank nodes contain simple KG triples, we focus on the additional complexity introduced by the more advanced KG constructs described in the “Description” column. The table highlights how different KR formalisms, ranging from basic structures like simple triples to more complex constructs such as rules and named graphs, require corresponding sophisticated KG representations.
<table><caption>Table 1.2: Mapping of narrative requirements to the interviews and their required KG representations. For a requirement *r* and interview *I*<sub>*i*</sub>, a ✓ indicates that *c* was identified as a requirement in *I*<sub>*i*</sub>. For a requirement *r* and a representation *r*, a ✓ indicates that *c* can be represented with *r*. Req: Requirements.</caption><thead><tr><th rowspan="2">Narrative Req.</th><th colspan="6">Interviews</th><th colspan="5">Syntactic Constructs</th><th rowspan="2">Total</th></tr><tr><th>I1</th><th>I2</th><th>I3</th><th>I4</th><th>I5</th><th>I6</th><th>Total</th><th>basic<br>n-ary<br>variables</th><th>higher-order<br>temporal</th><th>epistemic<br>causal</th><th>rules</th></tr></thead><tbody><tr><td>Agents</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>6</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>types</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>2</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>properties</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>2</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>roles</td><td>✓</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td>4</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>Objects</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>2</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>types</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>2</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>properties</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>2</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>roles</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>2</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>Locations</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>4</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>relations</td><td></td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>2</td><td>✓</td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>Provenance</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>3</td><td></td><td>✓</td><td></td><td></td><td>1</td></tr><tr><td>narrative text</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>3</td><td></td><td>✓</td><td></td><td></td><td>1</td></tr><tr><td>Events/Actions</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>6</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>4</td></tr><tr><td>types</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>2</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>properties</td><td>✓</td><td></td><td>✓</td><td></td><td></td><td></td><td>1</td><td>✓</td><td></td><td></td><td></td><td>1</td></tr><tr><td>relations</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>6</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>4</td></tr><tr><td>Dynamics</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>6</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>4</td></tr><tr><td>script</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>4</td><td>✓</td><td></td><td></td><td>✓</td><td>2</td></tr><tr><td>storyline</td><td></td><td></td><td>✓</td><td></td><td></td><td></td><td>1</td><td></td><td>✓</td><td></td><td></td><td>2</td></tr><tr><td>state</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>3</td><td></td><td>✓</td><td></td><td></td><td>1</td></tr><tr><td>goal</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td></td><td>2</td><td></td><td>✓</td><td>✓</td><td></td><td>2</td></tr><tr><td>outcome/effect</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td>1</td><td></td><td>✓</td><td>✓</td><td></td><td>2</td></tr><tr><td>perspective</td><td></td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>3</td><td></td><td>✓</td><td>✓</td><td></td><td>2</td></tr><tr><td>conditions</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td>3</td><td></td><td>✓</td><td></td><td>✓</td><td>2</td></tr></tbody><tfoot><tr><td>Total</td><td>14</td><td>5</td><td>18</td><td>18</td><td>5</td><td></td><td>14</td><td>6</td><td>3</td><td>8</td><td>2</td><td>4</td><td>2</td><td>3</td></tr></tfoot></table>

## 1.3 Can existing ontologies represent narratives?

To answer **RQ2** on assessing whether ontologies can represent narrative requirements, we conduct a systematic review on event-centric and narrative-centric ontologies. We then evaluate the ontologies based on how effectively they can represent narrative requirements and based on their vocabulary usage.
<table><caption>Table 1.3: Mapping of KG/Query/Rules representations to KR formalisms. NG: Named Graph, LPG: Labelled Property Graph.</caption><thead><tr><th>Name</th><th>Construct</th><th>Repr.</th><th>Description</th><th colspan="3">KR formalisms</th></tr><tr><td></td><td></td><td></td><td></td><td colspan="3"><strong>basic</strong><br><strong>n-ary</strong><br><strong>variables</strong><br><strong>higher-order</strong><br><strong>temporal</strong><br><strong>epistemic</strong><br><strong>causal</strong><br><strong>rules</strong></td></tr></thead><tbody><tr><td>Simple KG</td><td></td><td></td><td>simple (s, p, o) triples</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Blank Node</td><td></td><td>RDF</td><td>nodes with no IRI</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Singleton Property</td><td>KG</td><td></td><td>triples annotated</td><td>✓</td><td></td><td></td></tr><tr><td>Reification</td><td></td><td></td><td>statement nodes</td><td></td><td>✓</td><td>✓</td></tr><tr><td>NG</td><td></td><td>NG</td><td>triples grouped</td><td></td><td>✓</td><td></td></tr><tr><td>LPG</td><td></td><td>LPG</td><td>key-value pairs</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Hyperrelational Graph</td><td></td><td>RDF-Star</td><td>«s p1 o1» p2 o2</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Query</td><td>Query</td><td>-</td><td>-</td><td>✓</td><td></td><td></td></tr><tr><td>Rule</td><td>Rule</td><td>-</td><td>-</td><td></td><td>✓</td><td>✓</td></tr></tbody></table>

### 1.3.1 Systematic Review

We conducted a systematic review to compare existing ontologies based on their ability to represent narratives in a structured format. The process we used is illustrated in Figure 1.1. We used the query “narrative OR event-centric”, focusing on abstracts and titles. We then incorporated references from survey papers identified during the search or from other relevant sources, and included ontologies cited in the identified papers. Survey papers included surveys on narrative-centric ontologies [88] and event-centric ontologies [66, 76, 78]. We included only papers that presented a novel ontology relevant for representing a narrative graph, with inclusion criteria determined through abstract review.

<figure><img src="image_4.png" /><figcaption>Figure 1.1: Process used to retrieve papers for our comparison (PRISMA [63]). For the snowballing, we looked at paper references from the first iteration.</figcaption></figure>

This resulted in 61 papers: 45 were papers presenting an ontology, 1 presented both an ontology
<table><caption>Table 1.4: Resource Paper Description.</caption><thead><tr><th>Id</th><th>Resource</th><th>Ontology</th></tr></thead><tbody><tr><td>1</td><td>Structured Event Entity Resolution [42]</td><td>EO [71]</td></tr><tr><td>2</td><td>VLX-Stories [27]</td><td>VLX-Stories [27]</td></tr><tr><td>3,4</td><td>EventKG [20, 13]</td><td>SEM [75]</td></tr><tr><td>5</td><td>DIVE [20]</td><td>SEM [75]</td></tr><tr><td>6</td><td>ECG [28]</td><td>SEM [75]</td></tr><tr><td>7</td><td>Wind in Our Sails [77]</td><td>CIDOC-CRM [21]</td></tr><tr><td>8</td><td>BiographySampo [38]</td><td>CIDOC-CRM [21]</td></tr><tr><td>9</td><td>ArCo [6]</td><td>CIDOC-CRM [21]</td></tr><tr><td>10</td><td>SCUPLTEUR [3]</td><td>CIDOC-CRM [21]</td></tr><tr><td>11</td><td>Using Event Spaces [59]</td><td>CIDOC-CRM [21], LODE [78]</td></tr><tr><td>12</td><td>EventMedia [44]</td><td>CIDOC-CRM [21], LODE [78]</td></tr><tr><td>13</td><td>OceanGraph [94]</td><td>SOSA [39] (extended from DUL [31])</td></tr><tr><td>14</td><td>3cixty [86]</td><td>DUL [31], Schema [2]</td></tr><tr><td>15</td><td>History of Vienna [46]</td><td>Schema [2]</td></tr></tbody></table>

and a resource KG, and 15 were resources based on these ontologies. Among the 46 (45 + 1) ontologies, we further analysed the other ontologies on which they were built, such as DUL [31] or CIDOC-CRM [21]. This resulted in three categories of ontologies: (1) event-centric ontologies, which includes (1a) top-level ontologies and (1b) ontologies extended from ontologies from (1a), (2) narrative-centric ontologies, which includes (2a) narrative-centric ontologies and (2b) ontologies extended from ontologies from (2a), and (3) media-centric ontologies, which we discarded for our comparison since these were about video content that we did not have in our interviews. Table 1.4 and 1.5 describe the 15 resources and 46 ontologies respectively. Figure 1.2 shows the visual dependencies on the ontologies from category (1b) and (2b).

<figure><img src="image_5.png" /><figcaption>Figure 1.2: The connections between the various event-centric ontologies (1). The ones coloured in blue (from 1a) are the ones we kept for our comparison: EO [71], DUL [31], DOLCE [30], CIDOC-CRM [21] and EC [45].</figcaption></figure>

We refer to our supplementary online material<fnref n="9" /> for further details on the systematic review. The main document contains a description of our methodology and sources, and the folders contain the papers retrieved with the search engines. The ontologies highlighted in blue in Table 1.5 are the ones we retained for our comparison, i.e. the ontologies from categories (1a) and (2a). All the ontologies from (1b) and (2b) were extended from others from (1a) and (2a), so we focus on (1a) and (2a) ontologies instead. Table 1.6 describes the references found from the search engines: “all” refers to the number of results we got from the search queries, and “useful” refers to the number of references we kept.

<footnote n="9">⁹https://bit.ly/ekaw-2024-systematic-review</footnote>
<table><caption>Table 1.5: Ontology Paper Description. *Extended from* specifies the original ontology, if applicable, from which the listed *Ontologies* were derived.</caption><thead><tr><th>Type</th><th>Id</th><th>#</th><th>Ontologies</th><th>Extended from</th></tr></thead><tbody><tr><td>Event</td><td>1a</td><td>16</td><td>EO [71], DUL [31], DOLCE [30], CIDOC-CRM [21], EC [45], SEM [75], LODE [78], the Enslaved Ontology [79], ABC [48], EDM [23], VLX-Stories [27], E [92], SEO [26], FARO [72], ProppOntology [64], BBO [47]</td><td>-</td></tr><tr><td rowspan="7">Event</td><td rowspan="7">1b</td><td>1</td><td>Seismic Ontology [52]</td><td>EO [71]</td></tr><tr><td>3</td><td>F [76], SOSA [39], EPFDO [53]</td><td>DUL [31]</td></tr><tr><td>1</td><td>Curate [58]</td><td>DUL [31], CIDOC-CRM [21]</td></tr><tr><td>2</td><td>COMM [5], DOLCE-LITE</td><td>DOLCE [30]</td></tr><tr><td>1</td><td>FRBR [62]</td><td>FRBRoo [22]</td></tr><tr><td>1</td><td>the Jazz Ontology [69]</td><td>FRBR [62]</td></tr><tr><td>1</td><td>Extended EC [17]</td><td>EC [45]</td></tr><tr><td>Narrative</td><td>2a</td><td>7</td><td>RST [60], Mediation [41], Princess Lovely [84], StoryTeller [93], ODY-ONT [43], DL Ontology for Fairy Tale [65], Transmedia Fictional Worlds [11]</td><td>-</td></tr><tr><td rowspan="4">Narrative</td><td rowspan="4">2b</td><td>1</td><td>Drammar [19]</td><td>DOLCE-LITE</td></tr><tr><td>1</td><td>AO [18]</td><td>Drammar [19], FRBRoo [22]</td></tr><tr><td>4</td><td>FRBRoo [22], ArCo [6], Story-Foutain [57], RSO [61]</td><td>CIDOC-CRM [21]</td></tr><tr><td>1</td><td>Narrative Ontology [6]</td><td>CIDOC-CRM [21], EC [45]</td></tr><tr><td>Media</td><td>3</td><td>6</td><td>OntoMedia [49], SsVM [25], Eventory [91], DISC [32], VERL [29], RDF-Events [82]</td><td>-</td></tr></tbody></table>

### 1.3.2 Description of the retained ontologies

Some models we included come from a survey on narrative-centric ontologies [88], that reviewed 11 ontologies that explicitly mention the term “narrative” in their description. [60] proposes a storytelling ontology using RST [51] that describes events and objects, and how they are connected. Both [6] and [57] are extended from CIDOC-CRM [21], an ontology widely used in the cultural heritage domain. [6] focuses on representing core notions in narratology such as fabula, plot and narration, and [57] provides an interface to discover links between entities. A follow-up of [6] is the Narrative Ontology [10] to be used in Digital Libraries. The Mediation Ontology [41] is in the domain of international relations. The Labyrinth system [18] allows to discover narratives between entities in the domain of digital humanities and western archetypes. It re-uses ontologies like Drammar [50] and FRBRoo [22]. [84] presents a fabula model for emergent narratives, and re-uses the General Transition Network [85]. The main focus is to represent causal transitions between events. ODY-ONT [43] is used to query a literary text, and BKOnto [93] to represent biographical narratives. The CRBOnto ontology [65] relies upon Propp's morphology of folks stories [68]. [11] proposes an ontology to represent transmedia fiction such as the Marvel universe. [19] proposes a model based on Drammar [50] and DOLCE-LITE to represent actions motivated by heroes in the literature.

Event-centric ontologies encompass diverse domains such as: SEO [26] for scientific events,
<table><caption>Table 1.6: Provenance description of papers from the systematic review. N: Narrative, E: Event-centric. For K-CAP/N (useful), 5 papers were not accessible.</caption><thead><tr><th>Provenance</th><th>N (all)</th><th>N (useful)</th><th>E (all)</th><th>E (useful)</th></tr></thead><tbody><tr><td>SWJ</td><td>7</td><td>2</td><td>2</td><td>2</td></tr><tr><td>JOWS</td><td>21</td><td>2</td><td>147</td><td>7</td></tr><tr><td>ISWC</td><td>35</td><td>9</td><td>66</td><td>2</td></tr><tr><td>ESWC</td><td>32</td><td>2</td><td>106</td><td>1</td></tr><tr><td>EKAW</td><td>16</td><td>1</td><td>14</td><td>1</td></tr><tr><td>K-CAP</td><td>44</td><td>1</td><td>176</td><td>2</td></tr><tr><td>KGSWC</td><td>2</td><td>1</td><td>40</td><td>4</td></tr></tbody></table>

CIDOC-CRM [21] and EDM [23] for cultural heritage, SEM [75], LODE [78], ABC [48], EO [71], EC [45] and E [92] for more generic usages. CIDOC-CRM [21] is widely used for the cultural heritage domain, and several other ontologies have since been derived from, such as FRBR [62] for bibliographic records. DUL [31] is a combination and a simplification of DOLCE [30] and the DnS [31] ontologies. Several ontologies were extended from DUL, such as F [76], originally for emergency response and SOMA-NARR [23]. The Enslaved Ontology [79] integrates content about the historical slave trade from diverse sources, while VLX-Stories [27] structures text from media feeds into an event-centric ontology. ProppOntology [64] is built upon Propp's morphology of the folktale [68]. Lastly, BBO [47] represents dynamic BPMN process executions in structured format and FARO [72] focuses on event relations.

### 1.3.3 Evaluation of Narrative Ontologies

#### Background for Ontology Evaluation.

Tim Berners-Lee defined a 5-star ranking schema to assess the quality of Linked Data [1]. This scoring system was based on four rules which did not include any considerations with respect to the vocabulary that includes broader categories such as ontologies and thesauri. [40] therefore proposed a new 5-star system to evaluate the quality of the vocabulary, displayed in Figure 1.3. We re-use this 5-star system to compare the ontologies.

<figure><img src="image_6.png" /><figcaption>Figure 1.3: The 5-star system to evaluate ontologies from [40].</figcaption></figure>
**Ontologies with lower 5-star rating.** Some ontologies such as Mediation [41] or Emergent*<fnref n="10" /> [84] adhere less to established web standards, which not only lowers their 5-star scores but also significantly their interoperability and integration.

**Narrative Dynamics.** Fewer ontologies can represent the narrative dynamics, that are core to binding and driving the progression of the narrative elements. Among the 4-rated domain-specific ontologies, SEO [26] and Transmedia* [11] can represent scripts and storylines. Among the 4-rated generic ontologies, SEM [75] can represent perspectives through the sem:View class and blank nodes, and FARO [72] goals, states and conditions. The other ontologies that can represent narrative dynamics have lower 5-star ranking (cf. 1-score models in Table 1.3.3).

**Other.** The number of requirements an ontology can represent varies significantly, from 21 in DUL [31] and DOLCE [30] at the cost of complexity, to 2 in EC [45], although both with a high 5-star score. There is a positive correlation between key requirements, such as agent or event representation, and their presence in ontologies. It is important to note that the presence of a checkmark (✔) indicates the representation of a requirement but does not reflect the number of such relations an ontology can represent. For instance, some ontologies only include basic relationships between events, while others offer a more extensive list. Lastly, the simpler narrative requirements such as agents and objects are well covered by event-centric ontologies, whereas the more complex ones like narrative dynamics tend to be less covered or less adhering to web standards, highlighting the ongoing challenges of narrative representation in structured format.

# 1.4 Recommendations

To answer **RQ3** on the best ontology to represent a narrative, we provide generic recommendations on how to best identify a structured representation for a narrative. We lastly provide tailored recommendations from the interviews use case (cf. Tables 1.1 and 1.2).

## 1.4.1 Generic Recommendations

Based on the previous analyses on the narrative requirements and their KR formalisms (Section 1.2), and on how effectively ontologies fullfill these narrative requirements (Section 1.3), we recommend using the following steps to effectively build a narrative in a structured format for a specific application:

1. Refer to Table 1.2 to **identify the narrative requirements** that are relevant to your project. These tables categorise narrative requirements by their complexity, aiding in the selection process.

2. Based on the chosen narrative requirements, identify the **types of KR formalisms** needed from Table 1.2 and 1.3. This will help in understanding the level of complexity your model must support.

3. Choose **ontologies that can effectively represent** the identified narrative requirements. Use Table 1.3.3 to find ontologies that cover your narrative requirements and prioritise ontologies that have a high score on the 5-star schema ranking [40].

4. **Extend existing ontologies** by picking the one that fits best, and ensure it adheres to web standards to facilitate future integration, reuse and collaboration.

<footnote n="10">¹⁰Cf. Table 1.3.3 for the explanation on the asterisk.</footnote>
<table><caption>Table 1.7: Recommendations from interview use-cases.</caption><thead><tr><td>Id Domain</td><td>Recommendation</td></tr></thead><tbody><tr><td>I1 History</td><td>DOLCE [30]/DUL [31] if reification, FARO [72]</td></tr><tr><td>I2 Biomedical Science</td><td>PICO with FARO [72]</td></tr><tr><td>I3 Social Media Analysis</td><td>Same as I1 + SEM [75] for perspectives</td></tr><tr><td>I4 Robotic Kitchen</td><td>DUL [31] or derived ontologies (as per the literature)</td></tr><tr><td>I5 Arts & Literature</td><td>ProppOntology [68], extending Emergent* [84] if time permits</td></tr><tr><td>I6 Cognitive Science</td><td>Extend or create an ontology</td></tr></tbody></table>

5. Consider developing your own ontology if existing ones are insufficient, and ensure it adheres to web standards to facilitate future integration, reuse and collaboration.

### 1.4.2 Tailored Recommendations from interviews use cases

Table 1.7 provides an overview of the recommendations we give. **I1** and **I3**, within the History and Social Media domains respectively, share numerous narrative requirements. The primary distinction is that I1 includes more generic event descriptions, while I3 encompasses more narrative dynamics such as goals and perspectives. Our recommendations are tailored to the type of KG employed, each with its own advantages and drawbacks. For example, reification is compliant with RDF standards but may be inefficient. If using reification exclusively, we suggest employing the core classes of DOLCE [30] or DUL [31], whose ontologies and design patterns are based on n-ary relationships. For I3 and the perspectives, SEM [75] is recommended due to its existing implementation. Other KGs, such as Named Graphs, are more compact but can become overloaded with numerous identifiers, while RDF-star is more efficient but still not widely adopted. In these cases, we recommend using DOLCE [30] or DUL [31] for simple narrative requirements and extending them to accommodate higher-order relationships. Additionally, we advise considering FARO [72], which, to our knowledge, can represent a wide range of event relations, including the goals pertinent to I3.

**I2** and **I6**, within the Biomedical Science and Cognitive Science domains respectively, both have a limited number of narrative requirements. The primary difference is that the input data for I2 is already structured according to the PICO ontology<fnref n="11" />, whereas the input data for I6 is unstructured. Given their limited conceptual needs, we recommend targeting ontologies that precisely represent these requirements rather than selecting the most comprehensive ones. For I2, which requires agents and relationships between events and states, we suggest combining PICO with FARO [72]. FARO is capable of representing numerous relationships between events and states, while PICO provides domain-specific vocabulary for agents. For I6, which requires

<footnote n="11">¹¹https://linkeddata.cochrane.org/pico-ontology</footnote>
agents, relationships between events, and outcomes/effects, Table 1.3.3 indicates that this last requirement is not well-represented across ontologies. The options available are the generic DOLCE [30]/DUL [31] or the domain-specific Emergent* [84]. Designing new patterns over DOLCE/DUL may be too complex for those not specialized in the semantic web. Therefore, we recommend either extending existing ontologies by creating new classes or properties, or building a new one tailored to these specific requirements.

**I4** in the Robotic Kitchen domain has the most narrative requirements including objects and numerous narrative dynamics. From the literature, it is also much linked to sensory information from robots [7, 8]. Most work in this area like SOSA [39] has used or built upon DUL [31]. In line with the existing work, we would recommend continuing using DUL [31] or derived ontologies.

**I5** in the Arts and Literature domain overlaps with domains-specific ontologies in Table 1.3.3. We recommend starting with these ontologies. ProppOntology [68] covers most simple requirements and some complex ones like storyline. For other narrative dynamics such as outcome/effect or conditions, designing patterns over DOLCE [30]/DUL [31] is an option. However, given the domain-specific nature of this application, extending Emergent* [84] to cover more narrative dynamics and improve its sustainability is a preferable approach if time permits.

## 1.5 Discussion and Conclusion

In this paper, we map identified narrative requirements to appropriate knowledge representations. We conduct six interviews and identify core narrative requirements, as well as the KR formalisms they require. We find that requirements like objects or agents require simpler KR formalisms, while requirements such as events or narrative dynamics require more complex KR formalisms (RQ1). We only included domains relevant to MUHAI, limiting our findings to text-based or sensor-based narratives and potentially reducing applicability to other domains such as multimedia content, that could be explored in future work.

We then conduct a systematic review to extract event-centric and narrative-centric ontologies to represent narratives. We analyse the 24 retained ontologies with respect to how effectively they can represent narrative requirements and how well they are rated on the 5-star vocabulary system [40] (RQ2). We found that simple requirements are best represented by event-centric ontologies. Event-centric ontologies such as DUL [31] and DOLCE [30] cover a broad range of requirements but this comes at the cost of complexity [72], since these are upper-level ontologies, which means that one might need additional work to define templates. Lower-rated ontologies struggle with interoperability due to poor web standards adherence. Fewer ontologies effectively represent narrative dynamics, with SEM [75] and FARO [72] being notable exceptions with high ratings for goals, perspectives and conditions. Coverage of complex requirements is more limited and often poorly standardized.

We lastly provide recommendations on how to best pick a model to represent a narrative, and apply them to the interview s (RQ3). The interview subjects and domain-specific ontologies cover key domains of narrative representation, but the list is not exhaustive, potentially limiting generalizability. Our semantic-web focus may narrow the scope of our recommendations to semantic web-related techniques, but broader exploration could lead to more comprehensive guidance across various KR paradigms.

In future work, we aim to enhance the evaluation of models by integrating other metrics for ontology evaluation, and to improve the generalisability of this study. Various works [12, 14, 24, 36, 37, 70, 90] have attempted to define criteria to evaluate an ontology from various perspectives such as Software Engineering, verification, validation or semiotics. Most importantly, we are actively working on automating the generation of structured narratives from KGs. Our initial efforts included a prototype for generating a narrative on the French
Revolution [10] using the SEM ontology [75], where the data collection was performed manu-
ally. We are now developing a method to automatically retrieve relevant content to construct
a narrative in the form of a KG.
# Chapter 1: References

[AA1] Linked Data - Design Issues.

[AA2] Schema.org, 2024. Accessed: 2024-09-17.

[AA3] Matthew Addis, Mike Boniface, Simon Goodall, Paul Grimwood, Sanghee Kim, Paul Lewis, Kirk Martinez, and Alison Stevenson. SCULPTEUR: Towards a New Paradigm for Multimedia Museum Information Handling. In Dieter Fensel, Katia Sycara, and John Mylopoulos, editors, *The Semantic Web - ISWC 2003*, pages 582-596, Berlin, Heidelberg, 2003. Springer.

[AA4] James F Allen. Maintaining knowledge about temporal intervals. *Communications of the ACM*, 26(11):832-843, 1983.

[AA5] Richard Arndt, Raphaël Troncy, Steffen Staab, Lynda Hardman, and Miroslav Vacura. COMM: Designing a Well-Founded Multimedia Ontology for the Web. In Karl Aberer, Key-Sun Choi, Natasha Noy, Dean Allemang, Kyung-Il Lee, Lyndon Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux, editors, *The Semantic Web*, pages 30-43, Berlin, Heidelberg, 2007. Springer.

[AA6] Valentina Bartalesi, Carlo Meghini, and Daniele Metilli. Steps towards a formal ontology of narratives based on narratology. In *7th Workshop on Computational Models of Narrative (CMN 2016)*. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016.

[AA7] Daniel Beßler, Robert Porzel, Mihai Pomarlan, Michael Beetz, Rainer Malaka, and John Bateman. A formal model of affordances for flexible robotic task execution. In *ECAI 2020*, pages 2425-2432. IOS Press, 2020.

[AA8] Daniel Beßler, Robert Porzel, Mihai Pomarlan, Abhijit Vyas, Sebastian Höffner, Michael Beetz, Rainer Malaka, and John Bateman. Foundations of the socio-physical model of activities (soma) for autonomous robotic agents 1. In *Formal ontology in information systems*, pages 159-174. IOS Press, 2021.

[AA9] Inès Blin. Building a french revolution narrative from wikidata. In *SEM4NBU@IJCAI*, pages 1-5, 2022.

[AA10] Inès Blin. Building narrative structures from knowledge graphs. In *European Semantic Web Conference*, pages 234-251. Springer, 2022.

[AA11] Frank Branch, Theresa Arias, Jolene Kennah, Rebekah Phillips, Travis Windleharth, and Jin Ha Lee. Representing transmedia fictional worlds through ontology. *Journal of the Association for Information Science and Technology*, 68(12):2771-2782, 2017.

[AA12] Janez Brank, Marko Grobelnik, and Dunja Mladenic. A survey of ontology evaluation techniques. In *Proceedings of the conference on data mining and data warehouses (SiKDD 2005)*, pages 166-170. Citeseer, 2005.
[AA13] Jerome Bruner. The narrative construction of reality. *Critical inquiry*, 18(1):1–21, 1991.

[AA14] Andrew Burton-Jones, Veda C. Storey, Vijayan Sugumaran, and Punit Ahluwalia. A semiotic metrics suite for assessing the quality of ontologies. *Data & Knowledge Engineering*, 55(1):84–102, October 2005.

[AA15] Valentina Anita Carriero, Aldo Gangemi, Maria Letizia Mancinelli, Ludovica Marinucci, Andrea Giovanni Nuzzolese, Valentina Presutti, and Chiara Veninata. Arco: The italian cultural heritage knowledge graph. In *International Semantic Web Conference*, pages 36–52. Springer, 2019.

[AA16] Tommaso Caselli, Eduard Hovy, Martha Palmer, and Piek Vossen. *Computational Analysis of Storylines: Making Sense of Events*. Cambridge University Press, 2021.

[AA17] Iliano Cervesato, Massimo Franceschet, and Angelo Montanari. A guided tour through some extensions of the event calculus. *Computational Intelligence*, 16(2):307–347, 2000.

[AA18] Rossana Damiano and Antonio Lieto. Ontological representations of narratives: a case study on stories and actions. In *2013 Workshop on Computational Models of Narrative*, pages 76–93. Schloss Dagstuhl Leibniz-Zentrum für Informatik, 2013.

[AA19] Rossana Damiano, Vincenzo Lombardo, and Antonio Pizzo. The ontology of drama. *Applied Ontology*, 14(1):79–118, 2019.

[AA20] Victor De Boer, Johan Oomen, Oana Inel, Lora Aroyo, Elco Van Staveren, Werner Helmich, and Dennis De Beurs. Dive into the event-based browsing of linked historical media. *Journal of Web Semantics*, 35:152–158, 2015.

[AA21] Martin Doerr. The cidoc conceptual reference module: an ontological approach to semantic interoperability of metadata. *AI magazine*, 24(3):75–75, 2003.

[AA22] Martin Doerr, Chryssoula Bekiari, Patrick LeBoeuf, and Bibliothèque nationale de France. Frbroo, a conceptual model for performing arts. In *2008 Annual Conference of CIDOC, Athens*, pages 15–18, 2008.

[AA23] Martin Doerr, Stefan Gradmann, Steffen Hennicke, Antoine Isaac, Carlo Meghini, and Herbert Van de Sompel. The europeana data model (edm). In *World Library and Information Congress: 76th IFLA general conference and assembly*, volume 10, page 15. IFLA, 2010.

[AA24] Astrid Duque-Ramos, Jesualdo Tomás Fernández-Breis, Robert Stevens, and Nathalie Aussenac-Gilles. OQuaRE: A SQuaRE-based Approach for Evaluating the Quality of Ontologies. *Journal of Research and Practice in Information Technology*, 43(2), 2011.

[AA25] Ahmet Ekin, A Murat Tekalp, and Rajiv Mehrotra. Integrated semantic-syntactic video modeling for search and browsing. *IEEE Transactions on Multimedia*, 6(6):839–851, 2004.

[AA26] Said Fathalla, Sahar Vahdati, Christoph Lange, and Sören Auer. Seo: A scientific events data model. In *International semantic web conference*, pages 79–95. Springer, 2019.

[AA27] Dèlia Fernàndez-Cañellas, Joan Espadaler, David Rodriguez, Blai Garolera, Gemma Canet, Aleix Colom, Joan Marco Rimmek, Xavier Giro-i Nieto, Elisenda Bou, and Juan Carlos Riveiro. VLX-Stories: Building an Online Event Knowledge Base with Emerging Entity Detection. In Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtěch
Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, and Fabien Gandon, editors, *The Semantic Web – ISWC 2019*, pages 382–399, Cham, 2019. Springer International Publishing.

[AA28] Erwin Filtz, María Navas-Loro, Cristiana Santos, Axel Polleres, and Sabrina Kirrane. Events matter: Extraction of events from court decisions. *Legal Knowledge and Information Systems*, pages 33–42, 2020.

[AA29] Alexandre RJ Francois, Ram Nevatia, Jerry Hobbs, Robert C Bolles, and John R Smith. Verl: an ontology framework for representing and annotating video events. *IEEE multimedia*, 12(4):76–86, 2005.

[AA30] Aldo Gangemi, Nicola Guarino, Claudio Masolo, Alessandro Oltramari, and Luc Schneider. Sweetening ontologies with dolce. In *International conference on knowledge engineering and knowledge management*, pages 166–181. Springer, 2002.

[AA31] Aldo Gangemi and Peter Mika. Understanding the semantic web through descriptions and situations. In *OTM Confederated International Conferences *On the Move to Meaningful Internet Systems*”, pages 689–706. Springer, 2003.

[AA32] Joost Geurts, Stefano Bocconi, Jacco van Ossenbruggen, and Lynda Hardman. Towards Ontology-Driven Discourse: From Semantic Graphs to Multimedia Presentations. In Dieter Fensel, Katia Sycara, and John Mylopoulos, editors, *The Semantic Web - ISWC 2003*, pages 597–612, Berlin, Heidelberg, 2003. Springer.

[AA33] Simon Gottschalk and Elena Demidova. Eventkg: A multilingual event-centric temporal knowledge graph. In *European Semantic Web Conference*, pages 272–287. Springer, 2018.

[AA34] Simon Gottschalk and Elena Demidova. EventKG - the Hub of Event Knowledge on the Web - and Biographical Timeline Generation. volume 10, pages 1039–1070. IOS Press, 2019.

[AA35] Saiping Guan, Xueqi Cheng, Long Bai, Fujun Zhang, Zixuan Li, Yutao Zeng, Xiaolong Jin, and Jiafeng Guo. What is event knowledge graph: A survey. *IEEE Transactions on Knowledge and Data Engineering*, 2022.

[AA36] Asunción Gómez-Pérez. Ontology Evaluation. In Steffen Staab and Rudi Studer, editors, *Handbook on Ontologies*, pages 251–273. Springer, Berlin, Heidelberg, 2004.

[AA37] Hlomani Hlomani and Deborah Stacey. Approaches, methods, metrics, measures, and subjectivity in ontology evaluation: A survey. *Semantic Web Journal*, 1(5):1–11, 2014.

[AA38] Eero Hyvönen, Petri Leskinen, Minna Tamper, Heikki Rantala, Esko Ikkala, Jouni Tuominen, and Kirsi Keravuori. Biographysampo–publishing and enriching biographies on the semantic web for digital humanities research. In *European Semantic Web Conference*, pages 574–589. Springer, 2019.

[AA39] Krzysztof Janowicz, Armin Haller, Simon JD Cox, Danh Le Phuoc, and Maxime Lefrançois. Sosa: A lightweight ontology for sensors, observations, samples, and actuators. *Journal of Web Semantics*, 56:1–10, 2019.

[AA40] Krzysztof Janowicz, Pascal Hitzler, Benjamin Adams, Dave Kolas, and Charles Vardeman II. Five stars of Linked Data vocabulary use. *Semantic Web*, 5(3):173–176, January 2014. Publisher: IOS Press.
[AA41] VenuMadhav Kattagoni and Navjyoti Singh. Irevent2story: A novel mediation ontology and narrative generation. In *Text2Story@ ECIR*, pages 5–14, 2018.

[AA42] Mayank Kejriwal, Jing Peng, Haotian Zhang, and Pedro Szekely. Structured Event Entity Resolution in Humanitarian Domains. In Denny Vrandečić, Kalina Bontcheva, Mari Carmen Suárez-Figueroa, Valentina Presutti, Irene Celino, Marta Sabou, Lucie-Aimée Kaffee, and Elena Simperl, editors, *The Semantic Web – ISWC 2018*, pages 233–249, Cham, 2018. Springer International Publishing.

[AA43] Anas Fahad Khan, Andrea Bellandi, Giulia Benotto, Francesca Frontini, Emiliano Giovannetti, and Marianne Reboul. Leveraging a narrative ontology to query a literary text. In *7th Workshop on Computational Models of Narrative (CMN 2016)*. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016.

[AA44] Houda Khrouf and Raphaël Troncy. EventMedia: A LOD dataset of events illustrated with media. *Semantic Web*, 7(2):193–199, January 2016. Publisher: IOS Press.

[AA45] Robert Kowalski and Marek Sergot. A logic-based calculus of events. *New generation computing*, 4:67–95, 1986.

[AA46] Bernhard Krabina. Building a Knowledge Graph for the History of Vienna with Semantic MediaWiki. *Journal of Web Semantics*, 76:100771, April 2023.

[AA47] Franz Krause, Kabul Kurniawan, Elmar Kiesling, Heiko Paulheim, and Axel Polleres. On the Representation of Dynamic BPMN Process Executions in Knowledge Graphs. In Fernando Ortiz-Rodriguez, Boris Villazón-Terrazas, Sanju Tiwari, and Carlos Bobed, editors, *Knowledge Graphs and Semantic Web*, pages 97–105, Cham, 2023. Springer Nature Switzerland.

[AA48] Carl Lagoze and Jane Hunter. The abc ontology and model. 2002.

[AA49] K Faith Lawrence, Mischa M Tuffield, Mike O Jewell, Adam Prugel-Bennett, David E Millard, Mark S Nixon, Nigel R Shadbolt, et al. Ontomedia-creating an ontology for marking up the contents of heterogeneous media. 2005.

[AA50] Vincenzo Lombardo, Rossana Damiano, and Antonio Pizzo. Drammar: A comprehensive ontological resource on drama. In *International Semantic Web Conference*, pages 103–118. Springer, 2018.

[AA51] William C Mann and Sandra A Thompson. *Rhetorical structure theory: A theory of text organization*. University of Southern California, Information Sciences Institute Los Angeles, 1987.

[AA52] Valadis Mastoras, Alexandros Vassiliades, Maria Rousi, Sotiris Diplaris, Thanassis Mavropoulos, Ilias Gialampoukidis, Stefanos Vrochidis, and Ioannis Kompatsiaris. Towards a Framework for Seismic Data. In Fernando Ortiz-Rodriguez, Boris Villazón-Terrazas, Sanju Tiwari, and Carlos Bobed, editors, *Knowledge Graphs and Semantic Web*, pages 106–119, Cham, 2023. Springer Nature Switzerland.

[AA53] Amed Leiva Mederos, Doymer García-Duarte, Daniel Galvez Lio, Yusniel Hidalgo-Delgado, and José Antonio Senso Ruíz. An Ontological Model for the Failure Detection in Power Electric Systems. In Boris Villazón-Terrazas, Fernando Ortiz-Rodríguez, Sanju M. Tiwari, and Shishir K. Shandilya, editors, *Knowledge Graphs and Semantic Web*, pages 130–146, Cham, 2020. Springer International Publishing.

[AA54] Carlo Meghini, Valentina Bartalesi, and Daniele Metilli. Representing narratives in digital libraries: The narrative ontology. *Semantic Web*, 12(2):241–264, 2021.
[AA55] Albert Meroño-Peñuela, Ashkan Ashkpour, Marieke van Erp, Kees Mandemakers, Leen Breure, Andrea Scharnhorst, Stefan Schlobach, and Frank van Harmelen. Semantic technologies for historical research: A survey. *Semantic Web*, 6(6):539–564, January 2015. Publisher: IOS Press.

[AA56] Ben Miller, Antonio Lieto, Rémi Ronfard, Stephen Ware, and Mark Finlayson. Proceedings of the 7th workshop on computational models of narrative. In *7th Workshop on Computational Models of Narrative (CMN 2016)*, volume 53, 2016.

[AA57] Paul Mulholland, Trevor Collins, and Zdenek Zdrahal. Story fountain: intelligent support for story research and exploration. In *Proceedings of the 9th international conference on Intelligent user interfaces*, pages 62–69, 2004.

[AA58] Paul Mulholland, Annika Wolff, and Trevor Collins. Curate and Storyspace: An Ontology and Web-Based Environment for Describing Curatorial Narratives. In Elena Simperl, Philipp Cimiano, Axel Polleres, Oscar Corcho, and Valentina Presutti, editors, *The Semantic Web: Research and Applications*, pages 748–762, Berlin, Heidelberg, 2012. Springer.

[AA59] Paul Mulholland, Annika Wolff, Eoin Kilfeather, and Evin McCarthy. Using Event Spaces, Setting and Theme to Assist the Interpretation and Development of Museum Stories. In Krzysztof Janowicz, Stefan Schlobach, Patrick Lambrix, and Eero Hyvönen, editors, *Knowledge Engineering and Knowledge Management*, pages 320–332, Cham, 2014. Springer International Publishing.

[AA60] Arturo Nakasone and Mitsuru Ishizuka. Storytelling ontology model using rst. In *2006 IEEE/WIC/ACM International Conference on Intelligent Agent Technology*, pages 163–169. IEEE, 2006.

[AA61] Dominic Oldman and Diana Tanase. Reshaping the Knowledge Graph by Connecting Researchers, Data and Practices in ResearchSpace. In Denny Vrandečić, Kalina Bontcheva, Mari Carmen Suárez-Figueroa, Valentina Presutti, Irene Celino, Marta Sabou, Lucie-Aimée Kaffee, and Elena Simperl, editors, *The Semantic Web – ISWC 2018*, pages 325–340, Cham, 2018. Springer International Publishing.

[AA62] Edward T O'Neill. Frbr: Functional requirements for bibliographic records. *Library resources & technical services*, 46(4):150–159, 2011.

[AA63] Matthew J. Page, David Moher, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and Joanne E. McKenzie. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. *BMJ*, 372:n160, March 2021. Publisher: British Medical Journal Publishing Group Section: Research Methods & Reporting.

[AA64] Franziska Pannach, Caroline Sporleder, Wolfgang May, Aravind Krishnan, and Anusharani Sewchurran. Of lions and Yakshis. *Semantic Web*, 12(2):219–239, January 2021. Publisher: IOS Press.

[AA65] Federico Peinado, Pablo Gervás, and Belén Díaz-Agudo. A description logic ontology for fairy tale generation. In *Procs. of the Workshop on Language Resources for Linguistic Creativity*, LREC, volume 4, pages 56–61, 2004.
[AA66] Rajesh Piryani, Nathalie Aussenac-Gilles, and Nathalie Jane Hernandez. Comprehensive Survey on Ontologies about Event. In *ESWC Workshops on Semantic Methods for Events and Stories (SEMMES @ ESWC 2023)*, volume 3443, pages 1–15, Hersonissos, Greece, May 2023. ceur-ws.org. Backup Publisher: Pasquale Lisena and Ilaria Tiddi and Simon Gottschalk and Luc Steels.

[AA67] Robert Porzel. On formalizing narratives. In *JOWO*, 2021.

[AA68] Vladimir Propp. *Morphology of the Folktale*. University of texas Press, 1968.

[AA69] Polina Proutskova, Daniel Wolff, György Fazekas, Klaus Frieler, Frank Höger, Olga Velichkina, Gabriel Solis, Tillman Weyde, Martin Pfleiderer, Hèlène Camille Crayencour, Geoffroy Peeters, and Simon Dixon. The Jazz Ontology: A semantic model and large-scale RDF repositories for jazz. *Journal of Web Semantics*, 74:100735, October 2022.

[AA70] Joe Raad and Christophe Cruz. A Survey on Ontology Evaluation Methods. In *Proceedings of the International Conference on Knowledge Engineering and Ontology Development, part of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management*, Lisbonne, Portugal, November 2015.

[AA71] Yves Raimond and Samer Abdallah. The event ontology. 2007.

[AA72] Youssra Rebboud, Pasquale Lisena, and Raphael Troncy. Beyond causality: Representing event relations in knowledge graphs. In *International Conference on Knowledge Engineering and Knowledge Management*, pages 121–135. Springer, 2022.

[AA73] Marco Rospocher, Marieke van Erp, Piek Vossen, Antske Fokkens, Itziar Aldabe, German Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard. Building event-centric knowledge graphs from news. *Journal of Web Semantics*, 37:132–151, 2016.

[AA74] Carlo Romano Marcello Alessandro Santagiustina, Laura Spillner, Thomas Mildner, Robert Porzel, et al. Towards conflictual narrative mechanics. In *CEUR WORKSHOP PROCEEDINGS*, pages 19–27. CEUR-WS, 2022.

[AA75] Roger C Schank and Robert P Abelson. Scripts, plans, and knowledge. In *IJCAI*, volume 75, pages 151–157. New York, 1975.

[AA76] Ansgar Scherp, Thomas Franz, Carsten Saathoff, and Steffen Staab. F-a model of events based on the foundational ontology dolce+ dns ultralight. In *Proceedings of the fifth international conference on Knowledge capture*, pages 137–144, 2009.

[AA77] Stijn Schouten, Victor De Boer, Lodewijk Petram, and Marieke Van Erp. The wind in our sails: developing a reusable and maintainable dutch maritime history knowledge graph. In *Proceedings of the 11th on Knowledge Capture Conference*, pages 97–104, 2021.

[AA78] Ryan Shaw, Raphaël Troncy, and Lynda Hardman. Lode: Linking open descriptions of events. In *Asian semantic web conference*, pages 153–167. Springer, 2009.

[AA79] Cogan Shimizu, Pascal Hitzler, Quinn Hirt, Dean Rehberger, Seila Gonzalez Estrecha, Catherine Foley, Alicia M. Sheill, Walter Hawthorne, Jeff Mixter, Ethan Watrall, Ryan Carty, and Duncan Tarr. The enslaved ontology: Peoples of the historic slave trade. *Journal of Web Semantics*, 63:100567, August 2020.

[AA80] Luc Steels, Lara Verheyen, and Remi Van Trijp. An Experiment in Measuring Understanding. In Stefan Schlobach, María Pérez-Ortiz, and Myrthe Tielman, editors, *Frontiers in Artificial Intelligence and Applications*. IOS Press, September 2022.
[AA81] Lise Stork, Ilaria Tiddi, René Spijker, and Annette Ten Teije. Explainable Drug Repurposing in Context via Deep Reinforcement Learning. In Catia Pesquita, Ernesto Jimenez-Ruiz, Jamie McCusker, Daniel Faria, Mauro Dragoni, Anastasia Dimou, Raphael Troncy, and Sven Hertling, editors, *The Semantic Web*, volume 13870, pages 3–20. Springer Nature Switzerland, Cham, 2023. Series Title: Lecture Notes in Computer Science.

[AA82] Roland Stühmer, Darko Anicic, Sinan Sen, Jun Ma, Kay-Uwe Schmidt, and Nenad Stojanovic. Lifting Events in RDF from Interactions with Annotated Web Pages. In Abraham Bernstein, David R. Karger, Tom Heath, Lee Feigenbaum, Diana Maynard, Enrico Motta, and Krishnaprasad Thirunarayan, editors, *The Semantic Web - ISWC 2009*, pages 893–908, Berlin, Heidelberg, 2009. Springer.

[AA83] Fabian M Suchanek. The need to move beyond triples. In *Text2Story@ ECIR*, pages 95–104, 2020.

[AA84] Ivo Swartjes and Mariët Theune. A fabula model for emergent narrative. In *International Conference on Technologies for Interactive Digital Storytelling and Entertainment*, pages 49–60. Springer, 2006.

[AA85] Tom Trabasso, Paul Van den Broek, and So Young Suh. Logical necessity and transitivity of causal relations in stories. *Discourse processes*, 12(1):1–25, 1989.

[AA86] Raphaël Troncy, Giuseppe Rizzo, Anthony Jameson, Oscar Corcho, Julien Plu, Enrico Palumbo, Juan Carlos Ballesteros Hermida, Adrian Spirescu, Kai-Dominik Kuhn, Catalin Barbu, Matteo Rossi, Irene Celino, Rachit Agarwal, Christian Scanu, Massimo Valla, and Timber Haaker. 3cixty: Building comprehensive knowledge bases for city exploration. *Journal of Web Semantics*, 46-47:2–13, October 2017.

[AA87] Willem Robert Van Hage, Véronique Malaisé, Roxane Segers, Laura Hollink, and Guus Schreiber. Design and use of the simple event model (sem). *Journal of Web Semantics*, 9(2):128–136, 2011.

[AA88] Udaya Varadarajan and Biswanath Dutta. Models for narrative information: a study. *arXiv preprint arXiv:2110.02084*, 2021.

[AA89] Óscar Vilarroya. Somos lo que nos contamos. *Cómo los relatos construyen el mundo en que vivimos*., *Editorial Ariel, Barcelona*, 2019.

[AA90] Denny Vrandečić. Ontology Evaluation. In Steffen Staab and Rudi Studer, editors, *Handbook on Ontologies*, pages 293–313. Springer, Berlin, Heidelberg, 2009.

[AA91] Xiang-jun Wang, Swathi Mamadgi, Atit Thekdi, Aisling Kelliher, and Hari Sundaram. Eventory–an event based media repository. In *International Conference on Semantic Computing (ICSC 2007)*, pages 95–104. IEEE, 2007.

[AA92] Utz Westermann and Ramesh Jain. Toward a common event model for multimedia applications. *IEEE multimedia*, 14(1):19–29, 2007.

[AA93] Jian-hua Yeh. Storyteller: An event-based story ontology composition system for biographical history. In *2017 International Conference on Applied System Innovation (ICASI)*, pages 1934–1937. IEEE, 2017.

[AA94] Marcos Zárate, Pablo Rosales, Germán Braun, Mirtha Lewis, Pablo Rubén Fillot-trani, and Claudio Delrieux. OceanGraph: Some Initial Steps Toward a Oceanographic Knowledge Graph. In Boris Villazón-Terrazas and Yusniel Hidalgo-Delgado, editors,
*Knowledge Graphs and Semantic Web*, pages 33–40, Cham, 2019. Springer International Publishing.
# Chapter 2

## ChronoGrapher: Event-centric
Knowledge Graph Construction via
Informed Graph Traversal

Inès Blin*†, Ilaria Tiddi†, Remi van Trijp*, Annette ten Teije†

*Sony Computer Science Laboratories - Paris

†Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

### Abstract

Event-centric knowledge graphs help enhance coherence to otherwise fragmented and overwhelming data by establishing causal and temporal connections using relevant data. We address the challenge of automatically constructing event-centric knowledge graphs from generic ones. We present ChronoGrapher, a two-step system to build an event-centric knowledge graph from grand events such as the French Revolution. First, an informed graph traversal retrieves connected sub-events from large, open-domain knowledge graphs. We define event-centric filters to prune the search space and a heuristic ranking to prioritise nodes like events. Second, we combine a rule-based system and information extraction from text to build event-centric knowledge graphs. ChronoGrapher demonstrates adaptability across datasets like DBpedia and Wikidata, outperforming approaches from the literature. To evaluate the utility of these graphs, we conduct a preliminary user study comparing different prompting techniques for event-centric question-answering. Our results demonstrate that prompts enriched with event-centric knowledge graph triples yield more factual answers compared to those enriched with generic knowledge graph triples or base prompts, achieving groundedness scores of 2.85, 2.24, and 1.11 respectively, while preserving succinctness and relevance.

**Keywords** Event-centric KGs, Graph traversal, Knowledge graphs

## 2.1 Introduction

We tackle the problem of building event-centric knowledge graphs (KGs) automatically starting from generic KGs. An event-centric KG can be seen as a KG that focuses on events as central entities, rather than generic entities. An event is an occurrence or happening that can be described by its attributes and relationships to other entities within a knowledge graph. It includes information such as time, location, participants, and other relevant properties. Relating events together provide coherence to otherwise fragmented and overwhelming data by
establishing causal and temporal connections using relevant data. There are numerous scenarios in which systems could assist individuals in constructing event-centric KGs in their tasks: in news article retrieval for event-centric exploration [76], in the legal domain where experts must associate events that could bolster their case [15], or in the historical domain when generating hypotheses by retrieving valuable information from archives [50].

Let us imagine a historian seeking to understand the events that happened at the end of the French Revolution. Our system aims to produce an event-centric KG, as illustrated in Figure 2.1, which we manually created using mainly the SEM [75] ontology and Wikidata. Figure 2.1 shows that the *Coup of 18 Brumaire directlyPrecedes* the *French Consulate*, and that Napoleon was a *participant* in the *Coup of 18 Brumaire*, and *First Consul* during the *French Consulate*. We can therefore infer that the *Coup of 18 Brumaire* marked the beginning of the *French Consulate* and enabled Napoleon to become *First Consul*.

<figure><img src="image_7.png" /><figcaption>Figure 2.1: One example of an event-centric KG on the Coup of 18 Brumaire.</figcaption></figure>

Building an event-centric KG can be seen as a 2-step process: searching cues in a massive memory (step-1), and analysing and reasoning about these cues (step-2) [67]. KGs have been used as background knowledge to support the construction of event-centric KGs [23]. In this context, these two steps can be translated into identifying relevant content from a generic KG (step-1), and linking this content with additional information to create an event-centric KG (step-2). Most existing event-centric KG resources are however built from annotated text [42, 53, 28] or from generic KGs [20] (step-2), but do not focus on identifying relevant events for event-centric KG construction (step-1). Likewise, event subgraph extraction (step-1) has been used for downstream tasks such as question-answering (QA) [68, 35, 44], but does not focus on building event-centric KGs (step-2). We propose ChronoGrapher that both retrieves relevant events and generates an event-centric KG, as well as an evaluation of an event-centric KG constructed by ChronoGrapher in a QA setting in the form of a user study. We make our code openly available<fnref n="1" />.

Our objective is to extract sequences of events as a plausible answer for a question, such as summarising the French Revolution, or discussing whether Napoleon and Paul Barras were both involved in the Coup of 18 Brumaire. We define three research questions: (RQ1) *How to extract relevant events from a generic KG?* (step-1) (RQ2) *How to create an event-centric KG from these events?* (step-2) (RQ3) *How to evaluate the utility of the event-centric KG*

<footnote n="1">¹https://github.com/SonyCSLParis/graph_search_framework/tree/main</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 2.2: Overview of the diverse tasks we tackle to answer our research questions. The example on the bottom is with the French Revolution as starting node and DBpedia as the KG. ChronoGrapher integrates both a subgraph extraction step through a semantically informed best-first search, and a KG construction step through a rule-based system and information extraction from text. The downstream task is event-centric question-answering.</figcaption></figure>

*constructed by ChronoGrapher?* Figure 2.2 provides an overview of the different tasks we tackle, as well as the input and output of each components.

For **RQ1**, ChronoGrapher contains an informed graph traversal to retrieve connected sub-events from large, open-domain KGs. It is a *semantically-informed* best-first search. First, it uses *event-centric filters* to prune the search space and to guide the exploration of the graph, and it identifies which ones are the most relevant to build an event-centric KG. Second, it adds a ranking step to prioritise nodes, which, to the best of our knowledge, has not been used in search systems starting from one node in KGs. We define the ranking heuristics based upon syntactic and semantic properties of the graph.

For **RQ2**, ChronoGrapher combines a rule-based system on triples with information extraction from text to generate an event-centric KG. The rule-based systems converts the triples from the original KG into event-centric ones. Furthermore, since the KGs can contain valuable information in text format, we include information extraction from text. We evaluate ChronoGrapher and adapted methods from the literature [33, 18] against ground truth events from EventKG [20]. ChronoGrapher shows adaptability and efficiency across datasets like DBpedia and Wiki-data, outperforming existing methods. Concerning the technical novelty of the work, methods for representing events in KGs are underdeveloped, manually built and time-consuming, and ChronoGrapher is a method for building event-centric KGs automatically from a generic KG. Unlike most approaches that relied on manual efforts or solely on text extraction, ChronoG-
rapher integrates data from existing KGs and texts, and results in more concise and precise
outputs than baselines adapted from existing systems. The modularity of our approach ensures
flexibility and adaptability, while the novel combination and adaptation of previously uninte-
grated elements contributes to the field and improves the adapted baselines [33, 18].

For **RQ3**, we evaluate the event-centric KGs built with ChronoGrapher via a QA downstream task. A human evaluation is conducted to compare three types of zero-shot QA prompts (without explicit context, with triples from generic KGs, with triples from event-centric KGs) and six types of different questions. Our results demonstrate that prompts enriched with event-centric knowledge graph triples yield more factual answers compared to those enriched with generic knowledge graph triples or base prompts, achieving groundedness scores of 2.85, 2.24, and 1.11 respectively, while preserving succinctness and relevance.

## 2.2 Related Work

New approaches have emerged to build event-centric KGs with various applications (i). To the best of our knowledge, building event-centric KGs *from generic KGs automatically* is novel. We therefore adapt and improve existing work to harness the potential of event-centric KGs (ii). To assess the usefulness of event-centric KGs, we lastly present works that used KGs to improve the quality of answers generated by large language models (LLMs) (iii).

(i) **Event-centric KGs.** Recent research on KGs emphasises the need to have more event-centric structures to facilitate the analysis of sequence of events [61]. Event-centric KGs were built in various domains: news [52, 41, 53, 43], political events [5], novels [38], the biomedical domain [47], tourism [79], the legal domain [15], olfactory heritage [45] and plenary debates [31]. More particularly in the digital humanities and cultural heritage domain, knowledge graph resources are emerging to link objects to their events and underlying stories. This is for example the case for Arco, an Italian cultural heritage KG [6], that was built on XML date using the eXtreme Design methodology [4]. Another application is to better understand history, may it be through the history of a city like Amsterdam [54] or through historical archives [65, 74, 72, 51]. [74] focused on event extraction and modelling from text archive, and [72] proposes an agile cataloguing process to build a knowledge graph. The work that is the most closely related to ours is EventKG [13], an event-centric KG that was built manually starting from large, generic KGs, such as DBpedia or Wikidata.

We attempt to build EventKG automatically, and use this dataset as golden standard in our evaluation (see Section 2.4.2). The above resources extracted events without a focus on identifying relevant events for an event-centric KG. Our approach targets the extraction of relevant events locally.

There are different applications of using narrative graph representations. [25] worked on enhancing sequential pattern mining with time and reasoning, to analyse complex dynamical systems. Understanding such systems can help to acquire new knowledge but also to predict and control its behaviour. Event-centric representations can also help in decision-making, for instance in the epidemiological domain [3] or maritime transportation [12]. [13] lastly uses structured narrative representations to build hypotheses over knowledge bases.

(ii) **Methods for Event-centric KG construction.** We present different research domains that are relevant to building event-centric KGs: linked-traversal based methods, graph navigational languages and entity relatedness and summarisation. We also compare it to ChronoGrapher.

**Linked-traversal based methods** identify and extract the most relevant set of information
for a given task. One of the pioneering systems in this domain is LDSpider [33], which employs diverse search strategies such as breadth-first search or load-balancing to crawl web data. Subsequent systems aimed to reduce the search space during the traversal, encoding features like property selection [64] or basic graph patterns [73]. Systems like Linked Data-fu [69] and LiDAQ [73] added reasoning capabilities to their approaches. Other systems like SQUIN [27] focused on optimising query execution efficiency.

**Graph navigational languages** are used to explore Linked Data iteratively by providing instructions to retrieve nodes or subgraphs. There are two types of approaches, namely SPARQL extensions [17, 26, 60] and standalone languages [18, 28]. Most of these approaches use regular expressions, like SPARQLeR [39] and PSPARQL [1], and later nested regular expressions, like [80] and nSPARQL [56] to explore the graph. Although our approach is more similar to a link traversal-based method, we compare the scope and output of our method to those of navigational languages. Our approach can handle local graphs and any SPARQL endpoint, and the output of our method is regions, similar to GeL [16] and NautiLOD [18]. The most expressive system appears to be LDQL [28] that separates the pattern processing and the source selection. **Entity Relatedness** systems aim to determine the optimal path connecting two nodes in a KG [36]. It involves computing all paths with search heuristics and activation criteria to prune the search space and ranking paths [31]. Search heuristics include bidirectional search [13, 71], backward search heuristic [30], SPARQL queries [59], exploratory association search [10], and A* [11]. Activation criteria include entity clustering [10], metrics based on node degree [13, 52] or ontology [48], and similarity measures [30, 11] like Jaccard, Wikipedia link-based measures [78] or SimRank [34]. For ranking paths, metrics include PF-ITF [59], exclusivity-based relatedness [30] and point mutual information [32]. Table 2.1 provides an overview of the entity relatedness systems we described. **Entity Summarization** systems [46] take a KG as input and output a graph that keeps certain original properties of the graph. The methods used to select nodes are similar to the ranking ones from entity relatedness systems.

<table><caption>Table 2.1: Overview of entity relatedness systems.</caption><thead><tr><th>System</th><th>Path Search</th><th>Activation Criteria</th><th>Ranking</th></tr></thead><tbody><tr><td>[71]<br>Profiler [30]</td><td>Bidirectional search<br>Backward search heuristic</td><td>Primitives<br>Similarity measures</td><td>Cost functions<br>N/A</td></tr><tr><td>RECAP [59]<br>EXPLASS [10]</td><td>SPARQL queries<br>Exploratory association search</td><td>N/A<br>Clustering entities</td><td>Path informativeness<br>N/A</td></tr><tr><td>[11]<br>[52]</td><td>A*<br>Shortest paths</td><td>Jaccard-based metrics<br>Low-degree nodes</td><td>Top k paths<br>Top k paths</td></tr><tr><td>REX [13]<br>RelFinder [48]</td><td>Bidirectional search<br>All paths</td><td>Degree of node<br>Class, link and length filter</td><td>Interestingness measure<br>N/A</td></tr></tbody></table>

Our method traverses KGs to extract relevant events for event-centric KG building, akin to link traversal-based methods and graph navigational languages. It handles local graphs from any SPARQL endpoint, producing subgraphs (like [16] and [18]). Addressing limitations in expressiveness and navigation control of link-traversal methods, our approach provides enhanced control and reasoning. It employs filters for search space pruning without prior knowledge of predicates, unlike graph navigational languages requiring regular expressions. Additionally, we introduce a ranking step inspired by entity relatedness and summarization systems, prioritizing nodes based on a cost function. To the best of our knowledge, this feature is absent in existing navigational systems. Unlike entity relatedness systems that pre-compute all paths between entities, our system dynamically computes paths. Furthermore, our approach requires only one starting node as input, with target nodes defined as variables. It extracts subgraphs from
the original KG, aiming to retrieve all events of a given entity, disregarding specific property considerations as in entity summarization systems.

**(iii) Retrieval Augmented Generation (RAG).** Studies have shown that LLMs store factual knowledge in their parameters [58], e.g. in the form of implicit knowledge bases [62]. However, the mechanisms to retrieve such knowledge remain opaque. RAG systems [9] aim to provide a more interpretable way to access knowledge by combining a generative model and an explicit *knowledge retriever component*. Such systems tend to generate more specific, diverse and factual language [43], and have proven beneficial for downstream tasks such as QA [8] or reasoning [70]. Systems like ORQA [41] or REALM [24] learn the retriever and the reader jointly, whereas systems like DrQA [8] have a fixed document retrieval component. [37] proposes a dense passage retriever to index all documents in a latent space. [57] augments the reader with a retriever in an unsupervised way.

In our user study, we compare three types of zero-shot prompting for QA: one prompting without explicit context, and two prompting with triples from generic KGs and event-centric KGs respectively. Our *knowledge retriever component* is based on SPARQL queries, to retrieve triples from a knowledge base.

## 2.3 ChronoGrapher

As displayed in Figure 2.2 from Section 2.1, ChronoGrapher takes an event IRI as input, searches all sub-events of that event, and outputs an event-centric KG. It is flexible and can be applied to retrieve any type of node, such as people or places, and to any input KG. We present preliminary content (Section 2.3.1), the semantically-informed graph traversal method to extract relevant events (RQ1) (Section 2.3.2) and the methods to build an event-centric KG (RQ2) (Section 2.3.3).

### 2.3.1 Preliminaries

We first introduce the generic concepts of knowledge graph, event-centric knowledge graph and event.

**[Knowledge Graph]** A **knowledge graph** *KG* is an RDF graph where the nodes represent entities (subjects or objects) and literals (objects) and the edges represent the relationships (predicates) between these entities and literals. Formally, $KG = \{(s, p, o) | s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$, where $\mathcal{E}$ is the set of entities (nodes) in the KG, $\mathcal{L}$ is the set of literals in the KG, and $\mathcal{R}$ is the set of predicates (edges) representing relationships in the KG.

**[Event-centric KG and Event]** In this work, an **event-centric knowledge graph** is a KG whose vocabulary, classes and relationships originate from the SEM [75] and the NIF [18] ontologies. There are four core classes in SEM: **sem:Event** (what), **sem:Actor** (who), **sem:Place** (where) and **sem:Time** (when). The NIF ontology enables to integrate text data in KGs. An **event** is furthermore defined as an occurrence in time that can be characterised by attributes from the SEM ontology, such as actor and location.

We second formally define concepts that are useful for our link traversal-based method, before presenting ChronoGrapher into details: the ingoing and outgoing triples of a node in a KG, node expansion, best-first search algorithm, event retrieval path and relation patterns. Please note that these definitions are generic, and we later instantiate and use them to build the event-centric KGs (cf. Section 2.4.1 for the experimental settings).
[Ingoing triples] Let $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$, and $P(KG) = \{A|A \subseteq KG\}$ the set of all subsets of $KG$. The **ingoing triples** of a node $n$ are formally defined as:

$$ 
\begin{array}{l}
\text{ingoing}_{KG}: \mathcal{E} \to \mathcal{P}(KG) \\
n \mapsto \{(s, p, n) \mid \exists p \in \mathcal{R}, \exists o \in \mathcal{E}, \text{ such that } (s, p, n) \in KG\}
\end{array}
$$

[Outgoing triples] Let $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$, and $P(KG) = \{A|A \subseteq KG\}$ the set of all subsets of $KG$. The **outgoing triples** of a node $n$ are formally defined as:

$$ 
\begin{array}{l}
\text{outgoing}_{KG}: \mathcal{E} \to \mathcal{P}(KG) \\
n \mapsto \{(n, p, o) \mid \exists p \in \mathcal{R}, \exists o \in \mathcal{E} \cup \mathcal{L}, \text{ such that } (n, p, o) \in KG\}
\end{array}
$$

[Node Expansion] **Expanding a node** $n$ means retrieving its ingoing and outgoing triples.

[Best-first search algorithm] A **best-first search algorithm** expands unvisited nodes with the highest scores based on a predefined heuristic.

[Event-retrieval task] Given $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$ and an event $s$, the aim of the **event retrieval task** is to traverse $KG$ starting from $s$ to retrieve the sub-events of $s$, $e = \{e_1, ..., e_n\}$. For evaluation, $e$ is compared to ground truth sub-events.

[Relation Patterns and Node Satisfaction] Let $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$. We define the following relation patterns in $KG$ as Boolean functions:

* **Single-step outgoing predicate pattern**: Let a predicate $p \in \mathcal{R}$, we formally define $opp_{p,KG}$ as follows:

$$ 
\begin{array}{l}
opp_{p,KG}: \mathcal{E} \to \{0, 1\} \\
n \mapsto \begin{cases} 1 & \text{if } \exists o \in \mathcal{E} \text{ such that } (n, p, o) \in KG, \\ 0 & \text{otherwise.} \end{cases}
\end{array}
$$

* **Single-step ingoing predicate pattern**: Let a predicate $p \in \mathcal{R}$, we formally define $ipp_{p,KG}$ as follows:

$$ 
\begin{array}{l}
ipp_{p,KG}: \mathcal{E} \to \{0, 1\} \\
n \mapsto \begin{cases} 1 & \text{if } \exists s \in \mathcal{E} \text{ such that } (s, p, n) \in KG, \\ 0 & \text{otherwise.} \end{cases}
\end{array}
$$

* **Single-step outgoing predicate-object pattern**: Let a predicate $p \in \mathcal{R}$ and an entity $e \in \mathcal{E}$, we formally define $opop_{p,e,KG}$ as follows:

$$ 
\begin{array}{l}
opop_{p,e,KG}: \mathcal{E} \to \{0, 1\} \\
n \mapsto \begin{cases} 1 & \text{if } (n, p, e) \in KG, \\ 0 & \text{otherwise.} \end{cases}
\end{array}
$$

* **Single-step ingoing predicate-object pattern**: Let a predicate $p \in \mathcal{R}$ and an entity $e \in \mathcal{E}$, we formally define $ipop_{p,e,KG}$ as follows:

$$ 
\begin{array}{l}
ipop_{p,e,KG}: \mathcal{E} \to \{0, 1\} \\
n \mapsto \begin{cases} 1 & \text{if } (e, p, n) \in KG, \\ 0 & \text{otherwise.} \end{cases}
\end{array}
$$

For each pattern defined above, a node $n \in \mathcal{E}$ satisfies this pattern if its image is 1.
<figure><img src="image_9.png" /><figcaption>Figure 2.3: Description of one iteration during the informed graph traversal. For clarification, we denote as *DB* the set of triples in DBpedia. The type of ranking is predicate-object frequency. The *WHERE*, *WHAT* and *WHEN* filters are activated (formally defined as *e_filter*{*db:Location*}, *DB*,*rdf:type*, *p_filter*{*rdf:type*} and *t_filter*{*db:startDate*,1789-05-05,*db:endDate*,1799-12-31 respectively).</figcaption></figure>

2.3.2 Relevant Content Extraction (RQ1)

The first component of ChronoGrapher is a modified best-first search that extracts relevant content to build an event-centric KG. We define generic filters to prune the search space, and instantiate them as event-centric filters for the experimental settings (Section 2.4.1). We use a combination of ranking metrics and preference function. As illustrated in Figure 2.3, each iteration in the graph search process is composed of four stages. During Stage 1, nodes with the highest ranking are expanded, ie. their ingoing and outgoing nodes are retrieved. During Stage 2, we apply filters to remove newly visited ingoing and outgoing nodes from the search space. During Stage 3, we update the ingoing and outgoing nodes that were not discarded and the subgraph output. During Stage 4, we score and rank patterns. Both filters and ranking are detailed later in this Section.

Algorithm 2.3.2 presents our four-step method: (1) nodes with the highest ranking are explored in the graph, (2) a filtering step using event-centric filters removes newly visited nodes from the search space, (3) the pending nodes and subgraph output are updated in the third step, (4) nodes in the pending set are scored and ranked, and those with the highest score are selected for the next iteration. *nodes* is initialised in line 2 of the Algorithm ; it is the set of nodes that will be expanded at the next iteration and is considered the priority queue of the pending set of nodes *P* initialised in line 1.

<footnote n="2">²Images taken from https://www.flaticon.com/freeicons/watch and https://www.flaticon.com/freeicons/output</footnote>
**input:** $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$: knowledge graph to traverse,
$n_{start}, t_{start}, t_{end}$: node to start the search, its start and end dates $n$: number of iteration,
**to_extract**: type of nodes to extract,
filtering, preference function, scoring: defined in Section 2.3.

**output:** $e = \{e_1, ..., e_n\}$: events extracted with the traversal,
[1] initialise $\mathcal{P}$: list of pending nodes that could be expanded
initialise $\mathcal{nodes} = \{n_{start}\}$: list of nodes to be expanded

[h!]
initialise $\mathcal{N}_{expanded}$: nodes visited throughout the traversal
initialise $e$: events extracted with the traversal $i \in \{1, ..., n\}$ node $\in \mathcal{nodes}$ add node in $\mathcal{N}_{expanded}$
$\mathit{ingoing} = \mathit{Ingoing}(\mathit{node}, KG)$ Expanding
$\mathit{outgoing} = \mathit{Outgoing}(\mathit{node}, KG)$ Expanding
$\mathit{to\_add\_output}, \mathit{pending} = \mathit{Filter}(\mathit{ingoing}, \mathit{outgoing})$ Filtering
add $\mathit{pending}$ to $\mathcal{P}, \mathit{to\_add\_output}$ to $e$ Updating $\mathcal{nodes} = \mathit{selectNodesToExpand}(\mathcal{P})$ Scoring
and Preference Function
return $e$

## Event-centric filters.

We introduce the filters to prune the search space in Definition 2.3.2. The definitions of the filters are generic, but the examples we give and the filters we use in our experiments are event-centric because we define them following the SEM [75] ontology, further detailed in Section 2.3.1. Each of the filters is defined according to one of the four core classes of SEM. These filters could be implemented using filters in graph navigational languages like NautiLOD [18], or using SPARQL queries. However, the latter methods do not permit to rank patterns. In step 2 of Figure 2.3, filters function dynamically. Expanding the search to other KGs only requires identifying predicates for temporal data and classes for locations or people. We further detail the filters' definitions.

[Event-centric filters] We hereafter define three types of event-centric filters: predicate-based, entity-based and temporal-based. Each filter is a boolean function that enables to prune the search space in a knowledge graph $KG = \{(s, p, o)|s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$.

### Predicate-based filter.

Let $\mathcal{R}' \subset \mathcal{R}$ and $(s, p, o) \in KG$. The predicate-based filter discards $(s, p, o)$ from the search space if $p \in \mathcal{R}'$. It can be formally defined as:

$$p\_filter_{\mathcal{R}'}: KG \rightarrow \{0, 1\} \\ (s, p, o) \mapsto \begin{cases} 1 & \text{if } p \in \mathcal{R}' \\ 0 & \text{else} \end{cases}$$

* **WHAT-filter.** For example, $\mathcal{R}' = \{rdf:type\}$.

### Entity-based filter.

Let $\mathcal{E}' \subset \mathcal{E}, p \in \mathcal{R}, s \in \mathcal{E}$. The entity-based filter discards $s$ from the search space if $s$ is of any type in $\mathcal{E}'$. It can be formally defined as:

$$e\_filter_{\mathcal{E}', KG, p}: \mathcal{E} \rightarrow \{0, 1\} \\ s \mapsto \begin{cases} 1 & \text{if } \exists t \in \mathcal{E}', \text{ such that } (s, p, t) \in KG \\ 0 & \text{else} \end{cases}$$

* **WHO-filter.** For example, $\mathcal{E}' = \{dbo:Person\}, p = rdf:type$.

* **WHERE-filter.** For example, $\mathcal{E}' = \{dbo:Location\}, p = rdf:type$.
Temporal-based filter.

Let $\mathcal{L}_{date} \subseteq \mathcal{L}$ be the subset of literals corresponding to dates. Let $(ps, pe) \in \mathbb{R}^2$, $(ts, te) \in \mathcal{L}_{date}^2$. $ps$ and $pe$ are predicates denoting the start and end dates of an events, and $ts$ and $te$ are literal timestamps. The temporal-based filter discards $s \in \mathcal{E}$ from the search space if it ends before $ts$ or if it starts after $te$. It can be formally defined as:

$$ t\_filter_{ps,ls,pe,le}: \mathcal{E} \to \{0, 1\} \\ s \mapsto \begin{cases} 1 & \text{if } \exists t_1 \in \mathcal{L}_{date}, \text{ such that } (s, ps, t_1) \in \mathcal{KG} \text{ and } t_1 > le \\ & \text{or } \exists t_2 \in \mathcal{L}_{date}, \text{ such that } (s, pe, t_2) \in \mathcal{KG} \text{ and } t_2 < ls, \\ 0 & \text{otherwise.} \end{cases} $$

* **WHEN-filter.** For example, $ps = dbo:birthDate$, $pe = dbo:deathDate$, $ts = "1789-05-05"$, $te = "1799-12-31"$.

The above filters are parameters of the traversal, and can be activated on demand. In Figure 2.3, the filters we activated are WHAT, WHERE, and WHEN (formally defined as $e\_filter_{\{dbo:Location\},DB,rdf:type}$, $p\_filter_{rdf:type}$ and $t\_filter_{dbo:startDate,1789-05-05,dbo:endDate,1799-12-31}$ respectively). For WHAT, `dbo:MilitaryConflict` is discarded from the search space, since it is the object of the triple (`dbr:French_Revolution`, `rdf:type`, `dbo:MilitaryConflict`). Regarding WHERE, `Kingdom_of_France` is discarded since it is of type `dbo:Loca`

tion. Lastly, `Jean-Jacques_Rousseau` is discarded because Rousseau died before the French Revolution started, hence applying the WHEN filter.

tion. Lastly, `Jean-Jacques_Rousseau` is discarded because Rousseau died before the French Revolution started, hence applying the WHEN filter.

Scoring and Preference Function.

This step ranks patterns, and selects the nodes to expand. For the highest scored pattern, all nodes that satisfy the same pattern in the graph are expanded. If there is no remaining pattern and nodes to be expanded, the search stops. There are two core elements: scoring and preference function.

[Scoring] Let $\mathcal{G}'$ a subset of $\mathcal{KG}$.

We first define scoring metrics for ingoing and outgoing predicate patterns. Let $p, p \in \mathbb{R}$, and the patterns $ipp_p$ and $opp_p$ from Definition 2.3.1

* **Predicate Frequency (pf).** Edges with predicates that are often used in the dataset will be favoured.

$$ pf(ipp_p) = |\{s | \exists o \in \mathcal{E}, \text{ such that } (s, p, o) \in \mathcal{G}'\}| \quad (2.1) $$

$$ pf(opp_p) = |\{o | \exists s \in \mathcal{E}, \text{ such that } (s, p, o) \in \mathcal{G}'\}| \quad (2.2) $$

* **Entropy Predicate Frequency (epf).** Edges with the most informative predicates in the dataset will be favoured. Let $pp \in \{ipp_p, opp_p\}$.

$$ epf(pp) = -\frac{pf(pp)}{|\mathcal{G}'|} \times \log\left(\frac{pf(pp)}{|\mathcal{G}'|}\right) \quad (2.3) $$

* **Inverse Predicate Frequency (ipf).** Edges with predicates that are rarely used in the dataset will be favoured. Let $pp \in \{ipp_p, opp_p\}$.

$$ ipf(pp) = \frac{1}{pf(pp)} \quad (2.4) $$
We second define scoring metrics for ingoing and outgoing predicate-object patterns. Let $p$, $p \in \mathcal{R}$, $e$, $e \in \mathcal{E}$, and the patterns $ipop_{p,e}$ and $opop_{p,e}$ from Definition 2.3.1

* Predicate-Object Frequency ($pof$). This is similar to $pf$, but this metrics also differentiates between the objects.

$$pof(ipop_{p,e}) = |\{s | (s, p, e) \in \mathcal{G}'\}| \qquad (2.5)$$

$$pof(opop_{p,e}) = |\{o | (e, p, o) \in \mathcal{G}'\}| \qquad (2.6)$$

* Entropy Predicate-Object Frequency ($epof$). This is similar to $epf$, but this metrics also differentiates between the objects. Let $pop \in \{ipop_{p,e}, opop_{p,e}\}$.

$$epof(pop) = -\frac{pof(pop)}{|\mathcal{G}'|} \times \log\left(\frac{pof(pop)}{|\mathcal{G}'|}\right) \qquad (2.7)$$

* Inverse Predicate-Object Frequency ($ipof$). This is similar to $ipf$, but this metrics also differentiates between the objects. Let $pop \in \{ipop_{p,e}, opop_{p,e}\}$.

$$ipof(pop) = \frac{1}{pof(pop)} \qquad (2.8)$$

The scoring metric used in Figure 2.3 is $pof$. The score of the pattern $ipop_{dbo:isPartOfMilitaryConflict,dbr:French_Revolution}$ is 2 since there are two ingoing nodes of the $dbr:French_Revolution$ that are connected to the latter with the predicate $dbo:isPartOfMilitaryConflict$. If the scoring metric was $epof$, the score would be $-\frac{2}{3} \times \log \frac{2}{3} \approx 0.12$.

[Preference Function] The preference function is a boolean function that prioritises a pattern if it meets certain conditions. Let $KG = \{(s, p, o) | s \in \mathcal{E}, o \in \mathcal{E} \cup \mathcal{L}, p \in \mathcal{R}\}$ a KG. Let $d \in \mathcal{R}$ and $sc \in \mathcal{R}$ denoting predicates for domain and subclass information. Let $t_e \in \mathcal{E}$ a type of events to retrieve. Let $p \in \mathcal{R}$, $e \in \mathcal{E}$ and a pattern $pp$, $pp \in \{ipp_p, opp_p, ipop_{p,e}, opop_{p,e}\}$. We formally define:

$$ \begin{array}{l} pref\_fpp_{d,sc,t_e}: \{ipp_p, opp_p, ipop_{p,e}, opop_{p,e}\} \to \{0,1\} \\ \qquad pp \mapsto \left\{ \begin{array}{ll} 1 & \text{if } \exists n \in \mathcal{E}, \text{ such that } (p,d,n) \in KG \\ & \text{and}(n,sc,t_e) \in KG \\ 0 & \text{otherwise.} \end{array} \right. \end{array} $$

* For example, $d = rdfs:domain$, $sc = rdfs:subClassOf$, $t_e = dbo:Event$.

The preference function prioritises predicates with the highest scores from the scoring step, favouring nodes with the correct types. We use the ontological equivalent predicates of rdfs:domain and rdfs:range. In DBpedia, we have (dbo:isPartOf MilitaryConflict, rdfs:domain, dbo:MilitaryConflict) and (dbo: MilitaryConflict, rdfs:subClassOf*, dbo:Event). Consequently, in (s, dbo:isPartOfMilitaryConflict, o), s must be of type dbo:Event. With the preference function, the system prefers patterns with dbo:isPartOfMilitaryConflict over dbo:battle since the former has dbo:MilitaryConflict as its domain, a subclass of event type, while the latter lacks explicit information about its domain or range. In Figure 2.3, the preference function does not affect the selection of nodes to expand. Without the preference function, patterns are ranked in descending order of scores from the scoring step.
### 2.3.3 Event-centric KG Population (RQ2)

The second component of ChronoGrapher builds event-centric KGs by combining a rule-based system with information extraction from text. The rule-based system converts the triples into SEM triples based on the predicate labels, aiming to reproduce the KGs from EventKG [20]. An example can be seen in Figure 2.4. On the left side of the figure there is a subgraph taken from the Wikidata page of the Coup of 18 Brumaire (wd:Q620965). On the right side of the figure, there is event-centric KG following the SEM ontology.

<figure><img src="image_10.png" /><figcaption>Figure 2.4: Populating a SEM KG from the Coup of 18 Brumaire in Wikidata. wd is a prefix used in Wikidata for https://www.wikidata.org/wiki/.</figcaption></figure>

Table 2.2 shows the labels that were used to retrieve information from triples. If a predicate contained any of the labels, it would add information in the graph based on its narrative dimension and SEM predicate. As an example, let us take an input triple (s, p, o). If the string of p contains the substring “person”, then a triple (s, sem:hasActor, o) is added to the output graph<fnref n="3" />.

<table><caption>Table 2.2: Labels used to retrieve information from KG triples.</caption><thead><tr><th>Narrative dimension</th><th>SEM predicate</th><th>Labels</th></tr></thead><tbody><tr><td>Who</td><td>sem:hasActor</td><td>person, combatant, commander, participant</td></tr><tr><td>When (begin)</td><td>sem:hasBeginTimeStamp</td><td>start time, date, point in time</td></tr><tr><td>When (end)</td><td>sem:hasEndTimeStamp</td><td>end time</td></tr><tr><td>Where</td><td>sem:hasPlace</td><td>place, location, country</td></tr><tr><td>Part of</td><td>sem:subEventOf</td><td>partof, part of</td></tr><tr><td>Part of (inverse)</td><td>sem:hasSubEvent</td><td>has part, significant event</td></tr></tbody></table>

ChronoGrapher goes beyond EventKG and adds structured information from text. We mainly use the frame semantics theory [14], which introduce frames, that can be seen as templates with roles that need to be filled in. Some event-centric KGs built from news articles [42, 28] also used frames. Although EventKG contains text events and links them to entities, it does not provide the finer-grained semantic structure of frames. One example of frames is the *Change_of_leadership* frame<fnref n="4" />. It describes “the appointment of a *New_leader* or removal from office of an *Old_leader*”. Semantic roles such as *Old_leader* and *New_leader* are called frame elements.

The step of the extraction of frames and frame elements is done automatically. We use a pre-trained transformer-based model [7] to extract the frames and the frame elements. We use the NIF [18] ontology to encode text content, and DBpedia Spotlight [17] to extract the entities from frame elements. Each frame and frame element is added to the knowledge

<footnote n="3">³The rules can be found in the code, in the file "src/build ng/generic kb to ng.py"</footnote>

<footnote n="4">⁴https://framenet2.icsi.berkeley.edu/fnReports/data/frameIndex.xml?frame=Change_of_leadership</footnote>
graph. We lastly link these frames to Framester [16], and generate annotations using the same structure as Framester<fnref n="5" />,⁶.

An example is shown in Figure 2.5. It begins with the abstract from `dbr:Coup_of_18_Brumaire`, which is broken down into sentences like `ex:S1`. The frame annotation `ex:FNAnnot1` is connected to the frame template `frame:Cause_to_end` and further linked to roles such as `ex:E1`, representing *the French Revolution*. This literal is then connected to the IRI entity `dbr:French_Revolution` via DBpedia Spotlight. The frame knowledge in the KG population (from RQ2) is used to answer the event-centric questions in RQ3.

<figure><img src="image_11.png" /><figcaption>Figure 2.5: Example of frames extracted from text with the *Coup of 18 Brumaire*.</figcaption></figure>

## 2.4 Evaluation

We present the setup (Section 2.4.1), and results on the quality of the method (Section 2.4.2) and the usefulness of the constructed event-centric KGs (Section 2.4.3). The quantitative evaluation (Section 2.4.2, RQ1 and RQ2) focuses on evaluating the quality of the method, both for the traversal (RQ1) and the KG population (RQ2). The evaluation for RQ1 is done automatically and can be reproduced. The evaluation for RQ2 is partly done automatically, and partly manually assessed. For RQ3, we conducted a preliminary user study to assess the potential of event-centric KGs on a QA setting, and we found encouraging qualitative results.

### 2.4.1 Experimental Setting (RQ1, RQ2)

Our experiments first aim to assess the quality of ChronoGrapher, both for relevant event extraction (RQ1) and for event-centric KG population (RQ2). For RQ1, we investigate the effects of filtering and ranking on the graph traversal approach. We first experiment under different configurations, with six distinct parameters for the search process: (1) scoring metric, (2) preference function (cfr. Section 2.3.2), (3-6) filters pertaining to WHAT, WHERE, WHEN, and WHO (Definition 2.3.2). The scoring metrics is one of the six options described above (Definition 2.3.2), whereas all the other parameters are boolean. We ran our experiments on two Ubuntu machines with 40 CPUs, and 348GiB and 320GiB of memory.

<footnote n="5">⁵http://etna.istc.cnr.it/framesterpage/wsj/wsjpropnetannotations/CE_55094
⁶The code for this part is in the "src/buildNg/frameSemantic.py" file.</footnote>
**Baselines.** For RQ1 on relevant event extraction, we compare *chronographer* with three categories of system we implemented. We adapted LDSpider [33] and NautiLOD [18] to align with our method. The difference in performance between ChronoGrapher and the considered baselines is solely assessed on the event extraction part.

* random-5, random-10 and random-15: at each iteration, *n* nodes are randomly selected to be expanded. We used *n* ∈ {5, 10, 15}. Parameters (1-6) are non applicable.

* ldspider-m: inspired by LDSpider [33], this search is a breadth-first search with some predefined predicates to be pruned out of the search space. Parameters (1) and (2) are not applicable, and (3-6) are set to 0.

* nautilod-m: inspired by NautiLOD [18], this baseline works as a breadth-first search combined with more complex filters than ldspider-m. Parameters (1) and (2) are not applicable, but we experiment with (3-6).

**Evaluation.** We used EventKG 3.1 [22] as ground truth since it was manually constructed. For RQ1 on relevant event extraction, we compare the sub-events extracted with our method to the ones accessible with EventKG. We use SPARQL queries to retrieve the start and end dates, and the sub-events of all events. For RQ2 on event-centric KG population, we construct a KG from EventKG that is directly comparable to our event-centric KGs. We compute F1, Precision and Recall as metrics.

**Datasets.** Our experiments focus on DBpedia (2021-09) [2], Wikidata (2021-03-05) [77] and YAGO4 [55] because EventKG was built on them. We use the HDT version of DBpedia, which we obtained from TriplyDB<fnref n="7" />. We obtain Wikidata from rdfhdt<fnref n="8" /> and YAGO4 from their official website<fnref n="9" />.

Table 2.3 shows information on events and their sub-events across datasets. All types of events are taken into account, which mainly include historical events, sports events and political events such as elections. We choose to retain solely the historical events with more than ten sub-events, as they offer more than a simple list of events. The final distribution of historical events in EventKG and their average number of outgoing links are presented in Table 2.4. Events with a higher number of sub-events tend to have a higher average number of features. The limited number of outgoing links for YAGO4 resulted in traversals that would end after one iteration. YAGO4 was built on Wikidata with an improved schema, without the full content from Wikidata, and we found that the ingoing links were mostly about creative works, and the outgoing links about metadata, discarded for the search. This resulted in a limited number of features. We thus only used Wikidata and DBpedia.

<table><caption>Table 2.3: Statistics on all events across datasets.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="4">Nb. of sub-events</th></tr><tr><th>= 1</th><th>>1</th><th>>10</th><th>Final</th></tr></thead><tbody><tr><td>Wikidata</td><td>203,988</td><td>238,094</td><td>2,408</td><td>341 DBpedia</td></tr><tr><td>84,599</td><td>95,504</td><td>1,333</td><td>250</td><td></td></tr><tr><td>YAGO4</td><td>70,738</td><td>76,682</td><td>993</td><td>306</td></tr></tbody></table>

<footnote n="7">⁷https://triply.cc</footnote>

<footnote n="8">⁸https://www.rdfhdt.org/datasets/</footnote>

<footnote n="9">⁹https://yago-knowledge.org/downloads/yago-4</footnote>
<table><caption>Table 2.4: Number of filtered events and average number of features for events with more than *n* sub-events.</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="5">Nb. of sub-events</th><th colspan="5">Avg. nb. of features</th></tr><tr><th>&gt;10</th><th>&gt;20</th><th>&gt;30</th><th>&gt;50</th><th>&gt;100</th><th>&gt;10</th><th>&gt;20</th><th>&gt;30</th><th>&gt;50</th><th>&gt;100</th></tr></thead><tbody><tr><td>Wikidata</td><td>341</td><td>208</td><td>147</td><td>91</td><td>47</td><td>237</td><td>273</td><td>290</td><td>323</td><td>381</td></tr><tr><td>DBpedia</td><td>275</td><td>158</td><td>106</td><td>66</td><td>37</td><td>122</td><td>133</td><td>141</td><td>147</td><td>160</td></tr><tr><td>YAGO4</td><td>306</td><td>180</td><td>126</td><td>76</td><td>44</td><td>5</td><td>4</td><td>5</td><td>5</td><td>5</td></tr></tbody></table>

**Filters and preference function definition.** The filters from Definition 2.3.2 were generic with some event-centric examples for the WHAT, WHERE, WHEN and WHO filters. Table 2.5 describes the predicates we used for each of the filters. Each filter relates to one of the core classes of the SEM ontology [75], making them event-centric. However, other filters can also be implemented by instantiating other variables, making the method easily expandable and usable for other tasks.

## 2.4.2 Quantitative Evaluation (RQ1, RQ2)

### Parameter selection for the search (RQ1).

For *chronographer* and *nautilod-m*, we first experimented on 12 events to select the most meaningful parameters to run the graph traversal on all events. Table 2.6 shows the 12 events used for parameter selection for the search. There were 192 possible combinations of parameters ($6 \times 2^5$) for *chronographer*, and 16 ($2^4$) for *nautilod-m*.

We investigated on the correlation between our ranking and filtering parameters and the best F1 score. For *chronographer*, we found a strong positive correlation between *domain_range* and higher F1 scores. This suggests that prioritizing events based on their domain range significantly improves event retrieval performance. We observed a weak but statistically significant correlation between *when* and the F1 score, suggesting that adding temporal information could help the search process. For *nautilod-m*, we found a strong positive correlation between *what* and the best F1 score. The *what* filter indeed enables to remove more generic nodes from the search, which proves its efficiency here.

Table 2.7 shows the contributions of each filter individually. As an illustration, *what* = 1 means that only the experiments with this parameter activated, and all the others set to 0, are taken into account. For *nautilod-m*, the parameter that yields the best result is *what*, with an average F1 of 0.36. The best average F1 score is obtained when all parameters are activated. On average, *chronographer* performs better or comparably to *nautilod-m*, under all configurations. The parameter that yields the best results is *domain_range*.

For computational and performance reasons, we set the following parameters: **domain_range** = **when** = **what** = **where** = **who** = 1. We kept all metrics but the inverse ones for *chronographer* for the full experiments.

Lastly, Table 2.8 shows the results of all systems. We set a timeout of 10 hours for each experiment. Only 5 experiments from *ldspider-m* timed out. In these cases, we took the latest output of the search at that stage. Our systems achieve better result than the baselines. The search is furthermore more localised and efficient, since the output graphs and the runtime are smaller.
## Search Results (RQ1).

Table 2.9 shows the final results on the remaining 604 events for all systems. The three metrics tend to improve when moving from the *random* baselines to our *chronographer* methods. *random-5* has the worst F1 scores, and *chronographer-epof* has the best ones. The differences are the most visible in the Recall scores, which means that the *informed* systems leave out less false negative than the random baselines. Precision scores for Wikidata tend to be higher than the ones for DBpedia, whereas it is the reverse for Recall. Therefore, the content retrieved in Wikidata is less noisy but also less complete. The *random* baselines and *ldspider-m* yield significantly bigger graphs than *nautilod-m* and the *chronographer* baselines, and take more time to run. *nautilod-m* performs better than the above baselines, however the size of the graphs and the runtime remain higher than the *chronographer* baselines (×4).

To summarise findings on relevant event extraction (RQ1), adding event-centric filters (WHAT, WHERE, WHEN, WHO) and a ranking step with an entropy predicate-object frequency heuristics helps to select relevant events while being more efficient time-wise and while exploring smaller regions of the graphs, as shown by the better performance of the *chronographer* systems in Table 2.9.

## Event-centric KG Population Results (RQ2).

To generate our event-centric KGs, we use information from triples (i) and text (ii).

### (i) Constructing event-centric KG from triples.
We generate event-centric KGs for each event using all sub-events from the ground truth. We compare the number of common triples against EventKG. The results are shown in Table 2.10 ChronoGrapher achieves an overall F1 score of 67.2% and 76% for DBpedia and Wikidata. Precision scores for DBpedia are higher than for Wikidata, which means that the event-centric KGs are less noisy but less complete. For Wikidata, it is the opposite.

Table 2.11 shows the F1, Precision and Recall scores of the narrative graphs generated from the output of the search algorithm presented in Section 2.3.2. Our end-to-end system achieves an overall (for all predicates) F1 score of 51.7% and 49.2% respectively. As in Table 2.10, precision is higher for DBpedia, while recall tends to be higher for Wikidata. The results are furthermore lower than those in Table 2.10, which is expected since the output of the search also contains events that are not in the ground truth events. Consequently, for each of such event, all generated triples will not be in the ground truth from EventKG.

### (ii) Constructing event-centric KG from text.
We generate event-centric KGs from all the DBpedia events and sub-events that contain an abstract. The content that is extracted in this part is not present in EventKG. In total, this sums up to 9,438 events. On average, each event had 18 distinct frames and 48 instantiations of frames with at least one mapped role. The frames that appear the most in terms of number are: *Military* (26,681), *Hostile_encounter* (26,111), *Political_locales* (17,513), *Attack* (16,155), and *Leadership* (14,002). We are interested in events and relations between events, hence we focus on "causation" frames for a qualitative analysis, but many other frames are extracted. We manually annotate (1) whether there is a causation frame (2) whether the cause and the effect are correctly identified (3) whether the DBpedia entity is correct for 100 frame annotations. Out of 100 frames, 97 had at least one identified Cause or Effect. 85 were correctly identified as Causation frames. False positive causation frames often contain causative lemmas such as the noun "cause" that might have induced the model in error. Among the 171 retrieved frame elements, 151 (88.3%) were
correct. The majority of wrong frame elements resulted from false positive causation frames. Among the 94 DBpedia entities that were identified, 80 (85.1%) were correct.

To summarise findings on event-centric KG population (RQ2), we use both information ex-
traction from triples and text. For triples, our system achieves F1 scores of 67.2% and 76%
for DBpedia and Wikidata. For text, we conduct a qualitative evaluation on the “causation”
frames and find that the system is able to retrieve information with good performance.

2.4.3 User Study (RQ3)

To evaluate the utility of event-centric KGs on a QA setting (RQ3), we designed 6 types of
event-centric questions: summary of an event, causal explanations for an event, types of events
within a time-range, sub-events of an event, events in which an actor participated, and events
in which two actors both participated. We prompted GPT-4¹⁰ in three ways: (a) **base**: base
prompt, without context triples, (b) **db-kg**: **base** + context triples from DBpedia, and (c)
**ec-kg**: **base** + context triples from the event-centric KG built by ChronoGrapher.

We gathered 10 participants for our user studies. We reached the participants of the user study throughout our research labs. The researchers who participated are based in Paris or Amsterdam, and they have a background in computer science and/or computational linguistics. They were not overly familiar with the questions we had them evaluated, and were familiar with LLMs systems and their pitfalls.

For the context triples, we manually implemented the SPARQL queries to retrieve triples from DBpedia or the event-centric KG<sup>11</sup>. The participants were asked to assess the quality of the answers with these criteria: (1) granularity of the info, (2) relevance to the question (3) succinctness and (4) diversity in content, on a scale of 1 to 5. We furthermore manually assessed (5) grounding to factual events. A QA system must most importantly give faithful content, making the groundedness metric (5) a priority. The relevance metric measures how well the answer provides a plausible answer to the question, but not its actual truthfulness. We designed 2 forms<sup>12, 13</sup> with 6 questions each, and with 3 answers per question corresponding to the three prompt types. We split the 10 participants evenly across these 2 forms. Each participant rated 18 answers (6 question types × 3 prompt types). Each prompt was assessed 60 times for metrics 1-4 (6 × 10), and 12 times for metric 5 (6 × 2). Given the scale, we solely focused on the French Revolution.

Table 2.12 presents the average scores for each metric and prompt type. The **base** prompt has the lowest groundedness which undermines its reliability, and we found that it was prone to hallucinations, often about actors outside the LLM’s implicit knowledge base. This might be linked to its lowest granularity and highest succinctness scores, due to having less content to work with. For the remainder of the analysis, we discard the **base** prompt because of its low groundedness score.

When comparing the triples-based prompts, **ec-kg** stands out with the highest groundedness (+27%) and succinctness (+8%) scores, indicating that its generated answers are both more faithful and concise. Additionally, **ec-kg** scores slightly higher in relevance (+2.5%) compared to **db-kg**, though **db-kg** achieves a higher diversity (+3.8%) score, likely due to its more varied context triples, and a slightly higher granularity score (+0.5%). Overall, **ec-kg** emerges as the most promising approach, balancing faithfulness, relevance, and conciseness effectively.

This first small user study thus hints that event-centric KGs can help an LLM provide more

<footnote n="10">¹⁰https://openai.com/research/gpt-4
¹¹The queries can be found in the code: "experiments run/usageNg/sparql_queries.py"
¹²Form 1 link: https://forms.gle/1SJTtTMUUM4uh7KT6
¹³Form 2 link: https://forms.gle/Rtgj7ucpdC1jm3tc6</footnote>
factual and concise answers to event-centric questions (2.85) than a base prompt (1.11) or a
generic KG-based prompt (2.24), while maintaining succinctness.

## 2.5 Conclusion

We address the challenge of automatically constructing event-centric KGs from generic ones. We present ChronoGrapher that extracts relevant content and builds an event-centric KG. To extract relevant content (RQ1), ChronoGrapher contains an informed graph traversal strategy integrating event-centric filters to prune the search space and a ranking step for node prioritisation, yielding F1 scores of 0.78 and 0.68 for DBpedia and Wikidata. To generate event-centric KGs (RQ2), ChronoGrapher combines a rule-based system that achieves F1 scores of 0.67 and 0.76 for DBpedia and Wikidata, and information extraction from text. To evaluate the utility of event-centric KGs (RQ3), we conduct preliminary first experiments comparing different prompts on an event-centric QA setting, and show that prompts enriched with event-centric KG triples give more factual answers while maintaining succinctness (2.95 vs. 1.11 and 2.24). Future work will focus on improving ChronoGrapher in a number of directions, including expanding the search method to reach more sub-events using e.g. EventKG for additional type information; expanding the event-centric KG population beyond EventKG to tackle data noise and incompleteness; improving the modelling of complex entities; considering novel ontologies and vocabulary for the event representation (e.g. RDF-star); and use more sophisticated RAG-based methods for a larger user study. We also plan to work on additional quantitative and automatic metrics for a better evaluation of the event-centric KGs, since the long-term availability of the OpenAI service is uncertain and the human ratings are hard to reproduce.

**Supplemental Material Statement:** Source code for our system, the baselines we (re-)implemented and the experiments is available on Github<sup>14</sup>. The source code contains the predicate labels that were used to generate the event-centric KG (Section 2.3.3), pointers to download the datasets and more generic statistics on events and their sub-events (Section 2.4.1), the 12 events for parameter selection (Section 2.4.2), additional results on experiments on the end-to-end system combining the event extraction and the graph generation and the manual annotations for the causation frames (Section 2.4.2), and all code, prompts and data linked to the user study (Section 2.4.3).

<footnote n="14"><sup>14</sup>https://github.com/SonyCSLParis/graph_search_framework/tree/main</footnote>
<table><caption>Table 2.5: Instantiated variables used for each of the filters and the preference function.</caption><thead><tr><th>Name</th><th>Type</th><th>Dataset</th><th>Variable</th><th>Predicates</th></tr></thead><tbody><tr><td rowspan="2">WHAT</td><td rowspan="2">predicate filter</td><td>DB</td><td rowspan="2">R'</td><td>rdf:type</td></tr><tr><td>WD</td><td>wp:P31</td></tr><tr><td rowspan="4">WHO</td><td rowspan="4">entity filter</td><td>DB</td><td>ε'</td><td>dbo:Person</td></tr><tr><td rowspan="3">WD</td><td>p</td><td>rdf:type</td></tr><tr><td>ε'</td><td>wd:Q5</td></tr><tr><td>p</td><td>wd:P31</td></tr><tr><td rowspan="6">WHERE</td><td rowspan="6">entity filter</td><td>DB</td><td>ε'</td><td>dbo:Place<br>dbo:Location</td></tr><tr><td rowspan="5">WD</td><td>p</td><td>rdf:type</td></tr><tr><td>ε'</td><td>wd:P17<br>wp:P276<br>wd:Q6256</td></tr><tr><td>p</td><td>wp:P31</td></tr><tr><td>ps</td><td>dbp date<br>dbp:COMMON<br>dbp:BirthDate</td></tr><tr><td>pe</td><td>dbp:COMMON<br>dbp:DeathDate</td></tr><tr><td rowspan="6">WHEN</td><td rowspan="6">temporal filter</td><td>DB</td><td>ps</td><td>dbp:P585<br>dbp:P580<br>dbp:P569</td></tr><tr><td rowspan="3">WD</td><td>pe</td><td>dbp:P582<br>dbp:P570</td></tr><tr><td>d</td><td>rdfs:domain</td></tr><tr><td>sc</td><td>rdfs:subClassOf</td></tr><tr><td rowspan="3">Preference Function</td><td rowspan="3">-</td><td>DB</td><td>t<sub>e</sub></td><td>dbo:Event</td></tr><tr><td rowspan="3">WD</td><td>d</td><td>wd:Q21503250</td></tr><tr><td>sc</td><td>wp:P279</td></tr><tr><td></td><td></td><td>t<sub>e</sub></td><td>wd:Q13418847</td></tr></tbody></table>

<table><caption>Table 2.6: Events used for parameter selection.</caption><tr><td>World War I</td><td>American Indian Wars</td><td>Mediterranean and Middle East Theatre of World War II,</td></tr><tr><td>French Revolution</td><td>Cold War</td><td>European Theatre of World War II</td></tr><tr><td>Napoleonic Wars</td><td>Coalition Wars</td><td>European Theatre of World War I</td></tr><tr><td>Pacific War</td><td>Russian Civil War</td><td>Yugoslav Wars</td></tr></table>
<table><caption>Table 2.7: Mean F1 score for individual parameters. *no filter* and *all* means that all parameters are set to 0 and 1 respectively. *pf*, *pof*, *epf*, *epof*, *ipf* and *ipof* are the six possible scoring metrics (Section 2.4.1).</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">nautilod-m</th><th colspan="5">chronographer</th></tr><tr><th>pf</th><th>pof</th><th>epf</th><th>epof</th><th>ipf</th><th>ipof</th></tr></thead><tbody><tr><td>domain_range = 1</td><td>N/A</td><td>0.66</td><td><strong>0.77</strong></td><td>0.66</td><td><strong>0.77</strong></td><td>0.66</td><td>0.65</td></tr><tr><td>what = 1</td><td>0.36</td><td>0.50</td><td>0.48</td><td>0.37</td><td>0.39</td><td>0.25</td><td>0.27</td></tr><tr><td>where = 1</td><td>0.28</td><td>0.47</td><td>0.48</td><td>0.33</td><td>0.39</td><td>0.25</td><td>0.23</td></tr><tr><td>when = 1</td><td>0.27</td><td>0.47</td><td>0.48</td><td>0.45</td><td>0.43</td><td>0.29</td><td>0.30</td></tr><tr><td>who = 1</td><td>0.25</td><td>0.43</td><td>0.50</td><td>0.39</td><td>0.48</td><td>0.26</td><td>0.26</td></tr><tr><td>no filter</td><td>0.31</td><td>0.46</td><td>0.48</td><td>0.34</td><td>0.39</td><td>0.24</td><td>0.24</td></tr><tr><td>all</td><td>0.53</td><td>0.69</td><td><strong>0.77</strong></td><td>0.69</td><td><strong>0.77</strong></td><td>0.69</td><td>0.72</td></tr></tbody></table>

<table><caption>Table 2.8: Results comparison of various systems.</caption><thead><tr><th>System</th><th>F1</th><th>Precision</th><th>Recall</th><th>Nb_Nodes</th><th>Nb_Preds</th><th>Nb_Triples</th><th>Runtime</th></tr></thead><tbody><tr><td>random-5</td><td>0.24</td><td>0.68</td><td>0.16</td><td>40,356</td><td>70</td><td>40,915</td><td>38'00"</td></tr><tr><td>random-10</td><td>0.31</td><td>0.54</td><td>0.24</td><td>1,554</td><td>70</td><td><strong>2,027</strong></td><td>43'00"</td></tr><tr><td>random-15</td><td>0.28</td><td>0.57</td><td>0.21</td><td>11,890</td><td>89</td><td>12,744</td><td>25'00"</td></tr><tr><td>ldspider-m</td><td>0.23</td><td>0.70</td><td>0.23</td><td>126,015</td><td>137</td><td>241,049</td><td>5'20"</td></tr><tr><td>nautilod-m</td><td>0.53</td><td>0.51</td><td>0.63</td><td>5,096</td><td>121</td><td>13,755</td><td>23'00"</td></tr><tr><td>chronographer-pf</td><td>0.69</td><td>0.70</td><td>0.71</td><td>874</td><td>21</td><td>2366</td><td>1'51"</td></tr><tr><td>chronographer-pof</td><td><strong>0.78</strong></td><td><strong>0.79</strong></td><td>0.77</td><td>750</td><td><strong>17</strong></td><td>2090</td><td>2'13"</td></tr><tr><td>chronographer-epf</td><td>0.69</td><td>0.70</td><td>0.72</td><td>857</td><td>20</td><td>2353</td><td>1'35"</td></tr><tr><td>chronographer-epof</td><td><strong>0.77</strong></td><td>0.78</td><td><strong>0.77</strong></td><td><strong>726</strong></td><td><strong>17</strong></td><td>2082</td><td><strong>1'42"</strong></td></tr></tbody></table>

<table><caption>Table 2.9: Systems results on the 604 events. *DB*: DBpedia, *WD*: Wikidata.</caption><thead><tr><th rowspan="2">System</th><th colspan="2">F1</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th colspan="2">Nb_Nodes</th><th colspan="2">Nb_Preds</th><th colspan="2">Nb_Triples</th><th colspan="2">Runtime</th></tr><tr><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th></tr></thead><tbody><tr><td>random-5</td><td>0.65</td><td>0.63</td><td>0.72</td><td>0.79</td><td>0.68</td><td>0.59</td><td>22,185</td><td>427</td><td>23</td><td>9</td><td>23,221</td><td>530</td><td>23'00"</td><td>11'00"</td></tr><tr><td>random-10</td><td>0.65</td><td>0.64</td><td>0.72</td><td>0.78</td><td>0.69</td><td>0.60</td><td>30,078</td><td>1,033</td><td>27</td><td>10</td><td>33,264</td><td>1,470</td><td>18'56"</td><td>11'33"</td></tr><tr><td>random-15</td><td>0.64</td><td>0.63</td><td>0.71</td><td>0.78</td><td>0.68</td><td>0.60</td><td>41,868</td><td>1,139</td><td>27</td><td>11</td><td>53,561</td><td>1,582</td><td>18'26"</td><td>16'43"</td></tr><tr><td>ldspider-m</td><td>0.64</td><td>0.63</td><td>0.73</td><td>0.78</td><td>0.68</td><td>0.61</td><td>131,607</td><td>4,992</td><td>52</td><td>20</td><td>239,548</td><td>7,389</td><td>27'05"</td><td>44'05"</td></tr><tr><td>nautilod-m</td><td>0.70</td><td>0.66</td><td>0.72</td><td>0.81</td><td>0.74</td><td>0.62</td><td>481</td><td>4,787</td><td>23</td><td>14</td><td>1,150</td><td>7,321</td><td>2'02"</td><td>39'26"</td></tr><tr><td>chronographer-pf</td><td>0.75</td><td>0.67</td><td>0.75</td><td>0.82</td><td>0.80</td><td>0.63</td><td>108</td><td>661</td><td>5</td><td>6</td><td>213</td><td>1,146</td><td>26"</td><td>9'38"</td></tr><tr><td>chronographer-pof</td><td>0.77</td><td><strong>0.68</strong></td><td>0.76</td><td><strong>0.83</strong></td><td><strong>0.82</strong></td><td><strong>0.64</strong></td><td><strong>97</strong></td><td><strong>612</strong></td><td>5</td><td>6</td><td><strong>196</strong></td><td><strong>863</strong></td><td>25"</td><td>8'43"</td></tr><tr><td>chronographer-epf</td><td>0.75</td><td>0.67</td><td>0.75</td><td><strong>0.83</strong></td><td>0.80</td><td>0.62</td><td>108</td><td>670</td><td>6</td><td>6</td><td>212</td><td>963</td><td>32"</td><td>9'07"</td></tr><tr><td>chronographer-epof</td><td><strong>0.78</strong></td><td><strong>0.68</strong></td><td><strong>0.77</strong></td><td><strong>0.83</strong></td><td><strong>0.82</strong></td><td>0.63</td><td>99</td><td>752</td><td>5</td><td>6</td><td>199</td><td>1,093</td><td><strong>24"</strong></td><td><strong>8'16"</strong></td></tr></tbody></table>
<table><caption>Table 2.10: Our event-centric KGs against EventKG. *DB*: DBpedia, *WD*: Wikidata.</caption><thead><tr><th rowspan="2">Predicate</th><th colspan="2">F1</th><th colspan="2">Precision</th><th colspan="2">Recall</th></tr><tr><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th></tr></thead><tbody><tr><td>all</td><td>67.2</td><td>76.0</td><td>92.1</td><td>72.1</td><td>52.9</td><td>80.4</td></tr><tr><td>sem:hasActor</td><td>65.0</td><td>43.6</td><td>90.4</td><td>58.1</td><td>50.7</td><td>34.9</td></tr><tr><td>sem:hasBeginTimeStamp</td><td>77.7</td><td>81.9</td><td>90.5</td><td>71.7</td><td>68.0</td><td>95.5</td></tr><tr><td>sem:hasEndTimeStamp</td><td>77.7</td><td>75.9</td><td>90.5</td><td>70.4</td><td>68.1</td><td>82.4</td></tr><tr><td>sem:hasPlace</td><td>63.6</td><td>90.3</td><td>95.0</td><td>92.1</td><td>47.8</td><td>88.6</td></tr></tbody></table>

<table><caption>Table 2.11: Metrics of generated narrative graphs compared to EventKG.</caption><thead><tr><th rowspan="2">Pred</th><th colspan="2">F1</th><th colspan="2">Precision</th><th colspan="2">Recall</th></tr><tr><th>DB</th><th>WD</th><th>DB</th><th>WD</th><th>DB</th><th>WD</th></tr></thead><tbody><tr><td>all</td><td>51.7</td><td>49.2</td><td>79.3</td><td>41.1</td><td>38.4</td><td>61.4</td></tr><tr><td>sem:hasActor</td><td>48.5</td><td>27.6</td><td>76.5</td><td>35.1</td><td>35.5</td><td>22.8</td></tr><tr><td>sem:hasBeginTimeStamp</td><td>62.6</td><td>50.3</td><td>79.5</td><td>38.0</td><td>51.6</td><td>74.4</td></tr><tr><td>sem:hasEndTimeStamp</td><td>62.7</td><td>48.3</td><td>79.5</td><td>38.1</td><td>51.7</td><td>65.9</td></tr><tr><td>sem:hasPlace</td><td>48.2</td><td>59.2</td><td>81.6</td><td>50.7</td><td>34.2</td><td>71.2</td></tr></tbody></table>

<table><caption>Table 2.12: Average scores for metrics on the human evaluation. The Groundedness metric is the most important one as it assesses the faithfulness of the answers.</caption><thead><tr><th>Type Prompt</th><th>Groundedness</th><th>Granularity</th><th>Relevance</th><th>Succinctness</th><th>Diversity</th></tr></thead><tbody><tr><td>base</td><td>1.11 ± 1.47</td><td>(3.92 ± 0.96)</td><td>(4.18 ± 1.03)</td><td>(4.22 ± 0.92)</td><td>(3.83 ± 1.03)</td></tr><tr><td>db-kg</td><td>2.24 ± 1.6</td><td><strong>4.15</strong> ± 0.9</td><td>4.02 ± 1.07</td><td>3.35 ± 1.01</td><td><strong>3.97</strong> ± 1.09</td></tr><tr><td>ec-kg</td><td><strong>2.85</strong> ± 1.86</td><td>4.13 ± 0.96</td><td><strong>4.12</strong> ± 1.01</td><td><strong>3.62</strong> ± 0.99</td><td>3.82 ± 1.03</td></tr></tbody></table>
Chapter 2: References

[AB1] Faisal Alkhateeb, Jean-François Baget, and Jérôme Euzenat. Extending sparql with regular expression patterns (for querying rdf). *Journal of web semantics*, 7(2):57-73, 2009.

[AB2] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In *The semantic web*, pages 722-735. Springer, 2007.

[AB3] Johanne Bakalara, Thomas Guyet, Olivier Dameron, André Happe, and Emmanuel Oger. An extension of chronicles temporal model with taxonomies-application to epidemiological studies. In *HEALTHINF 2021-14th International Conference on Health Informatics*, pages 1-10, 2021.

[AB4] Eva Blomqvist, Valentina Presutti, Enrico Daga, and Aldo Gangemi. Experimenting with extreme design. In *International Conference on Knowledge Engineering and Knowledge Management*, pages 120-134. Springer, 2010.

[AB5] Elizabeth Boschee, Jennifer Lautenschlager, Sean O'Brien, Steve Shellman, James Starz, and Michael Ward. ICEWS Coded Event Data, 2015.

[AB6] Valentina Anita Carriero, Aldo Gangemi, Maria Letizia Mancinelli, Ludovica Marinucci, Andrea Giovanni Nuzzolese, Valentina Presutti, and Chiara Veninata. Arco: The italian cultural heritage knowledge graph. In *International Semantic Web Conference*, pages 36-52. Springer, 2019.

[AB7] David Chanin. Open-source frame semantic parsing. *arXiv preprint arXiv:2303.12788*, 2023.

[AB8] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open-Domain Questions. In Regina Barzilay and Min-Yen Kan, editors, *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics.

[AB9] Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 17754-17762, 2024.

[AB10] Gong Cheng, Yanan Zhang, and Yuzhong Qu. Explass: exploring associations between entities via top-k ontological patterns and facets. In *The Semantic Web-ISWC 2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings, Part II 13*, pages 422-437. Springer, 2014.

[AB11] Laurens De Vocht, Sam Coppens, Ruben Verborgh, Miel Vander Sande, Erik Mannens,
and Rik Van de Walle. Discovering meaningful connections between resources in the web of data. In *LDOW*, 2013.

[AB12] Géraldine Del Mondo, Peng Peng, Jérôme Gensel, Christophe Claramunt, and Feng Lu. Leveraging spatio-temporal graphs and knowledge graphs: Perspectives in the field of maritime transportation. *ISPRS International Journal of Geo-Information*, 10(8):541, 2021.

[AB13] Lujun Fang, Anish Das Sarma, Cong Yu, and Philip Bohannon. Rex: explaining relationships between entity pairs. *arXiv preprint arXiv:1111.7170*, 2011.

[AB14] Charles J Fillmore and Collin F Baker. Frame semantics for text understanding. In *Proceedings of WordNet and Other Lexical Resources Workshop, NAACL*, volume 6, 2001.

[AB15] Erwin Filtz, María Navas-Loro, Cristiana Santos, Axel Polleres, and Sabrina Kirrane. Events matter: Extraction of events from court decisions. 2020.

[AB16] Valeria Fionda and Giuseppe Pirrò. Explaining graph navigational queries. In *The Semantic Web: 14th International Conference, ESWC 2017, Portorož, Slovenia, May 28-June 1, 2017, Proceedings, Part I 14*, pages 19-34. Springer, 2017.

[AB17] Valeria Fionda, Giuseppe Pirrò, and Mariano P Consens. Querying knowledge graphs with extended property paths. *Semantic Web*, 10(6):1127-1168, 2019.

[AB18] Valeria Fionda, Giuseppe Pirrò, and Claudio Gutierrez. Nautilod: A formal language for the web of data graph. *ACM Transactions on the Web (TWEB)*, 9(1):1-43, 2015.

[AB19] Aldo Gangemi, Mehwish Alam, Luigi Asprino, Valentina Presutti, and Diego Reforgiato Recupero. Framester: A wide coverage linguistic linked data hub. In *Knowledge Engineering and Knowledge Management: 20th International Conference, EKAW 2016, Bologna, Italy, November 19-23, 2016, Proceedings 20*, pages 239-254. Springer, 2016.

[AB20] Simon Gottschalk and Elena Demidova. Eventkg: A multilingual event-centric temporal knowledge graph. In *European Semantic Web Conference*, pages 272-287. Springer, 2018.

[AB21] Simon Gottschalk and Elena Demidova. Eventkg-the hub of event knowledge on the web-and biographical timeline generation. *Semantic Web*, 10(6):1039-1070, 2019.

[AB22] Simon Gottschalk and Elena Demidova. Eventkg, April 2020.

[AB23] Saiping Guan, Xueqi Cheng, Long Bai, Fujun Zhang, Zixuan Li, Yutao Zeng, Xiaolong Jin, and Jiafeng Guo. What is event knowledge graph: A survey. *IEEE Transactions on Knowledge and Data Engineering*, 2022.

[AB24] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-Augmented Language Model Pre-Training, February 2020. arXiv:2002.08909 [cs].

[AB25] Thomas Guyet. *Enhancing sequential pattern mining with time and reasoning*. PhD thesis, Université de Rennes 1, 2020.

[AB26] Steve Harris, Andy Seaborne, and E Prud'hommeaux. Sparql 1.1 query language. w3c recommendation (2013). URL https://www.w3.org/TR/sparql11-query, 2013.

[AB27] Olaf Hartig. Squin: a traversal based query execution system for the web of linked data.
In *Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data*, pages 1081–1084, 2013.

[AB28] Olaf Hartig and Jorge Pérez. Ldql: A query language for the web of linked data. *Journal of Web Semantics*, 41:9–29, 2016.

[AB29] Sebastian Hellmann, Jens Lehmann, Sören Auer, and Martin Brümmer. Integrating nlp using linked data. In *The Semantic Web-ISWC 2013: 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II* 12, pages 98–113. Springer, 2013.

[AB30] José E Talavera Herrera, Marco Antonio Casanova, Bernardo Pereira Nunes, Giseli Rabello Lopes, and LAPP Leme. Dbpedia profiler tool: profiling the connectivity of entity pairs in dbpedia. In *Proceedings of the 5th International Workshop on Intelligent Exploration of Semantic Data (IESD 2016)*, pages 17–18. sn, 2016.

[AB31] José Eduardo Talavera Herrera. *On the connectivity of entity pairs in knowledge bases*. PhD thesis, Ph. D. Thesis, Department of Informatics, Pontifical Catholic University of ..., 2017.

[AB32] Ioana Hulpuş, Narumol Prangnawarat, and Conor Hayes. Path-based semantic relatedness on linked data and its use to word and entity disambiguation. In *The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I 14*, pages 442–457. Springer, 2015.

[AB33] Robert Isele, Jürgen Umbrich, Christian Bizer, and Andreas Harth. Ldspider: An open-source crawling framework for the web of linked data. In *Proceedings of the 2010 International Conference on Posters & Demonstrations Track*, volume 658, pages 29–32. Citeseer, 2010.

[AB34] Glen Jeh and Jennifer Widom. Simrank: a measure of structural-context similarity. In *Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining*, pages 538–543, 2002.

[AB35] Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. Complex temporal question answering on knowledge graphs. In *Proceedings of the 30th ACM international conference on information & knowledge management*, pages 792–802, 2021.

[AB36] Javier Guillot Jiménez, Luiz André P Paes Leme, and Marco A Casanova. Coepinkb: Evaluating path search strategies in knowledge bases. *Journal of the Brazilian Computer Society*, 28(1):13–25, 2022.

[AB37] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 6769–6781, Online, November 2020. Association for Computational Linguistics.

[AB38] Takahiro Kawamura, Shusaku Egami, Koutarou Tamura, Yasunori Hokazono, Takanori Ugai, Yusuke Koyanagi, Fumihito Nishino, Seiji Okajima, Katsuhiko Murakami, Kunihiko Takamatsu, et al. Report on the first knowledge graph reasoning challenge 2018. In *Joint International Semantic Technology Conference*, pages 18–34. Springer, 2019.

[AB39] Krys J Kochut and Maciej Janik. Sparqler: Extended sparql for semantic association
discovery. In *The Semantic Web: Research and Applications: 4th European Semantic Web Conference, ESWC 2007, Innsbruck, Austria, June 3-7, 2007. Proceedings* 4, pages 145-159. Springer, 2007.

[AB40] Hermann Kroll, Denis Nagel, and Wolf-Tilo Balke. Modeling narrative structures in logical overlays on top of knowledge repositories. In *International Conference on Conceptual Modeling*, pages 250-260. Springer, 2020.

[AB41] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics.

[AB42] Kalev Leetaru and Philip A Schrodt. Gdelt: Global data on events, location, and tone, 1979-2012. In *ISA annual convention*, volume 2, pages 1-49. Citeseer, 2013.

[AB43] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, April 2021. arXiv:2005.11401 [cs].

[AB44] Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng. Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs. *arXiv preprint arXiv:2106.00327*, 2021.

[AB45] Pasquale Lisena, Daniel Schwabe, Marieke van Erp, Raphaël Troncy, William Tullett, Inger Leemans, Lizzie Marx, and Sofia Colette Ehrich. Capturing the semantics of smell: The odeuropa data model for olfactory heritage information. In *European Semantic Web Conference*, pages 387-405. Springer, 2022.

[AB46] Qingxia Liu, Gong Cheng, Kalpa Gunaratna, and Yuzhong Qu. Entity summarization: State of the art and future challenges. *Journal of Web Semantics*, 69:100647, 2021.

[AB47] Xuli Liu, Jihao Jin, Qi Wang, Tong Ruan, Yangming Zhou, Daqi Gao, and Yichao Yin. Patienteg dataset: bringing event graph model with temporal relations to electronic medical records. *arXiv preprint arXiv:1812.09905*, 2018.

[AB48] Steffen Lohmann, Philipp Heim, Timo Stegemann, and Jürgen Ziegler. The relfinder user interface: Interactive exploration of relationships between objects of interest. In *Proceedings of the 15th international conference on Intelligent user interfaces*, pages 421-422, 2010.

[AB49] Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. Dbpedia spotlight: shedding light on the web of documents. In *Proceedings of the 7th international conference on semantic systems*, pages 1-8, 2011.

[AB50] A. Merono, A. Ashkpour, M.G.J. van Erp, K. Mandemakers, L. Breure, A. Scharnhorst, K.S. Schlobach, and F.A.H. van Harmelen. Semantic technologies for historical research: A survey. *Semantic Web*, 6(6):539-564, 2015.

[AB51] Albert Meroño-Peñuela and Rinke Hoekstra. What is linked historical data? In *EKAW*, 2014.

[AB52] Joshua L Moore, Florian Steinke, and Volker Tresp. A novel metric for information retrieval in semantic networks. In *The Semantic Web: ESWC 2011 Workshops: ESWC*
2011 Workshops, Heraklion, Greece, May 29-30, 2011, Revised Selected Papers 8, pages 65-79. Springer, 2012.

[AB53] Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besançon. A dataset for open event extraction in english. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)*, pages 1939-1943, 2016.

[AB54] Julia Noordegraaf, Marieke van Erp, Richard Zijdeman, Mark Raat, Thunnis van Oort, Ivo Zandhuis, Thomas Vermaut, Hans Mol, Nicoline van der Sijs, Kristel Doreleijers, Vincent Baptist, Charlotte Vrielink, Brenda Assendelft, Claartje Rasterhoff, and Ivan Kisjes. Semantic deep mapping in the amsterdam time machine: Viewing late 19th- and early 20th-century theatre and cinema culture through the lens of language use and socio-economic status. In Florian Niebling, Sander Münster, and Heike Messemer, editors, *Research and Education in Urban History in the Age of Digital Libraries - Second International Workshop, UHDL 2019, Dresden, Germany, October 10-11, 2019, Revised Selected Papers*, volume 1501 of *Communications in Computer and Information Science*, pages 191-212. Springer, 2019.

[AB55] Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. Yago 4: A reasonable knowledge base. In *European Semantic Web Conference*, pages 583-596. Springer, 2020.

[AB56] Jorge Pérez, Marcelo Arenas, and Claudio Gutierrez. nsparql: A navigational language for rdf. *Journal of Web Semantics*, 8(4):255-270, 2010.

[AB57] Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How Context Affects Language Models' Factual Predictions. February 2020.

[AB58] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics.

[AB59] Giuseppe Pirrò. Explaining and suggesting relatedness in knowledge graphs. In *The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I 14*, pages 622-639. Springer, 2015.

[AB60] Juan L Reutter, Adrián Soto, and Domagoj Vrgoč. Recursion in sparql. In *The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I 14*, pages 19-35. Springer, 2015.

[AB61] Mark O Riedl. Computational narrative intelligence: A human-centered goal for artificial intelligence. *arXiv preprint arXiv:1602.06484*, 2016.

[AB62] Adam Roberts, Colin Raffel, and Noam Shazeer. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 5418-5426, Online, November 2020. Association for Computational Linguistics.

[AB63] Marco Rospocher, Marieke van Erp, Piek Vossen, Antske Fokkens, Itziar Aldabe, Ger-
man Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard. Building event-centric knowledge graphs from news. *Journal of Web Semantics*, 37:132–151, 2016.

[AB64] Sebastian Schaffert, Thomas Kurz, Dietmar Glachs, Christoph Bauer, Fabian Dorschel, and Manuel Fernandez. The linked media framework. 2012.

[AB65] Jaspreet Singh, Wolfgang Nejdl, and Avishek Anand. History by diversity: Helping historians search news archives. In *Proceedings of the 2016 ACM on conference on human information interaction and retrieval*, pages 183–192, 2016.

[AB66] Laura Sinikallio, Senka Drobac, Minna Tamper, Rafael Leal, Mikko Koho, Jouni Tuominen, Matti La Mela, Eero Hyvönen, et al. Plenary debates of the parliament of finland as linked open data and in parla-clarin markup. In *3rd Conference on Language, Data and Knowledge, LDK 2021*. Schloss Dagstuhl-Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2021.

[AB67] Steven A Sloman. The empirical case for two systems of reasoning. *Psychological bulletin*, 119(1):3, 1996.

[AB68] Tarcísio Souza Costa, Simon Gottschalk, and Elena Demidova. Event-qa: A dataset for event-centric question answering over knowledge graphs. In *Proceedings of the 29th ACM International Conference on Information & Knowledge Management*, pages 3157–3164, 2020.

[AB69] Steffen Stadtmüller, Sebastian Speiser, Andreas Harth, and Rudi Studer. Data-fu: A language and an interpreter for interaction with read/write linked data. In *Proceedings of the 22nd international conference on World Wide Web*, pages 1225–1236, 2013.

[AB70] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph, March 2024. arXiv:2307.07697 [cs].

[AB71] Ilaria Tiddi, Mathieu d'Aquin, and Enrico Motta. Learning to assess linked data relationships using genetic programming. In *The Semantic Web-ISWC 2016: 15th International Semantic Web Conference, Kobe, Japan, October 17-21, 2016, Proceedings, Part I 15*, pages 581–597. Springer, 2016.

[AB72] Francesca Tomasi. Artchives: a linked open data native catalogue of art historians’ archives. 2021.

[AB73] Jürgen Umbrich, Aidan Hogan, Axel Polleres, and Stefan Decker. Improving the recall of live linked data querying through reasoning. *RR*, 7497:188–204, 2012.

[AB74] Chiel Van den Akker, Lora Aroyo, Agata Cybulska, Marieke Van Erp, Peter Gorgels, Laura Hollink, Cathy Jager, Susan Legene, Lourens van der Meij, Johan Oomen, et al. Historical event-based access to museum collections. *Applied Artificial Intelligence*, 25, 2010.

[AB75] Willem Robert Van Hage, Véronique Malaisé, Roxane Segers, Laura Hollink, and Guus Schreiber. Design and use of the simple event model (sem). *Journal of Web Semantics*, 9(2):128–136, 2011.

[AB76] Nikos Voskarides, Edgar Meij, Sabrina Sauer, and Maarten de Rijke. News article retrieval in context for event-centric narrative creation. In *Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval*, pages 103–112, 2021.
[AB77] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. *Communications of the ACM*, 57(10):78–85, 2014.

[AB78] Ian H Witten and David N Milne. An effective, low-cost measure of semantic relatedness obtained from wikipedia links. 2008.

[AB79] Jie Wu, Xinning Zhu, Chunhong Zhang, and Zheng Hu. Event-centric tourism knowledge graph—a case study of hainan. In *International conference on knowledge science, engineering and management*, pages 3–15. Springer, 2020.

[AB80] Harald Zauner, Benedikt Linse, Tim Furche, and François Bry. A rpl through rdf: Expressive navigation in rdf graphs. In *Web Reasoning and Rule Systems: Fourth International Conference, RR 2010, Bressanone/Brixen, Italy, September 22-24, 2010. Proceedings 4*, pages 251–257. Springer, 2010.
# Part II

## Understanding Society through Social Demography
# Chapter 3

# Semantic Technologies for Social Demography Research

Lise Stork*†, Richard L. Zijdeman*, Ilaria Tiddi†, Annette ten Teije†

†Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

*International Institute of Social History

**Abstract** A shift in scientific publishing from paper-based to knowledge-based practices promotes reproducibility, machine actionability and knowledge discovery. This is important for disciplines like social demography, where study indicators are often social constructs such as race or education, hypothesis tests are challenging to compare due to their limited temporal and spatial coverage, and research output is presented in natural language, which can be ambiguous and imprecise. In this work, we present a resource to aid researchers in their research workflow, and publish FAIR (Findable, Accessible, Interoperable, Reusable) findings. The MIRA-KG consists of: (1) an ontology for social demography research, (2) a method for automated ontology population by prompting large language models (LLMs), and (3) a knowledge graph populated in terms of the ontology by annotating a set of research papers on health inequality, and (4) a feasibility study along with a qualitative user study to measure perceived usefulness and veracity of the knowledge graph for literature-based discovery as part of the research process of a researcher studying our social and cultural history. The resource allows researchers to formally represent their social demography research hypotheses, discovering research biases and novel research questions.

**Keywords** Scientific Knowledge Graphs, Social Demography, Hypothesis Representation, Health Inequality, Knowledge Extraction

DOI: https://doi.org/10.5281/zenodo.10286846

URL: http://w3id.org/mira http://w3id.org/mira/ontology/

## 3.1 Introduction

Research on social demography focuses on the statistical study of human populations, with the aim of understanding and predicting social, cultural and economic trends across populations. Since the first known census taken by the Babylonian Empire in 3500 BCE [28], demographers have involved themselves with the task of explaining aggregate statistics of a population [16]. Specifically, the research cycle of a social historian consists of analysing observational data
about society to form novel hypotheses and theories about societal mechanisms (see left side of Figure 3.1). However, such studies tend to be restricted to specific time periods and regions, making comparison of hypothesis tests across the vast array of research papers difficult. Moreover, demographers test research hypotheses in which variables are often social constructs such as intelligence, ownership, or nationality. Unclarity and ambiguity reporting research findings in natural language complicates their precise understanding. Due to these complexities, demographers often describe rather than explain demographic phenomena [16], whereas causal explanations could help shape better policies for the future. Below we describe a common motivating scenario.

**Example 1.** A social historian aims to analyse a population census from the Netherlands between 1850-1922—consisting of certificates (births and marriages), occupational and survival data<fnref n="1" />—to better understand mechanisms of social inequality. For this, the historian has to carefully survey the literature for known theories and the datasets and statistics that support them. What was the evidence for a specific hypothesis? How did they measure social stratification? What were the outcomes? Which societal factors, that are (not) in my dataset, can moderate the effect between my study variables? *The historian may find it challenging to discover relevant papers or interpret research findings precisely.*

To address such challenges, the paradigm of scientific publishing is seeing a shift from document-oriented publishing towards knowledge-based publishing [1, 21], with the aim of making the infrastructure for scientific publication of research output more FAIR (Findable, Accessible, Interoperable, and Reusable).

Semantic technologies have been successfully employed in various domains to accommodate such a shift. Notable examples are the Open Science Knowledge Graph [1, 35], the Unified Medical Language System (UMLS) [3], or Biomedical knowledge graphs such as [18], but many other examples exist [36, 10, 12, 4, 37, 9, 10, 24, 22]. These sources promote reusability of research findings, as well as the machine-aided design of novel research questions. Moreover, by adopting formal representations of knowledge, such resources promote transparency and explainability.

In social demography, work has been done formally describing observational data, such as census data hosted as linked data at the International Institute of Social History (IISH)<fnref n="2" /> [27]. To the best of our knowledge, no studies formalise knowledge such as *research questions* and *findings* on social demography—and very few from social sciences in general [36, 24], whereas these are important in each of the steps of the scientific workflow of a social historian (see Figure 3.1). The research process, hypotheses and findings are mostly written up in scientific documents in natural language, which can be ambiguous and imprecise. Such fields can thus benefit from adopting the FAIR data principles, to reduce unclarity and ambiguity in the research workflow of a social demographer (see Figure 3.1).

In summary, there is a need for the improvement of the digital infrastructure underlying scientific publication in social demography and social history research, to stimulate a deep understanding, and reuse of existing research questions, methods and findings. To address this need, this work provides:

1. The MIRA ontology, which includes a set of classes, properties and axioms for capturing research findings (*observations, comparisons and explanations*) on social demography, as well as SHACL shapes following data quality criteria of [7], for data validation.

<footnote n="1">¹for example: https://datasets.iisg.amsterdam/dataverse/HSNDB-HSN</footnote>

<footnote n="2">²https://iisg.amsterdam/en</footnote>
2. A Knowledge Graph Construction (KGC) method, based on: (i) prompting a large language model to annotate paper abstracts using the ontology, (ii) mapping concepts to terms from NCBO BioPortal ontologies and GeoNames, and (iii) refining the final graph by a set of SHACL constraints, developed according to data quality criteria.

3. The *MIRA-KG*, a knowledge graph of machine-annotated paper abstracts on social health inequality *explanations* in terms of the MIRA ontology. The explanations are linked to Linked Open Data, and evaluated against SHACL shapes and data quality criteria.

In general, this work (i) supports the shift towards knowledge-based scientific publishing by contributing a novel method for KGC construction that can easily be adapted to accommodate other application domains, and (ii) contributes to a more FAIR infrastructure for social demography research.

<figure><img src="image_12.png" /><figcaption>Figure 3.1: The social demography research workflow. Blue boxes contain components of FAIR data management (e.g., for social history) https://www.go-fair.org/fair-principles/fairification-process/.</figcaption></figure>

## 3.2 Related Work

**Ontologies and Vocabularies for Scientific Knowledge.** Below we discuss examples of ontologies developed for various types of scientific knowledge:

**Upper-level science ontologies.** Examples of upper-level ontologies for scientific research are the Modern Science (ModSci) ontology [12], an upper ontology for modern science branches and related entities, and the Semanticscience Integrated Ontology (SIO) [11], a simple integrated ontology for rich description of scientific concepts and processes.

**Hypothesis representations.** Work has been done on formalising research questions, and notable examples are the SuperPattern ontology [4], which allows researchers to write up the main claims of scientific articles as statements in formal logic; the nanopublication model [17], which aims at publishing “core scientific statements with associated context”, or the DISK Hypothesis Evolution ontology [14], which captures the evolution of hypotheses over time. Many other representations exist, of which [14] provides an extensive comparative overview. An example of a domain-specific scientific ontology is the PICO
ontology (for Population, Intervention, Comparator, Outcome) for synthesis and querying of clinical trial experiments [26].

**Data representations.** The RDF Data Cube vocabulary<fnref n="3" /> allows researchers to publish their multi-dimensional data, and an example of a domain-specific model for publishing datasets is the Data Scopes model [19, 6], which captures how datasets are processed in social history research.

**Publication metadata.** Lastly, vocabularies have been created to capture publication metadata. The most notable ones are the bibliographic ontology (BIBO)<fnref n="4" />, the PRISM vocabulary<fnref n="5" />, and the Dublin Core<fnref n="6" />.

In this work, we formalise research questions and link them to data representations and publication metadata. We do so for the domain of social demography, which has not been done before, and make use of some of the ontologies and vocabularies enumerated above.

**Scientific Knowledge Graphs.** Knowledge Graphs that capture scientific knowledge have been created from unstructured texts such as research papers, in various domains like Computer Science and Artificial Intelligence [8, 9, 10], social science [33], biomedicine [18], plant sciences [22], social and behavioural sciences [24], or scientific articles in general [34, 1, 35]. Resources such as these support a variety of tasks that aid researchers in the development of their research questions and methods. Our work is most related to the work of [33] and [24], but instead proposes a semantic model of social demography research questions and findings, and a knowledge graph construction (KGC) method that does not rely on the knowledge-intensive task of expert semantic annotation and links extracted knowledge to other linked open data (LOD).

## 3.3 Knowledge Graph Construction

This section describes the knowledge graph construction pipeline (see Figure 3.2 for the entire pipeline). Ontology creation is discussed in Section 3.3.1, the resulting ontology in 5.3.1, ontology population in Section 3.3.3 and the resulting KG in Section 3.3.4.

<figure><img src="image_13.png" /><figcaption>Figure 3.2: The knowledge graph construction pipeline. Coloured artifacts are novel contributions; non-coloured artifacts represent external repositories.</figcaption></figure>

<footnote n="3">³https://www.w3.org/TR/vocab-data-cube/</footnote>

<footnote n="4">⁴https://www.dublincore.org/specifications/bibo/bibo/bibo.rdf.xml</footnote>

<footnote n="5">⁵https://www.w3.org/submissions/2020/SUBM-prism-20200910/psv-over.html</footnote>

<footnote n="6">⁶https://www.dublincore.org/</footnote>
### 3.3.1 Ontology Creation

Below, we describe ontology creation process (see ① of Figure 3.2). We follow the eXtreme Design (XD) method for ontology development [31], a collaborative, incremental, iterative method for pattern-based ontology design. We followed the main steps of the XD methodology: (i) project scoping, (ii) creation of requirement stories, (iii) a module development loop.

#### 1. Project initiation and scoping.

A first interview was set up with a domain expert on social history and humanities research from the International Institute of Social History (IISH)<fnref n="7" />. The outcome of the interview was used to define the task context, get a better understanding of the challenges, and write out a first list of ontology requirements (inspired by how a hypotheses is represented in DISK [14, 15]): *Article metadata (RQ1)*, *Hypothesis classification (RQ2)*, *Hypothesis statement (RQ3)*, *Hypothesis context (RQ4)*, *Hypothesis qualifier (RQ5)*, and *Hypothesis evidence (RQ6)*. For RQ2-6, we collected requirement stories.

#### 2. Requirement stories.

Within this project, requirement stories were collected based on insights retrieved during the project scoping phase about hypothesis types and their elements. We show one requirement story below (Example 2), which shows an explanatory question (asking whether economic developments can explain differences in socioeconomic inequality).

**Example 2.** "Is the influence of ascription on education larger than the influence of achievement, and is the effect moderated by economic developments?"

After a first elicitation round, academic papers were used as requirement stories. An example annotation, based on the paper "Industrialization and inequality revisited: Mortality differentials and vulnerability to economic stress in Stockholm, 1878–1926" [29] is used as an example in a visual schematic of the final ontology, figs. 3.3 to 3.5 (indicated by ■).

#### 3. Module development loop.

Ontological requirements were elicited by generalising terms from requirement stories and deriving competency questions (CQs), resulting in a categorisation of main entity types and relations derived from the requirement stories, see Table 3.1.

Moreover, contextual statements (CSs), and reasoning requirements (RRs) were derived. CSs informed the creation of SHACL constraints for validation, and RR informed the creation of OWL<fnref n="8" /> axioms. A collaborative environment in the form of a Wiki was set up to gather the CQs, CSs, RR and other outcomes, and a dump of the Wiki is published on Zenodo together with the dataset.

**Ontology reuse.** For representing article metadata (RQ1), we use BIBO, PRISM and Dublin Core (as discussed in Section 3.2). To classify research questions (RQ2), we use the classification (QT) from Table 3.1. Moreover, to write up hypothesis statements (RQ3) and their context (RQ4, based on (QL) from Table 3.1) we utilise the structure and properties of the SuperPattern Ontology [4]. However, we represent instances of the SuperPattern as individuals instead of classes as, in social demography, evidence often comes from specific time periods and regions. Later, class axioms can be used to collect all evidence belonging to a high-level

<footnote n="7">⁷https://iisg.amsterdam/en</footnote>

<footnote n="8">⁸https://www.w3.org/TR/owl-syntax/</footnote>
<table><caption>Table 3.1: Question Type (QT), Question Level (QL), Observation Type (OT), and Trend Type (TT) categorisations, based on requirement stories.</caption><thead><tr><th>QT</th><th>QL</th><th>OT</th><th>TT</th></tr></thead><tbody><tr><td><strong>Descriptive</strong><br>Describing phenomena,<br>e.g. temperature is<br>rising over time</td><td><strong>Micro</strong><br>Individuals<br>Human<br>populations</td><td><strong>Longitudinal</strong><br>Observations over<br>time</td><td><strong>Temporal trend</strong><br>Comparison of<br>time intervals</td></tr><tr><td><strong>Comparative</strong><br>Comparisons between sample<br>means, trends, or effects,<br>e.g. did temperature rise<br>more after industrialisation?</td><td><strong>Meso</strong><br>Social groups,<br>Organisations</td><td><strong>Intergenerational</strong><br>Observations over<br>generations</td><td><strong>Spatial trend</strong><br>Comparison of<br>regions</td></tr><tr><td><strong>Explanatory</strong><br>Defining a potential cause<br>of the outcome of a<br>comparison, e.g. did<br>human emission cause<br>the rise of temperature?</td><td><strong>Macro</strong><br>Geographical<br>regions</td><td><strong>Intersectional</strong><br>Observations from<br>a single point<br>in time</td><td><strong>Socioeconomic trend</strong><br>Comparison of<br>distribution across<br>social classes</td></tr></tbody></table>

claim. We additionally reuse classes and properties from the Semanticscience Integrated Ontology (SIO) [37] (e.g. trend line sio:SIO_000527 or human population sio:SIO_001062). For qualifiers (RQ5), we reuse qualifier categories from [7], as results can be translated to effect sizes. Lastly, to link research questions to their evidence (RQ6), we reuse the *RDF Data Cube Vocabulary*¹⁹, furthering reproducibility and machine actionability.

**Design Pattern (ODP) reuse.** Scientific research questions and claims often consist of complex relations between variables, relations, and findings, and as such there are various ways of modeling them. One typical way is to reify the relation-holding context as a node with binary relations for the subject, object and property, and a fourth binary relation indicating the context [13]. A downside of such a representation is that the Web Ontology Language (OWL)<fnref n="10" /> does not support reification. Some Ontology Design Patterns such as the Content Slices ODP<fnref n="11" /> address the issue, but such modeling hampers easy integration with paper metadata. Therefore, we choose for *reification* with the idea that the reified hypotheses can later be transformed to binary relations, should OWL reasoning be required.

### 3.3.2 The MIRA Ontology

The main concepts, relations and their domains and ranges of the MIRA Ontology are created in light of our requirements and categorisations (Table 3.1)<fnref n="12" />. These are shown in Figure 3.3. Figure 3.4 unfolds the *comparison* branch of the MIRA ontology. Descriptive statistics (such as ratios) can be compared, and are linked to their respective data cube slices. Figure 3.5 unfolds the *explanation* (main effect and interaction effect) branch of the MIRA ontology.

From the CSs and RR s enumerated during the ontology design process, we developed a set of OWL axioms and SHACL constraints for validation, which describe the semantic requirements of the hypotheses. An example of an OWL statement and axiom are: *is evidence for* is an *owl:transitiveProperty*, and *mira:InteractionEffect* *owl:subClassOf* *mira:Explanation* respectively. Examples of SHACL constraints are the use of *sh:minCount*

<footnote n="10">¹⁰http://www.w3.org/TR/vocab-data-cube/</footnote>

<footnote n="11">¹¹https://www.w3.org/TR/owl-ref/</footnote>

<footnote n="12">¹²http://ontologydesignpatterns.org/wiki/Submissions:Context_Slices</footnote>
<figure><img src="image_14.png" /><figcaption>Figure 3.3: Main classes of the MIRA ontology. A ★ points to the IRI of the class(es) or superclass. Elements retrieved from paper abstracts during the ontology population process (see Section 3.3) are indicated in blue. Commonly, paper abstracts from social demography do not contain statistical data and precise findings.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 3.4: The **Comparison** class and its links to the RDF Data Cube Vocabulary. Comparisons compare information content entities such as trend lines, centrality measures, and ratios, which are based on slices over multidimensional data cubes. Example instances and literals are based on the article “Industrialization and inequality revisited: Mortality differentials and vulnerability to economic stress in Stockholm, 1878-1926”, and indicated with ■</figcaption></figure>

and `sh:maxCount` to ensure that each paper has at least one study, each study has at least one explanation, each explanation has as exactly one context, subject, object, relation and qualifier (but no more than one moderator and mediator). Axioms and SHACL shapes can be found in the ontology and Github repository, respectively.

### 3.3.3 Ontology Population

Below, we describe the ontology population process from scientific papers (See ② of Figure 3.2). As a use case, we focus on articles related to health inequality.

#### 1. Retrieve papers.

Papers included were retrieved from Semantic Scholar using the Semantic Scholar API, and selected by the following steps:

- retrieving papers between 2020-2023, based on the keywords: ∈ {social, inequality, disparities,
<figure><img src="image_16.png" /><figcaption>Figure 3.5: The Explanation (Main Effect) class and its links to the RDF Data Cube Vocabulary. Explanations describe measured effects of dimensions in the dataset (such as ex:socialClass, instance of `qb:DimensionProperty`) on measurements in the dataset (instances of `qb:MeasureProperty` such as ex:mortalityRate). These properties can then be linked to concepts from ontologies using `qb:concept` (for example ex:mortalityRate `qb:concept stato:STATO_0000414`). The mira:correlatesWith property is used as a sub-property of mira:RelationalProperty. No causal link is yet stated, since more evidence from various regions and periods is required to state a causal relationship. Interaction Effects are subclasses of main effects. They are modeled in the same way, with the following exceptions: the range of mira:hasObject is an instance of mira:MainEffect; the range of mira:hasProperty is mira:moderates.</figcaption></figure>

mortality, population, socioeconomic, demographic, study}, and the fields of study $\in$
{Economics, History, Sociology}. These keywords were terms encountered in the titles
of the five papers used as requirement stories;

- filtering papers based on:

i *citation count*: removing articles with lower impact, with $\le 10$ citations;

ii *journal*: removing articles published in non peer-reviewed journals;

iii *abstract length*: including articles with abstract length $l = \mu - \sigma > l > \mu + \sigma$ (within plus-or-minus 1 standard deviation) as we found that it was easier to extract hypotheses from texts that were not too long nor too short due to conciseness and lack of information, respectively;

iv *doi*: whether or not articles included a Digital Object Identifier (DOI).

## 2. Extract Knowledge.

In social demography, the abstract of a paper commonly includes the explanatory question and the time period and location of the population sample. We thus annotate paper abstracts with these parts of the ontology (see the blue classes in figs. 3.3 to 3.5). In order to extract the relevant entities, relations and attributes in terms of the ontology, we prompt GPT-4 (OpenAI, 2023), see Table 3.2. Large language models (LLMs) can learn multiple tasks without any explicit supervision [32, 20] and can therefore take on the task normally performed through human annotators via crowdsourcing, without any training. Retrieved annotations were extracted directly as triples of the form `< subject, predicate, object >` using the Resource
<table><caption>Table 3.2: Prompt used to retrieve structured abstract annotations using GPT-4. Each prompt consisted of the paper abstract, the ontology (domains and ranges), and some instructions for formatting of individuals.</caption><thead><tr><th>Prompt template</th><th></th></tr></thead><tbody><tr><td><pre><code> Prompt template < abstract > Describe the claim of the abstract above only using RDF (the turtle syntax), and using the following ontology: < property > &lt; domain &gt; &lt; range &gt; < instructionnumber > &lt; instruction &gt; &lt; example &gt; Only return proper RDF, no free text comments</code></pre></td><td></td></tr><tr><td><pre><code> Instruction example Instruction: use rdfs:label to describe all blank nodes, also the geographic region. Use short descriptions, pieces of text verbatim from the abstract, and add language tags to all labels. Example: [] :hasSubject [rdfs:label "social class"@en]. Instruction: for instances of the class gn:Feature, find the URI for the place name in GeoNames (uri = https://www.geonames.org/&lt;code&gt;) Example: [] gn:locatedIn &lt;uri&gt;</code></pre></td><td><pre><code> Answer [] :hasSubject [rdfs:label "residential patterns and nutrition"@en] [] gn:locatedIn &lt;http://sws.geonames.org/2673730/&gt;</code></pre></td></tr></tbody></table>

Description Framework<fnref n="13" />

## 3. Process Knowledge.

After extracting a set of triples (a subgraph) from a paper abstract, we: (i) **reshape** the subgraph, as we prompted GPT-4 to return annotations in a somewhat simpler structure to avoid structural errors, (iii) **add paper metadata** by transforming paper metadata retrieved from SemanticScholar (abstract, DOI, citations, publication date, authors) to RDF, (ii) **link** study variables in the subgraph to Linked Open Data from BioPortal and GeoNames, (iii) **validate** the subgraph using the SHACL shapes, and (iv) **integrate** the subgraph with the MIRA-KG, and **postprocess** the merged graph by removing redundant entities.

Specifically, in the **link** step, instances of **qb:ComponentProperty** are linked to concepts from BioPortal<fnref n="14" /> using **qb:concept**. Prior to linking, a manual mapping study was performed (output can be found in the Zenodo repository) to see which BioPortal ontologies contained most of the useful study variables. Selected ontologies were HHEAR<fnref n="15" />, SIC<fnref n="16" />, IOBC<fnref n="17" /> and MESH<fnref n="18" />). Subsequently, we retrieve location metadata from GeoNames<fnref n="19" /> through FactForge<fnref n="20" />, as the knowledge extraction step already retrieves GeoNames IRIs. Location coordinates, for instance, can aid researchers in discovering regional biases, or temporal trends. For the **validate** step, SHACL shapes are used to check for schema and ontological correctness. One of the SHACL

<footnote n="13">¹³https://www.w3.org/RDF/</footnote>

<footnote n="14">¹⁴https://bioportal.bioontology.org/</footnote>

<footnote n="15">¹⁵https://bioportal.bioontology.org/ontologies/HHEAR/</footnote>

<footnote n="16">¹⁶https://www.ebi.ac.uk/ols/ontologies/sio</footnote>

<footnote n="17">¹⁷https://purl.bioontology.org/ontology/IOBC</footnote>

<footnote n="18">¹⁸https://purl.bioontology.org/ontology/MESH</footnote>

<footnote n="19">¹⁹https://www.geonames.org/</footnote>

<footnote n="20">²⁰http://factforge.net</footnote>
shapes, for instance, checks for syntactic validity of dates, by ensuring `sh:datatype xsd date`, and by checking for ontological correctness (begin before end-date, see SHACL shapes). For the **integrate** and **postprocess** steps, the subgraph is merged with the MIRA-KG. After processing all paper subgraphs, we remove duplicate BioPortal concepts and their labels via `owl:sameAs` smushing using RDFPRO<fnref n="21" />

The time it took to process a single paper abstract was 1 minute and 11 seconds (MacBook M1 Chip, 8 cores, 8gb RAM). A fully annotated paper abstract as machine-readable data can be found in the Github repository.

### 3.3.4 The MIRA-KG

Table 3.3 shows statistics of the final knowledge graph, formalising 398 paper abstracts. We remove paper metadata functional properties from this view (for example `dcterms:created`), as these counts are equal to the instance count for `bibo:AcademicArticle`. We present the average degree with and without types, as these are naturally densely connected hubs. More interesting are the densely connected papers via citations or BioPortal concepts and locations with high in-degrees. Nodes with the highest in-degree that were not classes of the MIRA ontology were: economics/economy (MESH:D004467, IOBC:200906008393315870, 222), mortality rate (MESH:D009026, 188), hygiene (MESH:D006920, 91), the United States (geonames:6252001, 90) health (MESH:D006262, 90), age (MESH:D006262, 83) and income (MESH:D007182, 47).

<table><caption>Table 3.3: Statistics of the MIRA-KG: Class instance count, property count, number of nodes and edges, and average degree.</caption><thead><tr><th>Classes</th><th>Instances</th><th>Properties</th><th>Count</th></tr></thead><tbody><tr><td>time:Instant</td><td>948</td><td>bibo:cites</td><td>13496</td></tr><tr><td>qb:DimensionProperty</td><td>701</td><td>qb:concept</td><td>4041</td></tr><tr><td>skos:Concept</td><td>587</td><td>rdfs:label</td><td>3133</td></tr><tr><td>mira:Explanation</td><td>582</td><td>dcterms:contributor</td><td>1176</td></tr><tr><td>SIO:SIO_000414 (geographic region)</td><td>524</td><td>SIO:SIO_000008 (has attribute)</td><td>1065</td></tr><tr><td>SIO:SIO_001050 (sample)</td><td>477</td><td>time:inXSDDate</td><td>946</td></tr><tr><td>time:TemporalEntity</td><td>474</td><td>mira:hasSubject</td><td>607</td></tr><tr><td>qb:MeasureProperty</td><td>477</td><td>mira:hasContext</td><td>607</td></tr><tr><td>SIO:SIO_000976 (observational study)</td><td>469</td><td>SIO:SIO_000205 (is represented by)</td><td>607</td></tr><tr><td>SIO:SIO_001062 (human population)</td><td>408</td><td>mira:hasObject</td><td>602</td></tr><tr><td>bibo:AcademicArticle</td><td>398</td><td>mira:hasEffectSize</td><td>589</td></tr><tr><td>gn:Feature</td><td>114</td><td>mira:hasRelation</td><td>583</td></tr><tr><td>mira:InteractionEffect</td><td>105</td><td>sem:hasPlace</td><td>479</td></tr><tr><td>SIO:SIO_000012 (organisation)</td><td>12</td><td>time:hasTime</td><td>477</td></tr><tr><td></td><td></td><td>gn:locatedIn</td><td>454</td></tr><tr><td></td><td></td><td>wgs84_pos:long</td><td>172</td></tr><tr><td>Number of Nodes</td><td>24281</td><td>wgs84_pos:lat</td><td>172</td></tr><tr><td>Number of Edges</td><td>32359</td><td>gn:name</td><td>114</td></tr><tr><td>Av. Degree</td><td>1.59</td><td>mira:hasMediator</td><td>77</td></tr><tr><td>Av. Degree (no types)</td><td>1.33</td><td>SIO:SIO_000061 (located in)</td><td>2</td></tr></tbody></table>

## 3.4 Evaluation

In this section, we first evaluate the ontology (Section 3.4.1), and then the whole knowledge graph via a use case and data quality measures (Section 3.4.2).

### 3.4.1 Ontology Evaluation

Here, we show one of the competency questions as a SPARQL query (for brevity we omit prefixes, but these can be found in Figure 3.3): The CQ retrieves a claim with a specific dependent variable and a potential mediator.

<footnote n="21">²¹https://github.com/dkmfbk/rdfpro</footnote>
```
PREFIX ...
SELECT DISTINCT ?ind_lab ?ind_var ?qual
           ?med_var ?med_lab WHERE {
?exp mira:hasSubject/qb:concept/rdfs:label ?ind_lab;
#what is the subject of the explanation
mira:hasSubject/qb:concept ?ind_var;
#the object is COVID-19 mortality
mira:hasObject/qb:concept mesh:D000086382,
                                MESH:D00902;
#what is the effectSize of the association
mira:hasEffectSize ?effectSize .
#which variables mediate the association
OPTIONAL{?exp mira:hasMediator ?med_var;
          mira:hasMediator/rdfs:label ?med_lab .}
}
```

<table><caption>(a) The SPARQL query for the CQ</caption><thead><tr><th>ind_lab</th><th>med_lab</th></tr></thead><tbody><tr><td>1</td><td>Residential Segregation</td></tr><tr><td>2</td><td>Economics</td></tr><tr><td>3</td><td>Ethnicity</td></tr><tr><td>4</td><td>Occupation</td><td>Ability</td></tr><tr><td>5</td><td>Occupation</td><td>Home</td></tr></tbody></table><table><thead><tr><th colspan="2">effectSize</th></tr></thead><tbody><tr><td>1</td><td>mira:largeMediumNegativeES</td></tr><tr><td>2</td><td>mira:largeMediumPositiveES</td></tr><tr><td>3</td><td>mira:largeMediumPositiveES</td></tr><tr><td>4</td><td>mira:smallPositiveES</td></tr><tr><td>5</td><td>mira:smallPositiveES</td></tr></tbody></table>

<caption>(b) The result for the CQ (first five rows)</caption>

Figure 3.6: **CQ1:** Have associations been found between a socioeconomic variable and COVID-19, and which explanations can be found for the inequality?

Table 3.6 shows the output of the query for the CQ (first five rows). It indicates that residential segregation, economics and ethnicity were all correlated with COVID-19 mortality. The association between COVID-19 and occupation was specifically mediated by the ability to work from home (ability, home). The rest of the competency questions can be found as SPARQL queries in the Github repository. They could all be answered using SPARQL queries over the MIRA ontology, although some limitations were discovered during ontology creation, see Section 3.4.2 below.

## Limitations.

The sessions with the expert showed that within papers there are various reformulations of the same hypothesis, which are at times not semantically the same. Such variations cannot be covered entirely without overly complicating the ontology, and cannot always be fully understood without access to the data and experiments. Moreover, effect sizes are often not clearly reported in the paper abstracts, hampering the quality of the extracted effect sizes.

## 3.4.2 Knowledge Graph Evaluation

In this section, we (1) test the quality via a set of data quality criteria, and we (2) evaluate the use of the MIRA-KG via a use case: understanding the state of the art of a topic by querying research questions.

### Data Quality Evaluation.

We use the RDF data quality criteria as defined by [7] to, on the one hand, validate the MIRA-KG as well as the KGC pipeline, and, on the other hand, refine the graph. These are published alongside the KG, but we summarise the most important ones for our case below.

**Accuracy.** Accuracy in terms of Linked Data deals with syntactic and semantic validity, as well as duplicate entries. Syntactic validity: the DatatypeConstraintComponent catches data type violations, such as the use of `rdf:XMLLiteral` where `xsd:-date` should have been used. Furthermore, Table 3.3 shows all retrieved GeoNames identifiers were correct identifiers, as metadata was retrieved for all of them from FactForge. Semantic validity: The LessThanConstraintComponent measures logical temporal violations (begin date before end date). Duplicate entries: from Table 3.4 we see that with eight violations, these were annotated
<table><caption>Table 3.4: **Left:** SHACL constraint violations for the MIRA-KG. **Right:** LLM retrieval errors for 20 research articles.</caption><thead><tr><th>Constraint violation</th><th>Count</th><th>LLM retrieval error (20 articles)</th><th>Count</th></tr></thead><tbody><tr><td>sh:LessThanConstraintComponent</td><td>8</td><td>Study variables</td><td>3</td></tr><tr><td>sh:DatatypeConstraintComponent</td><td>23</td><td>Location</td><td>2</td></tr><tr><td>sh:ClassConstraintComponent</td><td>38</td><td>Context</td><td>1</td></tr><tr><td>sh:MinCountConstraintComponent</td><td>122</td><td>Time period</td><td>7</td></tr><tr><td>sh:MaxCountConstraintComponent</td><td>4</td><td>Effect size</td><td>1</td></tr><tr><td>sh:XoneConstraintComponent</td><td>27</td><td></td><td></td></tr></tbody></table>

with high logical accuracy (98.3%). Lastly, duplicate BioPortal entities were handled during the postprocessing step of the KGC pipeline, as described in Section 3.3.3.

**Trustworthiness.** This dimension is defined as the degree to which the information is accepted to be correct, true, real and credible. As the MIRA-KG was created automatically using GPT-4, we analyse the first twenty research questions (annotations can be found on Zenodo). Errors with respect to the retrieval are shown in Table 3.4. *Study variables:* in case study variables were incorrect, GPT-4 used related terms (such as *time* instead of *age* as independent variable). *Time period:* most common errors were incorrectly retrieved dates. However it should be mentioned that, in most cases, dates were incorrectly extracted when no dates were mentioned in the abstract.

Even though errors appear, most terms appear to be retrieved correctly, even for complex research questions such as: *In the context of a human population, “Social inequality”, mediated through “Health-services structure”, is correlated with “COVID-19 mortality” with a large Medium Positive ES.* *Governmental response to COVID-19* moderates this correlation. Evidence comes from *“São Paulo residents”*, from *“São Paulo”* between the years *“2020-03-01”* and *“2020-09-30”*.

**Consistency.** Semantic consistency is the extent to which a collection uses the same values for conveying the same meanings throughout. SHACL constraints were applied to enforce schema-correctness (such as a hypothesis has at least one independent variable). Constraint violations can be found in Table 3.4 (*sh:Class*, *sh:MinCount*, *sh:MaxCount* and *sh:Xone-ConstraintComponent*). By looking at the exact violations, we see that common errors were errors in structure, such as a context which was linked to a study instead of an explanation, resulting in violations of both the *Class* as well as the *MinCount* constraint components. The graph was queried to retrieve and dissolve these common errors to further consummate the entire graph.

## Limitations

Due to the limitations mentioned in Section 3.4.1 (underspecification, ambiguity, and unclarity), it is challenging to extract research questions from scientific papers in a precise manner. The ontology will, however, allow researchers to publish their findings in a machine-readable manner alongside paper publication. Second, representing complex concepts, such as ‘the ability to work from home’, is non-trivial, as these are not always defined in ontologies or vocabularies and can be described in many different ways. For querying, linking such a variable to the concepts ‘ability’ as well as ‘home’ would suffice, but if precise reasoning is required, other solutions need to be considered.
PREFIX ...
SELECT ?long ?lat ?locName ?es ?geoId WHERE {
#the subject of the explanation is income
?exp mira:hasSubject/qb:concept MESH:D007182 ;
#the object of the exp. is mortality
mira:hasObject/qb:concept MESH:D00902 ;
#what is the effect size of the explanation
mira:hasEffectSize ?es ;
#what is the context of the explanation
mira:hasContext ?pop .
#which sample is used to represent the population.
?pop sio:SIO_000205 ?sample .
#where is the sample collected
?sample sem:hasPlace ?location .
#what is the GeoNamesId of the location
?location gn:locatedIn ?geoId .
?geoId rdf:type gn:Feature ;
#what is the name of the location
gn:name ?locName ;
#what is the longitude
wgs84_pos:long ?long ;
#what is the latitude
wgs84_pos:lat ?lat . }

(a) The SPARQL query for the use case

<table><thead><tr><th></th><th>long</th><th>lat</th><th>locName</th></tr></thead><tbody><tr><td>1</td><td>17.64</td><td>59.85</td><td>Uppsala</td></tr><tr><td>2</td><td>8.05</td><td>47.4</td><td>Aarau</td></tr><tr><td>3</td><td>24.93</td><td>60.17</td><td>Finland</td></tr><tr><td>4</td><td>-52.0</td><td>-13.84</td><td>Brazil</td></tr></tbody></table>

<table><thead><tr><th></th><th>geoId</th><th>es</th></tr></thead><tbody><tr><td>1</td><td>gn:2666199</td><td>largeMediumNegativeES</td></tr><tr><td>2</td><td>gn:2661881</td><td>largeMediumPositiveES</td></tr><tr><td>3</td><td>gn:660013</td><td>largeMediumPositiveES</td></tr><tr><td>4</td><td>gn:3469034</td><td>largeMediumPositiveES</td></tr></tbody></table>

(b) The query result for the use case
(first four rows)

<figure><img src="image_17.png" /><figcaption>Figure 3.8: Density plot for the **use case** query, showing a bias towards studies based on American population samples.</figcaption></figure>

## Use Case: Geographic Biases of Research Questions and Findings.

A researcher is studying the effect of income on mortality rates. They aim at discovering whether there is a geographic study bias for the research question, and whether the strength of the association differs among geographic regions. The researcher queries the MIRA-KG for all studies researching the influence of income (MESH:D007182) on mortality (MESH:D00902).

The output of the SPARQL query (first four rows shown in Table 3.7b) is visualised on a world map (see Figure 3.8), and shows a bias of research studies towards American samples for the specific correlation. Moreover, by aggregating the results, the historian notices that four studies that are based on American population samples measure a negative correlation, and four a positive.