### 3.4.3 The Knowledge Graph of Research Questions for Literature-based Discovery

To evaluate the feasibility and usefulness of the MIRA-KG to for literature-based discovery, we created a method to propose research questions based on the MIRA-KG, and, with three experts, we discussed the alignment of the tool with their scientific process as well as the perceived usefulness and veracity of the method and proposed research questions.

#### Framework

Here, we define literature-based discovery (LBD) as consisting of two tasks: (i) the task of extracting research questions $q \in Q$ from a set of articles $A$ called “the literature”, and (ii) the task of suggesting novel research questions $q_n \in Q_n$ from $Q$ based on some gap. We distinguish the following types:

1. **Novel research questions.** Novel research questions are new plausible combinations of study variables that have not been studied together before.

2. **Novel empirical evidence.** Studies in social demography, due to their observational nature, tend to be restricted to specific time periods and regions. As evidence for a specific research question accumulates, research findings consolidate, leading to a stronger and more robust body of evidence. Failing to do so leads to biases:

(a) **Geographical gaps.** Gaps in geographical evidence lead to geographical biases for certain research questions. For example, a focus on data samples from the United States.

(b) **Temporal gaps.** Gaps in temporal evidence lead to temporal biases for specific research questions. For example, a focus on data samples from before 2010.

An example of a novel research question, its explanation, and a possible study setup is shown in Figure 3.9.

#### Method

For our method, we rely on the following template to generate novel questions, where brackets indicate fillable **slots**:

**Research Question Template**

**Suggested RQ.** A study of the effect of {indicator} on {health outcome}, mediated by {indicator}, for {population} with/during a certain {indicator} in {region} between {period}.

**RQ. explanation.** The research question is interesting to examine because {explanation}.

**Suggested study.** {population} with a certain {attribute} living in {country} during {period}.

**Study explanation.** The specific population and context are interesting to examine because {explanation}.

The method is based on two steps: (1) research question synthesis, and (2) research question generation, as can be seen in 3.12.

We utilise capabilities of LLMs to generate research questions in fluid natural language and rely on structured Pydantic data models to generate the output.
<figure><img src="image_18.png" /><figcaption>Figure 3.9: Task of suggesting new research questions and studies by a set of articles from the body of literature. Colored terms indicate research question components, or 'slots' that can be filled.</figcaption></figure>

## KG-based discovery

To create new research questions, we further interlink the MIRA-KG by creating a subclass hierarchy of terms for study variables and other research question components, see Figure 3.10 and 3.11 for examples. Such a taxonomy can be used to better understand how various research questions are related to one another, in terms of closeness of their study variables in the hierarchy. The taxonomy was created by going over the MIRA-KG and manually grouping the variables together. At a later stage, we will curate the taxonomy with experts in the field of sociology.

<figure><img src="image_19.png" /><figcaption>Figure 3.10: Part of the subclass hierarchy of superclass socio-economic indicator.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Figure 3.11: Part of the subclass hierarchy of superclass health outcome.</figcaption></figure>

To propose novel research questions, we choose the following methods:

**Novel research question:** generate a novel research question:

1. retrieve an existing research question from the knowledge graph;

2. count the in-degree of its slots;
3. replace a slot with a high in-degree with one from the same superclass with a low in-degree.

**Address a spatiotemporal bias:** suggest novel evidence for a research question from the literature:

1. retrieve an existing research question from the knowledge graph;

2. calculate the total geographical coverage of the research question, and take the complement of that;

3. from the complement, select a novel country or period to address the research question's geographical or temporal bias.

To suggest a study sample and explanation, we use the generated research question and ingest it in the following prompt:

*Contextualize the population sample by describing the socioeconomic and political climate of the country being studied in light of the research question.*

## Qualitative user study

The user study was done at the International Institute of Social History (IISH) with three researchers: a social demographer, a researcher working on understanding cultural history using digital humanities techniques, and a data analyst and data engineer of (historical) demographic data. The user study was, first of all, meant to better understand user needs in terms of their scientific process, by studying the alignment between how they did research and the research generation framework that we proposed. Second, it was used to understand the usefulness of the KG and the research question generation tool within that scientific process.

## Research process alignment

The first part of the user study was tailored to understanding the research process of participants, by providing them with paper and markers, and giving them the task to visualise the discrete steps in their research:

1. *How do you start? Inspiration, datasets, journals, projects?*

2. *Do you make use of particular datasets and digital tools to design research questions and generate hypotheses?*

3. *How do you analyse your materials?*

4. *Do you use particular tools to refine your insights and your research questions and hypotheses?*

The main insight during the discussion that followed was that the way in which participants started their search for a project or testable research question were distinct. Ideas could be influenced by which societal issues were relevant, leading to a comparative literature study to find out information about the research question, i.e., which study variables are important and whether the question was posed in other time periods (most similar to the process described in this work). Ideas, however, could also come from encountering a problem in a census dataset, discussing and testing with colleagues whether the problem was known and already addressed, and thereafter doing a literature review. Moreover, personal research interests, funding themes, as well as availability of data, were indicated to impact, top down, which questions were addressed. Participants indicated that the main tools they used during research were used after the discovery of a general research question, for the purpose of understanding whether the
question was novel and placing it within the state of the art. Tools used after the discovery of an interesting questions were thus: scientific databases (i.e. Google Scholar) and colleagues.

**Evaluation of method** To evaluate the method, we presented participants with an intro-
ductory talk about language models and knowledge graphs, we walked participants through a
demo to generate novel research questions with the method, followed by a survey based on HI-
TAM constructs [25], in which they were each presented with the same four generated research
questions and explanations, see an example below.

Research question: A study of the effect of Employment status on Limiting Long-Term Illness rate

Geographic coverage: England

Temporal coverage: 1971-1991

Proposed geographic coverage: Foshan, China

Proposed temporal coverage: 1800-1801

Proposed study sample: Adults in Foshan, China between 1800 and 1801 with varying employment statuses

Explanation: The geographic coverage of the research question was limited to the region(s): England. Therefore, we suggest to test the hypothesis on a sample from the region Foshan, in China as well to mitigate the geographic bias. The temporal coverage of the research question was limited to the period between 1971 and 1991. Therefore, we suggest to test the hypothesis on a sample from the period between 1800 and 1801 as well to mitigate the temporal bias.

Explanation study sample: The proposed study sample consists of adults in Foshan, China between 1800 and 1801, a time characterized by the Qing Dynasty rule. Foshan in the early 19th century was known for its thriving economy, primarily driven by handicraft industries such as ceramics and silk production. The population during this period would likely have diverse employment statuses, ranging from artisans and traders to farmers and laborers. This setting provides a unique socio-economic context to study the relationship between employment status and limiting long-term illness rates, offering insights into historical health outcomes in a dynamic economic environment.

Literature used: DOI: https://doi.org/10.1093/ije/31.2.336

Participants indicated not to feel confident using the methods in their work (neutral (1) and
disagree (2) to Q: "I would feel confident using this application in my research"), yet all their
comments were positive (A: "If it would be more attuned to a wider set of research articles, and
less focused on historical demographics, for sure.", A: "Yes, I would certainly want to try it
out.", A: "Yes, if I am able to restrict it to a specific region and/or time period and can see
which journals were used as input.").

Moreover, many indicated to mostly use such an app during brainstorming Q: "In case I would like to come up with a novel hypothesis, I would use the app" (A: "I might use it during a brainstorm session.").

Concerning the trustworthiness and use of the research questions, answers were varied, as not all of the participants had the same research interests and worked on the same types of research questions. Therefore, answers to the question Q: "What do you feel is missing, in making these statements more trustworthy and useful in daily research practice?" mostly concerned the types of research questions (A: "I feel that the tool needs to be bounded by time period. Someone studying the modern age, is almost never interested in 1800" or A: "It's catered too strongly to specific types of articles."). Additionally, participants felt it difficult to judge whether research questions were feasible and interesting (A: "[...], a lot of expertise/insight is needed to judge/understand why a specific country is interesting for analyses". and A: "Feasibility, originality and added value.").), which is in line with findings from the research process alignment, in which it was found that most participants came up with research questions top down, from societally relevant topics or via interactions with colleagues. Both methods however did not generate research questions that had obvious problems.

**Quantitative analysis** In future work, a follow-up study will be conducted in which partici-
pants compare the KG-based method with an LLM-based method (summarisation and retrieval
augmented generation (RAG)) to understand better the merits of building a knowledge graph
for the automated generation of research questions. Specifically, we will look at two steps: (1)
research question synthesis, and (2) research question generation, see Figure 3.12

For step (1), a follow-up quantitative user study and semantic comparison will be conducted to
compare the LLM-based versus KG-based (which is based on a curated dataset) them in terms of
veracity. For step (2), we will conduct a user study, similarly based on HI-TAM constructs [25]. Moreover, we will do a semantic comparison in which we will use a set of automated metrics to compare research questions with the texts from which they were extracted: ROUGE-L [23], METEOR [2], and BLUE [23]. Moreover, we use precision, recall and F1 to assess whether all research questions are retrieved.

<figure><img src="image_21.png" /><figcaption>Figure 3.12: KG and LLM-based discovery</figcaption></figure>

## 3.5 Conclusions

Studies in (historical) social sciences, such as social demography, tend to be restricted to specific time periods and regions, making comparison of hypothesis tests across the vast array of research papers difficult. To assist social historians and demographers in the scientific process, this study describes MIRA, a method knowledge graph construction, and a resource consisting of a ontology and knowledge graph to capture research questions and findings in social demography.

This study shows that the MIRA ontology allows researchers to formulate their key questions, and research results in a structured and semantically sound way. Moreover, using over 400 abstracts from the field of social demography, our knowledge graph construction pipeline demonstrates that the knowledge-intensive task of semantic annotation can be (semi-)automated when employing a large language model, even without any training, validating data quality after automated annotation. Our qualitative user study indicated that even though researchers find literature-based discovery useful, it would take effort to integrate such tools into their scientific workflow due to a discrepancy between how our proposed methods generate research questions bottom-up, and how researchers come up with research questions, which appeared to be a combination between top-down discourse in society and within research groups, and a subsequent literature review.

Future work will focus on dealing with variations in natural language, and on expanding the MIRA-KG by application of the MIRA ontology on a larger set of studies in order for the field to reorganize itself, for instance by motivating researchers to publish their findings in a machine-readable manner. Moreover, future work will include a quantitative comparison between the KG and RAG-based method for literature-based discovery, as mentioned earlier in this work.
Chapter 3: References

[BA1] Sören Auer, Viktor Kovtun, Manuel Prinz, Anna Kasprzik, Markus Stocker, and Maria Esther Vidal. Towards a knowledge graph for science. In *Proceedings of the 8th international conference on web intelligence, mining and semantics*, pages 1–6, 2018.

[BA2] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In *Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization*, pages 65–72, 2005.

[BA3] Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. *Nucleic acids research*, 32(suppl_1):D267–D270, 2004.

[BA4] Cristina Iulia Bucur, Tobias Kuhn, Davide Ceolin, and Jacco Van Ossenbruggen. Expressing High-Level Scientific Claims with Formal Semantics. *K-CAP 2021 - Proceedings of the 11th Knowledge Capture Conference*, pages 233–240, 2021.

[BA5] Gustavo Candela, Pilar Escobar, Rafael C Carrasco, and Manuel Marco-Such. Evaluating the quality of linked open data in digital libraries. *Journal of Information Science*, 48(1):21–43, 2022.

[BA6] Victor de Boer, Ivette Bonestroo, Marijn Koolen, and Rik Hoekstra. A linked data model for data scopes. In *Metadata and Semantic Research: 14th International Conference, MTSR 2020, Madrid, Spain, December 2–4, 2020, Revised Selected Papers 14*, pages 345–351. Springer, 2021.

[BA7] Rosaline de Haan, Ilaria Tiddi, and Wouter Beek. Discovering research hypotheses in social science using knowledge graph embeddings. In *The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18*, pages 477–494. Springer, 2021.

[BA8] Danilo Dessí, Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, and Enrico Motta. Scicero: A deep learning and nlp approach for generating scientific knowledge graphs in the computer science domain. *Knowledge-Based Systems*, 258:109945, 2022.

[BA9] Danilo Dessí, Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, and Enrico Motta. Cs-kg: A large-scale knowledge graph of research entities and claims in computer science. In *The Semantic Web-ISWC 2022: 21st International Semantic Web Conference, Virtual Event, October 23–27, 2022, Proceedings*, pages 678–696. Springer, 2022.

[BA10] Danilo Dessì, Francesco Osborne, Diego Reforgiato Recupero, Davide Buscaldi, Enrico Motta, and Harald Sack. Ai-kg: an automatically generated knowledge graph of artificial intelligence. In *The Semantic Web-ISWC 2020: 19th International Semantic Web Conference, Virtual Event, October 23–27, 2020, Proceedings*, pages 1–6, 2020.
*Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part II* 19, pages 127-143. Springer, 2020.

[BA11] Michel Dumontier, Christopher JO Baker, Joachim Baran, Alison Callahan, Leonid Chepelev, José Cruz-Toledo, Nicholas R Del Rio, Geraint Duck, Laura I Furlong, Nichealla Keath, et al. The semanticscience integrated ontology (sio) for biomedical research and knowledge discovery. *Journal of biomedical semantics*, 5:1-11, 2014.

[BA12] Said Fathalla, Christoph Lange, and Sören Auer. An upper ontology for modern science branches and related entities. In *European Semantic Web Conference*, pages 436-453. Springer, 2023.

[BA13] Aldo Gangemi and Valentina Presutti. A multi-dimensional comparison of ontology design patterns for representing n-ary relations. In *SOFSEM 2013: Theory and Practice of Computer Science: 39th International Conference on Current Trends in Theory and Practice of Computer Science, Špindlerov Mlýn, Czech Republic, January 26-31, 2013. Proceedings* 39, pages 86-105. Springer, 2013.

[BA14] Daniel Garijo, Yolanda Gil, and Varun Ratnakar. The disk hypothesis ontology: Capturing hypothesis evolution for automated discovery. In *K-CAP Workshops*, pages 40-46, 2017.

[BA15] Yolanda Gil, Daniel Garijo, Varun Ratnakar, Rajiv Mayani, Ravali Adusumilli, Hunter Boyce, Arunima Srivastava, and Parag Mallick. Towards continuous scientific data analysis and hypothesis evolution. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 31, 2017.

[BA16] Elspeth Graham. Theory and explanation in demography: The case of low fertility in europe. *Population Studies*, 75(sup1):133-155, 2021.

[BA17] Paul Groth, Andrew Gibson, and Jan Velterop. The anatomy of a nanopublication. *Information services & use*, 30(1-2):51-56, 2010.

[BA18] Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. *Elife*, 6:e26726, 2017.

[BA19] Rik Hoekstra and Marijn Koolen. Data scopes for digital history research. *Historical Methods: A Journal of Quantitative and Interdisciplinary History*, 52(2):79-94, 2019.

[BA20] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. *Advances in neural information processing systems*, 35:22199-22213, 2022.

[BA21] Tobias Kuhn, Christine Chichester, Michael Krauthammer, Núria Queralt-Rosinach, Ruben Verborgh, George Giannakopoulos, Axel-Cyrille Ngonga Ngomo, Raffaele Viglianti, and Michel Dumontier. Decentralized provenance-aware publishing with nanopublications. *PeerJ Computer Science*, 2:e78, 2016.

[BA22] Pierre Larmande and Konstantin Todorov. AgroLD: A Knowledge Graph for the Plant Sciences. *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, 12922 LNCS:496-510, 2021.

[BA23] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In *Text summarization branches out*, pages 74-81, 2004.
[BA24] Ian H. Magnusson and Scott E. Friedman. Extracting Fine-Grained Knowledge Graphs of Scientific Claims: Dataset and Transformer-Based Results. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 4651-4658, 2021.

[BA25] Yaoli Mao, Janet Rafner, Yi Wang, and Jacob Sherson. A hybrid intelligence approach to training generative design assistants: partnership between human experts and ai enhanced co-creative tools. In HHAI 2023: Augmenting Human Intellect, pages 108-123. IOS Press, 2023.

[BA26] Chris Mavergames, Silver Oliver, and Lorne Becker. Systematic reviews as an interface to the web of (trial) data: using pico as an ontology for knowledge synthesis in evidence-based healthcare research. SePublica, 994:22-6, 2013.

[BA27] Albert Meroño-Peñuela, Ashkan Ashkpour, Christophe Guéret, and Stefan Schlobach. Cedar: the dutch historical censuses as linked open data. Semantic Web, 8(2):297-310, 2017.

[BA28] Spyros Missiakoulis. Cecrops, king of athens: the first (?) recorded population census in history. International Statistical Review, 78(3):413-418, 2010.

[BA29] Joseph Molitoris and Martin Dribe. Industrialization and inequality revisited: Mortality differentials and vulnerability to economic stress in stockholm, 1878-1926. European Review of Economic History, 20(2):176-197, 2016.

[BA30] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.

[BA31] Valentina Presutti, Enrico Daga, Aldo Gangemi, and Eva Blomqvist. extreme design with content ontology design patterns. In Proc. Workshop on Ontology Patterns, pages 83-97, 2009.

[BA32] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

[BA33] Giuliana Spadaro, Ilaria Tiddi, Simon Columbus, Shuxian Jin, Annette Ten Teije, CoDa Team, and Daniel Balliet. The cooperation databank: machine-readable science accelerates research synthesis. Perspectives on Psychological Science, 17(5):1472-1489, 2022.

[BA34] Markus Stocker, Tina Heger, Artur Schweidtmann, Hanna Ćwiek-Kupczyńska, Lyubomir Penev, Milan Dojchinovski, Egon Willighagen, Maria-Esther Vidal, Houcemeddine Turki, Daniel Balliet, et al. Skg4eosc-scholarly knowledge graphs for eosc: Establishing a backbone of knowledge graphs for fair scholarly information in eosc. Research Ideas and Outcomes, 8:e83789, 2022.

[BA35] Markus Stocker, Allard Oelen, Mohamad Yaser Jaradeh, Muhammad Haris, Omar Arab Oghli, Golsa Heidari, Hassan Hussein, Anna-Lena Lorenz, Salomon Kabenamualu, Kheir Eddine Farfar, et al. Fair scientific information with the open research knowledge graph. FAIR Connect, 1(1):19-21, 2023.

[BA36] Ilaria Tiddi, Daniel Balliet, and Annette ten Teije. Fostering scientific meta-analyses with knowledge graphs: a case-study. In The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings 17, pages 287-303. Springer, 2020.
[BA37] Susan Marie Viet, Jill C Falman, Lori S Merrill, Elaine M Faustman, David A Savitz, Nancy Mervish, Dana B Barr, Lisa A Peterson, Robert Wright, David Balshaw, et al. Human health exposure analysis resource (hhear): A model for incorporating the exposome into health studies. *International journal of hygiene and environmental health*, 235:113768, 2021.
# Part III

## Understanding Society through Social Media
# Chapter 4

## Towards Conflictual Narrative Mechanics

Laura Spillner*, Carlo Santagiustina†‡, Thomas Mildner*, Robert Porzel*

*Digital Media Lab, University of Bremen

†Venice International University, Italy

‡Ca’Foscari University of Venice, Italy

### Abstract

We propose a five steps methodology to retrieve, reconstruct and analyse conflict related narratives in a standardized and automated way. Our methodology combines AI and network analysis techniques to build a visual representation of key agents and entities involved in a conflict and to characterize their relations. Unlike the majority of existing methods, ours can be applied to any type of conflict, as, through two data downloading phases, it first generates a bird’s-eye representation and then a fine-grained map of any conflict. Given the broad applicability of the proposed methodology, we believe that this work moves the first steps towards a better understanding of conflictual narrative mechanics.

### 4.1 Introduction

Online social media have started as tools for people to connect with others all over the globe [38, 19, 26]. Meanwhile, platforms like Facebook, Reddit, and Twitter have been labeled ‘game-changers’ for entertainment [38, 11] while offering novel opportunities to advertisers and businesses [18, 16, 25]. Increasing the impact of social media even further, platforms take an important part of the everyday political discourse [22, 23, 30]. Twitter in particular is used not only by people who share their opinions and engage in political discussions, the platform is also used by official agencies as well as people working in governments to disseminate information more rapidly than via traditional news media.

When it comes to political and social conflicts, the surrounding online conversation is often characterized by a range of opinions. Discourse may evolve around conflict, parties involved therein, or actions undertaken while different ‘causal’ relationships between events and actors may be asserted in naïve manners. By being constantly exposed to these conversational dynamics, users’ opinions may be influenced by the narrative that is most popular based on general interest of their personal social media bubble. This effect is further amplified by tailored algorithms that elevate content predicted to be aligned with prior interest. This results in a
concerning problem for users as well as public discourse to experience alternative points of views.

In a recent crisis, this problematic tendency became blatantly real when Russia started its invasive war against Ukraine on February 14th, 2022, escalating a fiery situation that began with Russia's annexation of the Ukrainian peninsula Crimea in 2014. Soon after the war's outbreak, on the 28th of February, 2022, the social network Twitter decided to expand labeling policies for content as a counter to the spread of misinformation on behalf of Russia, adding the 'Russia state - affiliated media' tag to related posts. Showing an impact of the spread of Russia's media outlets [3], Twitter's decision moved Russia to block the platform in its nation, as well as Facebook who followed a similar strategy. Both interventions, of Twitter and of Russia, highlight how vulnerable public speech and information can become. In this particular case, Russia's government decided to cut off their country from major western media outlets, allowing them to precisely control news available to Russian's citizens.

Resulting conflictual narratives, occurring during crises like the one mentioned above, present an urgent need to understand underlying mechanics. In this paper, we present a methodology, as a work in progress, to study and investigate the online narratives surrounding conflicts and crisis. The methodology itself is not necessarily limited to conflict alone, but aims to discover different perspectives on social media while limiting any introduced researcher bias when constructing the corpus itself. This paper describes the idea behind the methodology. We aim to use the methodology to analyse Twitter conversations of Russia's invasive war against Ukraine, and we present here the results of the first (preliminary) part of this analysis.

Our hypothesis is that conflict narratives are strategically designed around recurrent storytelling patterns and frames that assign a set of (asymmetric and stereotyped) roles to involved parties. We plan to ultimately utilize the methodology presented herein to investigate this hypothesis on the case-study of the Russo-Ukraine war; the results of this investigation will be presented in future work.

When trying to capture the context and viewpoint of narratives, one of the main difficulties is not to introduce researcher bias to sampled data: By collecting, for example, only posts written in English, by defining which terms to query for, which conflict-related narrative frames are being looked up, or which actors in the conflict are of interest apart from the main parties, any resulting corpus represents only a certain fraction of the discourse.

Therefore, we define a two-phase approach to investigate conflict narratives based on online conversations on Twitter:

1. A dataset is collected of tweets that explicitly mention the conflict at hand through the names of the main parties or neutral conflict-related terms. From this, the entities involved in the conflict, such as actors, events, and locations, either actively or passively, are discovered and the most important entities and their co-occurrence are identified.

2. More fine-grained data is then collected by searching for all the main entities and capturing frames used to characterize these entities as well as possible relationships between them.

So far, we have conducted the first phase of the analysis on the conflict between Russia and Ukraine, as characterized by a sample of the Twitter conversation in the spring of 2022. In order to explain the proposed methodology, we present the results of this first step, detail the steps planned for the second phase analysis, and discuss which challenges remain to be solved in order to expand the methodology to a greater scope and other case studies.
## 4.2 Related Work

The following section presents a brief summary of recent literature on conflict-related narratives with a particular focus on how these narratives are constructed and diffused through online social media.

### 4.2.1 Conflict narratives

Conflicts and their escalation in the physical and narrative space are generally the by-product of heterogeneous beliefs [27], asymmetric information [33], cognitive biases [17], like the availability heuristic and the confirmation bias, as well as complex entanglements [2, 6] of competing interests, strategies, and objectives, which are often opaque being difficult to elicit and model.

Conflict narratives, and more in general debates about conflicts through which these narratives evolve and spread, have long been studied in the social and political sciences under many different approaches and perspectives [28, 35, 4]. For example, the study of conflict framing [36], factional discourse design [39], and other polarizing communication strategies [34] is key for understanding any consensus-building and group-mobilization process when there are competing views or interests at stake.

The narrative dimensions of conflicts are generally more visible in non-authoritarian countries, where partisan narratives [29] are constructed and employed in relation to public support-building and policy justification. Conflict narratives operate at all levels of national and international governance processes. Especially democratic countries aim to mitigate and resolve potential or actual conflicts through transparent, informed, and participatory deliberative processes. From local debates about public policies to be implemented in response to a pandemic, like COVID-19, to the renegotiation of trade agreements between parties that compete for the control on strategic natural resources and technologies, like conflicts for rare earths or for the control over semi-conductor industry technologies, the emergence of conflict narratives is an ubiquitous phenomenon in contemporary times. This is particularly evident in decentralized and multi-directional online communication mediums such as social media [1, 12], which have become the default propagation medium for (popular) narratives, including conflict-related ones.

### 4.2.2 Conflict modeling

Recent attempts to model conflicts, like [10], have shown that conflicts are not necessarily the outcome of diverging material interests among individuals and groups, as modeled in early game-theoretic works [15]. They can also be the product of: (i) differently-biased or competing world views used to decipher events and to comprehend the intentions underlying specific actions or communications by other individuals and groups [8]; or (ii) to the deterioration of inter-group trust [31]. As a result, conflicts may emerge and exacerbate even when the material interests of the different parties converge from a rational (i.e., utilitarian) perspective. For this reason, the presence of extrinsic incentives [24] may not suffice to mitigate or resolve ongoing conflicts grounded on incompatible belief-systems or on the lack of trust, like during the cold-war [13]. For example, this is the case in attrition wars, which may be represented as negative-sum games. Rather than immediately negotiating a mutually advantageous agreement, two parties are ready to bear the material and humanitarian costs of a long-lasting conflict if that gives them the opportunity to punish the other, by reciprocating harm and causing a similar or larger damage to their opponents.
### 4.2.3 Conflict-related narratives extraction and events reconstruction

The incessant growth of online social media and their communities [9], together with the increasing availability of computational power and advanced linguistic analysis methods [4, 4, 5] for big textual datasets offers an opportunity to capture and model conflict narrative dynamics on an unprecedented scale.

Recent works [36, 29, 41] have used posting activities on social media and online newspaper articles, including comment sections, for capturing and mapping partisan or faction-specific arguments and their dynamics across time through a combination of AI and network analysis methods. A further step towards the automated mapping of conflicts and of their key actors and events has been done by [13] and [21] who combined large-scale knowledge graphs with semi-structured sources in an event KG RDF-representation.

The former branches of research open the way to a new AI-augmented research field on all kinds of conflicts. Such research could combine the potentials of NLP, network analysis, and computational linguistic methods with the semantic web, serving as interfaces for the real-time observation and understanding of conflict-related narratives.

## 4.3 Methodology

To analyze the narratives that surround certain conflicts, we developed a methodology consisting of two phases which include two cycles of data collection. The reasons behind this are twofold: Firstly, it allows the discovering of actors and entities of interest that are involved in the conflict dynamically from the online conversation surrounding it, instead of defining them a-priori. Thus, we hope to reduce the bias that would otherwise be introduced during the corpus construction. Secondly, this also makes it possible to employ the same methodology to investigate extremely different topics and types of conflicts.

Although in this paper, we focus on the Russian war against Ukraine, the aim for this approach is to be independent of any specific topic in order to enable other works to also re-construct narratives from different crisis, whether violent or non-violent, where the parties involved are not clearly defined, may change over time, or where the definition of parties depends on specific viewpoints.

In the following sub-sections, we present how we are currently using this two phase approach to identify which narratives are shaping the online conversation on Titter concerning the ongoing war in Ukraine. Phase 1 describes the analysis that has been conducted so far as well as the results of this, while Phase 2 discusses the steps that we plan to conduct next.

#### 4.3.1 Phase 1 - Discovery

In this first discovery phase, the goal is to collect tweets from a specified time interval which explicitly mention the conflict in question. We thereby collect any tweet that meets two conditions: Tweets that (1) either contain the name of at least one of the directly involved parties or the name of a specific event and that (2) contain a generic term like conflict, tension, or crises, denoting that the tweet refers to a conflict related to the selected event or involved parties. From these tweets, we identify all actors and entities involved in the conflict which are mentioned most often in this context. We are then able to construct a network based on their co-occurences.
## Step 1: Corpus construction

To construct the first corpus on the current war in Ukraine, we collected all tweets

* between the first of January and the first of May 2022

* which contain the words: “Ukraine”, “Russia”, and any of “conflict(s)”, “tension(s)”, “crises” or “crisis”

* which are not retweets.

As of now, we have only conducted a preliminary trial in order to test and validate our methodology. For this, we have only searched for these terms in English. Hence, the collected corpus only represents a perspective from English-speaking Twitter users. However, we want to also collect tweets containing these key terms in other languages, including Ukrainian and Russian, in order to generate a multilingual corpus that represents a broader view and contains differing narratives that include those from both of the opposing parties.

An advantage of this two phase approach is that at the beginning, only a small number of terms needs to be defined, which are compatible with any type of conflict. We plan to employ the help of native speakers of other languages to translate these terms and validate the results of the first phase. However, we hope that we will not require the help of experts in the domain which are also native speakers of the additional languages. Apart from the definition of this small set of neutral conflict-related starting terms, the collection of the data (across both phases) is entirely automatic. We chose to go with only these three terms (conflict, tension and crisis) firstly in order to limit the scope of the twitter query, and secondly because we believe that other synonyms of those terms generally connote either violent or non-violent conflicts (e.g. synonyms of “conflict” on ConceptNet include terms such as “battle” or “disagreement”).

For this preliminary analysis, we collected a total of 724400 tweets. In a first pre-processing step, we removed special UTF-8 characters, like emojis, emoticons, and URL links.

## Step 2: Entity Recognition

After generating this dataset, we use this broad-coverage overview of the conflict at hand to discover from it which actors or other entities are relevant to the conflict. These will constitute the terms which we will explicitly teach for in the second phase. Of interest are

* named entities such as known persons, organizations, countries or peoples, e.g. in this case “Putin”, “Ukraine”, “EU”, “Russians”.

* noun phrases which include these same terms as signifiers, such as “Ukrainian president”, “russian army”, “Ukraine war”, “russia-ukraine conflict”, etc.

Therefore, instead of defining a term for the conflict and thus characterizing it ourselves – based on the different connotations of “war”, “unrest”, “invasion”, “operation”, etc. – we are able to discover which terms are used in online conversations. The same holds for other entities involved, e.g. a person being referred to as a “president” or “dictator”. Either variant will be collected if it appears in a noun phrase together with one of the named entities.

In this step, we use the open source NLP library spaCy for part-of-speech tagging, dependency parsing, and named entity recognition. From this, we identify all noun phrases which include a token that was classified as a named entity. Such identified entities would include only geopolitical entities such as countries, nationalities or religious/political groups, organizations, persons, events, and locations. However, these entities would exclude instances such as cardinals.
<figure><img src="image_22.png" /><figcaption>Figure 4.1: Entities network for the Ukraine-Russia conflict. Network filtered by node weighted degree: 0.999 percentile; and then by edge weight: 0.99 percentile.</figcaption></figure>

Identified noun phrases can then be consolidated in entities, by linking them to named entities in a known knowledge graph such as WikiData (consolidating e.g. "the dictator Putin", "Vladimir Putin" and "the russian president", etc.). For the preliminary analysis that follows, the consolidation step has yet to be implemented. In future versions of this work this will be done through name matching and name substring matching, without disambiguation, or through more advanced distributional semantics and ML methods, like OpenTapioca [14].

### Step 3: Co-occurence network

In the third step, identified entities are mapped through an undirected network based on their co-occurences in the corpus. Each node represents an entity, with edge weights denoting how often two entities occur together in one tweet in the dataset. By so doing we obtain a weighted and undirected network containing 933081 nodes and 10871807 edges. With an average degree of nodes equal to 23.303 and an average weighted degree of nodes equal to 46.162.

Network nodes are then filtered based on their weighted degree centrality metric; alternative centrality measures, like pagerank, betweenness or eigen-centrality may also be used for this purpose. This filtration is done in order to remove entities that are less influential (more peripheral) and have a marginal role in the entity network for the selected conflict. As shown in Figure 4.2, this filtration step removes those entities which are mentioned rarely in the dataset, like entities weekly related to the conflict and other irrelevant noun phrases. This might include entities such as sports teams or their fans who might be involved in a metaphorical conflict, misspellings, or other noun phrases that do not play a relevant role in online conflict-related narratives.

### 4.3.2 Phase 2 - Analysis of Narratives

In the second phase, we plan to retrieve thinner grained data about the conflict on the selected social media, in order to (re-)construct and analyze the narratives surrounding specific relational blocks. As a starting point for this will serve the key actors/entities and relations (dyads of
<figure><img src="image_23.png" /><figcaption>Figure 4.2: Entities network for the Ukraine-Russia conflict. Network filtered by node weighted degree: 0.9999 percentile; and then by edge weight: 0.9 percentile.</figcaption></figure>
actors), which were identified in Phase 1 based on the chosen centrality metric, together with
the largest/heaviest cliques of order 3+.

Step 4: Second corpus construction

At this point, we have identified a number of key entities and relations, as well as different
reference terms which are used to refer to them. These will then be used to construct a new
set of queries from which a second corpus of tweets is collected. In this second phase, we are
going to search explicitly for the entities that the first phase identified as being perceived as
important actors in the conflict, using the terms that were discovered to be used to refer to
them, by twitter users.

While the first corpus included only tweets that reference the conflict itself, thus providing a
broader view of the online conversation surrounding it, this second corpus will include tweets
referring specifically to one or more of the relevant entities and allow for a more fine-grained
analysis. By collecting these tweets, we aim to identify how the entities themselves, as well
as the relationships between them, are characterized by different people online, for example
through adjectives and verbs qualifying the relation between two key entities.

Our hypothesis is that the narratives surrounding the actors involved in the conflict are based
on recurring phrase fragment patterns and frames that assign specific roles and attributes to
the involved parties. In many cases, these roles are a-symmetric and mirror certain stereotypes
that are common to (almost) all conflict narratives, like the role of the victim and that of the
perpetrator. We plan to use the methodology we describe herein to investigate this hypothesis
on the corpus we are currently collecting about the war in Ukraine. Using the tweets from this
Phase 2 corpus, we aim to collect and assign a number of different frames to the actors and
relations, which are essential constituents of the conflict narrative. For example, we expect that
we will find

* verbs related to asymmetric roles in the conflict, like: aggression / protection, offense / defense, attack / counter-attack, ownership claims, “deserviness” claims, resisting/ surrendering, etc.

* conflict related nouns and adjectives, like: aggressor / aggressed, invader / invaded, liberator / liberated, oppressor / oppressed, strong / weak, winning / losing, perpetrator / victim, etc.

* characterizations of the conflict or its escalation, like: justified / unjustified, legitimate / illegitimate, necessary / unnecessary, explainable / unexplainable, expected / unexpected, hot / cold, violent / non-violent, verbal / physical, ideological, political, economic, financial, military, etc.

* equivalently, characterizations of

– the peoples or populations involved in the conflict

– the leaders of the factions involved in the conflict

– the countries involved in the conflict

– the factions, armies or soldiers involved in the conflict

– etc.
Step 5 - Analysis

From the entities, relations and frames used to characterize them, which are collected in the
previous step, a second, fine-grained and dynamic network will be constructed. This network,
which could be based of RDF-star or a labelled property graph, is an in-depth representation of
the narratives describing the relations between the key actors and their characteristics. For this
step, we plan to again employ the part-of-speech recognition and dependency tree identified
using the spaCy toolkit to reconstruct this network.

4.4 Discussion

The network we have already constructed from the Phase 1 dataset shows an interesting bird's-
eye perspective on the online narratives surrounding the selected conflict, that is, the war
between Ukraine and Russia. However, this pipeline for automatically reconstructing conflict
narratives is still at a very early development stage. Open questions remain concerning the
advantages and disadvantages of the centrality measures and AI methods to be used at each
stage of the process, for instance, filtering the actors network. A future aim is to explore and
benchmark these alternative procedures and metrics, as well as their impact on the results,
throughout future works, employing multiple conflict datasets. Here follows a brief discussion
about the criticalities that we have identified at the current state of the work and that we will
address as a priority in the next development stages.

Firstly, we plan to evaluate the most useful way to construct the co-occurence network. Currently, the network is based on the counts of co-occurences of identified entities and noun phrases in tweets. More precisely, the more often two entities appear in the same tweet, the higher the weight of that edge is. An alternative approach is to take the frequency of any two connected nodes into consideration when having to weigh the relation between two entities (as more frequently occurring entities are also more likely to appear together by mere chance). This metric would also allow us to differentiate between pairs of nodes that appear together more often than random, and pairs of nodes that appear together less often than random. Similar approaches have been used in other fields of studies, for example, for species probabilistic co-occurrence analysis. In connection with this, we are also considering a number of different options to calculate the centrality of a node, which is currently based on its degree.

The second part which we will further investigate is the representation of conflict dynamics over time. The current network is based on the entire set of tweets collected from a defined time frame. During the next iteration, we plan to firstly collect tweets from a larger time interval and secondly to slice the network into smaller time-chunks. This will enable us to visualize and analyze how the network changes over time. We expect an interesting perspective to be added to both the co-occurence network of the first phase as well as to the more fine-grained network.

Thirdly and finally, we plan for the next iteration of this dataset to analyze more of the context
data surrounding the tweets themselves: Many tweets include metadata about the country of
origin, time zone, location of users, and their language, signifying emojis, flags and hashtags,
as well as the metadata connected to the user's account. Considering users' biographies or
geolocation might make it possible to look at which narratives are more prominent among
which groups of users or locations. This is also connected to our previous goal of adding more
languages to the corpus.
Chapter 4: References

[CA1] Mona Abdel-Fadil. The politics of affect: The glue of religious and identity conflicts in social media. *Journal of Religion, Media and Digital Culture*, 8(1):11–34, 2019.

[CA2] James M Acton. Escalation through entanglement: How the vulnerability of command-and-control systems raises the risks of an inadvertent nuclear war. *International security*, 43(1):56–99, 2018.

[CA3] Jesús Aguerri, Mario Santisteban, and Fernando Miró-Llinares. The fight against disinformation and its consequences: Measuring the impact of “russia state-affiliated media” on twitter. *OSF Preprints*, 2022.

[CA4] Sanam Naraghi Anderlini, Vasu Gounden, Ed Garcia, Barbara Harff, Maha Khan, Deepa Khosla, Michael S Lund, David Nyheim, Luis Alberto Padilla, Thomas Schmальberger, et al. *Journeys through conflict: Narratives and lessons*. Rowman & Littlefield, 2001.

[CA5] Nabiha Asghar. Automatic extraction of causal relations from natural language texts: a comprehensive survey. *arXiv preprint arXiv:1605.07895*, 2016.

[CA6] Greg Austin. Unwanted entanglement: the Philippines' spratly policy as a case study in conflict enhancement? *Security Dialogue*, 34(1):41–54, 2003.

[CA7] Katrien Beuls, Paul Van Eecke, and Vanja Sophie Cangalovic. A computational construction grammar approach to semantic frame extraction. *Linguistics Vanguard*, 7(1), 2021.

[CA8] Colin F Camerer. Progress in behavioral game theory. *Journal of economic perspectives*, 11(4):167–188, 1997.

[CA9] Umit Can and Bilal Alatas. A new direction in social network analysis: Online social network analysis problems and applications. *Physica A: Statistical Mechanics and its Applications*, 535:122372, 2019.

[CA10] Samuel Charap and Timothy J Colton. The negative-sum game and how to move past it. *Adelphi Series*, 56(460):151–184, 2016.

[CA11] Stuart Cunningham and David Craig. Being ‘really real’ on YouTube: authenticity, community and brand culture in social media entertainment. *Media International Australia*, 164(1):71–81, August 2017. Publisher: SAGE Publications Ltd.

[CA12] Gjalt de Graaf and Albert Meijer. Social media and value conflicts: An explorative study of the dutch police. *Public Administration Review*, 79(1):82–92, 2019.

[CA13] Anne Deighton. *The Impossible Peace: Britain, the Division of Germany, and the Origins of the Cold War*. Oxford University Press, 1993.

[CA14] Antonin Delpeuch. Opentapioca: Lightweight entity linking for wikidata, 2019.
[CA15] Melvin Dresher. Theory of games of strategy. Technical report, RAND CORP SANTA MONICA CA, 1956.

[CA16] Carsten Ellwein and Benedikt Noller. Social Media Mining: Impact of the Business Model and Privacy Settings. In *Proceedings of the 1st ACM Workshop on Social Media World Sensors*, SIdEWayS '15, pages 3–8, New York, NY, USA, September 2015. Association for Computing Machinery.

[CA17] Hans-Peter Erb, Gerd Bohner, Knut Schmilzle, and Susanne Rank. Beyond conflict and discrepancy: Cognitive bias in minority and majority influence. *Personality and Social Psychology Bulletin*, 24(6):620–633, 1998.

[CA18] Lukas P. Forbes and Lukas P. Forbes. Does Social Media Influence Consumer Buying Behavior? An Investigation Of Recommendations And Purchases. *Journal of Business & Economics Research (JBER)*, 11(2):107–112, January 2013. Number: 2.

[CA19] Lisa M. Given, Denise Cantrell Winkler, and Kathryn Hopps-Wallis. Social Media for Social Good: A Study of Experiences and Opportunities in Rural Australia. In *Proceedings of the 8th International Conference on Social Media & Society*, #SMSociety17, pages 1–7, New York, NY, USA, July 2017. Association for Computing Machinery.

[CA20] Simon Gottschalk and Elena Demidova. Eventkg—the hub of event knowledge on the web—and biographical timeline generation. *Semantic Web*, 10(6):1039–1070, 2019.

[CA21] Simon Gottschalk, Endri Kacupaj, Sara Abdollahi, Diego Alves, Gabriel Amaral, Elisavet Koutsiana, Tin Kuculo, Daniela Major, Caio Mello, Gullal S Cheema, et al. Oekg: The open event knowledge graph. In *CLEOPATRA@ WWW*, pages 61–75, 2021.

[CA22] Alfred Hermida, Fred Fletcher, Darryl Korell, and Donna Logan. Share, Like, Recom mend. *Journalism Studies*, 13(5-6):815–824, October 2012. Publisher: Routledge_eprint: https://doi.org/10.1080/1461670X.2012.664430.

[CA23] Kristoffer Holt, Adam Shehata, Jesper Strömbäck, and Elisabet Ljungberg. Age and the effects of news media attention and social media use on political interest and participation: Do social media function as leveller? *European Journal of Communication*, 28(1):19–34, February 2013. Publisher: SAGE Publications Ltd.

[CA24] Amelia Hoover Green. The commander's dilemma: Creating and controlling armed group violence. *Journal of Peace Research*, 53(5):619–632, 2016.

[CA25] Arunesh Mathur, Arvind Narayanan, and Marshini Chetty. Endorsements on Social Media: An Empirical Study of Affiliate Marketing Disclosures on YouTube and Pinterest. *Proceedings of the ACM on Human-Computer Interaction*, 2(CSCW):1–26, November 2018. arXiv: 1809.00620.

[CA26] Thomas Mildner and Gian-Luca Savino. Ethical User Interfaces: Exploring the Effects of Dark Patterns on Facebook. In *Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems*, pages 1–7. Association for Computing Machinery, New York, NY, USA, May 2021.

[CA27] Antony Millner and Hélène Ollivier. Beliefs, politics, and environmental policy. *Review of Environmental Economics and Policy*, 2020.

[CA28] Josiah Ober. Political conflicts, political debates, and political thought. *The Shorter Oxford History of Europe I: Classical Greece*, pages 111–38, 2000.

[CA29] Andrew J Polsky. Partisan regimes in american politics. *Polity*, 44(1):51–80, 2012.
[CA30] Eugenia Ha Rim Rho and Melissa Mazmanian. Political Hashtags &amp; the Lost Art of Democratic Discourse. In *Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems*, CHI '20, pages 1–13, New York, NY, USA, April 2020. Association for Computing Machinery.

[CA31] Dominic Rohner, Mathias Thoenig, and Fabrizio Zilibotti. War signals: A theory of trade, trust, and conflict. *Review of Economic Studies*, 80(3):1114–1147, 2013.

[CA32] Carlo Romano Marcello Alessandro Santagiustina and Massimo Warglien. The architecture of partisan debates: The online controversy on the no-deal brexit. *PLoS one*, 2022 (forthcoming).

[CA33] Alastair Smith and Allan C Stam. Bargaining and the nature of war. *Journal of Conflict Resolution*, 48(6):783–813, 2004.

[CA34] Murat Somer, Jennifer L McCoy, and Russell E Luke. Pernicious polarization, autocoratization and opposition strategies. *Democratization*, 28(5):929–948, 2021.

[CA35] Özden Melis Uluğ, Brian Lickel, Bernhard Leidner, and Gilad Hirschberger. How do conflict narratives shape conflict-and peace-related outcomes among majority group members? the role of competitive victimhood in intractable conflicts. *Group Processes & Intergroup Relations*, 24(5):797–814, 2021.

[CA36] Emma van der Goot, Sanne Kruikemeier, Jeroen de Ridder, and Rens Vliegenthart. Online and offline battles: Usage of different political conflict frames. *The International Journal of Press/Politics*, page 19401612221096633, 2022.

[CA37] Francois van Schalkwyk, Jonathan Dudek, and Rodrigo Costas. Communities of shared interests and cognitive bridges: The case of the anti-vaccination movement on twitter. *Scientometrics*, 125(2):1499–1516, 2020.

[CA38] Anita Whiting and David Williams. Why people use social media: a uses and gratifications approach. *Qualitative Market Research: An International Journal*, 16(4):362–369, August 2013.

[CA39] Fredel M Wiant. Exploiting factional discourse: Wedge issues in contemporary american political campaigns. *Southern Journal of Communication*, 67(3):276–289, 2002.

[CA40] Tom Willaert, Sven Banisch, Paul Van Eecke, and Katrien Beuls. Tracking causal relations in the news: data, tools, and models for the analysis of argumentative statements in online media. *Digital Scholarship in the Humanities*, 2021.

[CA41] Tom Willaert, Paul Van Eecke, Jeroen Van Soest, and Katrien Beuls. An opinion facilitator for online news media. *Frontiers in big Data*, 4:46, 2021.
# Chapter 5

## OKG: A Knowledge Graph for
Fine-grained Understanding of Social
Media Discourse on Inequality

Inès Blin*†, Lise Stork†, Laura Spillner‡, Carlo Santagiustina△△

*Sony Computer Science Laboratories - Paris

†Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

‡Digital Media Laboratory, University of Bremen

△ Venice International University, Italy

△ Ca'Foscari University of Venice, Italy

### Abstract

In recent years, social media platforms such as Twitter have allowed people to voice their opinions by engaging in online discussions. The availability of such discussions has garnered interest amongst researchers in analyzing the dynamics on critical topics, such as inequality. Most of the current strategies are, however, limited with respect to conveying the fine-grained opinions of users, focusing on tasks such as sentiment analysis or topic modeling that extract coarse categorizations. In this work, we address this challenge by integrating a Twitter corpus with the output of finer-grained semantic parsing for the analysis of social media discourse. To do so, we first introduce the *OBservatory Integrated Ontology (OBIO)* that integrates social media metadata with various types of linguistic knowledge. We then present the *Observatory Knowledge Graph (OKG)*, a knowledge graph in terms of the ontology, populated with tweets on inequality. We lastly provide use cases showing how the knowledge graph can be used as the backbone of a social media observatory, to facilitate a deeper understanding of social media discourse.

**Keywords** Information extraction, Semantic networks, Ontology engineering

### 5.1 Introduction

In recent years, the proliferation of social media platforms such as Twitter has allowed people to voice their opinions by engaging in online discussions. The accessibility of several of these online resources has piqued the interest of researchers and policymakers. They are now eager to capture and discover perspectives and impactful narratives circulating throughout society on critical topics like migration, war, vaccination, inequality, or climate change. Many works
have addressed this interest by publishing dashboards [35], social media datasets [8, 13, 14], and by leveraging automated natural language processing (NLP) strategies for the discovery and analysis of online debates [23, 29, 30, 36].

Traditional NLP strategies for the understanding of online debates have focused commonly on sentiment analysis, opinion mining and topic modeling [26, 34]. Such strategies are limited in their capabilities to convey fine-grained analysis of the opinions of individuals or social groups in the form of narratives, arguments or claims, given that statistical strategies are used to label natural language texts that are vague and imprecise in nature. The complexity of Twitter posts, like the usage of slang and acronyms, further complicates precise interpretation.

Fine-grained text analysis techniques, e.g., named entity recognition [12], relation extraction [39], semantic role labeling [21], frame extraction [32], and dependency parsing [19] can provide fine-grained insights into the specific stances communities take in a debate or narrative. Such techniques allow researchers to retrace the provenance of their findings [38], ensuring the validity of results, reproducibility of experiments, and fostering transparency in research. We argue that the integration of tweet metadata with the output of both fine and coarse-grained NLP analyses into a single integrative network, allows researchers to perform increasingly complex analyses to better understand online debates.

With this goal in mind, this work introduces the *Observatory Knowledge Graph (OKG)*, a knowledge graph in terms of the *OBservatory Integrated Ontology (OBIO)*, which integrates tweet metadata with various types of linguistic knowledge and Linked Open Data (LOD), such as named entities, dependencies, and PropBank rolesets. We present use cases that demonstrate how the OKG can aid researchers in understanding online discussions on inequalities [43], e.g., perceived causes, driving factors and effects of inequalities, as well as the relations among mentioned entities (people, places, events, organizations, etc.) Thus, the contributions of this paper are twofold:

1. the *OBservatory Integrated Ontology (OBIO)*<fnref n="1" /> which integrates tweet metadata with linguistic analysis data.

2. the *Observatory Knowledge Graph (OKG)*, which integrates a Twitter corpus with the output of both fine-grained and coarse-grained NLP analyses, and present relevant use cases. Code and the OKG are available via a github library² and Zenodo³ respectively. Upon request, access to the SPARQL endpoint⁴, maintained both by Triply⁵ and the IISG⁶, can be provided.

Such a holistic picture can offer valuable insights into public perceptions, social trends, and the perceived effectiveness of inequality mitigation efforts. It can inform decision-making processes, guide empirical research efforts in the field, and contribute to a more informed debate about inequality.

## 5.2 Related Work

**Discourse on social media.** We focus on related work that analyzes discourse in social media, and more particularly on Twitter. First, we present work that is the closest to ours,

<footnote n="1">¹https://www.w3id.org/okg/obio-ontology/
²https://github.com/muhai-project/okg_media_discourse
³https://doi.org/10.5281/zenodo.10034210
⁴https://api.druid.datalegend.net/datasets/lisestork/OKG/services/OKG/sparql
⁵https://triply.cc
⁶https://iisg.amsterdam</footnote>
ie., that uses knowledge graphs as intermediate representations. Second, we present work that uses NLP techniques to get insights from social media data, not necessarily in graph format.

The work that is the most similar to ours is TweetsKB [14] and TweetsCOV19 [13], that is a subset of TweetsKB containing Covid-related tweets. They created a RDF(S) model for describing metadata and annotation information for a collection of tweets, and they presented useful use cases such as entity-centric data exploration. The authors of [10] used TweetsKB as a starting point to analyze public attitudes towards controversies, specifically on the topic of migration, and propose the following components in their pipeline: topic modeling, sentiment analysis, hate/speech detection and entity linking to Wikidata. We extend these models with further fine-grained linguistic information retrieved from tweet texts. More specifically, we add information on semantic roles of entities, as well as sentence grammar.

More recently, a lot of attention has been given to analysing social narratives about Covid19 on Twitter. [8] released a multilingual dataset containing tweets about the coronavirus, and [37] monitored the mood of India during the Covid pandemic starting from tweets. Lastly, [2] analyzed trending hashtags on Twitter with a specific use-case on Covid-19, and mapped the output to knowledge bases like Framester [16].

Tweet2story [6] is an NLP pipeline to automatically extract narratives from tweets in the form of simple graphs. Their pipeline includes: actor entity extraction, time entity extraction, event entity extraction, link extraction and semantic role extraction. Our work is complementary, providing a complete ontology for the knowledge graph output. The authors of [22] used the Dutch vaccination debate on Twitter to identify online communities, narratives and interactions. [26] combines an NLP pipeline with network analysis to extract conflicting narrative mechanics from Twitter data.

**Knowledge Graph from text.** We first present ontologies that were used to model textual data, and more specifically focused on social media data. We then present existing resources.

The NLP Interchange Format (NIF) ontology [18] was designed to integrate text into knowledge graphs, whereas the NLP Annotation Format (NAF) ontology [15] additionally focused on linking linguistic annotations. The OntoLex ontology [24] models lexical data in the semantic web. To represent social media data in the form of a knowledge graph, TweetsKB [14] reused existing ontologies such as the Semantically Interlinked Online Community (SIOC) [5]. The Influence Tracker ontology [27] integrates tweet data with quality metrics about Twitter users. Framester [16] is a frame-based ontological resource that bridges major linguistic resources such as FrameNet and PropBank. In our model and graph, we reuse and extend the existing TweetsKB model, and integrate it with (i) text analysis using NIF, (ii) new metrics that are not in the Influence Tracker ontology, and (iii) Framester PropBank rolesets.

In terms of existing resources, [28] proposes an NLP pipeline to build an event-centric knowledge graph from news data, using frame semantics. Throughout the BioSampo project, researchers extracted knowledge graphs from plenary debates, to analyze parlementary language and culture [31]. TakeFive [1] transforms texts into a frame-oriented knowledge graph, and FRED [17] also parses natural text into linked data.

## 5.3 Knowledge Graph Construction

Here, we describe the OBIO ontology (Section 5.3.1), and the construction and validation of the OKG in terms of the ontology (Section 5.3.2).
### 5.3.1 The Ontology

The goal of the ontology we build is the inclusion of fine-grained semantics, as a semantic layer on top of the tweet texts. We show five example tweets, use these to describe our ontological requirements which then inform the ontology creation process.

#### Motivating Examples.

We present 5 sentences from tweets in Table 5.1 to show the type of analyses we aim to do with our ontology. Relevant content for the understanding of these sentences can be divided in three categories: (1) **tweet metadata**, (2) **meaning** and (3) **grammar**.

<table><caption>Table 5.1: Sentences from tweets on inequality.</caption><thead><tr><td>Id</td><td>Sentence Content</td></tr></thead><tbody><tr><td>1</td><td>“We see the end of the <strong>trauma</strong> created by our brutal system of race and <strong>gender inequality</strong>”</td></tr><tr><td>2</td><td>“These <strong>unregulated systems</strong> could <u>cause</u> <strong>discrimination</strong> on a massive scale says <strong>Buolamwini</strong>”</td></tr><tr><td>3</td><td>“How’re we expected to <u>behave rationally</u> in the face of <strong>brutality, inequality, and racism</strong> when they get “scared” of a black person reaching for their wallet at a traffic stop.”</td></tr><tr><td>4</td><td>“Nothing wrong with <u>wanting to end inequality</u>.”</td></tr><tr><td>5</td><td>“I’m glad you think we should <u>judge</u> people on <strong>merit</strong>.”</td></tr></tbody></table>

The **tweet metadata** category would include information such as the user who posted the content, information on the user, the date of the tweet, etc. It would also include standards metrics such as the number of likes.

The **meaning** category would typically include entities from sentences, such as *inequality* or *Buolamwini*. In our approach, we aim to go beyond the mere enumeration of entities in a tweet, and to add another semantic layer on top, specifically extracted rolesets: verbs and their arguments. PropBank [20] details provide such fine-grained analyses. PropBank, short for the Proposition Bank, is a linguistic resource that associates verbs with their arguments, also called semantic roles. An instantiated ensemble with a verb and its filled arguments is called a roleset. For example: sentence 2 from Table 5.1, the verb “cause” triggers a roleset, that has “unregulated systems” as subject and “discrimination” as object. Likewise in sentence 5, “think” triggers a roleset that has “we should judge people on merit” as object.

Lastly, the **grammar** category represents the grammatical structure of the sentences, more specifically the dependency relationships. Other than facilitating the entity extraction process, the analysis can provide interesting insights like the nesting of PropBank rolesets, as is the case in sentence 4, which triggered two rolesets: “wanting” and “end”.

#### Ontological Requirements (ORs).

We derive three high-level ontological requirements from the examples above, directly linked to the three types of analyses that our ontology should enable.

**OR1:** Model the metadata of the social media ecosystem.
1. Distinguish between a regular tweet, a repost and a reply.

2. Each tweet should have a date.

3. Metrics: sentiment, polarity, subjectivity, number of repost, number of likes.

4. User attributes: account verified, number of followers, number of accounts followed, location.

**OR2:** Model the meaning of the tweets' content.

1. Link tweets to extracted entities, and link them to external ontologies.

2. Link tweets to PropBank extracted rolesets: triggering verb and its arguments.

**OR3:** Model the grammar of the tweets' content.

1. Tweet content should be chunked down by sentences and tokens.

2. Dependency relations should be added between tokens.

3. Include: lemma, part-of-speech tag, token index.

## Ontology creation

Following best practices in ontology development [11], we aim to re-use existing models and extend them with classes and object properties. Our core starting point is TweetKB [14], that uses the SIOC [5] ontology. We create new classes, data and object properties if they do not already exist. Apart from TweetsKB, we mainly integrate two other ontologies: NIF [18] for integrating text with KGs and Framester [16] for PropBank-related information. The prefix for the OBIO is `obio`[^7].

**Ontology Presentation**. We show the ontology in Figures 5.1 and 5.2. Figure 5.2 mainly covers OR2-2, while Figure 5.1 covers the rest. The prefixes are given in the Figures. We describe below how we encode the ontological requirements:

**OR1-1:** We introduce `obio:RePost` and `obio:Reply` as sub-classes of `sioc:Post` for reposts and replies. A repost, or a retweet in the Twitter context, is to share another user's tweets to your followers, while a reply is a direct response to another tweet.

**OR1-2:** We re-use TweetsKB for this part, with `dc:created`.

**OR1-3:** We introduce the `obio:postmetrics` data property for metrics related to tweets. We define sub-properties of this property for the different metrics that are listed: `obio:sentiment_label`, `obio:polarity_score`, `obio:subjectivity_score`, `obio:nb_repost`, and `obio:nb_like`.

**OR1-4:** We introduce various data properties to describe the following attributes for a user: `obio:is_verify`, `obio:status`, `obio:location`, `obio:follower`, and `obio:following`.

**OR2-1:** We re-use TweetsKB with `schema:mentions`.

**OR2-2:** We integrate text data with Propbank annotations and Framester. We attempt to stay as close as possible to the original Propbank annotations in Framester, with minor modifications[^8]. For one annotation of a Propbank roleset, we re-use the `wsj:CorpusEntry`.

<footnote>[^7]: Short for https://www.w3id.org/okg/obio-ontology/</footnote>

<footnote>[^8]: See http://etna.istc.cnr.it/framesterpage/wsj/wsjpropnetannotations/CE_64700</footnote>
class. The main difference is that instead of using blank nodes to describe mapped roles, we use classes from the NIF ontology, such as `nif:Word` and `nif:Phrase`. Lastly, similarly to the existing Datatype Property `wsj:onLemma`<fnref n="9" />, we create an Object Property `obio:onToken` that goes from a `wsj:CorpusEntry` to a `nif:String`. To enforce a better integration between OR2-1 and OR2-2, we add additional `nif:superString` links, as is shown in Figure 5.2.

**OR3-1:** We re-use the classes `nif:Word` and `nif:Sentence` from the NIF ontology.

**OR3-2:** We add `obio:dependency_relation` as a sub-property of `nif:inter` to describe the dependency relations between two words in a sentence. Each dependency relation is then added as a sub-property of `obio:dependency_relation`.

**OR3-3:** We mostly re-use content from the NIF ontology, and add the data property `obio:hasTokenIndex` to link a token to its index in the original tweet.

<figure><img src="image_24.png" /><figcaption>Figure 5.1: The Observatory Integrated Ontology.</figcaption></figure>

Together with the code that we submit with this paper, we add a more detailed documentation and visualization of our ontology.

## 5.3.2 Ontology Population and Validation

### Tweet, metadata and grammar extraction (OR1 & OR3).

The acquisition of social media data about inequality from the Twitter platform was done through the “academictwitteR” R library [2], using the Full-Archive API V2¹⁰ with the following query parameters: “*(inequality OR inequalities) lang:en*”. We downloaded the data before the changes of policies¹¹ in June 2023. We retrieved tweets and retweets published from the end

<footnote n="9">⁹See http://etna.istc.cnr.it/framesterpage/wsj/onLemma
¹⁰https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet
¹¹https://developer.twitter.com/en/developer-terms/policy#4-d</footnote>
<figure><img src="image_25.png" /><figcaption>Figure 5.2: Framester integration.</figcaption></figure>

(30th) of May 2020 to the beginning (1st) of May 2023. In this paper, we use a sample published from May 30th to August 27th, 2020. To be compliant with the Twitter policies, we remove user metadata and the texts of tweets and tweet sentences. We also replace user IDs with skolem IRIs through skolemization<fnref n="12" />.

For the grammar of the tweet content, we used the output of spaCy¹³, that covers all requirements for OR3. We used the `en_core_web_sm` model¹⁴. For the metadata of the tweets, all requirements of OR1 except OR1-3 were provided by the Twitter API: type of tweet (OR1-1), date (OR1-2) and user attributes (OR1-4). For the sentiment extraction (OR1-3), we use the RoBERTa-base model trained on around 124M tweets from January 2018 to December 2021 directly from Hugging Face¹⁵, which achieved 69.1% accuracy for sentiment analysis. For the polarity and subjectivity metrics (OR1-3), we use TextBlob¹⁶.

## Meaning extraction (OR2).

**Entities from text (OR2-1).** We use the same methodology as [26]. We first extract two types of named entities from the output of spaCy: named entities and noun phrases. A named entity refers to a proper name, whereas a noun phrase is a grammatical construction that includes a noun and its modifiers. These entity mentions are then consolidated into `obio:Entity`, and the mapping to DBpedia is done through DBpedia Spotlight [17].

**PropBank Rolesets (OR2-2).** To link the tweets to the PropBank rolesets (OR2-2), we use an extended version of the PropBank grammar developed by [4], that uses computational construction grammar to extract semantic frames from text corpora. Such a grammar uses as

<footnote n="12">¹²https://www.w3.org/TR/rdf11-concepts/#section-skolemization
¹³https://spacy.io
¹⁴Details on accuracy performance can be found at https://spacy.io/models/en
¹⁵https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest
¹⁶https://textblob.readthedocs.io/en/dev/</footnote>
a basis constructions, that are structured meaning-form pairs. The grammar output the verbs and their semantic roles. We then link the extracted frames to Framerster [16]. As an example, the frame protest.01 from [4] corresponds to the Framerster entity pbdata:protest.01. For each frame, we create a wsj:CorpusEntry as illustrated in Figure 5.2.

## Validation and statistics

**KG statistics.** The KG we present in this paper contains 9,243,293 triples and 1,084,882 unique entities. Out of the 10,613 obio:Entity entities, 3,592 (33.8%) had a mapping to DBpedia. The average node indegree and outdegree are 8.4 and 8.5 respectively. The minimum, mean and maximum number of entities per tweets are 1, 2.3 and 11 respectively. 2,398 distinct frames were extracted across all tweets. The top 10 extracted frames were do.02 (8,029), pandemic.01 (2,067), need.01 (1,675), work.01 (1,674), see.01 (1,583), do.01 (1,268), solve.01 (1,242), address.02 (1,212), say.01 (1,179), and fight.01 (1174).

We describe the distribution of class types in Table 5.2. The most prevalent classes in OKG come from the NIF ontology, which is expected since each tweet was chunked down into sentences and tokens. OKG introduces 136,391 new corpus entries for PropBank rolesets. There are 62,015 original posts, 34,551 reposts and 4,837 replies for a total number of 42,108 different users.

The KG contains 22,852 tweets with a negative label, 19,070 with a neutral one and 3,927 with a positive one. This represents a ratio of around 6 between the negatively labeled tweets and the positively labeled ones. Furthermore, the average numbers of reposts are 7,015, 827 and 2,769 for the negatively, neutral and positively labeled tweets respectively, which represent a ratio of around 3 between the positive and negative ones. Likewise, the average numbers of like are 2.8, 1.8 and 2.1 respectively, with a ratio of 1.3. We observe the negatively labeled tweets tend to be more numerous, to get more repost and more likes than the other tweets.

<table><caption>Table 5.2: KG Class Distributions.</caption><thead><tr><td>Class</td><td>Number</td></tr></thead><tbody><tr><td>nif: String</td><td>787,187</td></tr><tr><td>nif: Word</td><td>573,377</td></tr><tr><td>wsj: MappedRole</td><td>277,502</td></tr><tr><td>wsj: CorpusEntry</td><td>136,391</td></tr><tr><td>nif: Phrase</td><td>107,756</td></tr><tr><td>nif: Sentence</td><td>106,054</td></tr><tr><td>sioc: Concept</td><td>104,123</td></tr><tr><td>sioc: Post</td><td>62,015</td></tr><tr><td>sioc: Forum</td><td>45,472</td></tr><tr><td>sioc: Container</td><td>45,472</td></tr><tr><td>sioc: User</td><td>42,108</td></tr><tr><td>foaf: OnlineAccount</td><td>42,108</td></tr><tr><td>obio: RePost</td><td>34,551</td></tr><tr><td>obio: Entity</td><td>10,613</td></tr><tr><td>obio: Reply</td><td>4,837</td></tr></tbody></table>
<table><caption>Table 5.3: Top 10 entities in tweets, grouped per label. Freq. corresponds to the frequency of occurrence, and Perc. refers to the percentage of occurrence compared to the total number of tweets.</caption><thead><tr><th colspan="3">Negative</th><th colspan="3">Neutral</th><th colspan="3">Positive</th></tr><tr><th>Entity</th><th>Freq.</th><th>Perc.</th><th>Entity</th><th>Freq.</th><th>Perc.</th><th>Entity</th><th>Freq.</th><th>Perc.</th></tr></thead><tbody><tr><td>Economic Inequality</td><td>609</td><td>2.7</td><td>Economic Inequality</td><td>231</td><td>1.2</td><td>Racism</td><td>75</td><td>1.9</td></tr><tr><td>Racism</td><td>458</td><td>2.0</td><td>Racism</td><td>193</td><td>1.0</td><td>Economic Inequality</td><td>51</td><td>1.3</td></tr><tr><td>Poverty</td><td>234</td><td>1.0</td><td>Poverty</td><td>76</td><td>0.4</td><td>Poverty</td><td>25</td><td>0.6</td></tr><tr><td>America</td><td>117</td><td>0.5</td><td>Pandemic</td><td>63</td><td>0.3</td><td>Gender Inequality</td><td>21</td><td>0.5</td></tr><tr><td>Institutional Racism</td><td>113</td><td>0.5</td><td>Scientific American Mind</td><td>52</td><td>0.3</td><td>Black Lives Matter</td><td>20</td><td>0.5</td></tr><tr><td>Pandemic</td><td>111</td><td>0.5</td><td>Gender Inequality</td><td>51</td><td>0.3</td><td>Institutional Racism</td><td>19</td><td>0.5</td></tr><tr><td>Donald Trump</td><td>99</td><td>0.4</td><td>Severe Acute Respiratory Syndrome Coronavirus 2</td><td>49</td><td>0.3</td><td>Coronavirus Disease</td><td>19</td><td>0.5</td></tr><tr><td>Gender Inequality</td><td>96</td><td>0.4</td><td>Black Lives Matter</td><td>49</td><td>0.3</td><td>Podcast</td><td>14</td><td>0.4</td></tr><tr><td>Capitalism</td><td>94</td><td>0.4</td><td>Institutional Racism</td><td>48</td><td>0.3</td><td>United Kingdom</td><td>13</td><td>0.3</td></tr><tr><td>Social Inequality</td><td>88</td><td>0.4</td><td>United Kingdom</td><td>48</td><td>0.3</td><td>Web Conferencing</td><td>13</td><td>0.3</td></tr></tbody></table>

**Data validation.** The resulting graph was validated against a set of data quality criteria [7], specifically *accuracy* and *consistency* given that data was integrated via automated scripts. To check the graph we used SPARQL queries and SHACL shapes. For *accuracy*, we checked the syntactic validity of all literals using a set of SHACL shapes, and a SPARQL ASK query was created to check whether matches to DBpedia were valid URIs, whether all words were indeed included in their superstrings (`nif:superString`), and whether instances of `pbschema:ARG1` and `pbschema:ARG2` were valid Framester IRIs. For *consistency*, we checked for schema correctness using a set of SHACL shapes, e.g., every tweet has exactly one creator. Errors found through SPARQL and SHACL validation were corrected to improve the quality of the graph. The SPARQL queries and SHACL shapes can be found in the Github repository<fnref n="17" />

## 5.4 Use Cases

In this section, we present five use cases that reflect examples of questions researchers and policymakers can answer with the OKG. We first present relevant questions that can be answered using sentiment analysis and named entity recognition, similarly to other Twitter resources, e.g., TweetsKB [14]. Second, we present questions solely facilitated by the OKG, through the integration of tweet metadata with finer-grained semantic layers such as PropBank rolesets and dependency relationships. For each use case, we describe their relevance and show the SPARQL queries used to to each use case. For readability we omit prefixes, which can be found in Figures 5.1 and 5.2.

### 5.4.1 Top Entities Grouped per Tweet Sentiment

The following SPARQL query lists the frequency of sentiment labels per entity mention:

```sql
SELECT ?entity (COUNT(?t) as ?nb_t) WHERE {
    ?t schema:mentions ?entityMention ;
    obio:sentiment_label ?label .
    ?entity nif:anchorOf ?entityMention .
}
GROUP BY ?entity ?label
ORDER BY DESC(?nb_t)
```

<caption>Listing 5.1: Top entities per tweet label.</caption>

<footnote n="17">¹⁷https://github.com/muhai-project/okg_media_discourse</footnote>
<table><caption>Table 5.4: Top 5 pair of entities co-occurring in tweets, grouped per sentiment label. Freq. corresponds to the frequency of occurrence.</caption><thead><tr><th colspan="3">Negative</th><th colspan="3">Neutral</th><th colspan="3">Positive</th></tr><tr><th>Entity 1</th><th>Entity 2</th><th>Freq.</th><th>Entity 1</th><th>Entity 2</th><th>Freq.</th><th>Entity 1</th><th>Entity 2</th><th>Freq.</th></tr></thead><tbody><tr><td>Economic Inequality</td><td>Racism</td><td>42</td><td>Scientific American Mind</td><td>Terms</td><td>45</td><td>Coronavirus Disease</td><td>Economist</td><td>11</td></tr><tr><td>Economic Inequality</td><td>Poverty</td><td>27</td><td>Scientific American Mind</td><td>Happiness</td><td>45</td><td>Institutional Racism</td><td>Sociology</td><td>5</td></tr><tr><td>Capitalism</td><td>Economic Inequality</td><td>23</td><td>Scientific American Mind</td><td>Health</td><td>45</td><td>Hurricane Floyd</td><td>Minnesota</td><td>4</td></tr><tr><td>Poverty</td><td>Racism</td><td>21</td><td>Health</td><td>Terms</td><td>45</td><td>130 Crore Indians</td><td>British Association For Immediate Care</td><td>4</td></tr><tr><td>Economic Inequality</td><td>Institutional Racism</td><td>20</td><td>Happiness</td><td>Terms</td><td>45</td><td>British Association For Immediate Care</td><td>Modi Govt</td><td>4</td></tr></tbody></table>

Table 5.3 shows the top 10 entities mentioned in tweets per label. Some entities are mentioned across all sentiment labels, such as “Economic Inequality”, “Racism”, or “Poverty”. Other entities from the top 10 only appear in one type of tweets, like “Donald Trump” or “Capitalism” for the negatively labeled tweets, or “Coronavirus Disease” for the positive ones.

## 5.4.2 Entities co-occurrence grouped per tweet sentiment

The following SPARQL query lists co-occurrence of entity mentions, grouped per sentiment label:

```sql
SELECT ?sent_label ?ent_1 ?url_1 ?ent_2 ?url_2
  (COUNT(?t) as ?nb_t) WHERE {
    ?t schema:mentions ?ent_m_1, ?ent_m_2 ;
    obio:sentiment_label ?sent_label .
    ?ent_1 nif:anchorOf ?ent_m_1 .
    ?ent_2 nif:anchorOf ?ent_m_2 .
    OPTIONAL (?ent_1 nee:hasMatchedURL ?url_1 .)
    OPTIONAL (?ent_2 nee:hasMatchedURL ?url_2 .)
    FILTER (STR(?ent_1) < STR(?ent_2))
  }
GROUP BY ?sent_label ?ent_1 ?url_1 ?ent_2 ?url_2
ORDER BY DESC(?nb_t)
```

<caption>Listing 5.2: Pairs of entities co-occurring.</caption>

Table 5.4 shows the top 5 pairs of entities that co-occur in tweets, grouped per sentiment label. Among the negatively labeled tweets, “Economic Inequality” appears in nearly all the pairs, related to other abstract concepts such as “Racism” and “Poverty”. In neutral tweets, entities that appear the most are “Scientific American Mind”, “Happiness” and “Health”. In positive tweets, the set of entities is more diverse, with no clear repetitions.

As presented in Section ??, our ontology was extended from TweetsKB [14]. OKG can consequently be used for similar use cases than the ones that make use of both metadata information and entities extracted from tweets. For the next uses, we rather focus on the analysis that are enabled by the integration of the PropBank rolesets and the dependency relationships.

## 5.4.3 PropBank Rolesets per tweet sentiment

The following SPARQL query lists rolesets found in tweets, grouped per sentiment label:

```sql
SELECT ?sent_label ?rs_pb
  (COUNT(?rs_inst) as ?nb_rs) WHERE {
    ?rs_inst wsj:onRoleSet ?rs_pb ;
  }
```
```
obio: onToken ?lu_token .
?sent nif: word ?lu_token .
?t nif: sentence ?sent ;
obio: sentiment_label ?sent_label .
}
GROUP BY ?sent_label ?rs_pb
ORDER BY DESC(?nb_rs)
```

<caption>Listing 5.3: Number of PropBank rolesets per sentiment label.</caption>

Table 5.5 shows the top 10 frames that appear in tweets, grouped per sentiment label. There are differences across the sentiment labels. Whereas the frames for the negative labels relate more to (needed) actions or observations, such as `solve.01` or `need.01`, the frames for both the neutral and positive labels seem more motivational, such as `support.01`, `fight.01` or `thank.01`. Some frames most frequently appear regardless of the sentiment label, such as `do.02` and `work.01`.

<table><caption>Table 5.5: Top 10 PropBank rolesets from tweets, grouped per sentiment label. Freq.:frequency.</caption><thead><tr><th colspan="2">Negative</th><th colspan="2">Neutral</th><th colspan="2">Positive</th></tr><tr><th>Roleset</th><th>Freq.</th><th>Roleset</th><th>Freq.</th><th>Roleset</th><th>Freq.</th></tr></thead><tbody><tr><td>do.02</td><td>7369</td><td>do.02</td><td>1290</td><td>do.02</td><td>582</td></tr><tr><td>pandemic.01</td><td>1690</td><td>need.01</td><td>1002</td><td>work.01</td><td>361</td></tr><tr><td>see.01</td><td>1256</td><td>address.02</td><td>809</td><td>thank.01</td><td>280</td></tr><tr><td>do.01</td><td>1230</td><td>work.01</td><td>777</td><td>fight.01</td><td>278</td></tr><tr><td>solve.01</td><td>1120</td><td>fight.01</td><td>746</td><td>attack.01</td><td>277</td></tr><tr><td>need.01</td><td>1052</td><td>support.01</td><td>699</td><td>admire.01</td><td>276</td></tr><tr><td>work.01</td><td>961</td><td>use.01</td><td>605</td><td>see.01</td><td>257</td></tr><tr><td>expect.01</td><td>933</td><td>change.01</td><td>573</td><td>support.01</td><td>222</td></tr><tr><td>cut.02</td><td>932</td><td>pandemic.01</td><td>570</td><td>love.01</td><td>200</td></tr><tr><td>say.01</td><td>913</td><td>do.01</td><td>569</td><td>help.01</td><td>189</td></tr></tbody></table>

### 5.4.4 Semantic Roles linked to Entities

The following SPARQL query lists rolesets linked to entity mentions:

```
SELECT ?rs_pb ?ent ?pbarg
(COUNT(?rs_inst) as ?nb_rs) WHERE {
VALUES ?link_ss {nif:word nif:subString}
?rs_inst wsj: onRoleSet ?rs_pb;
obio: onToken ?lu_token ;
wsj: withmappedrole ?role_string .
?sent ?link_ss ?role_string .
?t nif: sentence ?sent .
?role_string wsj: withpbarg ?pbarg .
```
```sql
?t schema:mentions ?ent_m .
?role_string nif:superString ?ent_m .
?ent nif:anchorOf ?ent_m .
}
GROUP BY ?rs_pb ?ent ?pbarg
ORDER BY DESC(?nb_rs)
```

<caption>Listing 5.4: Linking semantic roles to entities.</caption>

The output of the query is shown in Table 5.6 which shows the semantic roles that contain the most entities. For instance, “Global Warming” is associated to the roleset `change.01` 112 times in the dataset, with argument `ARG1`. In PropBank, the argument role `ARG1` often represents the “Theme” or “Patient” of a predicate, whereas `ARG0` typically represents the “Agent” or “Experiencer”. Most of the entities are strongly related to issues having to do with inequality: various domains of inequality, such as housing or economic inequality, global warming and racial segregation. Lastly, the number of occurrences is not that high compared to the size of OKG, and in particular the size of the corpus entries and the entities. One explanation is that the integration of the entities and the semantic roles can be further refined, as there are cases where entities are substrings of arguments that are not yet linked<fnref n="18" />

<table><caption>Table 5.6: Top 10 entities included in PropBank rolesets. Freq.: frequency, Neg./Neut./Pos.: negative/neutral/positive.</caption><thead><tr><th>Roleset</th><th>Entity</th><th>Pbarg</th><th>Freq.</th><th>Neg.</th><th>Neut.</th><th>Pos.</th></tr></thead><tbody><tr><td>change.01</td><td>Global Warming</td><td>ARG1</td><td>112</td><td>64</td><td>38</td><td>10</td></tr><tr><td>act.02</td><td>United States Congress</td><td>ARG0</td><td>38</td><td>38</td><td>0</td><td>0</td></tr><tr><td>right.05</td><td>Human Rights</td><td>ARG1</td><td>32</td><td>20</td><td>8</td><td>4</td></tr><tr><td>understand.01</td><td>I Understand 1941 Song</td><td>ARG0</td><td>24</td><td>16</td><td>6</td><td>2</td></tr><tr><td>end.01</td><td>Ibm</td><td>ARG0</td><td>22</td><td>8</td><td>14</td><td>0</td></tr><tr><td>grow.01</td><td>Economic Inequality</td><td>ARG1</td><td>20</td><td>18</td><td>2</td><td>0</td></tr><tr><td>segregate.01</td><td>Racial Segregation</td><td>ARG3</td><td>18</td><td>10</td><td>8</td><td>0</td></tr><tr><td>house.01</td><td>Housing Inequality</td><td>ARG1</td><td>18</td><td>8</td><td>6</td><td>4</td></tr><tr><td>warm.01</td><td>Global Warming</td><td>ARG1</td><td>16</td><td>10</td><td>6</td><td>0</td></tr><tr><td>work.01</td><td>All Facial Recognition Work</td><td>ARG1</td><td>16</td><td>6</td><td>10</td><td>0</td></tr></tbody></table>

### 5.4.5 Relationships between rolesets

The following SPARQL query lists pairs of rolesets and their dependency relations:

```sql
SELECT ?rs_pb_1 ?rs_pb_2 ?depProp
(COUNT(?sent) as ?nb_s) WHERE {
?rs_inst_1 wsj:onRoleSet ?rs_pb_1 ;
  obio:onToken ?lu_token_1 .
?rs_inst_2 wsj:onRoleSet ?rs_pb_2 ;
  obio:onToken ?lu_token_2 .
?sent nif:word ?lu_token_1, ?lu_token_2 .
?depProp rdfs:subPropertyOf obio:dependency_relation .
?lu_token_1 ?depProp ?lu_token_2 .
FILTER (!CONTAINS(str(?depProp), "dependency_relation"))
```

<footnote n="18">¹⁸We plan to release a larger scale dataset later, which will include more rolesets.</footnote>
<table><caption>Listing 5.5: Dependency relationships between rolesets.</caption><tr><td>FILTER (!CONTAINS(str(?depProp), "aux"))</td></tr><tr><td>GROUP BY ?rs_pb_1 ?rs_pb_2 ?depProp</td></tr><tr><td>ORDER BY DESC(?nb_s)</td></tr></table>

Table 5.7 shows the frames that were appearing the most in tweets with a direct dependency relationship, with the number of occurrences and their distribution across the sentiment labels. Unlike Table 5.6, where most of the content came from the negative tweets, Table 5.7 shows that some rolesets are specifically associated with positive or neutral tweets, despite their lower number. This is the case for **study offers**, **get** and **provide**, and **offering suggestions**.

In this section, we provided examples of use cases enabled by OKG. Use case 5.4.1 and 5.4.2 first provided examples of use cases analyzing entities, co-occurrences of entities and sentiment analysis, outlining which actors/objects appear the most in tweets, and are more likely to play an important roles in the tweets' narratives.

<table><caption>Table 5.7: Most frequent relationships between rolesets.</caption><thead><tr><th>Roleset 1</th><th>Roleset 2</th><th>Dependency</th><th>Freq.</th><th>Neg.</th><th>Neut.</th><th>Pos.</th></tr></thead><tbody><tr><td>right.05</td><td>human.02</td><td>amod</td><td>62</td><td>40</td><td>16</td><td>6</td></tr><tr><td>see.01</td><td>build.01</td><td>ccomp</td><td>44</td><td>32</td><td>12</td><td>0</td></tr><tr><td>take.01</td><td>act.02</td><td>dobj</td><td>44</td><td>22</td><td>12</td><td>10</td></tr><tr><td>cut.02</td><td>educate.01</td><td>compound</td><td>40</td><td>40</td><td>0</td><td>0</td></tr><tr><td>offer.01</td><td>study.01</td><td>nsubj</td><td>26</td><td>0</td><td>4</td><td>22</td></tr><tr><td>get.01</td><td>provide.01</td><td>conj</td><td>26</td><td>0</td><td>4</td><td>22</td></tr><tr><td>rise.01</td><td>call.02</td><td>compound</td><td>26</td><td>0</td><td>24</td><td>2</td></tr><tr><td>help.01</td><td>move.01</td><td>ccomp</td><td>26</td><td>0</td><td>4</td><td>22</td></tr><tr><td>offer.01</td><td>suggest.01</td><td>dobj</td><td>26</td><td>0</td><td>4</td><td>22</td></tr><tr><td>address.02</td><td>issue.02</td><td>dobj</td><td>26</td><td>12</td><td>12</td><td>2</td></tr></tbody></table>

We then provided use cases to gain a deeper understanding of opinions about entities using finer-grained semantic layers like PropBank rolesets and dependency relationships. In use case 5.4.3 we analyzed commonly used rolesets in tweets, revealing that neutral and positive tweets often contain motivational verbs. In use case 5.4.4 we explored the entities frequently appearing in PropBank rolesets, uncovering connections to various forms of inequality, such as housing inequality and global warming. In use case 5.4.5 we examined the most frequent dependency relationships between PropBank rolesets, discovering links between specific rolesets in positive tweets, despite their low proportion. These use cases demonstrate how integrating tweets with finer-grained parsing allows us to analyze the roles of specific entities in events. Similar insights can be obtained for real-world use cases such as the understanding of online debates on COVID-19 vaccination [9].
5.5 Conclusion

We first present the *OBservatory Integrated Ontology (OBIO)* that integrates social media
metadata with various types of linguistic knowledge such as entities and PropBank rolesets.
We then populate this ontology with the *Observatory Knowledge Graph (OKG)*, with tweets
extracted on the topic of inequality. We lastly present several use cases that show how adding
finer-grained semantic layer can help improve the overall understanding on social media dis-
course. The paper focuses on the topic of inequality but the method is generic and can be
applied to other topics.

The work we present is based on a small sample of tweets to emphasise the usefulness of semantic layers. We plan to release a larger-scale KG in the future. Moreover, we aim at further curating the output of the natural language processing, as well as to evaluate the utility of the OKG in a real-world use case with expert users, such as social scientists.

Lastly, we plan to integrate better the entities and the PropBank rolesets to extract further
information from the latter. Since we use the NIF ontology, some parts of the OKG remain in
text format, hence we aim to improve the current representations within the KG.
Chapter 5: References

[CB1] Mehwish Alam, Aldo Gangemi, Valentina Presutti, and Diego Reforgiato Recupero. Semantic role labeling for knowledge graph extraction from text. *Progress in Artificial Intelligence*, 10:309-320, 2021.

[CB2] Mehwish Alam, Manuel Kaschura, and Harald Sack. Apollo: Twitter stream analyzer of trending hashtags: A case-study of# covid-19. In *ISWC (Demos/Industry)*, pages 64–69, 2020.

[CB3] Christopher Barrie and Justin Chun-ting Ho. academictwitter: an r package to access the twitter academic research product track v2 api endpoint. *Journal of Open Source Software*, 6(62):3272, 2021.

[CB4] Katrien Beuls, Paul Van Eecke, and Vanja Sophie Cangalovic. A computational con-
struction grammar approach to semantic frame extraction. *Linguistics Vanguard*,
7(1):20180015, 2021.

[CB5] John G Breslin, Stefan Decker, Andreas Harth, and Uldis Bojars. Sioc: an approach to connect web-based communities. *International Journal of Web Based Communities*, 2(2):133–142, 2006.

[CB6] Vasco Campos, Ricardo Campos, Pedro Mota, and Alípio Jorge. Tweet2story: a web app to extract narratives from twitter. In *European Conference on Information Retrieval*, pages 270–275. Springer, 2022.

[CB7] Gustavo Candela, Pilar Escobar, Rafael C Carrasco, and Manuel Marco-Such. Evaluat-
ing the quality of linked open data in digital libraries. *Journal of Information Science*,
48(1):21–43, 2022.

[CB8] Emily Chen, Kristina Lerman, Emilio Ferrara, et al. Tracking social media discourse about the covid-19 pandemic: Development of a public coronavirus twitter data set. *JMIR public health and surveillance*, 6(2):e19273, 2020.

[CB9] Qingqing Chen and Andrew Crooks. Analyzing the vaccination debate in social media data pre-and post-covid-19 pandemic. *International Journal of Applied Earth Observation and Geoinformation*, 110:102783, 2022.

[CB10] Yiyi Chen, Harald Sack, and Mehwish Alam. Analyzing social media for measuring public attitudes toward controversies and their driving factors: a case study of migration. *Social network analysis and mining*, 12(1):135, 2022.

[CB11] G Cota et al. The landscape of ontology reuse approaches. *Appl. Practices Ontol. Des., Extraction, Reason*, 49:21, 2020.

[CB12] Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. Analysis of named entity
recognition and linking for tweets. *Information Processing & Management*, 51(2):32–49, 2015.

[CB13] Dimitar Dimitrov, Erdal Baran, Pavlos Fafalios, Ran Yu, Xiaofei Zhu, Matthäus Zloch, and Stefan Dietze. Tweetscov19-a knowledge base of semantically annotated tweets about the covid-19 pandemic. In *Proceedings of the 29th ACM International Conference on Information & Knowledge Management*, pages 2991–2998, 2020.

[CB14] Pavlos Fafalios, Vasileios Iosifidis, Eirini Ntoutsi, and Stefan Dietze. Tweetskb: A public and large-scale rdf corpus of annotated tweets. In *The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15*, pages 177–190. Springer, 2018.

[CB15] Antske Fokkens, Aitor Soroa, Zuhaitz Beloki, Niels Ockeloen, German Rigau, Willem Robert Van Hage, and Piek Vossen. Naf and gaf: Linking linguistic annotations. In *Proceedings 10th Joint ISO-ACL SIGSEM Workshop on Interoperable Semantic Annotation*, pages 9–16, 2014.

[CB16] Aldo Gangemi, Mehwish Alam, Luigi Asprino, Valentina Presutti, and Diego Reforgiato Recupero. Framester: A wide coverage linguistic linked data hub. In *Knowledge Engineering and Knowledge Management: 20th International Conference, EKAW 2016, Bologna, Italy, November 19-23, 2016, Proceedings 20*, pages 239–254. Springer, 2016.

[CB17] Aldo Gangemi, Valentina Presutti, Diego Reforgiato Recupero, Andrea Giovanni Nuzolese, Francesco Draicchio, and Misael Mongiovì. Semantic web machine reading with fred. *Semantic Web*, 8(6):873–893, 2017.

[CB18] Sebastian Hellmann, Jens Lehmann, Sören Auer, and Martin Brümmer. Integrating nlp using linked data. In *The Semantic Web-ISWC 2013: 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II 12*, pages 98–113. Springer, 2013.

[CB19] Rupinder Paul Khandpur, Taoran Ji, Steve Jan, Gang Wang, Chang-Tien Lu, and Naren Ramakrishnan. Crowdsourcing cybersecurity: Cyber attack detection using social media. In *Proceedings of the 2017 ACM on Conference on Information and Knowledge Management*, pages 1049–1057, 2017.

[CB20] Paul Kingsbury and Martha Palmer. Propbank: the next level of treebank. In *Proceedings of Treebanks and lexical Theories*, volume 3. Citeseer, 2003.

[CB21] Xiaohua Liu, Kuan Li, Ming Zhou, and Zhongyang Xiong. Collective semantic role labeling for tweets with clustering. In *Twenty-Second International Joint Conference on Artificial Intelligence*, 2011.

[CB22] Roel O Lutkenhaus, Jeroen Jansz, and Martine PA Bouman. Mapping the dutch vaccination debate on twitter: Identifying communities, narratives, and interactions. *Vaccine: X*, 1:100019, 2019.

[CB23] Thomas Marcoux and Nitin Agarwal. Narrative trends of covid-19 misinformation. In *Text2Story@ ECIR*, pages 77–80, 2021.

[CB24] John P McCrae, Julia Bosque-Gil, Jorge Gracia, Paul Buitelaar, and Philipp Cimiano. The ontolex-lemon model: development and applications. In *Proceedings of eLex 2017 conference*, pages 19–21, 2017.

[CB25] Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. Dbpedia spot-
light: shedding light on the web of documents. In Proceedings of the 7th international conference on semantic systems, pages 1–8, 2011.

[CB26] Alexander Pak, Patrick Paroubek, et al. Twitter as a corpus for sentiment analysis and opinion mining. In LREc, volume 10, pages 1320–1326, 2010.

[CB27] Gerasimos Razis and Ioannis Anagnostopoulos. Semantifying twitter: The influence tracker ontology. In 2014 9th International Workshop on Semantic and Social Media Adaptation and Personalization, pages 98–103. IEEE, 2014.

[CB28] Marco Rospocher, Marieke van Erp, Piek Vossen, Antske Fokkens, Itziar Aldabe, German Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard. Building event-centric knowledge graphs from news. Journal of Web Semantics, 37:132–151, 2016.

[CB29] Hassan Saif, Yulan He, and Harith Alani. Semantic sentiment analysis of twitter. In The Semantic Web-ISWC 2012: 11th International Semantic Web Conference, Boston, MA, USA, November 11-15, 2012, Proceedings, Part I 11, pages 508–524. Springer, 2012.

[CB30] Carlo Romano Marcello Alessandro Santagiustina and Massimo Warglien. The architecture of partisan debates: The online controversy on the no-deal brexit. PLoS one, 17(6):e0270236, 2022.

[CB31] Laura Sinikallio, Senka Drobac, Minna Tamper, Rafael Leal, Mikko Koho, Jouni Tuominen, Matti La Mela, Eero Hyvönen, et al. Plenary debates of the parliament of finland as linked open data and in parla-clarin markup. In 3rd Conference on Language, Data and Knowledge, LDK 2021. Schloss Dagstuhl-Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing, 2021.

[CB32] Anders Søgaard, Barbara Plank, and Hector Alonso. Using frame semantics for knowledge extraction from twitter. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.

[CB33] Laura Spillner, Carlo R. M. A. Santagiustina, Thomas Mildner, and Robert Porzel. Towards conflictual narrative mechanics. In Proceedings of the IJCAI/ECAI Workshop on Semantic Techniques for Narrative-based Understanding, 2022.

[CB34] Lara Tavoschi, Filippo Quattrone, Eleonora D'Andrea, Pietro Ducange, Marco Vabanesi, Francesco Marcelloni, and Pier Luigi Lopalco. Twitter as a sentinel tool to monitor public opinion on vaccination: an opinion mining analysis from september 2016 to august 2017 in italy. Human vaccines & immunotherapeutics, 16(5):1062–1069, 2020.

[CB35] Ming-Hsiang Tsou, Chin-Te Jung, Chris Allen, Jiue-An Yang, Jean-Mark Gawron, Brian H Spitzberg, and Su Han. Social media analytics and research test-bed (smart dashboard). In Proceedings of the 2015 international conference on social media & society, pages 1–7, 2015.

[CB36] Andranik Tumasjan, Timm Sprenger, Philipp Sandner, and Isabell Welpe. Predicting elections with twitter: What 140 characters reveal about political sentiment. In Proceedings of the international AAAI conference on web and social media, volume 4, pages 178–185, 2010.

[CB37] Akhila Sri Manasa Venigalla, Sridhar Chimalakonda, and Dheeraj Vagavolu. Mood of india during covid-19-an interactive web portal based on emotion analysis of twitter data. In Conference companion publication of the 2020 on computer supported cooperative work and social computing, pages 65–68, 2020.
[CB38] Tom Willaert, Paul Van Eecke, Katrien Beuls, and Luc Steels. Building social media observatories for monitoring online opinion dynamics. *Social Media+ Society*, 6(2):2056305119898778, 2020.

[CB39] Changmeng Zheng, Junhao Feng, Ze Fu, Yi Cai, Qing Li, and Tao Wang. Multimodal relation extraction with efficient graph alignment. In *Proceedings of the 29th ACM International Conference on Multimedia*, pages 5298–5306, 2021.

[CB40] Florian Zollmann, Jeffery Klaehn, Tina Sikka, Kristin Comeforo, Daniel Broudy, Mandy Tröger, Elizabeth Poole, Alison Edgley, and Andrew Mullen. The propaganda model and intersectionality: integrating separate paradigms. *Media Theory*, 2(2):213–239, 2018.
# Chapter 6

# Mapping and Exploring the Dynamics
of Inequality Narratives Through
Social Media

Laura Spillner*, Carlo Santagiustina†‡, Thomas Mildner*, Robert Porzel*

*Digital Media Lab, University of Bremen

† Venice International University, Italy

‡ Ca'Foscari University of Venice, Italy

**Abstract** In this work, we apply a hybrid-AI framework to analyse online discourse about in-equality on Twitter. Our approach integrates knowledge from natural language understanding, knowledge graph-based semantics, and network analysis to identify narratives around inequal-ity and analyse their changes during the COVID pandemic. We present a two-step approach: A bird's-eye perspective on the overall discourse network around inequality maps how the enti-ties, concepts, and events of the narratives are connected by linking these to existing knowledge graphs and can be filtered on tweet metadata such as time. Two connected entities can then be explored in more detail in a fine-grained analysis of how the entities and their relationship are characterised by social media users, highlighting the diffusion of different perspectives on a given topic related to inequality, such as gender, education, and poverty.

**Keywords** social media, social networking sites, network analysis, natural language understanding, knowledge graphs

## 6.1 Introduction

Humans engage in various types of collaborative activities, some intentional – as the construc-tion of the pyramids, some unintentional – as creating traffic jams – and some that are in-between – such as creating a global discourse via social media. In the course of this collab-orative effort, so-called *narratives* emerge, spread, and compete with each other. Narratives play a key role in the conceptualization, identification, and comparison of social groups and in the perception of inequalities. They impact how people perceive situations in a wide range by creating, manipulating or diffusing social myths. They can justify or question socio-economic differences and target power structures and hierarchies. Narratives impact how people perceive their surroundings while influencing social antagonisms among social groups. Consequently, they carry social categories containing the diversity in which stories are being told [28]. Social
media platforms such as Twitter can nowadays be considered a viable proxy for understanding
narratives by citizens, interest group leaders, experts, advocates, and other stakeholders [20].

This work aims to identify and visualize narratives around inequality by integrating natural
language analysis of social media posts with formalized knowledge from open-source knowledge
graphs (KG). We propose that the *fabula*, — the ground truth about relevant events, entities,
and concepts in the narrative — is captured in the KG; and that social media users express from
their personal point of view the narratives they use to understand these (such as the role that
actors play, or causal links between concepts or events). While there have been many works
in recent years that aim to extract narratives from text [11, 14], we, instead, consolidate the
information from a large corpus of short-form texts into one narrative network, thus linking
many different points of view that exist about the same entities.

Our approach builds upon the framework proposed by Spillner et al. to analyse conflict nar-
ratives [26]. To analyse inequalities, the proposed framework consists of two parts. First, a
bird's-eye discourse mapping tool for extracting and mapping entities, including their relations
for a specific targeted issue. Second, a fine-grained analysis method that allows us to categorise
previously identified entity pairs and their relations to the topic of interest. The latter part
includes the different perspectives communicated on social media platforms.

We present an interactive tool that allows users to explore narrative networks both from a
bird's-eye view as well as on a fine-grained level, which extends both steps of the original
framework. Our method consolidates and validates entities identified in the text through named
entity recognition (NER) and linking (NEL). Linking the entities to the DBpedia KG [15]
allows us to enrich the narrative network further and, thereby, facilitate its understanding and
interpretation through general world knowledge. In addition, the information learned from the
natural language texts provides information about how these entities are viewed by people as
part of the common discourse on inequality, which is not captured in the KG.

This paper presents how this method can be applied to discussions about inequality on Twitter
during the COVID pandemic. Online discussions about inequality are extremely varied and
are dependent on the political, economic, social and cultural context. We are interested in how
people understand and talk online about multiform inequalities and intersectionality [43], and
want to understand which are the perceived causes, driving factors and effects of inequalities,
as well as the relations among mentioned entities (people, places, events, organizations, coun-
tries, etc.). The chosen time period and the variety of posts on Twitter allow us to explore
how inequality narratives developed during a number of political and social changes. Moreover,
analysing short and informal social media texts presents a unique challenge in natural language
understanding: One tweet usually only presents a small part of an overarching narrative of its
author. Understanding these texts requires a large amount of general world knowledge, knowl-
edge about current events, and the ability to abstract from the given information to understand
the point of view from which the text was likely written. In connecting the information from
a large corpus of texts with that of existing KGs, we show that our approach can reconstruct
these overarching narratives.

The contribution of this work is two-fold: The approach we implemented integrates knowledge
from natural language understanding with semantic knowledge from KGs, allowing users to
explore inequality narratives on social media on two levels - an overview of the narrative net-
work, as well as a detailed view that enriches individual connections to show different points
of view. The created tool can be used by the community to get a better understanding of the
diverging narrative strands and analyse its components. It further allows anyone to subtract
the diverse viewpoints during an ongoing narrative, which we believe can help news media and
other sources to observe how the perception and impact of inequality changes over time.
## 6.2 Methods and Prior Work

### 6.2.1 Narrative Understanding

While the term *narrative* is used in the literary sciences to mean the recounting of a series of events in a story, in cognitive science, narratives are understood to be a central means of how we make sense of the world [9]. Because of this, there are very different approaches to narratives in computer science [33]. There is a growing body of work that aims to convert natural language texts (e.g. fairy tails, news stories, or social media posts) into a formal representation of events and related actors [4, 38, 22, 1, 6, 18, 42]<fnref n="1" />, or to find connections between the textual narrative and an event KG [16]. Meghini et al. [16] define the event graph as the *fabula*, which is the basis for the (textual) narrative. In these works, the analysis is generally on the level of a single narrative or a single text per narrative. They presuppose that a narrative is always told from a certain perspective depending on the author or storyteller but do not capture this perspective in formal representation.

In robotics, however, a narrative is a formal representation that describes an event or episode from a certain point of view [23, 24], thus linking events (in terms of physical actions) to the concepts they evoke or actors to the role they play. This kind of narrative thus formalizes a level that is “in-between” the fabula and the textual description of it. A similar narrative model has also been proposed by Kroll et al. [13], who differentiate between factual relations in a KG (the fabula) vs hypothetical, narrative relations connecting them. Such formal narratives can be used to generate different natural language descriptions of the same event from different perspectives [24]. Similarly, Mensio et al. [18] proposed a framework for natural language understanding in which news articles are compared based on their narrative framing [8] (e.g. based on word choices), so that similar articles can be linked and compared in terms of their narratives.

In the context of understanding social media discourse, we consider the construction of nar- ratives to be a collaborative process between many people. A single tweet does not in and of itself present a complete and independent narrative but rather represents one puzzle piece that can be used to understand a larger network of interconnected, shared narratives. This aligns with the framework proposed in [26], where a *narrative network* is constructed, connecting all the entities which appear in the fabula across the corpus of texts.

### 6.2.2 Narrative Network Approach

Our analysis² results in two kinds of graphs, which can be visualized and interactively explored to gain insight into the inequality discourse. First, we build a *bird's-eye network* which visualizes the co-occurrences of different entities in the Twitter discourse, showing what role the entities play in the narratives surrounding inequality. Secondly, pairs of these entities that are linked in the bird's-eye network can be analysed in more detail, revealing more of the information learned from natural language analysis in another *fine-grained analysis* graph which “zooms in” on their relationship. Here, we look at how each entity is characterized in the context of the inequality discourse, revealing different points of view that Twitter users have on these entities (descriptors, associated actions, ownership ascriptions, etc.). Additionally, while the bird's-eye graph only shows the relationship between two entities in terms of their co-occurrences in the tweets, the fine-grained graph reveals how the relationship between them is understood (one entity acting on the other, similarities and contrasts, influences, etc).

<footnote n="1">¹See also the Text2Story Workshop [11]

²The code can be accessed under [we will add the link to our repository here; removed for anonymity].</footnote>
## Dataset

We collected a large corpus of social media posts about inequality that were published on Twitter from the 1st of January 2020 to the 1st of October 2022. For this purpose, using the academictwitteR R package [2], we have downloaded from the Twitter API V2 all posts (excluding retweets) in English published in that time whose text explicitly matches at least one of the following query terms: 'inequality' OR 'inequalities' OR 'inequal'. By choosing generic query terms, we voluntarily apply a very neutral and wide range data capturing strategy, which does not focus on a particular type of inequality, because we are interested in understanding how the focus and views of inequalities of different types change over the investigated period.

This study focuses on Twitter because it is a generalist social media news platform [26], it was among the most used media in those years, and it provides an API for downloading the data, which is accessible for academic research. However, the implemented approach can be employed with data from other sources and social media platforms. Using Twitter data enforces a common data format and provides access to the metadata provided by the platform, including timestamps and locations, referenced links or media, related posts and reactions (retweets and likes), the popularity of the post, as well as information about its author. Tweets and their authors are already a form of network graph that has previously been analysed through graph analysis [7, 19, 30], and many related studies analyse social media posting activity to understand the dynamics of online arguments and discourse [36, 29, 41].

## Natural Language Processing

First, tweet texts are processed using the spaCy NLP toolkit [10]. For the case study presented in Section 6.3, we processed a random sample of 100.000 tweets from the entire corpus, from which the co-occurrence network is constructed. We employed spaCy for part-of-speech (POS) tagging and dependency parsing together with the DBpedia Spotlight tool [17] to link named entities to DBpedia [15]. Social media-specific tokens such as hashtags are not removed but are instead part of the analysis.

## Named Entity Recognition and Linking

NER and NEL machine learning models can be used to identify mentions of named entities such as people, organizations, and locations and link them to KGs. However, instead of building a co-occurrence graph directly from the named entities recognized by the trained model, we instead use its prediction to identify two kinds of entities that are of interest for our analysis. First, the named entities are cross-references with the noun phrases found in the syntactic analysis of the tweets and all those noun phrases of which an entity is a subset of its tokens are collected. Then, the root token of the noun phrase is identified. Consider the following two noun phrases:

(1) "president Joe Biden": Here, the token span "Joe Biden" is identified by the NER as a named entity and linked to the KB entry concerning the person Joe Biden. The root of the noun phrase is the token "Biden".

(2) "the american president": Here, "american" is recognized as a named entity and is linked to the KB entry of the *United States of America*. The root of the noun phrase, however, is "president".

In Example (1), the entire noun phrase refers to the entity that was recognized, with the surrounding phrase describing it. The same entity could just as well be described as "Joe Biden" or "Presidential Candidate Biden" or "my neighbour Joe". In cases like this, all noun
phrases referring to the same entity are consolidated into one node in the co-occurrence network, which we call a KB entity. The different descriptions characterize these entities. In contrast, in Example (2), the entity identified by the NEL model is not, in fact, the entity that the entire phrase refers to. Depending on the context, it likely refers either to the office of the president of the United States of America or to the person holding the office at a certain point in time (e.g. Joe Biden). In some cases, the entity that the phrase refers to is part of the KG but is not identified correctly through NEL; in other cases, the phrase describes an entity or concept which is not (yet) part of the KG. One example of this is the concept of “vaccine inequality”, which appeared during the COVID pandemic. Thus, noun phrases following Example (2) are collected separately as “noun phrase entities” (NP entity) and are not consolidated.

### 6.2.3 Birds-Eye Network

#### Co-occurence Network

The bird's-eye network is constructed of a co-occurrence network of both the NP entities and the consolidated KG entities. Co-occurrences are counted on the level of tweets. From the sample of 100.000 tweets, we collect 201494 occurrences of 73901 unique entities, which means that on average, each tweet contains about two entities. Thus, the resulting graph contains 73901 nodes connected by 157773 edges, with a mean nodal degree of 4.270 (SD = 43.116).

#### Filtering

Visualization of large and highly-connected graphs is a difficult problem [37], and not all co-occurrences in this network represent meaningful information. Thus, the network has to be filtered to remove less relevant connections. First, spurious connections, that is, edges of weight 1, are removed, splitting the graph into multiple connected components and many unconnected nodes. All but the largest connected component are removed, as these contain parts of the discourse which are only tangentially related to the main inequality narrative.

For large and complex weighted networks, such as the one at hand, it is often difficult to identify the right trade-off between the level of filtration and the amount of relevant information to be preserved in the simplified network representation. In many real-world applications, the probability distribution $P(\omega)$ that any given edge $e_{ij}$ is carrying a weight $w(e_{ij}) = \omega$ is not constant across nodes [31]. When this is the case, a global cutoff based on edge weight does not consider the intrinsic diversity of edges connecting nodes with different structural roles (e.g, hub entities $V_s$ bridging entities $V_s$ peripheral entities). In the co-occurrence network, edge weights are dependent on the topology of an entity's neighbourhood. In particular, when the number and distributional properties of the weights of incident edges are not uniformly or identically distributed across nodes, a change in the focus toward a local filtration method is useful.

For this reason we have developed an ad-hoc filtration strategy that prunes edges based on their relative relevance for their incident nodes, maintaining relations that are important for both entities connected by the edge. Call $E$ the set of edges of the entities network $G$ and $V$ the set of nodes (i.e., entities) of the entities network $G$, the proposed filtration algorithm works as follows:

1. For all edges $e_{ij} \in E$:

(a) If the weight of the edge $w(e_{ij})$ is above the global threshold, add the edge $e_{ij}$ to the set $K$ of edges to be kept;

(b) If not; then for the two incident nodes $i$ and $j$ of $e_{ij}$:
<figure><img src="image_26.png" /><figcaption>Figure 6.1: Example of the dependency structure of a tweet. The text of this tweet was shortened to visualize the example; the full text was: “She is switching cause and effect. Supply chain issues are not causing inflation. Inflation and money printing are and have been disrupting the allocation of means and is therefore causing supply chain issues among other things like extreme income inequality”. The graphic was generated using spaCy’s dependency visualisation tool.</figcaption></figure>

i. compute the percentile of the weight of the selected edge $w(e_{ij})$ with respect to those of all edges incident to that node;

ii. if the percentile is above the local threshold for both incident nodes of the selected edge, add the edge $e_{ij}$ to the set $K$

2. remove all edges $e_{ij}$ s.t. $e_{ij} \in E \land e_{ij} \notin K$.

## Network Clustering

The filtered view of the network contains 683 nodes and 1392 edges (nodal degree: mean = 4.076, SD = 14.349). This subgraph is clustered using the Louvain algorithm, which identifies clusters based on the graph connectivity. The algorithm does not necessarily identify topic clusters, as the clustering is not based on natural language analysis. The filtered subgraph includes 22 connected components, including one large, central component as well as 21 small components of 3-5 nodes. The Louvain algorithm results in 38 clusters: the 21 small connected components and the main component, which is divided into 17 clusters.

### 6.2.4 Fine-grained view

In the second network view, one can “zoom in” on pairs of entities connected in the bird’s-eye graph. This view extends the more general and (intended to be) impartial knowledge included in KGs with the aggregated perspectives from social media posts. Both the semantic information that can be gained from the tweets in the form of semantic embeddings, as well as the syntactic structure of the tweet, is taken into account. Using this information, the subgraph of a given entity pair is extended with additional nodes containing different characterizations of each entity, and the edge between them is “split apart” into a number of paths that show different ways in which people understand the relationship between the two entities.

#### Characterization of entities

The first goal of the fine-grained narrative is to understand how two entities are characterized in the context of inequality and of each other. This includes, for instance, the role that is assigned to them, qualities associated with them, terms used to refer to them, or how they are understood to compare to others or change over time. These characterizations are extracted from the tweet
text based on its dependency structure. Figure 6.1 shows the syntactic structure of an example tweet as a dependency graph. The children of the root of each entity noun phrase and their respective subtrees in the dependency tree are aggregated across all occurrences of this entity. In the example shown in Figure 6.1, the entity “Inflation” has two such characterizing subtrees, “and” and “money printing”. For consolidated KB entities which are referred to in different terms, those distinct descriptions here become unique characterizations. The characterizations collected for each entity are sorted by their number of occurrences, weighted with a score s that is calculated as the ratio of tokens that belong to a more descriptive part of speech, e.g. adjectives, as opposed to those belonging to a less descriptive one, e.g. determiners.:

$$s = \frac{|\{t \in T | \text{pos}(t) \in \{\text{noun}, \text{verb}, \text{adjective}, \text{adverb}\}\}|}{|T|} \qquad (6.1)$$

with T the set of tokens in the phrase and pos(t) the part of speech of a token.

Only characterisations with s greater than the mean are shown in the visualisation. The exception to this rule is if (for comparatively uncommon entities) less than 10 unique characterizations have been found, in which case all of them are shown, or if less than 20 have been found, in which case the top 50% are shown.

Next, we utilize GloVe embeddings [21]<fnref n="3" /> to compare characterizations on a semantic level. We calculate cosine similarity scores between all pairs of characterizations of an entity. For those with similarity ≥ 0.7, an invisible edge is added, which acts in the physics simulation of the graph, pulling the two characterization nodes closer together the more similar they are.

Moreover, for an entity pair (A, B), characterizations of A might occur in tweets with only A, or in tweets where both A and B appear. Thus, each characterization is also scored with the ratio of how often it occurs when A is in the context of B, weighted by how often A itself occurs in the context of B out of all its occurrences. For a given characterization c of entity A, let $c_{AB}$ be the number of times it occurs in tweets with both entities A and B, and $c_A$ the number of times it occurs in tweets with just A. Let $t_A$ be the number of tweets with only A, and $t_{AB}$ the number of tweets with both entities, then we calculate this score $r_c$ as:

$$r_c = \frac{\frac{c_{AB}}{c_A+c_{AB}} * \frac{t_A+t_{AB}}{t_{AB}}}{\frac{c_{AB}}{c_A+c_{AB}} * \frac{t_A+t_{AB}}{t_{AB}} + \frac{c_A}{c_A+c_{AB}} * \frac{t_A+t_{AB}}{t_A}} \qquad (6.2)$$

This score is used to indicate whether a characterization of A is more likely to occur in the context of the second entity B than would be expected based on the relative frequency of the two entities, based on which the characterization nodes are coloured.

## Relationship Pathways through Tweets

We propose that the dependency tree form of a sentence encodes the relationship that the author describes between them in a structured form, which can be visualized in the graph view as a path connecting the two entity nodes through a number of words. The predicate of a sentence is at the root of its dependency tree; predicates of subclauses are children of this root verb. From there, there are three possibilities of how the two entities can appear in one sentence in respect to each other:

<footnote n="3">³These static vector embeddings have been shown to provide a useful representation of word meaning and preserve relations between different words in their relative position in the vector space. In contrast to more recent language models, static embeddings are consistent for unique words independent of the text in question; so that the same term as it appears in different tweets will be represented by the same numerical vector.</footnote>
<figure><img src="image_27.png" /><figcaption>Figure 6.2: A framework of the impact of inequality through common discourse on social media.</figcaption></figure>

* one entity is the subject executing the action, the other the object that is being acted upon

* both of them are objects of the same action

* the two entities belong to different subclauses

The shortest path between the two entities in the dependency tree (converted for this step to an undirected graph) then provides a path connecting the noun phrases of the two entities via that part of the text that is pertinent to their relationship. If one of the entities is the subject of its clause, then by switching the direction of edges between a verb and the noun subject, the path can be followed from the subject via the verb to the object; or in the case of more complex relationships through a relative clause to the entity in that clause. In the example shown in Figure 6.1, the path is very simple: (Inflation) -> (causing) -> (Economic inequality).

From all the tweets that mention both entities, all unique relationship paths are collected, and those seen only once are removed. The remaining paths are consolidated to combine partial paths shared among different relationships: The nodes are numbered from the centre of the path, that is, the node from which all edges are outgoing. Only those edges with two identical incident nodes both in the text and in the index of these nodes in the path are combined into one edge.

### 6.2.5 Filtering on Tweet Meta-Data

The filtered view of the big picture network presents a general overview of the inequality discussion on Twitter during the time from which we collected the corpus (2020-2022), showing the most important and most commonly co-occurring entities. Alternatively, it is possible to filter based on the characteristics of the tweets themselves. Thus, the occurrences of entities can be limited e.g. based on the location, the number of likes or retweets, or the account's verification status. We analysed the changes in the network throughout the given time period by filtering the entities' occurrences based on the time-stamp and constructing subgraphs for individual months. From these subgraphs, spurious connections (edges with weight 1) are removed, but no further filtering is applied. Each subgraph is also clustered using the Louvain Algorithm.

## 6.3 Case Study: Inequality

Studying the public discourse about inequalities is important to understand how these are perceived, described and faced by different groups in different countries or contexts and to map
how these views are redefined across time in relation to events, policies and collective actions that may reshape people's attention and aversion to specific forms of inequality. Analysing the public discourse is also key to understanding advocacy actions and citizens' demand for policies [3] aimed at reducing specific inequalities. For example, during the COVID-19 pandemic attention towards education inequality increased [27]. Similarly, after the murder of George Floyd, there has been a growing demand for public action and policies for reducing those inequalities that are disproportionately affecting ethnic minorities [35]. Besides its theorised, material and social implications, like its effect on economic growth [5], discrimination and related inequalities may indirectly harm also the ones that may derive some material benefits from having a privileged position in an unfair society [40]. Concerns for socio-economic inequality that are expressed through social media often go beyond preferences for wealth (re)distribution and for equal opportunities, extending to people's life and work activities and motivation, self-perceived identity and social role, sense of belonging and willingness to cooperate and reciprocate altruistic behaviour in the socio-economic systems in which they operate. Figure 6.2 summarises the framework we have developed as to how perceptions of inequality and the narratives built upon them are able to shape, through the medium of social media discourse, public policy, and ultimately impact beliefs about inequality as well.

The mere feeling of living in an unfair society can produce a widespread psychological and social malaise and a generalized sense of injustice that may reduce one's willingness to cooperate and trigger the emergence of social tensions, polarisation and partisanship, which can lead to further discrimination, societal conflicts, uprisings and revolts. Therefore, tools that allow researchers to assess almost in real time the attitude of people towards material or immaterial differences between groups are important for both research and policy making purposes.

### 6.3.1 Bird's-eye Network

#### Entities and Co-Occurences

The unfiltered bird's-eye network contains 73901 nodes (unique entities). However, 81.9% of these entities occur only once in the entire dataset. The mean number of occurrences per entity is 2.727 (SD = 46.172). The four most commonly occurring entities are four concepts related to inequality: *economic inequality* (10139 occurrences), *poverty* (3874 occurrences), *racism* (3217 occurrences) and *gender inequality*; which the fifth most common entity being the *pandemic* (1705 occurrences), which is closely followed by *global warming* and *Capitalism*.

Out of all those pairs of entities that co-occur together, most (87.9%) only co-occur once, leaving 9307 entities connected by 19132 edges after the first filtering step. Those pairs of nodes that co-occur most frequently usually include very frequently mentioned entities, with a few exceptions: Among the top nine strongest connections, the ones that include entities not among the most frequent ones listed above are *economic inequality* and of: *Donald Trump*, *Capitalism*, *gilded age* and *inflation*. The next three most frequent co-occurrences include a triple of two terms and a hashtag that do not appear very frequently in general, but that always appear together: *campaign finance*, *gridlock* and #fixcongress.

Figure 6.3 shows the filtered bird's-eye network.

<footnote n="4">⁴We attach interactive HTML files of the bird's-eye network and the fine-grained graphs in the supplementary material. There, smaller nodes can be explored in more detail, and the clusters can be selected individually. After review, these graphs will also be available in our repository.</footnote>
<figure><img src="image_28.png" /><figcaption>Figure 6.3: The filtered bird's-eye narrative network of inequality. Colours indicate clusters. Node size indicates the number of occurrences of the entities on a logarithmic scale. The layout of the graph was calculated using the force-directed layout algorithm.</figcaption></figure>
## Types of Nodes

Through the connection to the existing KGs, the general knowledge available about each entity can be accessed, such as the type of the entities. However, as DBpedia collates information from many sources, there are many entities for which several different type classifications are available (following e.g. OWL, Schema, Wikidata, DBO, etc.), and others for which there is only the top-level OWL class "Thing" (which includes all entities which can be considered "things" or "concepts", e.g. *sexism*, *coronavirus disease*, *stock market*, or *podcast*).

Next to this largest group, there are among those entities which co-occur at least twice: 1910 persons, 1271 organizations, 1362 places, 70 events, 69 languages, 157 mass media channels (television/radio stations and websites), 50 things considered products, and 50 sports teams. The last group is usually a mismatch: mentions of smaller countries are matched to the DBpedia entry of this country's football team. Similarly, there are a number of music albums or other entertainment media - however, in these cases, it is often a phrase that is the title of a piece of media and was incorrectly matched (e.g. "status quo"). We noted that many events were only captured as NP entities and not linked to DBpedia.

## 6.3.2 Analysis by Cluster

The clusters are derived from the structure of the network. They do not show which entities "belong" to the same topic semantically, but rather which entities are most strongly discussed in connection to each other. Therefore, there are often instances, depending on the filtering, where two apparently different topics (e.g. *racism* and *gender inequality* or the *pandemic* and *poverty*) belong to the same cluster, as well as instances where terms referring to the same overall topic (e.g. *pandemic*, *coronavirus disease 2019* and *severe acute respiratory syndrome coronavirus*) are each at the centre of their own cluster.

The bird's-eye network is divided into 17 clusters. The seven largest of these are:

1. 164 (mostly small) nodes surrounding *economic inequality*. The vast majority of the nodes in this cluster are not or only weakly connected with other areas, with only 6 of them having a degree $>= 3$ in the filtered network.

2. 87 nodes surrounding *poverty*. This cluster also contains some of the other largest nodes of the network, which are strongly interconnected and thus not split into separate clusters.

3. 63 nodes surrounding *racism*. The smaller nodes are not all linked to the central concept but rather form smaller cliques inside this cluster.

4. 53 nodes surrounding *gender inequality*.

5. 45 large-ish, sparsely connected nodes which are not so much connected to a central topic as they are descriptive of political concerns in the United States. These include (as the largest node) *social inequality*, *Donald Trump*, *China*, the *Republican Party*, other American politicians, as well as political and social problems which are here discussed specifically in reference to the United States (*police brutality in the U.S.*, *wealth inequality in the U.S.*, and the group *Black Lives Matter*).

6. 37 nodes including *capitalism*, *democracy* and the *planet*.

7. one noticeably thin and sparse cluster of 30 comparatively large nodes, which appear situated in between other very frequent entities. This cluster can be visually divided into two sides, one containing *SARS-CoV-2*, the *digital divide* and *educational inequality* (close to the cluster on poverty and the pandemic), the other containing *inflation*, *distribution*
<figure><img src="image_29.png" /><figcaption>Figure 6.4: Examples of four of the clusters in the bird's-eye narrative network. Pictures have been cropped to show more details. However, the relative sizes of the four pictures are the same.</figcaption></figure>
*of wealth, progressive era, as well as the American president and congress.* The reason why these entities co-occur often enough to make up their own cluster appears to be that they occur more frequently in a common interval of time: around the most recent election in the United States. Thus, in this case, the part of the narrative identified in this cluster provides a perspective of a point in time.

Figure 6.4 shows four of these clusters, three of which focus on a specific reason for or axis of inequality: poverty, racism, and gender inequality. All three of these clusters include several larger nodes or stronger internal connections, as well as smaller surrounding entities.

The cluster on *poverty* (2) also includes the *pandemic* and *global warming*. Many of the smaller nodes in this cluster refer to issues of health in relation to poverty, such as *public health*, *vaccines* and the *Lockdown*; as well as *life expectancy*, *obesity*, *disability* and *mental health*, but also *homelessness* and *gun violence*. Very close to it with many edges connecting them is the *gender inequality* cluster (4). This includes many names of countries and regions, with a focus on the global south. In addition, different forms of sex-based inequality or sex-based violence are linked to other clusters: *domestic violence* is strongly connected to the COVID pandemic, while the topic of *medicine* mediates between this cluster and the one focused on *capitalism*.

Cluster (3), which surrounds the concept of *racism*, has some internal cliques. These show focused discussions surrounding topics such as the history of *slavery in the United States*, or *gentrification* in cities; and recent political events such as the murder of George Floyd and the storming of the U.S. *Capitol*; as well as the relations between different forms of oppression/dis- crimination (*racism, sexism, misogyny, homophobia*). The comparatively sparse cluster around *capitalism* (6) includes mostly nodes also connected to large nodes elsewhere: concepts such as *socialism, neoliberalism, oligarchy*, and even *feudalism*; illustrating the point of view from which capitalism is discussed here.

### 6.3.3 Changing Narratives Over Time

We explored subnetworks for each month and identified periods of interest shown in Figure 6.5.

#### The COVID Pandemic

As Figure 6.5a shows, the biggest topics of concern were economic inequality and poverty, and the pandemic. The latter takes up much of the network so that it is split into distinct clusters (blue, orange, purple, and pink). These entities are discussed in different contexts and thus tend to create a cluster with other entities more commonly mentioned in this context. “The pandemic”, “COVID”, or “coronavirus” appear to be used more heavily in informal discourse by non-experts and when the topic is not medical, while official information channels, medical experts and scientific discussions tend to employ more formal phrases. Clustering the network can thus distinguish between narratives that impact or are shared among different social groups. In the context of the pandemic, the move to remote work also brought the *digital divide* to the forefront, showing how groups are affected differently and how existing inequalities are exacerbated by such crises.

#### Black Lives Matter

In June 2020, protests broke out across the United States under the banner of “Black Lives Matter” after the killing of George Floyd. In Figure 6.5b, the purple cluster is subsumed by the subnetwork on racism, and the COVID pandemic now makes up the orange cluster. The largest increase in the number of related tweets was in June, and by July, the size of the subnetwork
<figure><img src="image_30.png" /><figcaption>Figure 6.5: Examples of four of the clusters in the bird's-eye narrative network. The images have been cropped to improve the readability of the node names (only names of larger and, thus, more frequently occurring entities are shown). The relative sizes of nodes across the images are preserved, meaning that the changes in the size of a particular node represent it appearing more or less frequently in the respective months.</figcaption></figure>
has returned to a level comparable to April. Yet, the discussion surrounding racism has not
disappeared and will persist in the future, showing the lasting effect of this event in highlighting
inequality related to racism.

The 2020 United States Presidential Election

Social media discussion on the U.S. election was widespread in October 2020, and in many cases, issues of inequality were mentioned in this context. This is one case where different topics are contained in one cluster (racism, global warming, the pandemic, etc in the purple cluster). It appears that here, these are more commonly discussed together, and there is more focus on how they impact each other. This is also the only month in which the economic inequality cluster contains more than one large node: *America* and the *U.S. Congress*, and a number of phrases (“gilded age”, “secret ballot”, and “progressive era”). Clearly, the issue of economic inequality was perceived to be an important factor in deciding the outcome of the election, which is also evidenced by the fact that the cluster of poverty (red) is much larger than before.

International Women’s Day

At the end of the two-year period, in March 2022, COVID-related concepts have become tan-
gential to other issues. There is less connection between clusters, with conversations around
gender inequality (green), racism (purple), global warming and health (orange), poverty (red)
and capitalism (pink) all separated. A new cluster includes Ukraine and Russia (light blue),
due to the invasion of Ukraine in February 2022. Otherwise, this month is dominated by the
discussion surrounding gender inequality, as March 8th is International Women's Day.

6.3.4 Fine-Grained Analysis

Figure 6.6 shows four examples of the fine-grained network<fnref n="5" />. We picked one more general example (*global warming in relation to poverty*, two cases related to the COVID pandemic (gender and educational inequality), and one example related to a more recent development (economic inequality in the context of inflation).

Poverty and Global Warming

The *poverty* group is much larger, with many descriptions and characterizations. The group of co-located characterizations in the bottom left contains terms related to inequality. The characterizations of poverty which are more common when it is discussed in connection with *global warming* include “global”, “abject”, “extreme”, and “victorian”, as well as “homelessness” and “food”. In comparison, the descriptions which are common for poverty without the connection to global warming (“great”, “high”, “endemic”, “persistant”, “systemic”; but also “lower”) are noticeably less emotional. Similarly, in the context of poverty, global warming is more likely to be characterized as “catastrophic”, “irrevocable”, and “existential”. The network highlights the causal relationship between global warming and poverty, in particular through war and conflict due to climate change. The role of inequality as an intermediary variable between the two is particularly interesting, showing that people also believe that climate-change effects on poverty are catalysed by inequality.

<footnote n="5">⁵The interactive graphs can be viewed in the HTML files included in the supplementary material.</footnote>
<figure><img src="image_31.png" /><figcaption>Figure 6.6: Four examples of the fine-grained analysis. The position and colour of the entities is random. However, the characterizations that surround the entities are coloured to indicate how strongly a given characteristic is associated with the entity in question in general vs with it in the context of the other entity. In the relationship paths, the direction of edges visualized the direction of the relationship from the active to the passive party.</figcaption></figure>
## COVID and Gender Inequality

The SARS-CoV-2 entity is characterized by concerns relating to “health inequality”, “social mobility”, and “unemployment” – it is not itself described with many adjectives other than “disastrous”. In contrast, *gender inequality* is almost exclusively characterised by the adjectives with which it is described, which denote scale (“massive”, “stark”), recent increase (“biggest”, “highest”, “greatest”), and its extent or context (“pervasive”, “national”, “digital”). More frequent, however, is the characterization of it appearing in a pair together with racial inequality. The relation paths between the two entities in the network evidence the indirect relation between the COVID pandemic and gender inequality. There is a prevailing narrative that the pandemic has exacerbated, increased and deepened gender inequality. Interestingly, the connection channels through which this relation produced its effect, such as job losses during the lockdowns disproportionately affecting women, are not mentioned explicitly. The effect that the pandemic has can also be seen in how gender inequality is characterized in this specific context, as the adjective descriptions (such as “massive” or “highest” gender inequality) are all more likely to appear in the context of the COVID pandemic than without it.

## COVID and Educational Inequality

*Educational inequality* was not as frequently discussed as gender inequality, and thus there are fewer interesting characterizations. Especially in the context of the pandemic, it is related to economic and technical issues and is described as “severe” more commonly than outside of the COVID pandemic. There is a clear narrative of the COVID Pandemic widening existing education inequalities. This is supported by what we now know of the effects of the COVID pandemic on children’s psychological well-being and schooling during the lockdowns of 2020 and 2021 [25, 39]. One important point appears to be the impact of technology and the digital divide in education.

## Inflation and Economic Inequality

Between 2020 and 2022, *inflation* has been described as “high”, more often than “low” or “lower”. It has also been called “runaway” or “rampant”, and has been located on a “global” scale more commonly in connection to *economic inequality*. The latter is also characterized by many descriptions of scale, showing a perception of greater economic inequality in the context of inflation. It is connected to “income” and “wealth”, but also to racial and gender concerns and is mentioned both in the global/international as well as a local context (focusing on the U.S.). The most prominent narratives connecting the two entities follow a well-known relation between inflation and economic inequality [12, 34]. For example, high inflation for primary goods, like food, tends to disproportionately affect those communities and families that are already economically and financially fragile. Interestingly there is also a link in the opposite direction, that is, economic inequality causing inflation, which is a hypothesis that has little evidence in economic literature.

## 6.4 Discussion and Conclusion

In this work, we have implemented the two-phase narrative analysis framework proposed in [26] in a visualization tool that can be used to interactively explore the narrative network. Building on this framework, we show how information from natural language understanding and knowledge representation can be integrated to extend both parts of the analysis.

On the one hand, the general knowledge from KGs can be used to consolidate actors and events
that take part in the narratives, making it possible to highlight the different ways in which these
entities are viewed in different kinds of narratives and in different contexts. In the future, we
hope to extend this narrative exploration tool with more information from knowledge sources.
Firstly, by allowing users to access this information directly from the nodes of the network, and
secondly, by analysing the KG in order to find “ground truth” connections between two entities
that can be compared to their relationship as described in the social media discourse.

On the other hand, the information from social media posts can be used to extend the KG. Our approach identifies NP entities that are concepts not currently captured by the KG but that clearly play an important role in the inequality discourse, such as *vaccine inequality, green card inequality* or *student load debt*. In many cases, these terms appear on social media before they are eventually added to more formal knowledge sources. Other terms can be linked to a general concept in the KG but, in fact, highlight a current event, development, or perspective related to the concept: the frequent occurrence of *inflation* or *web conferencing* does not refer to the general concept itself, but rather mirrors the recent increase in inflation or the importance of remote work tools during the pandemic. Linking events to KGs is more difficult than identifying people or locations, as evidenced by the fact that only very few entities in the bird's-eye network are linked to events in the KG. Even though a Wikidata entry regarding the current inflation exists<fnref n="6" />, matching references to it is difficult, and the entity does not currently contain a lot of information such as that it is an event in time. However, combining references to inflation from many texts, the characterizations extracted from them, and the metadata of the tweets could make it possible to improve the linking and extend the KG information.

In exploring the bird's-eye and the fine-grained network, one interesting problem related to NER and NEL becomes clear: there are many cases in which ML-based entity linking fails due to the context in which the entity appears, the shortness of the texts and the informal language used on social media. This includes not only vague references to events (which require context about the time of writing to understand), but also: missed links due to informal spelling conventions or shortened names ("us" meaning U.S., "fed" referring to the Federal Reserve System in the United States); frequently repeated phrases relating to concepts ("the status quo") or events ("I stand with ...") being matched to the titles of music albums; entities that share a common name with other, more typical entities, where correct understanding requires the larger context of the conversation<fnref n="7" />; and incorrect links that appear to mirror a perspective on the entity that exists in the language, but influences the linking due to the semantic encoding<fnref n="8" />.

In the work presented here, we used the proposed approach to explore the narratives shared
as part of the common social media discourse surrounding inequality, specifically during the
time period from 2020 to 2022, which was heavily characterised by the COVID pandemic and
its socio-economic implications. As the results suggest, Twitter discourses on inequality are
dynamic and multifaceted. Sometimes inequality-related claims are based on statistical data,
while at other times, they are based on erroneous beliefs, which is why being able to capture
and analyse them is crucial for a broader understanding of the phenomenon, its cultural and

<footnote n="6">⁶https://www.wikidata.org/wiki/Q110281455, which is described as “ongoing global surge of higher-than-average inflation”.</footnote>

<footnote n="7">⁷In the training data for the NEL, the names of small countries in Europe likely usually refer to a football team. However, after the invasion of Ukraine in 2022, there was a marked increase in mentions of different European countries on English language social media, without explicit reference to the invasion but clearly understandable to human readers, given the overall context. In this case, NEL could be improved by comparing, e.g. whether the Latvian football team or the country Latvia is closer connected in the KG to other entities mentioned in the tweet.</footnote>

<footnote n="8">⁸An example of this case is that they are many occurrences in which NEL links the phrase “russia” to the KG entry of the *Russian Empire* instead of the country *Russia*, often when it is mentioned in the context of aggression or invasion.</footnote>
political implications, and its impact on people's (material and psychological) well-being. The case study presented in Section 6.3 highlights how researchers' connections of interest can be found interactively and can be analysed in more detail by "zooming in" on a given relation (i.e., edge). Next, we plan to extend this tool and evaluate it together with domain experts. We intend to develop an interface that will allow non-computer scientists to filter the data and retrieve KG information, offering them to perceive the multiple viewpoints of any narrative. At the moment, the presented case study is limited in scale due to the requirements of natural language processing and requests to the KG for entity linking. Therefore, we hope to scale up the architecture to enable new data to be retrieved and processed on the fly, allowing users to continuously observe how the inequality discourse develops over time. While this tool offers a novel approach for assessing and analysing diverse narratives interesting to this community, we hope to reach people beyond scientific fields, such as news media and policy-makers, to observe the bigger picture behind inequality narratives.
Chapter 6: References

[CC1] Kiran Kumar Bandeli, Muhammad Nihal Hussain, and Nitin Agarwal. A framework towards computational narrative analysis on blogs. In *Text2Story@ ECIR*, pages 63–69, 2020.

[CC2] Christopher Barrie and Justin Chun-ting Ho. academictwitter: an r package to access the twitter academic research product track v2 api endpoint. *Journal of Open Source Software*, 6(62):3272, 2021.

[CC3] W Lance Bennett. The personalization of politics: Political identity, social media, and changing patterns of participation. *The annals of the American academy of political and social science*, 644(1):20–39, 2012.

[CC4] William Croft, Pavlina Kalm, and Michael Regan. Decomposing events and storylines. *Computational Analysis of Storylines: Making Sense of Events*, page 67, 2021.

[CC5] Christophe Ehrhart. The effects of inequality on growth: a survey of the theoretical and empirical literature. *ECINEQ WP*, 107, 2009.

[CC6] Mark A Finlayson. Recent advances in narrative natural language processing. *Text2Story@ ECIR*, page 3, 2020.

[CC7] Maksym Gabielkov, Ashwin Rao, and Arnaud Legout. Studying social networks at scale: macroscopic anatomy of the twitter social graph. In *The 2014 ACM international conference on Measurement and modeling of computer systems*, pages 277–288, 2014.

[CC8] Erving Goffman. *Frame analysis: An essay on the organization of experience*. Harvard University Press, 1974.

[CC9] David Herman. Narrative theory and the cognitive sciences. *Narrative Inquiry*, 2001.

[CC10] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.

[CC11] A Jorge, Ricardo Campos, Adam Jatowt, and S Nunes. First international workshop on narrative extraction from texts (text2story 2018). In *ECIR*, pages 833–834, 2018.

[CC12] Venoo Kakar and Gerald Eric Daniels Jr. Role of cash and costs of inflation for different income groups in the us. *Economic Modelling*, 80:303–319, 2019.

[CC13] Hermann Kroll, Denis Nagel, and Wolf-Tilo Balke. Modeling narrative structures in logical overlays on top of knowledge repositories. In *International Conference on Conceptual Modeling*, pages 250–260. Springer, 2020.

[CC14] Kai-Robin Lange, Matthias Reccius, Tobias Schmidt, Henrik Müller, Michael WM Roos, and Carsten Jentsch. Towards extracting collective economic narratives from texts. Technical report, Ruhr Economic Papers, 2022.
[CC15] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. *Semantic web*, 6(2):167–195, 2015.

[CC16] Carlo Meghini, Valentina Bartalesi, and Daniele Metilli. Representing narratives in digital libraries: The narrative ontology. *Semantic Web*, 12:1–24, 12 2020.

[CC17] Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. Dbpedia spotlight: shedding light on the web of documents. In *Proceedings of the 7th international conference on semantic systems*, pages 1–8, 2011.

[CC18] Martino Mensio, Harith Alani, and Alistair Willis. Towards a cross-article narrative comparison of news. In *Proceedings of the Text2Story’20 Workshop*. CEUR WS, 2020.

[CC19] Seth A Myers, Aneesh Sharma, Pankaj Gupta, and Jimmy Lin. Information network or social network? the structure of the twitter follow graph. In *Proceedings of the 23rd International Conference on World Wide Web*, pages 493–498, 2014.

[CC20] Ruth Page. *Narratives online: Shared stories in social media*. Cambridge University Press, 2018.

[CC21] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*, pages 1532–1543, 2014.

[CC22] Jakub Piskorski, Fredi Šaric, Vanni Zavarella, and Martin Atkinson. Exploring machine learning techniques for linking event templates. *Computational Analysis of Storylines: Making Sense of Events*, 2021.

[CC23] Robert Porzel. On formalizing narratives. In *JOWO*, 2021.

[CC24] Robert Porzel, Mihai Pomarlan, Laura Spillner, John Bateman, Thomas Mildner, and Carlo Santagiustina. Narrativizing knowledge graphs. 2021.

[CC25] Fernando M Reimers. Learning from a pandemic. the impact of covid-19 on education around the world. In *Primary and secondary education during COVID-19*, pages 1–37. Springer, Cham, 2022.

[CC26] Richard Rogers. Debanalizing twitter: The transformation of an object of study. In *Proceedings of the 5th annual ACM web science conference*, pages 356–365, 2013.

[CC27] Pasi Sahlberg. Does the pandemic help us make education more equitable? *Educational Research for Policy and Practice*, 20(1):11–18, 2021.

[CC28] Carlo Romano Marcello Alessandro Santagiustina. From narrative economics to economists’ narrative. In Luc (ed.) Steels, editor, *Foundations for Meaning and Understanding in Human-centric AI*, chapter 2, pages 266–290. Venice International University, Venice, Italy, 2022.

[CC29] Carlo Romano Marcello Alessandro Santagiustina and Massimo Warglien. The architecture of partisan debates: The online controversy on the no-deal brexit. *PloS one*, 17(6):e0270236, 2022.

[CC30] Daniel Thilo Schroeder, Konstantin Pogorelov, and Johannes Langguth. Fact: a framework for analysis and capture of twitter graphs. In *2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS)*, pages 134–141. IEEE, 2019.
[CC31] M Ángeles Serrano, Marián Boguná, and Alessandro Vespignani. Extracting the mul-
tiscale backbone of complex weighted networks. *Proceedings of the national academy of sciences*, 106(16):6483-6488, 2009.

[CC32] Laura Spillner, Carlo R. M. A. Santagiustina, Thomas Mildner, and Robert Porzel. Towards conflictual narrative mechanics. In *Proceedings of the IJCAI/ECAI Workshop on Semantic Techniques for Narrative-based Understanding*, 2022.

[CC33] Luc (ed.) Steels. *Foundations for Meaning and Understanding in Human-centric AI*. Venice International University, Venice, Italy, 2022. This is the first volume in a series exploring how AI systems can construct rich models of problem situations for deliberative intelligence.

[CC34] Eleftherios Thalassinos, Erginbay Ugurlu, and Yusuf Muratoglu. Income inequality and inflation in the eu. 2012.

[CC35] Kadi Toure, Etienne V Langlois, Mehr Shah, Lori McDougall, and Helga Fogstad. How george floyd and covid-19 are highlighting structural inequities for vulnerable women, children and adolescents, 2021.

[CC36] Francois van Schalkwyk, Jonathan Dudek, and Rodrigo Costas. Communities of shared interests and cognitive bridges: The case of the anti-vaccination movement on twitter. *Scientometrics*, 125(2):1499-1516, 2020.

[CC37] Tatiana Von Landesberger, Arjan Kuijper, Tobias Schreck, Jörn Kohlhammer, Jarke J van Wijk, J-D Fekete, and Dieter W Fellner. Visual analysis of large graphs: state-of-the-art and future research challenges. In *Computer graphics forum*, volume 30, pages 1719–1749. Wiley Online Library, 2011.

[CC38] Piek Vossen, Tommaso Caselli, and Roxane Segers. A narratology-based framework for storyline extraction. *Computational Analysis of Storylines: Making Sense of Events*, page 125, 2021.

[CC39] Katharina Werner and Ludger Woessmann. The legacy of covid-19 in education. 2021.

[CC40] Richard G Wilkinson. *The impact of inequality: How to make sick societies healthier.*
Routledge, 2020.

[CC41] Tom Willaert, Sven Banisch, Paul Van Eecke, and Katrien Beuls. Tracking causal relations in the news: data, tools, and models for the analysis of argumentative statements in online media. *Digital Scholarship in the Humanities*, 2021.

[CC42] Iqra Zahid, Hao Zhang, Frank Boons, and Riza Batista-Navarro. Towards the automatic analysis of the structure of news stories. In *Text2Story@ ECIR*, pages 71–79, 2019.

[CC43] Florian Zollmann, Jeffery Klaehn, Tina Sikka, Kristin Comeforo, Daniel Broudy, Mandy Tröger, Elizabeth Poole, Alison Edgley, and Andrew Mullen. The propaganda model and intersectionality: integrating separate paradigms. *Media Theory*, 2(2):213–239, 2018.
# Part IV

## Understanding Society through Art
# Chapter 7

## Integrative Narrative Networks for Art Interpretation

Sofia Baroncini*, Luc Steels†, Remi van Trijp<fnref n="1" />, <fnref n="2" />

* Leibniz Institute of European History Mainz, [baroncini@ieg-mainz.de](mailto:baroncini@ieg-mainz.de)

† Venice International University, [steels@arti.vub.ac.be](mailto:steels@arti.vub.ac.be)

◦ Sony Computer Science Laboratories, Paris Lab, [remi.vantrijp@sony.com](mailto:remi.vantrijp@sony.com)

### Abstract

Recent advances in AI have steadily supplied Digital Humanities with new tools to analyse and contextualise artworks using techniques from computer vision, pattern recognition, ontology engineering, natural language processing, and the semantic web. These tools are important because they may unlock more insightful art descriptions and facilitate access to a vast range of online materials like catalogue descriptions, art history texts and thesauri. However, to obtain the full potential of these tools we need to tackle two issues: (i) how to integrate the fragmented and sometimes contradictory information these various tools provide, and (ii) how to make it much easier for art historians, curators, and artists to use and extend these tools.

This paper explores how these issues can be addressed through the use of Integrative Narrative Networks (INN), a data structure that formally represents the process of understanding in terms of narrative questions and answers, and that integrates information from various knowledge sources such as vision, pattern recognition and language. For this purpose, we present a case study on a Late Renaissance Italian painting by Lorenzo Lotto, which is notoriously challenging for the artist's complex symbology. We built two different scenarios, according to the type of source information leveraged. First, we approached art interpretation by extracting entities from the museum's catalogue entry through Natural Language Processing (NLP), and linking them to Wikidata. In the second scenario, the process of understanding starts using as a source information the artwork itself, and progressively deepens the knowledge about it by answering narrative questions using information from available Knowledge Graphs.

**Keywords** Art interpretation, AI tools, Integrative Narrative networks, NLP, Knowledge Graphs.

<footnote n="1">¹S. Baroncini is responsible for Sections 7.1.2 and 7.3. L. Steels is responsible for sections 7.1.1 and 7.2.1. R. van Trijp is responsible for Sections 7.2.2 and 7.4. All the authors are responsible for the remaining sections. The current paper is an extended and revised version of [5].</footnote>

<footnote n="2">²Supported by the EU Pathfinder project MUHAI (EU grant 951846) and by the international exchange program of Bologna University.</footnote>
# 7.1 Narrative-based Art Interpretation

## 7.1.1 Understanding as question-answering

Recent advances in AI have offered exciting new possibilities for analysing and contextualising artworks using techniques from computer vision, pattern recognition, ontology engineering, natural language processing, and the semantic web. However, these tools also offer fragmented pieces of insight that still need to be combined into coherent interpretations that are easy to access and extend for art historians and other viewers. In this paper, we propose to address this challenge through **narrative-based understanding** [23].

Narratives are an essential tool of the human sensemaking process [1] in which we take otherwise fragmented events and pieces of information, and turn them into a coherent *plot* – a causal chain or network of events [2]. When interpreting an artwork, the viewer tries to construct a narrative that integrates the visual depictions, memory of past experiences, knowledge of artistic styles, the personal history and prior work of the artist, and general world knowledge, in order to answer a whole series of questions, such as: Who is depicted? What are the characters doing? Why are they doing this and why have these actions been shown? What is the background scene? Why has it been chosen? What perspective has been adopted and why? etc. The viewer also tries to put an artwork in context. Who is the painter? Where and when did s/he paint this work? What was the function of the work? What kind of style was common at the time? These are called **narrative questions** that give the viewer a sense of direction; and answering these questions helps them to drive the plot of their narrative forward.

One major challenge is that we cannot conceptualize the process of interpretation and that of constructing a narrative as a linear pipeline of subprocesses in which one feeds into another, as is common in data-driven AI, because of a paradox known as the *hermeneutic circle* [15]: to understand the whole we need to understand the parts but to understand the parts we need to understand the whole [14]. Instead, narrative-based understanding has to be conceived of as a spiraling process. Starting from an initial examination of some input elements with a lot of ambiguity, uncertainty and indeterminacy, hypotheses of the whole are constructed, which then provide top-down expectations to be tested by a more detailed examination of the same or additional elements, leading to a clearer view of the whole, which then leads back to the examination of additional elements, etc., until the narrative ‘makes sense’ and resonates with the personal episodic memory of the viewer, reaching a state known as **narrative closure** in literature studies [10, 25].

When considering partial automation or digital support for interpretation we need not only work on the individual knowledge sources that can answer or raise questions, such as pattern recognition algorithms that can detect iconic elements or find the meaning of these elements in a database linking icons and their significance. We also need to work on the integration process, and this requires, as a first, fundamental step a data structure that supports integration.

This paper leverages such a data structure, called an **Integrative Narrative Network (INN)**, previously introduced in [25], which acts as a kind of blackboard on which different knowledge sources write partial descriptions and through which they can consult information provided by other knowledge sources to advance understanding.

This data structure potentially supports several functions: (i) share the information between knowledge sources so that there can be both bottom-up and top-down information flow, (ii) orchestrate semi-automatically the process of calling on different knowledge sources based on which information would be of most interest to enrich the interpretation, (iii) drive a user-interface that allows art historians to organise their work by making it possible for them to
actively invoke knowledge sources, and (iv) drive user interfaces to support the progressively deepening experience of a new artwork.

### 7.1.2 Case Study

We have already applied the methodology and techniques described in this paper to the interpretation of a contemporary artwork 'Secrets' by the painter Luc Tuymans. This experiment was the subject of an exhibition at the BOZAR cultural center in Brussels in 2011.<fnref n="3" /> For that project we focused on integrating results coming from pattern recognition and image processing as reported in [1] and from NLP techniques applied to text from the catalogue and other textual resources.

In this paper, we explore historical artworks from the late Renaissance period by the Venetian painter Lorenzo Lotto and we focus on integrating results from semantic web resources, specifically knowledge graphs, with the outputs of NLP processing. More concretely, we report on a use case for the painting 'Venus and Cupid' by Lorenzo Lotto at the Metropolitan Museum in New York<fnref n="4" /> (see Figure 7.1). This painting was chosen as a feasible case study for art understanding since the lack of textual documentation and its complex iconography constitute an interpretative challenge for art historians [11].

At first glance, the observer's attention is driven by some visual indicators, such as the brightness contrast, to the body of the main characters, i.e., a putto (little angel) and a woman. If the observer is familiar with the art themes depicted during the Italian Renaissance, s/he will recognise immediately these figures respectively as Cupid and his mother Venus, thanks to the identification of their attributes. Nevertheless, the actions performed (e.g. Cupid's act of urinating through a wreath, with a playful expression) and the presence of other objects (e.g. the snake, the cone shell, etc.) may lead the observer to raise new questions: 'What exactly is depicted?', 'Why are these objects represented?', 'What is the scene telling us?'.

These questions may be further enhanced by the observer's past experience with other paintings by Lotto, who often depicted symbols or visual anagrams in his artworks. At this point, the observer, particularly if s/he is an art historian, will start to look up the possible meanings of the iconic objects through visual and textual sources and verify if the newly discovered knowledge about it is coherent with the context.

Narrative closure is reached when answers to all these questions are coherently integrated with each other and grounded in external and past experiences. For example, the art historian Keith Christiansen recognises that the curious action made by Cupid and the majority of the objects (e.g. the red cloth, the snake, the myrtle wreath, the incense burner) belongs to the sphere of marital love, concluding that the artwork expresses a wedding wish.<fnref n="5" />

Note that the focus of the interpreter has gone beyond the main focal point of the painting (i.e., Venus and Cupid), to explore secondary details, which are now seen as signs, triggering a deeper interpretation of the scene.

An online user or museum visitor has usually a wealth of information available which s/he can use to understand the artwork, without going through the full hermeneutic process described above. For example, s/he can read the textual apparatus curated by museum's practitioners and art historians to progressively understand the painting. Following the narrative of the text,

<footnote n="3">³https://readymag.com/u3083945729/secrets-guide/</footnote>

<footnote n="4">⁴Catalogue entry available at https://www.metmuseum.org/en/art/collection/search/436918</footnote>

<footnote n="5">⁵A more thorough explanation of this interpretation can be found in the artwork catalogue entry of the Metropolitan museum.</footnote>
<figure><img src="image_32.png" /><figcaption>Figure 7.1: Lorenzo Lotto, *Venus and Cupid*, 1520s, Metropolitan Museum of Art, New York.
Public domain</figcaption></figure>

s/he will recall the personal knowledge to understand the concepts explained, and observe the
painting to relate the textual description with the depicted items. For example, if the text
says that the painting shows Venus, the observer will leverage his/her background knowledge
to search for a female figure in the artwork, if s/he is familiar with such a subject. In this
process, the observer may not know all the knowledge treated by the text, and retrieve more
information from external sources, for example through a web source, to answer his/her open
questions (e.g., ‘what is an epithalamium?’). In this case, the understanding process is guided
by the narrative that the text’s author adopted in order to effectively communicate to the
museum’s visitors the knowledge previously acquired. The observer follows the text guide, and
occasionally distances him/her from it to relate such information to the observed features of
the painting through the aid of background knowledge. Open questions for which the answer
is not provided by the text may be answered through access to external sources. The process
ends when the observer reaches a satisfactory understanding of the painting and the text.

In the remainder of this paper, we first discuss the technical solutions that we adopted for
providing support to the interpretation process sketched out above. Then we illustrate how
this can be applied to the Lotto's painting, in both cases in which 1) the knowledge is acquired
from the catalogue entry or 2) when no previous knowledge of the painting is available.
## 7.2 Integrative datastructure (INN)

### 7.2.1 Narrative questions and answers

As explained, we frame art interpretation as a process whereby the agent raises narrative questions, finds answers to those questions, and interlinks narrative questions and answers. We focus in particular on which knowledge can be automatically extracted from the text and better understood through the aid of semantic web sources, and how these can help to raise questions and find answers to them.

Whereas we refer to the previous work by the authors in which the INN structure was extensively defined [25],<fnref n="6" /> we here resume the main features of the data structure to allow the understanding of its application to the art history domain.

**What are the questions?** Questions are computationally operationalized as variables. Following AI tradition, the name of the question is written as a symbol with a question mark in front, as in ?question-name, where the question-name is a symbol that is chosen on the basis of the nature of the answer to the question. The name is meant to be meaningful for us. For example, we write ?Where instead of ?Question-540 to be suggestive that this question is about where the artwork was produced. Note however that question-names (and also names for entities) are only formal symbols that obtain their function solely from their connections to other symbols, not from the name they happen to have.

**What are the answers?** The answers to a question are entities in the domain of discourse. Entities are objects, events or (reified) concepts. They are designated with a symbol, called the entity-id, within angular brackets. The entity-id can be a proper name for a constant (e.g. <LorenzoLotto>) or an instance of a certain type of entity. Entities either refer to real-world observational data (for example a physical painting, a region in an image), to virtual entities (which may or may not exist in reality), or to entities in a knowledge graph in which case we use the URI (Universal Resource Identifier) as unique identifier.

INN makes the relation between a question and an answer computationally operational in terms of a binding between questions (which are technically variables) and identifiers of entities or constants. Bindings are a familiar concept in computer science. Bindings can be set explicitly or indirectly in a pattern matching process (as in logic programming languages like Prolog for example).

**Where do questions from?** We use a frame-based approach which was pioneered in research on knowledge representation and object-oriented programming, starting in the mid-seventies with the proposal by [18] and technical realisations such as KRL [7], KRS [22] or CLOS [13]. In the present INN implementation we use CLOS. In Minsky's original conception, a frame is a bundle of important questions to be asked about a particular type of object. A frame has a set of slots with values, which are in effect the questions that can be asked about a particular entity represented by a frame and respectively the answers to these questions. An entity which is described with a certain frame is called a frame-instance of that frame.

**Where do answers come from?** Answers, more concretely default answers, may already be contained in the frame itself. For example, the default value for the ?Period slot in the [renaissance-painter] frame would already be filled with the value <Renaissance>. In the current case study, answers consist of the information that it is possible to retrieve through

<footnote n="6">⁶The definition of INN is also supported by an open-source software implementation of integrative narrative networks (Apache 2.0 license) as part of the Babel cognitive software suite [17, 24] which can be downloaded at https://emergent-languages.org/</footnote>
queries to knowledge graphs or from NLP information extraction. In the network, there is no
specific order in which information must be available to perform a computation, and in the
examples the user is free to guide the development of the network.

7.2.2 Interactive Visualization

The open-source software implementation of Integrative Narrative Networks interfaces with the browser-based visualization library vis.js (community edition; https://visjs.org) for displaying an INN in a web browser.<fnref n="7" /> The implementation also allows users to make these visualizations interactive, which makes it possible to interact with and modify the networks directly in the web browser (e.g. by removing or adding nodes and links).

By default, narrative questions are represented as diamond shapes that are coloured red for unanswered questions, and green for answered questions. Instances, which may be the answers to questions, are represented with squares, representing frame-instances or constants, such as a number or a boolean value.

The user is also free to extend the types of nodes in the network. For instance, directed edges can be used for indicating a particular path in the network (e.g. for representing the plot of the narrative); or different colours can be used for differentiating the type of knowledge source.

Finally, for users who prefer other visualization libraries, networks can be exported so they can
be used in other programming environments and network tools.

7.3 Case study for the Lotto painting

This section illustrates the INN and its use in interpretation for a concrete case study on
the painting *Venus and Cupid* by late Renaissance painter Lorenzo Lotto introduced earlier
(Section 7.1.2). The case study has been implemented in two scenarios of understanding. The
first one reconstructs the case in which an observer tries to better understand the painting
through the aid of the catalogue entry provided by the museum cutators. In this case, we
experiment whether NLP tools can help in extracting relevant knowledge from the text, and
connect it to external sources to allow the user further navigation and discovery. The building
of the network follows the text narrative. In the second case, the scenario presupposes the sole
knowledge of the artwork, and the user conducts an exploratory search about it on available
semantic knowledge graphs, progressively extending the network according to the questions
answered and information retrieved.

7.3.1 Leveraging the catalogue entry

Recent advances in text processing with various AI techniques (e.g., Natural Language Pro-
cessing (NLP) and Large Language Models (LLMs)) have shown an increasing potential in text
processing to structure information from free text. In the art history domain, a wealth of infor-
mation about artworks is provided in the form of texts, such as the museums' catalogue entries,
scholars' writings, auction books, and archival records. Some studies have shown promising, but
still challenging attempts in structuring information provided by art historical texts. Among
them, it is worth considering Viewsari, the Knowledge Graph generated from Vasari's *Vite*, the
main historiographical source for Early Modern painters in Italy, by extracting main entities

<footnote n="7"><sup>7</sup>Our paper is supported by an open-source software implementation of Integrative Narrative Networks for Art Interpretation (Apache 2.0 license) as part of the Babel cognitive software suite [17, 24] which is available at https://github.com/SonyCSLParis/art-network</footnote>
such as places, people and their biographies [19]. Further studies extracted knowledge from multiple types of art historical texts with an open information extraction approach [16] and entity linking resolutions between the artwork's description provided by the dataset Artpedia<fnref n="8" /> and Wikidata [21]. Although relevant, all these studies highlight the challenges raised by correctly identifying and disambiguating highly domain-specific and context-dependent knowledge and describing it with accurate modeling. Given such current challenges, it is relevant to verify the extent to which AI text processing techniques can be combined with other sources, such as semantic ones, to enhance a meaningful structured representation of the information extracted from the textual sources.

In our study, we experimented with how the knowledge provided by the catalogue entry of Lotto's painting⁹ can help in advancing human and machine understanding through the extraction of information from the text and its linkage to existing knowledge bases. The retrieval of more information from knowledge graphs would help users to better understand unknown concepts, and experts in quickly retrieving more information about the same topic. Nevertheless, the quantity and quality of the information that can be collected highly depends on the performance of text-processing tools and on the quality and (specialistic) detail provided by the knowledge sources.

In our study, the text of the catalogue entry was processed through Textrazor¹⁰, a tool combining NLP and AI methods not only to parse and analyze text but also to create semantic metadata, generating mappings with Wikidata entities [26]. The approach adopted consisted of launching the text on the Textrazor interface and examining the results. Textrazor automatically subdivides the text into phrases, and, for each phrase, recognizes several entities, among which are Wikidata identifiers. We first manually corrected the subdivision of the phrases. Following, we checked the results of the Wikidata entities identified, evaluating which alignments were correct.

Each phrase was represented as a node in a graph. The NLP agent seeks to answer, for each phrase, the question ‘what is this sentence about?’. Therefore, we relate ?What nodes to each phrase, which are marked as solved (i.e., visualized in geen) when correct entities are retrieved. In this way, the network represents the narrative with which the discourse is conducted by the art historian who wrote the catalogue entry, and it is possible to identify the important entities repeatedly cited.

The fact that the identified entities are Wikidata ones allows the connection of multiple sources and the exploration of further information in the knowledge base. In particular, we queried the type of Wikidata entities through the relation ‘instance of’ (wdt:P31) or ‘subclass of’ (wdt:P279). Such information is the answer to ?What narrative questions evoked by the Wikidata entities retrieved. Therefore, we visually represented them as related to question nodes liked to the respective entities.

## Results

Out of the 51 text sections identified, 46 could be related to at least one Wikidata entity (see Table 7.1)¹¹. The 5 phrases for which an entity couldn't be retrieved are related to a red narrative question, as shown in Figure 7.2. The error percentage of the entities retrieved through TextRazor is 9.7% of the entity occurrences. The majority of the errors lie in the

<footnote n="8">⁸Available at https://doi.org/10.48610/8a1ccdf [21]

⁹available at https://www.metmuseum.org/art/collection/search/436918 in the ‘calatog entry’ section

¹⁰https://www.textrazor.com/

¹¹The complete list of results is made available at https://inn-painting-art-interpretation.streamlit.app/Catalogue_ENTRY_Narrative_Network</footnote>
<figure><img src="image_33.png" /><figcaption>Figure 7.2: Network generated by processing the painting’s catalogue entry. Each blue node is a sentence linked to the ?What questions raised when the entity extraction with Textrazor is performed. The blue edges link one sentence to the next one, highlighting the narrative of the text. When a Wikidata correct entity is retrieved (yellow nodes), the entity is bound to the question node, which color changes from red to green. The figure shows the network after the NLP processing</figcaption></figure>
<figure><img src="image_34.png" /><figcaption>Figure 7.3: Network generated by processing the painting’s catalogue entry network after the type of each Wikidata entity is retrieved through the relations wdt:P31 or wdt:P279 to answer the further raised ?What questions</figcaption></figure>
<table><caption>Table 7.1: Overview of the results of entity linking to Wikidata through Textrazor</caption><thead><tr><td>Type</td><td>Results</td></tr></thead><tbody><tr><td>Percentage of correct entity occurrences</td><td>90.3%</td></tr><tr><td>Total of unique correct entities</td><td>100</td></tr><tr><td>Total of unique entities (correct and wrong)</td><td>109</td></tr><tr><td>Phrases without a Wikidata entity</td><td>5</td></tr><tr><td>Total of phrases</td><td>51</td></tr></tbody></table>

wrong identification of the string 'fig.', referring to the image, with the plant 'ficus', or to entities that were too general for the context in which they were used (e.g., the object 'paining' was associated with the concept 'art of painting').

The Wikidata entities majorly linked, as they present the highest number of incoming edges, are 'Venus' (10), 'Cupid' (8), 'Lorenzo Lotto' (8), Venice (6), and 'wreath' (6).

A second iteration retrieved from Wikidata the type of the extracted entities. As all of the entities had at least one type specified, all the ?What nodes related to them were marked as satisfied (see Figure 7.3). This information gave further hints concerning the nature of the entities treated in the text, showing common types among 35 entities out of the 100 retrieved. The most related one is the node 'human', with 10 incoming edges, showing that at least 10 different people were cited in the text.

Results of the second iteration show that the retrieval of further information from Wikidata is useful not only to better understand the nature of the concepts introduced in the text, but also to identify the ones having a shared type, providing more insights of a quantitative overview of the subjects treated in the text.

In the current scenario, as the main source of information for the understanding process is the catalogue entry, the INN structure is also used as a means of visualizing the narrative of the text, linking each phrase, represented by a blue node, with the next one and with the identified Wikidata entities. In this way, it is possible to understand the central entities of the text, and to reconstruct which textual passages evoke them. The visualization is further enhanced by a dynamic visualization to observe how the network expands phrase by phrase<fnref n="12" />.

The experiment has shown that the integration of NLP tools with semantic web sources allows to deepen the understanding of the entities cited by the text and helps in establishing links among them. Nevertheless, the retrieval of further information was limited to the ?What question as other information would require a costumed retrieval system according to the type of entity. Furthermore, the experiment would benefit from further testing of other NLP tools and inquiries in how the knowledge retrieved from Wikidata could help to establish further semantic relations among the retrieved entities.

### 7.3.2 Queries to knowledge sources

In this scenario, the only source of information initially provided is the sole artwork. Consequently, the case study has been implemented in the sense that all the access to diverse knowledge sources is done through exploratory queries to knowledge bases and then integrated into the INN.

We formalised a set of exploratory narrative questions raised during art interpretation with a

<footnote n="12">¹²Available at https://inn-painting-art-interpretation.streamlit.app/Catalogue_ENTRY_Narrative_Network</footnote>
<table><caption>Table 7.2: Knowledge bases included in the case study.</caption><thead><tr><td>Knowledge Base</td><td>Description</td></tr></thead><tbody><tr><td>Wikidata [26]</td><td>Open, collaborative KB hosting structured data from other Wiki-media projects</td></tr><tr><td>Arco [9]</td><td>Italian Cultural Heritage</td></tr><tr><td>Zeri&amp;Lode [12]</td><td>The photographs about XVI Century art of Federico Zeri's photographic collection</td></tr><tr><td>The Iconology Dataset [3]</td><td>Iconological interpretations about ca 400 artworks, mainly by the art historian Erwin Panofsky</td></tr><tr><td>HyperReal [20]</td><td>Symbols and symbolic meanings in different contexts</td></tr></tbody></table>

top-down approach from the didactic art literature<sup>13, 6</sup>. The suggested questions were grouped as sub-questions of five wh-questions, namely Who? What? Where? When? Why?. In this way, we proposed a definition of the possible questions that a frame 'visual artwork' may raise when the intention is to interpret the piece. The questions were then expressed as SPARQL queries to be performed over an initial selection of knowledge bases (illustrated in table 7.2). For example, to answer the question about ?Who (i.e., who is the painter?) the following SPARQL query can be launched to the Wikidata knowledge base<sup>14</sup>.

```sql
SELECT ?author ?authorLabel
WHERE {
  wd:Q4009580 wdt:P170 ?author.
  ?author rdfs:label ?authorLabel.
}
```

Table 7.3 summarizes the questions in natural language that could be expressed as SPARQL queries. The answer from a query (if successful) is then integrated as an answer bound to the question ?Who in the INN. The criteria for selecting knowledge bases are the possible presence of data about artworks, their content, and/or possible meanings, in compliance with the results obtained by the study in [4]. Furthermore, results obtained from computer vision analysis were integrated into the Narrative Network through a test on Google API Cloud Vision<sup>15</sup>.

More concretely, we searched for the painting ID on the KBs through SPARQL queries looking for the artist's name and title or looking for alignments with the ID previously found (e.g. Wikidata's ID). We then performed the queries expressing the 5W questions on each KB. During the process, results were represented according to the INN data structure, showing a graph which progressively expanded from the initial artwork node. Nodes retrieved from different KBs, which were possibly referring to the same concept (e.g., the representation of the character 'Venus' in Wikidata and in Zeri&Lode), were detected through a label similarity fuzzy ratio, and through Wordnet synonyms, and through the presence of common URI to which they were already aligned.

## Results

In the initial phase, we retrieved the artwork ID, its title and its eventual image from the selected knowledge bases, starting from Wikidata. The artwork was described only in 3 of them, namely Wikidata, the Iconology Dataset, and Zeri&Lode. The first node, representing the artwork,

<footnote n="13"><sup>13</sup>University of Manchester, Art historical research. Url: https://library.wcupa.edu/_art_history</footnote>

<footnote n="14"><sup>14</sup>The SPARQL queries performed are available at https://inn-painting-art-interpretation.streamlit.app/Narrative_Questions_and_Queries</footnote>

<footnote n="15"><sup>15</sup>https://cloud.google.com/vision</footnote>