Relation Extraction for Constructing Knowledge Graphs: Enhancing the Searchability of Community-Generated Digital Content (CGDC) Collections

Authors: martin  marinov
youcef  benkhedda
ewan  hannaford
marc  alexander
goran  nenadic
riza  batista-navarro
Venue: KDD_org_2024_Workshop_DL4KG
Published: 2024-05-27 10:51:56

Abstract: Much of people's understanding of their cultural heritage is facilitated by the curation and preservation of community-generated digital content (CGDC): archival collections that were created for, with and by local communities. However, communities employ their own conventions in storing and publishing their content. Given this and the fact that semantic information tends to be buried within textual descriptions, CGDC archives are currently siloed and obscured, thus making it difficult for end-users (e.g., members of the public and researchers) to search for fine-grained information (e.g., "Where did Alfred Edward Julian work?"). In this paper, we propose to represent the information within CGDC archives in the form of knowledge graphs. To enable the construction of such knowledge graphs at scale, we developed a zero-shot approach for relation extraction, which we cast as a natural language inference (NLI) problem. Specifically, for each of the 20 relation types drawn from Wikidata that we have identified as relevant to CGDC, we created a premise-hypothesis pair that is presented to an NLI model that determines whether entailment (and thus the relation type) holds. The premise is a sentence from the natural language description and the hypothesis is automatically generated using a template based on each of the relation types. We present the results of comparing and combining three different transformer-based models that were already fine-tuned for the NLI task, namely, DeBERTa, BART and T5.

Main paper: 
 introduction: 
 Cultural heritage is preserved and passed on to future generations through archival collections and their digitalisation. While enormous efforts (e.g., The National Archives of the UK, Europeana) have been put into making such collections available to the public, a significant part of people's cultural heritage is represented only in community-generated digital content (CGDC): digital-born archive collections developed for, with and by communities [1] . For example, in the UK, around 5000 community history projects have been funded by the National Lottery Heritage Fund (NLHF), allowing many communities to explore and preserve their history and heritage, leading to the proliferation of CGDC collections.These communities tend to follow their own conventions to store and represent their CGDC collections. As a result, most of the rich semantic information contained within these collections remain buried within textual metadata, such as the titles and descriptions of CGDC items. The description of a photograph of a local landmark, for instance, might mention the person or organisation who was responsible for building that landmark-information that is potentially of interest to researchers or members of the public and yet obscured within text. In this work, we propose to transform the information within the textual metadata of CGDC collections into a knowledge graph, in order to make CGDC more searchable and queryable. To this end, we investigated zero-shot approaches to relation extraction (RE) as a means for automatically curating such knowledge graphs, thus eliminating the need for any training data. Specifically, we cast RE as a natural language inference (NLI) task, allowing us to take advantage of three pre-trained transformer-based language models: DeBERTa [2] , BART [3] and T5 [4] . Although RE has been addressed using zero-shot approaches in other domains [5] , to the best of our knowledge, the use of such has been under-explored in the cultural heritage domain. Cetoli [6] and Chen and Li [7] proposed zero-shot RE approaches for the general domain. Meanwhile, Tang et al. [8] investigated various pre-trained transformer-based language models for RE in ancient Chinese history documents; however, their models required some form of training (either by fine-tuning or chain-of-thought prompting). Most similar to our work is that of Tan et al. [9] , who also constructed knowledge graphs based on entity relations extracted from Chinese cultural heritage texts. However, their work is rule-based and did not explore the use of transformer-based models. 
relation-annotated dataset: 
 To support our experimentation with transformer-based RE models, we constructed a dataset of CGDC textual metadata where relationships between named entities were manually annotated.The work of Benkhedda et al. [10] considered the following five entity types as relevant to CGDC: Person, Organisation, Location, Miscellaneous and Date. Drawing inspiration from their work, we decided to focus on relation types involving any of these entity types. For this purpose, instead of creating our own relation types, we leveraged Wikidata properties [11] . Initially, a total of 29 Wikidata properties (relation types) were identified as relevant to CGDC. Our annotation process (described in Section 2.2), however, revealed that nine of these types were very rarely encountered in our dataset. Thus, in the end, after keeping only the types with at least three labelled examples, 20 relation types were retained. We refer the reader to Appendix A for a full listing of these relation types.In our work, we utilised the CGDC dataset that was annotated with named entities (and their corresponding identifiers in Wikidata) constructed by Benkhedda et al. [10] . The subset of their dataset that was made publicly available consists of 100 documents, where each document corresponds to the concatenation of the title and description of a CGDC item. Half of the dataset (50 documents) came from the Morrab Library Photographic Archive comprised of thousands of digitised photographs that capture Cornish culture and history. 1 The other 50 documents were drawn from People's Collection Wales (PCW), 2 an online platform for gathering various media (e.g., photographs, documents, audio and video recordings) from individuals and community groups who wish to contribute items related to Welsh culture and history.We then designed a simple relation annotation scheme, where we defined a relation as consisting of a head and tail entity, which are linked according to a relation type. For each relation type, our scheme specifies the possible entity types for each of the head and tail entities. For example, for the notable_work relation type, there are three possible head-tail entity combinations: Person-Miscellaneous (applies when a person created a miscellaneous entity such as a work of art), Organisation-Miscellaneous (if the creator is an organisation) and Person-Location (applies when a person is known for building or designing a location such as a church). Appendix A provides all the valid head-tail entity type combinations for each CGDC relation type.The brat tool [12] was configured to enable the manual annotation of our relation types of interest, with the entity type constraints for each relation type specified. Two annotators (the second and last authors of this paper) independently labelled entity relations in all of the 100 documents. The inter-annotator agreement (IAA) between the two annotators was determined to be 0.48 in terms of Cohen's Kappa [13] , which is considered to be substantial agreement [14] , considering that there are 20 possible classes. The annotated set of 100 documents was then expanded by labelling the named entities and relations in 137 further documents drawn from the PCW collection. Similar to the first set of documents, two annotators (the third and last authors of this paper) independently annotated the entity relations in all of the 137 documents and a similar level of IAA (i,e., 0.49 in terms of Cohen's Kappa) was obtained. In our experiments for evaluating the performance of transformer-based RE models, we utilise all the 237 relation-labelled CGDC documents. 
methods: 
 The RE task can be defined as follows: given a sentence 𝑠 and a pair of entities 𝑒1 and 𝑒2 contained within that sentence, as well as a set of classes 𝐶, an RE model should identify one class 𝑐 ∈ 𝐶 that best describes the relationship between 𝑒1 and 𝑒2 based on 𝑠. In this work, we are concerned with 20 relation types that are relevant to CGDC. However, apart from these, we also include an additional class that we refer to as None Of The Above or NOTA, to which pairs of entities that are not related according to any of our 20 CGDC types belong. Thus, there are a total of 21 classes in 𝐶.A number of transformer-based models that were already fine-tuned for the NLI task were employed to extract relations between entities in a zero-shot manner. Each of these models is described below.DeBERTa (which stands for Decoding-Enhanced BERT with disentangled Attention) improves upon the encoder-only BERT model by employing a disentangled attention mechanism [15] . In this work, we employed version 3 of the DeBERTa model [2] that was fine-tuned on NLI datasets. 3 This particular model can be employed in a zero-shot classification manner, whereby the model classifies an input sequence according to a given set of classes (labels), also providing a probability for each class.BART is an encoder-decoder model that combines a BERT-like encoder and an auto-regressive decoder similar to GPT. It has demonstrated satisfactory performance on both text generation and comprehension tasks (such as text classification) [3] . We employed a version of the BART model that was fine-tuned for NLI. 4 Similar to our chosen DeBERTa model, this model can be employed in a zero-shot classification manner and provides probabilities together with predicted labels.T5 stands for Text-to-Text Transfer Transformer [4] , which casts many downstream NLP tasks (including NLI) as a sequence-to-sequence modelling problem, following an encoder-decoder architecture. An extra extra large (XXL) version of T5 that was fine-tuned for the NLI task was employed in our experiments [16] . 5 Unlike the DeBERTa and BART models, this model does not provide any probability values together with its predicted labels.Importantly, we investigated an ensemble model, henceforth referred to as Ensemble, that is a combination of the above three models. The prediction of this ensemble model was determined by taking the majority vote amongst the constituent transformer-based models.As mentioned above, we cast RE as an NLI task, whereby two sentences are provided to a model as input: a premise and a hypothesis. If based on the premise, the hypothesis is true, then the model should detect that an entailment relation holds between the two sentences. Otherwise, there is no entailment relation between the two sentences. To frame RE as an NLI task, an input sentence 𝑠 is considered to be the premise, while a hypothesis is automatically generated by populating a sentence template-that is predefined for a particular relation type-with input entities 𝑒1 and 𝑒2. The premise-hypothesis pair is then presented to an NLI model. If the model detects entailment, then we say that the relation type (corresponding to the template) holds between the input entities. In Table 1 , we provide examples of premise-hypothesis pairs, in which the hypothesis was generated based on a template. The complete set of templates is provided as part of our codebase. 6In preparation for applying the NLI models in a zero-shot manner, we firstly segment each document (in the evaluation dataset) into individual sentences. For each of these input sentences, all possible pairs of entities (contained within a sentence) are created. For every entity pair, a hypothesis is generated for each relation type. Each of these generated hypotheses is then paired up with the premise (the input sentence). Finally, our NLI models take every premise-hypothesis pair as an input sample that is then classified as being characterised by entailment or not. If entailment is not detected (with a probability of at least 0.40 in the case of the BART and DeBERTa models) for any of the relation types, we assign the NOTA label to the input sample. In cases where entailment is detected by the model for more than one class (relation type), we simply take the class with the highest probability value, as provided by the BART and DeBERTa models. As the T5 model does not output any probability values, we implemented post-processing rules that specify which relation types should take precedence in case of ties. 
evaluation and error analysis: 
 In this section, we report the results of applying our chosen NLI models to the RE task in the zero-shot manner described in the preceding section. As previously mentioned, entity pairs were exhaustively created based on our evaluation data (the 237 CGDC documents) to form the input samples. This resulted in 5938 entity pairs that the NLI models need to classify according to relation type (or the lack thereof, in which case the NOTA class applies). Out of those, 5435 (or 92%) belong to the NOTA class.In Table 2 , we report the performance obtained by each of our models in terms of the standard metrics of precision, recall and F1-score. To avoid skewing the results towards the over-represented NOTA class, we report the performance separately for the 20 CGDC relation types, and the NOTA class. Weighted macro-averaging was employed in reporting combined performance for the 20 CGDC relation types.With respect to the 20 CGDC relation types, the Ensemble model obtained the best weighted macroaveraged F1-score of 0.487. Considering the IAA of 0.48-0.49 that we obtained (see Section 2.2), the model comes close to upper-bound performance. Meanwhile, the models obtained F1-scores as high as 0.914 (for T5) on the NOTA class. This is particularly impressive considering that it is well-known that handling NOTA cases is a challenging task [17] .We manually examined some cases where the models made incorrect predictions. For instance, in the sentence "She was born on 10th March 1833 at Market Rasen, Lincolnshire, the eldest daughter of Henry Albert Browne and Frances Margaret Nicholson", no relation (NOTA) holds between the entities "Frances Margaret Nicholson" and "Henry Albert Browne", according to the gold standard annotations. Both T5 and BART predicted the spouse label for this entity pair (whereas DeBERTa successfully predicted NOTA). One can, however, argue that even the predictions by T5 and BART are not necessarily incorrect.The interpretability of such results are rather subjective and can be considered as either correct or incorrect based on people's opinion or the intended use of the models. Overall, it was observed that the models tended to detect or infer implicit relations that human annotators might otherwise miss. 
knowledge graph curation: 
 Taking the predictions of the Ensemble model over the 237 documents in our dataset, we populated a knowledge graph (KG) whereby vertices represent named entities and edges represent any relations detected between them. Here, only entities that were manually assigned Wikidata IDs (as part of gold standard entity linking annotations) were included, to ensure that entities in the resulting KG are normalised.To create the knowledge graph, we utilised the Neo4j framework7 , a graph database management tool. Using the Cypher query language, we were able to query the resulting knowledge graph. Figure 1 shows a visualisation of the results of an example query for a use case where the relationships of a particular person of interest (King Edward) with other entities have been retrieved.  
conclusions: 
 In this paper, we demonstrate how a zero-shot approach based on NLI can be employed to extract entity relations in CGDC textual metadata. Our findings show that, based on evaluation using a dataset of 
 
 Appendix: The relation types relevant to CGDC, their equivalent human-readable label, their Wikidata identifiers (linked to a page with their definitions and synonyms) and the types of head and tail entities involved. https://photoarchive.morrablibrary.org.uk/https://www.peoplescollection.wales/https://huggingface.co/cross-encoder/nli-deberta-v3-largehttps://huggingface.co/facebook/bart-large-mnlihttps://huggingface.co/google/t5_xxl_true_nli_mixtureOur code and annotations (for the 237 in CGDC documents in our dataset) are available at https://github.com/ OurHeritageOurStories/cgdc_re.https://neo4j.com/ 



Reviews: review: title: title: Review
main_review: main_review: The paper proposes a relation extraction approach based on zero shot learning. It uses masked language models like BART DeBERTA, T5. The paper targets very interesting problem of knowledge graph construction with an application towards digital humanities which is already under explored.

- It would be good to justify the choice of language models.
- Did authors try out large language models: https://aclanthology.org/2023.acl-long.868.pdf 
- The inter-annotator agreement is a bit low 0.48
title: title: Review paper Relation Extraction for Constructing Knowledge Graphs: Enhancing the Searchability of Community-Generated Digital Content (CGDC) Collections
main_review: main_review: This (short) paper presents a zero-shot approach for extracting relations from archives, as a basis for the construction of a knowledge graph (very shortly described in the paper). The aim is to make such archives more searchable and queryable. 20 types of relations from Wikidata have been considered in experiments which have been run on three different transformer-based methods (DeBERTa, BART and T5) along with their combination. 

The literature on knowledge extraction in cultural heritage is huge and the authors claim that the use of a zero-shot approach has been fully exploited in the domain. The proposed approach is however quite classical (template-based). The results are not concluding and the authors do not go further on their discussion and future developments. Some details are also missing: how the relevantance of the 20 types of the CGDC has been established? how to improve inter-annotator agreement? how the sentence templates has been constructed?
title: title: The authors propose a reasonable approach to use existing techniques for creating KGs from unstructured data within the domain of CGDC Collections.
main_review: main_review: The authors describe a method to construct KGs via Relation Extraction (RE). Specifically, they propose performing RE as a Natural Language Inference (NLI) Task. For every relation type, the authors construct a template which they use to construct Hypotheses for the NLI task. The description of CGDC elements are used as Premises. By performing the NLI task with this setup, the correct relation type for the entities in the CGDC descriptions are extracted. This setup has the advantage that it can be performed with fine-tuned NLI models, without requiring extra training.
The presented approach is a combination of established techniques, but still provide a sufficient amount of novelty. The presented techniques show potential for creating KGs from unstructured sources in general. The approach is clearly described and the presentation makes it easy to follow the line of arguments.
Cons:
-	Limited evaluation (no comparison to different approaches or datasets)
Pros:
-	Showcasing the application of established techniques on new domains
-	Clear presentation