title: D4.7: Evaluation of plurilingual corpora and knowledge extraction methods (V1.0)
title: Software report DRAFT
titleType: Subtitle
Creators: name: Tripodi, Rocco
name: Graciotti, Arianna
name: Lazzari, Nicolas
Publisher: name: Zenodo
Year: 2024

description: Deliverable D4.7, "Evaluation of Plurilingual Corpora and Knowledge Exaction Methods", provides a final evaluation of Knowledge Extraction (KE) methods and tools developed within Work Package 4 (WP4).Extracting knowledge from texts, particularly in historical contexts, remains a complex endeavour. The Polifonia project’s focus on Musical Heritage (MH) preservation adds unique challenges to analyzing textual documents. In fact, the Polifonia Textual Corpus1 [1] comprises historical documents, which cannot fully benefit from the most recent advancements in Natural Language Processing (NLP). Transformer architectures [2] like BERT [3] or Large Language Models (LLM) like ChatGPT [4] have been primarily trained and evaluated on contemporary and clean documents. As a consequence, peculiar linguistic variations related to historical documents make it problematic for pre-trained language models to encode words or recognize syntactic structures as effectively as for contemporary documents. Also, when dealing with NLP models, it is important to consider not only linguistic variations but also the impact of Optical Character Recognition (OCR) errors. These errors can lead to unintended behaviour in LLMs, which are typically trained on clean text.Moreover, a challenge in reusing current NLP models on historical documents lies in the different distribution of named entities between historical and contemporary texts. In fact, advanced NLP techniques for Entity Linking (EL), namely the process of connecting mentions in a text to their corresponding entries in a knowledge base like Wikipedia, have limitations. These models may produce inconsistent associations due to the fact that their training sets contain mostly contemporary documents [5] or due to the lack of representation of minor historical entities in the reference knowledge base. Consequently, a named entity mention of an XVIII-century person occurring in a document published in XVIII century might be linked to a XX-century character, leading to erroneous results.Furthermore, the performance of current NLP models on open-domain Question Answering (QA) tasks is not satisfactory when the questions are centred around named entities that are less frequent in the models’ training sets, such as the ones occurring in historical texts. LLMs, including the largely popular conversational agent chatGPT [6], demonstrate a stronger tendency to hallucinate or to reply incorrectly when prompted with factual questions eliciting long-tail knowledge on less popular entities. When dealing with such types of questions, augmenting the prompt sent as input to LLMs with retrieved evidence appears to improve performance [7].Given these issues, our deliverable focuses on evaluating the enhancements that we brought to our KE pipeline (Polifonia Knowledge Extractor2) to make it more suitable for accurately processing historical texts:• Release of an advanced historical named entities recognition, classification and linking benchmark in English language;• Introduction and evaluation of a novel EL model;• Introduction of an adaptation of a widely utilised neural EL model that filters out implausible candidates based on time and type information from Wikidata;• Introduction of a factual QA benchmark focused on historical, long-tail named entities;• Presentation of a methodology for evaluating and comparing KE models along with the discussion of the results of its application.The evaluation methods and tools designed and tested in this deliverable contribute to the general goal of WP4: making musical heritage knowledge available through reliable knowledge graphs that can be easily queried.