# A systematic review of relation extraction task since the emergence of Transformers

Celian Ringwald, Fabien Gandon, Catherine Faron, Franck Michel, Hanna Abi Akl

► To cite this version:

Celian Ringwald, Fabien Gandon, Catherine Faron, Franck Michel, Hanna Abi Akl. A systematic review of relation extraction task since the emergence of Transformers. 2025. hal-05349792v2

HAL Id: hal-05349792

https://hal.science/hal-05349792v2

Preprint submitted on 21 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A systematic review of relation extraction task since the emergence of Transformers

CÉLIAN RINGWALD*, Université Côte d'Azur, Inria, CNRS, I3S, France

FABIEN GANDON, Université Côte d'Azur, Inria, CNRS, I3S, France

CATHERINE FARON, Université Côte d'Azur, Inria, CNRS, I3S, France

FRANCK MICHEL, Université Côte d'Azur, Inria, CNRS, I3S, France

HANNA ABI AKL, Data ScienceTech Institute, Inria, France

This article presents a systematic review of relation extraction (RE) research since the advent of Transformer-based models. Using an automated framework to collect and annotate publications, we analyze 34 surveys, 64 datasets, and 104 models published between 2019 and 2024. The review highlights methodological advances, benchmark resources, and the integration of semantic web technologies. By consolidating results across multiple dimensions, the study identifies current trends, limitations, and open challenges, offering researchers and practitioners a comprehensive reference for understanding the evolution and future directions of RE.

The resulting annotated 📚 Zotero bibliography has been made openly available and reusable, together with the developed software 🌐 SciLEx, which provides a unified framework for collecting and analysing papers in the context of systematic literature reviews.

CCS Concepts: • Information systems → Content analysis and feature selection: Resource Description Framework (RDF); • Computing methodologies → Information extraction: Natural language processing.

Additional Key Words and Phrases: Relation Extraction, Semantic Web

**ACM Reference Format:**

Célian Ringwald, Fabien Gandon, Catherine Faron, Franck Michel, and Hanna Abi Akl. 2025. A systematic review of relation extraction task since the emergence of Transformers. 1, 1 (November 2025), 37 pages. https://doi.org/10.1145/nnnnnn.nnnnnnn

## 1 INTRODUCTION

The increasing digitalisation of daily life has created vast volumes of textual data that organisations and individuals must manage. Transforming this unstructured information into structured knowledge is a key challenge for natural language processing (NLP) and information systems. Relation Extraction (RE) — the task of identifying semantic relations between entities in a text — addresses this challenge by producing triples of the form (*subject*, *relation*, *object*). This structured representation enables more effective indexing, search, and retrieval, and provides the foundation for populating and augmenting knowledge bases (KBs) in general and knowledge graphs (KGs) in particular.

Knowledge graphs usage were popularised by Google as well as open-resources like Freebase [14], YAGO [108], DBpedia [9], and Wikidata [120]. They have become critical infrastructures in both academia and industry. They support applications ranging from search engines to recommendation systems and scientific discovery. However, maintaining

Authors' addresses: Célian Ringwald, celian.ringwald@inria.fr, Université Côte d'Azur, Inria, CNRS, I3S, Sophia-Antipolis, France; Fabien Gandon, fabien.gandon@inria.fr, Université Côte d'Azur, Inria, CNRS, I3S, Sophia-Antipolis, France; Catherine Faron, catherine.faron@inria.fr, Université Côte d'Azur, Inria, CNRS, I3S, Sophia-Antipolis, France; Franck Michel, Université Côte d'Azur, Inria, CNRS, I3S, Sophia-Antipolis, France; Hanna Abi Akl, Data ScienceTech Institute, Inria, Sophia-Antipolis, France.
and expanding these resources remains labour-intensive. Automatic relation extraction offers a promising mechanism for enriching KGs at scale, making RE both a foundational research task and a practical necessity.

*Research Context and Objectives:* Research on RE spans more than three decades. A key milestone was the *Fourth Message Understanding Conference* (MUC-4) in 1992, which introduced standard evaluation metrics (*precision* and *recall*) and the first RE system, *Fastus* [48]. Since then, RE has evolved from rule-based systems, to statistical learning, to neural architectures. The advent of Transformer-based pre-trained language models [17, 28] has marked a decisive turning point, delivering significant performance gains across a broad spectrum of NLP tasks [123] and revitalising research in knowledge acquisition and information extraction [86, 93, 96, 157].

*Positioning with Respect to Previous Surveys:* Numerous surveys on RE have appeared in the past decade, but only a small subset adopt a rigorous systematic review methodology. Among them a great overview of the field was given by Martinez-Rodriguez et al. [76], but it covers pre-2020 research and focuses on pipelines and rule-based methods. Subsequent works addressed narrower aspects, such as grammar-checking, where RE is a subtask [79], temporal relation extraction [145], or dataset creation and annotation practices [12]. More recent studies examine knowledge graph construction in general [66, 103], but devote comparatively less attention to RE itself. These contributions remain valuable, but they are limited either in scope or methodology, leaving the field of RE without a comprehensive, systematic, and up-to-date synthesis.

*Intended Audience:* This survey aims to give a concrete picture of the impact of Transformers on the Relation Extraction task, from researchers, students, but also any private researchers. It provides a reference for researchers and practitioners at the intersection of NLP and knowledge graphs, looking for a concrete understanding of the main research directions proposed to solve this task since the language models ramp-up. This survey also provides a first analysis that could be easily extended in specific directions since it is based on a public corpus composed of two levels: the complete set of papers collected, and a refined, annotated version thereof.

*Survey purpose and Contributions:* This article presents an extended, systematic literature review (SLR) of RE from text, focusing on the period 2019–2024, which corresponds to the rapid rise of Transformer-based models [118], and combining methodological rigour with comprehensive coverage. The main contributions of this review are: (1) An updated and detailed overview of RE research, highlighting the transformative impact of large pre-trained language models; (2) An application of the *SciLEx* framework, an extended SLR methodology ensuring transparency, reproducibility, and open availability of results; (3) A fine-grained annotation along more than thirty analytical dimensions across resources, models, and surveys, released openly to support future investigations.

# 2 BACKGROUND

## 2.1 Knowledge Representation

A knowledge graph (KG) is a graph linking entities (the nodes of the graph) with different types of binary predicates (the edges of the graph), also called relations or properties. Each edge of the graph links a subject entity to an object entity or literal value using a specific predicate. A graph can be viewed as a set of such triples (subject, predicate, object) that form the fundamental units of a knowledge.

Entities represent real-world objects or abstract concepts. Examples include people, locations, organisations, events, or ideas. When an entity can be uniquely identified, it is typically referred to as a *named entity*. Detecting entities from text generally relates to *Named Entity Recognition* (Section 2.2.5). Attaching a unique identifier to an entity relates to
the Entity Linking task (Section 2.2.2). Identifiers of entities are often IRIs (Internationalised Resource Identifiers)—an extension of URIs (Uniform Resource Identifiers) that supports Unicode.

A label is a human-readable string that serves as the descriptive name of an entity—e.g., “Paris”. Unlike identifiers, labels are not unique and may refer to multiple entities. Trying to identify labels of entities in a text is a task known as Mention Detection (Section 2.2.1).

A relation connecting two entities is called an *object property*; a relation connecting an entity to a literal value (such as a date, string, or number), is called a *datatype property*. Identifying which relations are expressed by two entities in a text segment is generally part of the Relation Identification task (Section 2.2.6); attaching a unique identifier to a relation refers to the relation classification task (Section 2.2.7). Extracting triples from text is the special focus of our surveys and is generally referred as the Relation extraction task (Section 2.2).

Entities may also be associated with types, called classes or concepts. For instance Paris belongs to the class City or is of type City. Classes and properties are typically defined in an ontology, a schema, or more generally a vocabulary. Attaching a type to an entity is part of the NER task (Section 2.2.5), and is especially referred as the Entity Typing task (Section 2.2.4).

## 2.2 Relation Extraction from Text

Let $D_b \subseteq W \times G$ be a dual dataset, where W denotes a set of textual documents and G represents the corresponding set of knowledge graphs. The relation extraction (RE) task can be formalized as learning an extraction function $E_{D_b}: W \to G; w \mapsto g$, where w is an input text, and g is the graph representing the relations and entities implied by w. It can be decomposed into the following subtasks:

2.2.1 **Mention Detection:** This subtask consists in identifying specific spans of text within a document. Mention detection is a common task in natural language processing (NLP). In the context of RE, it includes recognising entity labels or extracting particular datatype properties. **Example:** detecting the mentions "Victor Hugo" and "Besançon" in the text "Victor Hugo is born in Besançon".

2.2.2 **Entity Linking:** This subtask consists in assigning a unique identifier from a knowledge base (such as DBpedia or Wikidata) to a detected text span. **Example:** "Paris is beautiful in spring." contains the mention "Paris" for which several entities are candidate ($e_1$ = Paris, France, $e_2$ = Paris, Texas, $e_3$ = Paris Hilton). The absence of "Hilton" in neighbouring tokens increases the likelihood of $e_1$ leading to the link EL("Paris") = http://dbpedia.org/resource/Paris.

2.2.3 **Coreference Resolution:** This subtask consists in identifying all expressions within a document that refer to the same entity and determining whether different mentions (such as names, pronouns, or noun phrases) are co-referential. **Example:** "Marie Curie was a brilliant scientist. She discovered radium with her husband Pierre." contains the mentions $m_1$ = "Marie Curie", $m_2$ = "She", and $m_3$ = "her", leading to a coreference cluster {$m_1, m_2, m_3$} referring to Marie Curie.

2.2.4 **Entity Typing:** This subtask consists in assigning a semantic type to an entity. This type may be a class defined in an ontology or be drawn from a limited set of classes, e.g., {Person, Place, Organisation}.

2.2.5 **Named Entity Recognition (NER):** In the literature, NER often refers to mention detection combined with basic entity typing. However, some definitions also encompass entity linking and coreference resolution.
2.2.6 **Relation Identification:** Given a document and a tuple of previously identified entities, this subtask consists in determining whether a relation exists between the entities within the text. *Example:* "Henri Matisse lived in Nice for many years." contains the mentions $m_1$ = "Henri Matisse", $m_2$ = "Nice" and a relation identification output would include $RI(m_1, m_2)$ = livedIn.

2.2.7 Relation Classification: This is a more advanced task than relation identification, where the specific type of relation expressed is identified, typically by assigning an identifier from a relation ontology or vocabulary. Following the previous example, a relation classification output could be livedIn ≡ (dbo:residence, wdt:P551, schema:homeLocation), with relation identifiers from three well-known ontologies.

Other associated RE tasks include:

2.2.8 **Slot Filling:** Given a template with a missing value (a slot) and an input text, this task consists in extracting the correct value to complete the template. *Example:* Victor_Hugo - bornIn - ? applied to the text "Barack Obama was born in Honolulu, Hawaii." leads to bornIn(Barack Obama, Honolulu).

2.2.9 Probing: Popularised by the use of language models, this task involves completing an incomplete triple provided as input, often without specific context. Probing serves as a diagnostic task in NLP to evaluate what linguistic or factual knowledge is encoded in pretrained language models (e.g., BERT, GPT) by testing their performance on simple prediction tasks. *Example:* What country is "Paris" in? in the text "Paris is the capital of France." should lead to the output France.

2.2.10 Question Answering (QA): Closely related to knowledge graph research, this task aims to produce an answer from a given question in natural language. This may involve generating queries, in SPARQL in the case of RDF knowledge graphs, which are subsequently executed against a knowledge graph to retrieve the answer.

Relation Extraction (RE) is therefore the task of identifying relationships between entities in a text, typically represented as triples of the form ($e_1, r, e_2$). In a **closed RE** setting, the task involves retrieving relations from a predefined set $\mathcal{R}$, and optionally restricting entities to those referenced in a knowledge graph (KG) from a defined set $\mathcal{E}$. This approach is well-suited for completing KGs but requires an ontology or, at least, a predefined set of relation types. It is typically implemented using supervised learning, which are unable to extract relations outside the fixed set $\mathcal{R}$ (and optionally $\mathcal{E}$). Conversely, in an **open RE** setting, the model is expected to produce triples using textual predicates, generally expressed as verbs or verb phrases extracted directly from the text: OpenRE(s) → {($e_1, r_{text}, e_2$)}. This design is well-suited for bootstrapping and generating new knowledge from large corpora, and it is particularly useful in low-resource scenarios. However, the resulting data tends to be noisier and more redundant.

3 SCILEX METHODOLOGY

The SciLEx<fnref n="1" /> (Science Literature Exploration) review framework is a Python toolbox that implements and extends the methodology defined by [16, 60]. Starting from a user-defined keyword list, SciLEx automates the construction of a collection of relevant papers by generating and executing all possible combinations of queries derived from this keyword list across multiple digital libraries. This automation facilitates the paper collection process, ensures traceability, and

<footnote n="1">¹https://github.com/Wimmics/SciLEx</footnote>
supports the aggregation and deduplication of search results. Additionally, SciLEx enriches the resulting corpus through
the integration with external services such as PaperWithCode<fnref n="2" />, CrossRef [47], and Opencitation [91]. PaperWithCode,
was intended for the Machine Learning community and aimed at connecting research articles to their corresponding
methods, implemented code, evaluation results on standard datasets, and initial paper annotations. OpenCitation
enables the retrieval of citations and references for a given paper, which can be used both to filter papers by impact and
to expand the corpus through citation snowballing. Finally, SciLEx exports all gathered information into a Zotero [78]
collection[自], facilitating collaborative management, selection, and annotation of the corpus.

We began our work with an initial "tertiary study" [59] (a review of reviews), followed by the systematic review of
the models and datasets proposed to solve a RE task.

## 3.1 Digital Libraries Search Plan and Scope and Automated Metadata Enrichment

We used SciLEx to collect articles on RE from nine digital libraries: HAL, Istex, arXiv, DBLP, Semantic Scholar, OpenAlex, Scopus, IEEE Xplore, and Springer Nature. Our search strategy relies on two keyword sets: $K_{wRE} = \{\text{"Relation extraction"}, \text{"Relation Classification"}, \text{"Triplet Extraction"}, \text{"Slot Filling"}, \text{"KG-to-Text"}, \text{"Text-to-data extraction"}\} \text{ and } K_{wSurvey} = \{\text{"Survey"}, \text{"State-of-the-art"}, \text{"Review"}, \text{"Study"}\}$

For the tertiary study, we generated all possible combinations of RE keywords ($K_{wRE}$) and survey-related terms ($K_{wSurvey}$). In the subsequent phase, the RE keywords alone ($K_{wRE}$) were used to retrieve articles related to models and datasets without further distinction. This comprehensive search strategy resulted in a total of $(6_{K_{wRE}} \times 6_{Years} \times 9_{APIs}) + (6_{K_{wRE}} \times 4_{K_{wSurvey}} \times 6_{Years} \times 9_{APIs}) = 2160$ queries executed via SciLEx. We deduplicated the retrieved articles by title and imported them into a Zotero library for further management. Ultimately, we collected more than 3,800 papers, including 3,567 proposed models, 250 datasets, and 77 survey articles.

We enriched the resulting corpus with metadata from PaperWithCode<fnref n="3" /> and LinkedPaperWithCode<fnref n="4" />, restricting inclusion to papers specifically related to the following RE sub-tasks: $K_{wPC} = \{\text{"Relation extraction"}, \text{"Relation Classification"}, \text{"Zero-shot Relation Triplet Extraction"}, \text{"Zero-shot Relation Triplet Classification"}, \text{"Zero-shot Slot Filling"}, \text{"Data-To-Text"}, \text{"Text-To-Data"}\}$. From these sources, we obtained code repositories, benchmark results, and tags, which formed the initial basis for developing our annotation guidelines. Subsequently, we retrieved the DOIs for all previously collected papers via CrossRef, and then gathered the citation network for each article using OpenCitations.

## 3.2 Selection Criteria

We filtered the enriched corpus using inclusion criteria based on article metadata (see Tab. 1). We reduced the corpus to a consistent set of 1,476 models, 75 datasets and 48 surveys.

Because the corpus was still large, we applied additional impact-based selection criteria to retain only articles with
significant influence in the scientific community:

* IC1. Ranking of model Papers:

- IC1.1 Journal publication: Impact factor greater than 10⁵

- IC1.2 Conference publication: ranked A* or A<fnref n="6" />

* IC2. Citation of model papers: at least 1 citation per year since its publication

<footnote n="2">²available until may 2025 https://paperswithcode.com now redirects to Hugging Face</footnote>

<footnote n="3">³https://paperswithcode.com</footnote>

<footnote n="4">⁴https://linkedpaperswithcode.com</footnote>

<footnote n="5">⁵We consolidated SciMago bibliometric data in our github</footnote>

<footnote n="6">⁶According to CORE 2023</footnote>
<figure><img src="image_2.png" /><figcaption>Table 1. Detailed inclusion criteria</figcaption></figure>

<table><thead><tr><th>Criteria</th><th>Exclusion criteria (EC)</th><th>Inclusion criteria (IC)</th><th>Exceptions (Ex)</th></tr></thead><tbody><tr><td>C1 - Publication period</td><td>Published between 2019 and 2024</td><td>Not published between 2019 and 2024</td><td>Inclusion: the paper is published before 2019 but is still relevant (often the case of dataset papers)</td></tr><tr><td>C2 - Coherence</td><td>Does not contain any survey term or task related term<br>neither in the abstract nor in the title</td><td>Contains in the abstract OR in the title at least one of the survey terms AND at least one of the task-related terms</td><td>Inclusion: the paper does not contain any identified terms but is related to our RQs ⇒ identification of new relevant terms</td></tr><tr><td>C3 - Language</td><td>in another language than English</td><td>the paper is written in English</td><td>/</td></tr><tr><td>C4 - Access</td><td>The access of the paper is not free</td><td>the paper could be read without any paywall</td><td>Inclusion: somewhere accessible (partially open)</td></tr><tr><td>C5 - Publication process</td><td>the paper was not peer-reviewed</td><td>the paper was peer-reviewed</td><td>Inclusion: the paper was not peer-reviewed but very relevant<br>Exclusion: the paper was peer-reviewed but not yet accessible</td></tr><tr><td>C6 - Referenceability</td><td>the paper is not identified by a DOI</td><td>the paper has an associated DOI</td><td>Inclusion: the paper does not have a DOI but is very relevant</td></tr><tr><td>C7 - Duplicates</td><td>It exists a more recent version of this paper</td><td>The paper is the last version published</td><td>if a paper is presenting both a model and a dataset it will be annotated twice</td></tr></tbody></table>

* IC3. Usage of dataset papers: used at least 2 times by one of the model papers studied.

* IC4. A dataset paper extending a dataset paper meeting criterion IC3 is accepted.

* IC5. Survey papers impact: at least 3 citations.

## 3.3 Annotation of the Collected Papers

After selecting our corpus, we refined and extended the PaperWithCode tags to better address our research questions. The research team refined this process over four rounds, during which the annotation guidelines were discussed and iteratively improved. A first round of annotation was conducted by one member of the team. This initial round provided a preliminary vocabulary of values for each dimension and served as a basis for discussions with the whole research team, whose members were later asked to cross-annotate a subset of each subcorpus. To evaluate the quality of our method and the consistency of the annotation dimensions proposed, we conducted a cross-annotation study on 10% of Manuscript submitted to ACM
the articles of each subcorpus: we randomly selected a subset of 4 datasets, 4 surveys and 10 models among the articles already pre-annotated. This cross annotation validation implied 5 annotators, who each annotated 3 datasets, 2 surveys, and 6 models, in order to get 3 annotators per paper. We organized a three-hour meeting to launch the annotation process, where a short tutorial on the annotation platform and grid was presented to the annotators who could discuss during the process and were provided with Annotation Guidelines<fnref n="7" /> including an annotation grid, a vocabulary of annotation dimensions, borderline examples and guidelines for solving ambiguous cases.

The annotation was prepared on a Zotero library, which enabled collaborative annotation and auto-completion. Each annotator was asked to spend at most 20 minutes per paper. They could add labels, update information, or delete labels as needed. Each expected annotation was indicated by a specific tag. All annotations were filled in as tags. They can be checked on the Zotero web interface by selecting a publication and opening the *Tags* tab.

We considered the agreement of the annotation dimensions on the subset level (dataset/models/surveys). To our knowledge, no agreement metric considers the multi-annotators multi-possible answers scenario. The case of multiple possible answers was already studied on a one-by-one basis among inter-annotators. We propose in this work to consider each possible tag set given by the annotators and by annotation dimension as a possible answer to compute the Fleiss Kappa agreement on every annotation dimension. The Fleiss Kappa score is close to 1 when annotators fully agree, and a value close to -1 indicates a strong disagreement. The detailed results<fnref n="8" /> show that annotators are closer to agreement on the data related to the dataset, followed by survey data and finally the model one. We can note a perfect agreement on the type of survey (Survey methodology), and substantial agreement on some other dimensions among which the Pretrained model used (PTM) and the Number of types of relation represented by datasets or targeted by the models. Other dimensions benefit from a moderate agreement, like the Use of negative examples, the Granularity, the Task and the Input type. Overall, as shown in Fig. 2, our annotations have a good level of agreement.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Number of annotation dimensions by Fleiss Kappa agreement level</figcaption></figure>

A resolution stage allowed us to manually align and aggregate conflicts from the annotations. Conflicts were discussed in regular meetings until full agreement was reached, and the guidelines were adjusted to clarify how to handle complex or ambiguous cases.

<footnote n="7">⁷https://github.com/datalogism/SciLex-RE_SystLitReview/blob/main/doc/AnnotationGuidelinesV6.pdf</footnote>

<footnote n="8">⁸Fleiss Kappa results available on github</footnote>
The guidelines were revised in the light of all the issues reported during the cross-annotation study. On this basis a unique expert annotator was asked to complete the annotation on the full corpus. This step required significant effort, as annotating a paper took about 10 minutes on average, totaling more than 37 hours of work.
<table><caption>Table 2. Detailed Annotation Grid</caption><thead><tr><th>Dimension</th><th>Desc.</th><th>Datatype</th><th>Surveys</th><th>Datasets</th><th>Models</th></tr></thead><tbody><tr><td>Benchmark type</td><td>Qualify the type of survey: qualitative or quantitative</td><td>Quali</td><td>✓</td><td></td><td></td></tr><tr><td>Survey Methodo</td><td>Is the survey following a systematic review process?</td><td>Bin</td><td>✓</td><td></td><td></td></tr><tr><td>SemanticWebTechno</td><td>Is the survey introducing the semantic web technologies in the context of RE?</td><td>Bin</td><td>✓</td><td></td><td></td></tr><tr><td>Lang</td><td>Which language is targeting the paper?</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Source</td><td>From which source is derived the paper?</td><td>Quali</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Domain</td><td>Is the article focused on a limited domain? (Finance, news, encyclopedic...)</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>ManualAnnotation</td><td>Did the conducted research involve manual annotation?</td><td>Bin</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>UseNER</td><td>Is the proposed process integrating tools to solve NER?</td><td>Quali</td><td>✓</td><td></td><td></td></tr><tr><td>SynthGeneration_Bin</td><td>Is the paper using/creating synthetic data?</td><td>Bin</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NbTypeRel</td><td>How many types of relation are covered?</td><td>Quanti</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NbEntity</td><td>How many entities are covered?</td><td>Quanti</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NbTypeEntity</td><td>How many classes/types of entities are covered?</td><td>Quanti</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NbTriples</td><td>How many triples in total are considered in the paper?</td><td>Quanti</td><td>✓</td><td></td><td></td></tr><tr><td>NbDoc</td><td>How many documents are considered in the corpus? (if granularity=doc)</td><td>Quanti</td><td>✓</td><td></td><td></td></tr><tr><td>NbSent</td><td>How many sentences are considered in the corpus? (if granularity=sent)</td><td>Quanti</td><td>✓</td><td></td><td></td></tr><tr><td>Granularity</td><td>Are the discussed, proposed or used resources given at the document level or paragraph/sentence level?</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Task</td><td>Which task is considered by the resource?</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Dataset</td><td>Which datasets are used in the paper?</td><td>Quali</td><td>✓</td><td></td><td></td></tr><tr><td>DatatypeProp</td><td>Which datatype are explicitly used in the object of triples? (String/Values/Dates...)</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>DatasetSplit</td><td>How was the dataset splitted? (Randomly, TimeAware...)</td><td>Quali</td><td>✓</td><td></td><td>✓</td></tr><tr><td>Input</td><td>Which content is taken in the proposed model as input? (Text/Graph/Images...)</td><td>Quali</td><td>✓</td><td></td><td>✓</td></tr><tr><td>ObjectProperties_Bin</td><td>Are the objects of triples URIs?</td><td>Bin</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>UseNegativeExample_Bin</td><td>Are the resources containing or using negative examples?</td><td>Bin</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>NbDataset</td><td>Number of datasets considered in the paper</td><td>Quanti</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>LinearizedGraph_Bin</td><td>Is the model explicitly defining a linearisation to represent graph as text?</td><td>Bin</td><td>✓</td><td></td><td>✓</td></tr><tr><td>NbModel</td><td>Number of models considered in the paper</td><td>Quanti</td><td>✓</td><td></td><td>✓</td></tr><tr><td>Archi</td><td>Which architectural layer is considered in the proposed work?</td><td>Quali</td><td>✓</td><td></td><td>✓</td></tr><tr><td>PTM</td><td>(Convolutional Network, MultiLayer Perceptron, Transformer...)</td><td>Quali</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>LearningMethod</td><td>Which pretrained model is used/described in the paper?</td><td>Quali</td><td>✓</td><td></td><td>✓</td></tr><tr><td>DecodingMethod_Bin</td><td>Which method is used to train the proposed model?</td><td>Quali</td><td>✓</td><td></td><td>✓</td></tr><tr><td>CostEval_Bin</td><td>Is the model explicitly updating the method used to generate the output?</td><td>Bin</td><td>✓</td><td></td><td>✓</td></tr><tr><td>LossUpdate_Bin</td><td>Is the model evaluating the cost of the training process?</td><td>Bin</td><td>✓</td><td></td><td>✓</td></tr><tr><td></td><td>Is the proposed model using a custom Loss function during the training?</td><td>Bin</td><td>✓</td><td></td><td>✓</td></tr></tbody></table>
In the end, we selected 32 annotation dimensions to describe our collected corpus of surveys, datasets, and models, among which 5 of them are derived from PaperWithCode. They are presented in Table 2. They may take three types of values: boolean (True/False), quantitative (expressed in powers of 10, e.g., $10^0 = [1 - 10]$, $10^1 = [10 - 100]$), and qualitative (open-ended values identified during the first annotation round). These dimensions are detailed in the Annotation Guideline.

# 4 DATA ANALYSIS OF THE COLLECTED PAPERS

We collected 3,894 papers, gathering 77 surveys, 250 datasets papers and 3,567 model papers. We first annotated them with all the available basic bibliographic data. Applying selection criteria C1–C7 then reduced the corpus to 1,500 papers. Although this subset still included many model papers, their number was drastically reduced by applying the Impact Criteria. Annotation and manual verification ultimately narrowed the scope to 202 papers, among which 34 surveys, 64 datasets, and 104 models.

<figure><img src="image_4.png" /><figcaption>Fig. 4. Publication types of surveys</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 5. Referencing of surveys</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 6. Accessibility of surveys</figcaption></figure>

<figure><img src="image_7.png" /></figure>

*Surveys*: the 67 collected surveys were published between 2010 and 2024. Fig. 3 shows that publications increased significantly after 2020, reaching 10 papers per year and the double in 2022. The subset analyzed in this survey is more evenly distributed between 2018 and 2024 and follows the same trend. As shown in Fig. 4, most collected surveys were published in journals, with about half as many appearing in conferences. Preprints formed a minor portion of the collected surveys and were excluded by our selection criteria. The selected surveys came from a diverse set of journals.
ACM Computing Surveys is well represented as the journal is specialised in publishing systematic reviews. There are also journals specialised in Database (Data Technologies and Applications, Foundations and Trends in Databases), Semantic Web (Semantic Web Journal, Transactions on Graph Data and Knowledge) or general AI (IEEE Transaction on Neural Networks, Learning Systems, IEEE Access). The main venues publishing surveys are computational linguistics conferences such as the Annual Meeting of the ACL and the conference on Empirical Methods in NLP (EMNLP). Fig. 5 shows that surveys are well-referenced, allowing the retrieval of a large part of the citation network associated to them. Please note that our selection process only kept surveys that were identified by a DOI. Finally, Fig. 6 shows that about half of the articles were either partially open or closed access.

<figure><img src="image_8.png" /><figcaption>Fig. 8. Publication types of dataset papers</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 9. Referencing of dataset papers</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 10. Accessibility of dataset papers</figcaption></figure>

<figure><img src="image_11.png" /></figure>

*Datasets:* as shown in Fig. 7, the 250 collected dataset papers span from 1990 to 2024. This wide temporal range reflects the enduring relevance of some benchmark datasets, which are still reused in later research. Dataset papers spiked around 2020, mirroring the increase in survey papers. In contrast to surveys, dataset papers are predominantly introduced at conferences, especially through dedicated resource tracks (Fig. 8). The 64 dataset papers analysed in this systematic review are representative of both publication patterns and dataset characteristics. Several dataset papers are released outside traditional venues, often in workshops or working groups on specific research challenges. As shown in Fig. 9 most dataset papers have DOIs and many are indexed in OpenCitations. Approximately half of the dataset papers are listed on PaperWithCode, often linking to associated GitHub repositories. However, as shown in Fig. 10, dataset
access is inconsistent and may be restricted, similarly to surveys. Analysing the publication venues for the 64 dataset papers reviewed, we observe a substantial prevalence of computational linguistics conferences such as EMNLP and ACL. Additionally, workshops and shared tasks—most notably SemEval—play a significant role in dataset dissemination. When published in journals, datasets are usually related to biosciences and appear in journals venues such as Database (Oxford), Journal of Biomedical Informatics, or Briefings in Bioinformatics.

*Models:* as shown in Fig. 11, the 3,567 model papers collected were published between 2009 and 2024. A clear increase in RE publications is observed: about 200 papers were published between 2014–2017, rising to 400 in 2019, and peaking at over 600 in 2021. Fig. 12 shows that models are rarely published in journals; most appear in conferences such as ACL, EMNLP, IJCAI, or AAAI. Regarding their indexing and referencing (see Fig. 13), not all models are associated with a DOI, and citation network data could only be retrieved for approximately half of the papers. According to PaperWithCode metadata, all collected models are listed, but only a minority provide access to source code or benchmark datasets. In terms of accessibility, most articles are openly available, as shown in Fig. 14.

<figure><img src="image_12.png" /><figcaption>Fig. 12. Publication types of model papers</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 13. Referencing of model papers</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 14. Accessibility of model papers</figcaption></figure>

<figure><img src="image_15.png" /></figure>

# 5 ANALYSIS OF SURVEYS ON RELATION EXTRACTION

Our analysis relies on the curated collection of 34 surveys on the relation extraction (RE) task published between 2019 and 2023. This represents 44% of the corpus of surveys initially collected. Surveys on RE vary considerably in scope, methodology and depth. Out of the 34 surveys in our curated corpus, 12 are purely qualitative. These are typically

Manuscript submitted to ACM
concise papers, often published on arXiv and covering a limited number of datasets (less than ten) while discussing between 10 and 100 models. Their primary purpose is to provide broad overviews of research activity rather than detailed methodological or benchmarking analyses. By contrast, thirteen surveys attempt to combine performance results with descriptive analysis of models, while five focus exclusively on benchmarking. From a methodological perspective, only seven surveys (20% [自]) apply principles consistent with a Systematic Literature Review (SLR), representing a minority of the corpus. These are directly compared with our work in Table 3. A few studies on RE have been published after our corpus was finalized, among which [156] that converges on some aspects with our survey, but is more limited in terms of paper annotation and does not rely on a detailed methodology.

<table><caption>Table 3. Comparison of our work with previous systematic literature reviews. To gain in readability, we refer to tasks using abbreviations: NER, for Named Entity Recognition, CR: Coreference Resolution, ET: Entity Typing, EL: Entity Linking, RC: Relation Classification. Also, Sent and Doc refer to sentence and of document granularity, respectively. "?" refers to the case where the information is not available or clearly stated in the survey. Ø indicates that the survey is not concerned by the analysis dimension.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Survey</th><th rowspan="2">Description</th><th colspan="6">Task</th><th rowspan="2">Architecture</th><th rowspan="2">PTM</th><th rowspan="2">Language</th><th rowspan="2">Granularity</th><th rowspan="2">Domain</th><th rowspan="2">Nb Models</th><th rowspan="2">Nb Datasets</th><th rowspan="2">Sem. Web</th></tr><tr><th>NER</th><th>CR</th><th>ET</th><th>EL</th><th>RC</th><th>RE</th><th>Other</th></tr></thead><tbody><tr><td>2020</td><td>[76]</td><td>Complete overview of research before 2020</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Pipeline, RuleBased</td><td>Ø</td><td>En</td><td>Doc-Sent</td><td>Multi</td><td>10<sup>1</sup></td><td>10<sup>1</sup></td><td>✓</td></tr><tr><td>2021</td><td>[79]</td><td>RE in grammar checking</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Transformer, SVM</td><td>Elmo</td><td>Multi</td><td>Document</td><td>Multi</td><td>10<sup>1</sup></td><td>10<sup>1</sup></td><td></td></tr><tr><td>2021</td><td>[145]</td><td>Temporal relation extraction</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>NeuralNet, RandomForest</td><td>Bert</td><td>Multi</td><td>Doc-Sent</td><td>Medical</td><td>10<sup>2</sup></td><td>10<sup>1</sup></td><td></td></tr><tr><td>2022</td><td>[12]</td><td>Dataset creation and annotation</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>NeuralNet</td><td>Bert, Glove, Scikit</td><td>EN+FR</td><td>Doc-Sent</td><td>Multi</td><td>10<sup>1</sup></td><td>10<sup>1</sup></td><td></td></tr><tr><td>2022</td><td>[103]</td><td>Global overview of research output</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Knowledge Augmented LM</td><td>ERNIE, K-BERT, KEPLER, COMET</td><td>EN+Multi</td><td>Doc-Sent</td><td>Multi</td><td>?</td><td>0</td><td></td></tr><tr><td>2023</td><td>[66]</td><td>Horizontal overview of KG+NLP</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Transformer, NeuralNet</td><td>BERT, Elmo, LUKE</td><td>Multi</td><td>Doc-Sent</td><td>Multi</td><td>10<sup>1</sup></td><td>10<sup>1</sup></td><td></td></tr><tr><td>2025</td><td>Our work</td><td>Horizontal + vertical overview of KG+NLP</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>NeuralNet, Transformer</td><td>Many</td><td>Multi</td><td>Doc-Sent</td><td>Multi</td><td>10<sup>2</sup></td><td>10<sup>1</sup></td><td>✓</td></tr></tbody></table>

The Systematic reviews in our corpus describe core RE subtasks (e.g., NER, Coreference Resolution, Entity Linking, Relation Classification), as well as dimensions such as language, granularity and application domain. However, none of them provide direct access to their annotated corpora; such information must generally be inferred from the text. By contrast, our work openly releases an extended analytical grid with more than thirty dimensions, enabling fine-grained analysis and reproductibility. In addition, most surveys focus on classical neural architectures with embeddings, which are increasingly outdated in light of the latest developments in transformer-based and graph neural models. The analysis presented in the surveys also tend to remain limited to small samples of roughly twenty models and datasets. Furthermore, they generally present datasets and models separately, without investigating their combined effect on the task formalisation. Our review addresses these gaps by jointly analysing models, datasets and their interactions, while also considering the role of Semantic Web resources, as initially proposed by [76]. Finally, our survey comes with a complete access to the collected bibliography<fnref n="9" />, as well as the tools used to conduct it<fnref n="10" />, and a direct access to the data analysed<fnref n="11" />.

Some surveys are linked to a benchmark. Eighteen surveys in our corpus report performance metrics, but only two offer substantial benchmarking depth. For example [80] enriches 4 datasets (NYT11-HRL, TACRED, WebNLG and SemEval-2010 Task 8) with results for 36 models, while [143] extends the evaluation on the NYT dataset with 11 models. We build on these efforts by consolidating results from PapersWithCode (PWC), ensuring broader coverage and consistent comparisons across datasets and models. The detail of the results is available online<fnref n="12" />.

## 6 ANALYSIS OF PAPERS PRESENTING DATASETS FOR THE RELATION EXTRACTION TASK

Our analysis relies on the curated collection of 64 data papers published between 2019 and 2024, addressing the relation extraction (RE) task. This represents 25% of the corpus of data papers initially collected.

<footnote n="9">⁹https://github.com/datalogism/SciLex-RE_SystLitReview</footnote>

<footnote n="10">¹⁰https://github.com/Scilex</footnote>

<footnote n="11">¹¹https://www.zotero.org/groups/6070963/scilex_re_syslitreview/library</footnote>

<footnote n="12">¹²https://github.com/datalogism/SciLex-RE_SystLitReview/blob/main/data/benchmark_data/Benchmark_consolidated.csv</footnote>
**Granularity of Annotation:** A key observation concerns the granularity at which relations are annotated. Half of the datasets (33 out of 64) are designed exclusively for paragraph-level RE, while 11 target the document level. Here are the dataset papers dealing with document-level RE [自] and the those with sentence-level RE [自]. Restricting relation extraction to individual sentences limits the capture of context-long and interdependent sentential relations. To address this limitation, the remaining 20 datasets support both paragraph- and document-level annotations, allowing for a broader and more realistic representation of semantic relations in text.

**Language Coverage:** The vast majority of datasets (approximately 80%) are dedicated to English [自], indicating a clear research focus on monolingual RE in high-resource settings. Nevertheless, there has been a growing effort to expand linguistic coverage, with several datasets incorporating additional languages such as Arabic [18, 31, 122], Chinese (in 5 papers, including [23, 33]), and Spanish (in 4 papers, including [3, 117]) alongside English. This trend reflects an increasing awareness of the importance of multilingual information extraction.

**Annotation Strategy:** Given the need of deep learning models for large-scale training data, **distant supervision** has become a popular annotation strategy. While this approach enables rapid dataset construction, it often introduces noise in labels. To mitigate this, several initiatives have been proposed to annotate and/or correct datasets built using distant supervision. In our corpus, 12 datasets were partially annotated to assess data quality [自], whereas 29 were entirely manually annotated [自].

**Source Material and Knowledge Base Integration:** Wikipedia emerges as a dominant source of text, underpinning 23 datasets in our collection [自]. These are typically paired with structured knowledge bases such as DBpedia, YAGO, Wikidata, or Freebase, which provide the ground truth for relation labels. In earlier datasets, relations were commonly expressed as raw strings, particularly when constructed without knowledge base alignment. In the case of Wikidata, where entities are represented by opaque machine-readable identifiers (Q-ids), identifiers are generally replaced by human-readable labels to facilitate extraction and interpretation.

**Types of Extracted Values:** While string-based relations remain the most common in the RE datasets (54 datasets) [自], there is notable attention to date-related relations (13 datasets) [自]. Only a minority (7 datasets) [自] focus on extracting numerical values. Aside it we can note that 28 datasets explicitly use object properties [自] accounting for only 1/3 of the datasets proposed.

**Domain:** Half of the datasets contain encyclopedic knowledge, and a quarter are news-related datasets. Scientific and biology-related datasets are also well represented in the corpus we analysed. Some other popular datasets are related to specific contexts, such as conversational data (ACE) and cybersecurity (CASIE [102]). There are also datasets spanning multiple domains, among which CROSS-RE [10], a fully annotated resource, and MultiCROSSRE [11], an augmented version created through translation.

## 6.1 Encyclopedic Datasets

33% of RE datasets are derived from Wikipedia, valued for its multilingual scope and rich meta-annotations such as page links. Most of these datasets are paired with Wikidata (16 of 23 cases [自]), with additional efforts drawing on DBpedia [11, 34, 42] and, earlier, Freebase [97, 129] (see Fig. 15). The first large-scale attempt was the **WIKI-KBP** dataset introduced by [97], which aligned Freebase triples with Wikipedia text through distant supervision. It operated under the assumption that if a paragraph contains two entities, it potentially expresses the relation linking them in the knowledge base (KB). Entity mentions were extracted using the Stanford NER tagger, focusing on four types: Person, Organisation, Place and Other. This first proposal was followed by **GoogleRE** [27], a small partially annotated collection focused on person-related relations, and later by **WIKI-KB** [105], which linked Wikipedia with Wikidata
<table><caption>Table 4. Details on Encyclopedic datasets. To gain in readability, we use abbreviations to refer to tasks: NER: for Named Entity Recognition, CR: Coreference Resolution, ET: Entity Typing, EL: Entity Linking, RC: Relation Classification and GtoT: Graph-to-Text, as well as to some recurring sources: WP: Wikipedia, WD: Wikidata, DBP: DBPedia and FB: FreeBase. Also, Sent and Doc refer to sentence and of document granularity, respectively. "?" refers to cases where data is not available or clearly stated in the articles.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Short Title</th><th colspan="8">TASK</th><th rowspan="2">GRANULARITY</th><th rowspan="2">LANG</th><th rowspan="2">SOURCE</th><th colspan="2">Datatype</th><th rowspan="2">ManualAnnot.</th><th rowspan="2">UseNERTools</th></tr><tr><th>NER</th><th>CR</th><th>ET</th><th>EL</th><th>RC</th><th>RE</th><th>GtoT</th><th>Other</th><th>ObjProp</th><th>DitProp</th></tr></thead><tbody><tr><td>2011</td><td>WIKI-KBP[97]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>FB,NYT</td><td>✓</td><td>Str</td><td></td><td>StandfordNER</td></tr><tr><td>2015</td><td>GoogleRE[27]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WP</td><td>✓</td><td>Str</td><td>Partial</td><td></td></tr><tr><td>2017</td><td>WebNLG[42]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>DBP</td><td>✓</td><td>Date,Str</td><td>✓</td><td></td></tr><tr><td>2017</td><td>WIKI-KB[105]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Date,Str</td><td>Partial</td><td>Standfordnlp.Wikilinks</td></tr><tr><td>2018</td><td>TREX[34]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP,DbP</td><td>✓</td><td>Str,Date,Val</td><td>Partial</td><td>Dbpediaspotlight</td></tr><tr><td>2018</td><td>FewRel[46]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Str</td><td>✓</td><td>Spacy</td></tr><tr><td>2018</td><td>GDS[52]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>Sent</td><td>EN</td><td>GOOGLERE</td><td></td><td>Str</td><td>✓</td><td></td></tr><tr><td>2019</td><td>DocRED[140]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Date,Str,Val</td><td>Partial</td><td>Spacy</td></tr><tr><td>2019</td><td>FewRel 2.0[41]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP,FewRel, PubMed, UMLS</td><td>✓</td><td>Str</td><td>✓</td><td>?</td></tr><tr><td>2019</td><td>X-WikiRE[3]</td><td></td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>X-WikiRE[3]</td><td>EN,FR,DE,ES</td><td>WD,WP</td><td></td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2019</td><td>LIFELONGFEWREL[126]</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>FewRel,SimpleQuestions</td><td></td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2019</td><td>WIKI-NRE/GEO-NRE[116]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Str</td><td></td><td>Wikilinks</td></tr><tr><td>2019</td><td>KnowledgeNet[77]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>DiffHotKG,WD,WP</td><td></td><td>Date</td><td>✓</td><td></td><td></td></tr><tr><td>2020</td><td>KILT[92]</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>Doc-Sent</td><td>EN</td><td>Many</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2021</td><td>REBEL[50]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>Doc-Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Date,Str</td><td></td><td>Wikilinks</td><td></td></tr><tr><td>2021</td><td>HaRED[23]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>CN,EN</td><td></td><td>Str</td><td>Partial</td><td>toolkit TexSmart</td></tr><tr><td>2021</td><td>WikiGraphs[129]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>FB,WikiText-103</td><td>✓</td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2021</td><td>CoRED[139]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc-Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>None</td><td>✓</td><td></td><td></td></tr><tr><td>2021</td><td>WIKI-EVENTS[63]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>WP</td><td>✓</td><td>Str</td><td>✓</td><td></td><td></td></tr><tr><td>2021</td><td>WIKI-ZSL[19]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>Wiki-Kb,WD,WP</td><td>✓</td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2022</td><td>ReDocRED[112]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>Wiki-Kb,WD,WP</td><td></td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2022</td><td>HyperRED[25]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Date,Val</td><td>✓</td><td>DBPSpotlight,Spacy</td><td></td></tr><tr><td>2022</td><td>MAVEN-ERE[130]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>WP</td><td>✓</td><td>Date,Str</td><td>Partial</td><td></td><td></td></tr><tr><td>2022</td><td>WDV[8]</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>Sent</td><td>EN</td><td>WEBNLG,WP</td><td>✓</td><td>Date,Str,Val</td><td>Partial</td><td></td><td></td></tr><tr><td>2023</td><td>DocRED-FE[125]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc</td><td>EN</td><td>WD,WP</td><td>✓</td><td>Str</td><td>✓</td><td></td><td></td></tr><tr><td>2023</td><td>RELD[5]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Doc-Sent</td><td>EN</td><td>Many</td><td></td><td>Str</td><td></td><td></td><td></td></tr><tr><td>2023</td><td>REDIm[18]</td><td></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>WD,WP</td><td></td><td>Date,Str,Val</td><td>Partial</td><td>Wikilinks</td><td></td></tr></tbody></table>

<figure><img src="image_16.png" /><figcaption>Fig. 15. Encyclopedic Datasets Family Tree</figcaption></figure>

using Wikilinks for entity detection. The **WEBNLG** [42] dataset was created by verbalising (generating simple sentences expressing) triples from DBpedia using crowdsourcing. It was proposed as a benchmark for text generation from triples, but also used to evaluate text-to-triples models.
T-REX [34] is a milestone in terms of scaling, aligning 10 million triples from DBpedia with Wikipedia content. This resource was later integrated into the KILT benchmark [92], which evaluates models on multiple knowledge-intensive tasks grounded in Wikipedia content at the document level. Subsequent efforts broadened the scope: manually annotated corpora such as GDS [52] and KnowledgeNet [77]; few-shot learning related datasets such as FewRel [46], later expanded into FewRel 2.0 [41], or WIKIZSL [19]. Also datasets designed to solve lifelong learning as LifelongFewRel [126]; document-level corpora, among which DocRED [140], which was corrected into RE-DocRED [112] and also refined into DocRED-FE [125]. More recent initiatives introduced new dimensions, including HyperRED [25] for n-ary relations, textbfCodRED [139] for cross-document relations, and REBEL [50], which covered over 1,000 relation types with millions of triples and led to a multilingual benchmark known as REDFm [18]. Generative approaches also recently appeared, such as the WDV [8] dataset built from Wikidata triples. Specialised event-extraction corpora were recently published [63, 130], diversifying the landscape of available datasets. In terms of dataset volume, resources such as REBEL, T-REX, and CodRED contain millions of instances, while meta-benchmarks like KILT or RELD reach 10⁶ documents. Sentence-level corpora, such as X-WikiRE, include up to 10⁷ short paragraphs. Encyclopedic datasets describe the largest number of relations (up to 10⁴ in RELD) and entity types (up to 10³ classes). By comparison, major knowledge bases such as DBpedia, Wikidata, YAGO and Freebase contained between 10⁸ and 10⁹ triples in 2016, and the English version of Wikipedia held about 10⁶ articles [35] in 2016. RELD is the most recent initiative gathering previous datasets built on top of Wikipedia and covers the entire dual base of textual data from Wikipedia and triples from DBpedia.

## 6.2 News Datasets

<table><caption>Table 5. Details on News datasets. To gain in readability, we use abbreviations to refer to tasks: NER: for Named Entity Recognition, CR: Coreference Resolution, ET: Entity Typing, EL: Entity Linking, RC: Relation Classification and GtoT: Graph-to-Text, as well as to some recurring sources: WP: Wikipedia, WD: Wikidata, DBP: DBPedia and FB: FreeBase. Also, Sent and Doc refer to sentence and of document granularity, respectively. "?" refers to cases where data is not available or clearly stated in the articles.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Short Title</th><th colspan="6">TASK</th><th rowspan="2">GRANULARITY</th><th rowspan="2">Lang</th><th rowspan="2">Source</th><th colspan="2">Datatype</th><th rowspan="2">ManualAnnot</th><th rowspan="2">NER Tools Used</th><th rowspan="2">SynthGen</th><th rowspan="2">Neg.Ex</th></tr><tr><th>NER</th><th>CR</th><th>ET</th><th>EL</th><th>RC</th><th>RE</th><th>Other</th><th>ObjProp</th><th>DTPProp</th></tr></thead><tbody><tr><td>2003</td><td>CONLL03 [114]</td><td>✓</td><td></td><td></td><td></td><td></td><td></td><td>Doc</td><td>EN,DE<br>EN,CN.AR</td><td>Reuters<br>?</td><td>✓</td><td>Str</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>2004</td><td>ACE2004 [31]</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td>✓</td><td>Sent</td><td>EN</td><td>CognitiveCompGroup</td><td></td><td>Str</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>2004</td><td>Conll04 [64]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td>Doc+Sent</td><td>EN</td><td>CN.AR</td><td>LDC</td><td></td><td>Str</td><td>✓</td><td></td><td></td></tr><tr><td>2006</td><td>ACE2005 [122]</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>EN</td><td>Doc+Sent</td><td>LDC</td><td>?</td><td>Str</td><td>✓</td><td></td><td></td></tr><tr><td>2006</td><td>TIMEBANK [94]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>EN</td><td></td><td></td><td></td><td>Date,Str</td><td>✓</td><td></td><td></td></tr><tr><td>2010</td><td>NYT10 [97]</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td>Sent</td><td>Multi</td><td>FB,NYT</td><td></td><td>Str</td><td></td><td>StandfordCRF-NER</td><td></td></tr><tr><td>2011</td><td>WIKI-KBP [97]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>FB,NYT</td><td></td><td>Str</td><td>Partial</td><td>StandfordNER</td><td></td></tr><tr><td>2013</td><td>TempEval [117]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>Nsp</td><td>AquaintTimebank</td><td></td><td>Date,Str</td><td>Partial</td><td></td><td></td></tr><tr><td>2014</td><td>HIVE [43]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>EN</td><td></td><td>GraphMe</td><td></td><td>Str</td><td>✓</td><td>?</td><td>✓</td></tr><tr><td>2015</td><td>TAC-KBP [33]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN,CN,ES</td><td>WebBaseKB.FB,NYT,Xinhua</td><td></td><td>Str</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>2016</td><td>NYT-FB [75]</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>FB,NYT</td><td></td><td>Str</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>2017</td><td>TACRED [106]</td><td>✓</td><td>✓</td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>Tac-Kbp</td><td></td><td>Str</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>2018</td><td>NYT-MULTI [151]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>Multi</td><td>FB,NYT</td><td></td><td>Str</td><td></td><td>StandfordCRF-NER</td><td></td></tr><tr><td>2018</td><td>MATRES [83]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>EN</td><td></td><td>TBDense</td><td></td><td>Date,Str</td><td>✓</td><td></td><td></td></tr><tr><td>2019</td><td>NYT1HRL,NYT1-HRL [111]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>EN</td><td></td><td>FB,NYT</td><td></td><td>Str</td><td>✓</td><td>StandfordNER</td><td></td></tr><tr><td>2020</td><td>TACREV [7]</td><td></td><td></td><td></td><td></td><td>✓</td><td></td><td></td><td>Sent</td><td>EN</td><td>Tacred</td><td></td><td>Str</td><td>Partial</td><td></td><td></td></tr><tr><td>2020</td><td>CASIE [102]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>APTRepoint.WD</td><td></td><td>Str</td><td>✓</td><td>CoreNLP,DBPSpotlight</td><td>✓</td></tr><tr><td>2020</td><td>NYT-H [1]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>FB,NYT</td><td></td><td>Str</td><td>✓</td><td>StandfordNER</td><td>✓</td></tr><tr><td>2021</td><td>DWIE [147]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>Deutschwelle.WD</td><td></td><td>Str</td><td>✓</td><td></td><td></td></tr><tr><td>2021</td><td>RETACRED [106]</td><td></td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>TACRED</td><td></td><td>Str</td><td>Partial</td><td></td><td></td></tr><tr><td>2021</td><td>FLTACRED [100]</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>EN</td><td>Tac-Kbp</td><td></td><td>Str</td><td>✓</td><td></td><td>✓</td></tr><tr><td>2023</td><td>RELD [5]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>Doc+Sent</td><td>Many</td><td></td><td>Str</td><td></td><td></td><td>✓</td></tr><tr><td>2023</td><td>MultiCrossRE [11]</td><td>✓</td><td></td><td></td><td></td><td>✓</td><td>✓</td><td></td><td>Sent</td><td>Multi</td><td>WP.DBP</td><td></td><td>Str</td><td></td><td>✓</td><td></td></tr></tbody></table>

In contrast to encyclopedic datasets, news-based resources were initially developed from curated collections of news articles, fully annotated by communities of linguists. Many of these initiatives emerged from conference challenges. A prominent example is the CoNLL shared tasks, which initially targeted the Named Entity Recognition (NER) subtask [114] before expanding to include coreference resolution and relation extraction [64]. Similarly, the Automatic Content Extraction (ACE) corpora [31, 122], produced by the Linguistic Data Consortium (LDC), were explicitly designed to support relation extraction research.
<figure><img src="image_17.png" /><figcaption>Fig. 16. News Datasets Family Tree</figcaption></figure>

Due to their content, news corpora are inherently linked to events, which explains why event extraction challenges were introduced early in this domain. Notable datasets in this area include **TimeBank** [94] and **HiEve** [43], both aimed to model temporal and event-based relations. These early resources generally involved fewer than ten types of relations and entity categories. Nevertheless, the multilingual dimension was considered from the outset. This consideration was also reflected in the **ACE2004** and **ACE2005** as well in the **TAC-KBP** [33] datasets, which include English, Chinese and Arabic content.

As with encyclopedic datasets, a major leap in scalability came with the application of distant supervision. This was typically based on two canonical resources: Freebase and the *New York Times* (*NYT*) corpus. The **NYT10** [97] dataset was derived from these two, giving rise to a new generation of news-based relation extraction datasets. Over time, **NYT10** underwent multiple adaptations and refinements: **NYT Multi** [151] extended the dataset to a multilingual setting; **NYT-HRL** [111] adapted it for reinforcement learning experiments; and **NYT-H** [1] scaled and corrected the original data. One persistent challenge in this context has been the large-scale detection of entities present in Freebase, which is typically addressed using the Stanford NER toolkit.

Another landmark resource in this category was introduced by the LDC in 2015: the **TAC-KBP** dataset. This corpus combined several sources—including the NYT, Freebase, Xinhua News, and WebBase—to meet the growing need for multilingual relation extraction. Fully annotated, TAC-KBP was subsequently extended by **TACRED** [106], which in turn underwent two major corrections: **TACREV** [7] and **Re-TACRED** [106]. These revisions highlighted the difficulty of achieving high-quality annotation and the direct impact of annotation quality on downstream usage.

Beyond relation extraction alone, news-based datasets have also supported event extraction research. Notable examples include **SemEval-2013** [117], **MATRES** [83] and **CASIE** [102], alongside more recent contributions such as **DWIE**[147], which proposes a novel set of articles finely described by the entity-centred graph from the *Deutsche Welle*.

## 6.3 Biology and Science-Related Datasets

Another significant domain contributing to the production of datasets for relation extraction is the field of biology, biochemistry and biomedical research. This is a vast scientific area in which experimental work is often costly, making it highly valuable to explore and leverage existing scientific literature. An illustrative example is *GenIA* [85], the earliest dataset in our corpus, which comprises fully annotated documents designed to address multiple aspects of
molecular biology relation extraction. These tasks include NER, Coreference detection and Relation Identification and Classification.

Further initiatives, such as those reported in [61, 62], focused on identifying the effects of chemical compounds on diseases. In these cases, relations were extracted from either medical reports or scientific articles sourced from PubMed and enriched with the *Medical Subject Headings (MeSH)* taxonomy. In five out of seven datasets, annotations are provided at the document level. Regarding scale, these datasets typically cover more than $10^3$ documents and aligned triples. The largest one, **GDA** [133], contains more than $10^5$ documents with over $10^5$ triples, and was specifically designed to support deep learning models [133].

A key characteristic of datasets in this domain is the relatively small number of relation types, generally fewer than ten per dataset. Finally, several datasets targeting more general scientific content have also been developed, such as *SciREX* [51], *SemEval 2018* [39] and, more recently, *SciERC* [70]. These datasets similarly feature a limited number of predefined relations and entity types.

# 7 ANALYSIS OF PAPERS PRESENTING MODELS FOR THE RELATION EXTRACTION TASK

Our review includes a curated collection of 104 papers presenting models for the relation extraction (RE) task and published between 2019 and 2024. This represents 51% of our curated corpus but a very small part (3%) of the 3,567 model papers initially collected between 2019 and 2024. As explained in Section 4, we chose to focus on the most influential papers in order to be able to conduct an in depth analysis, rather than conducting a more superficial analysis of the entire research produced on RE models.

*Granularity of Annotation:* Models are generally specialized for a specific granularity: 58 model papers in our corpus address RE at the sentence level only, 32 address RE at the document level only, and 13 at both levels. Here are the model papers dealing with document-level RE [自] and the those with sentence-level RE [自]. *Language Coverage:* All models but one are trained on English data (103 papers [自]). A small number extend to multilingual settings: 15 papers report training on both English and Chinese [自] and, in 9 cases, additional training was conducted on Arabic datasets [自]. The analysed model papers do not cover all the languages considered in the previously discussed multilingual data papers, like REDfm [18] or X-WikiRE [3]. In particular, no model paper in our corpus deals with Spanish, a language that is considered by several data papers.

*Resources Used:* All together, the surveyed models were trained or evaluated on approximately 100 distinct datasets. In 15 cases, models were trained on a newly created dataset developed specifically for the need of a specific proposed research question. Examples include **DocRED**, **REBEL**, **SynthIE** [58] and **GenIE** [57]. We can pinpoint from Fig. 17 some canonical datasets that were used many times by the proposed models. As a salient example, **DocRED** was used in around 20% of the model papers in our corpus.

*Property Diversity:* Most proposed models handle fewer than 100 relations. As shown in Figure 18, models trained and evaluated on datasets containing fewer than $10^2$ distinct relations account for more than 85% of the studies. Models designed to manage a diversity of properties similar to those in knowledge graphs represent approximately 25% of the proposed approaches. Only four initiatives attempt to perform on datasets with $10^3$ properties: **ERICA** [95], **CTN** [55], **BERT-GT** [159] and **Table-Sequence** [127].

## 7.1 Tasks

Before examining the architectural details of the proposed models, it is necessary to clarify the tasks they aim to solve. A key observation, illustrated in Figure 19, concerns the distribution of research efforts: approximately 70% of the Manuscript submitted to ACM
<figure><img src="image_18.png" /><figcaption>Fig. 18. Number of distinct properties managed by the proposed models</figcaption></figure>

<figure><img src="image_19.png" /></figure>

articles address Relation Classification [ ], while the remaining 30% approaches End-to-End RE models [ ]. Other tasks include: Named Entity Recognition (21 papers [ ]), Relation Identification (13 papers [ ]), Coreference Resolution (7 papers [ ]), Question Answering [95, 136, 141, 155], Entity Typing [63, 95, 136], Graph-to-Text Generation [32, 101], and Entity Linking [29, 119].

We also note the emergence of tasks related to Evidence Extraction, explored in three papers: DREEAM [72], SIRE [149], and Eider [135]. Finally, several marginal but noteworthy tasks appear in the literature, such as Machine Reading Comprehension (MRC) [155], Reasoning [160], *implicit relation detection* [153], and *Event Extraction* [63, 74].

## 7.2 Input

From the input perspective, the majority of RE models take raw text as input, although in some cases, embeddings of the text are used directly. Entity information is explicitly integrated into the input in 32 papers [ ], often together with *Entity Type* annotations [101, 107, 158]. In five works, *Part-of-speech (POS)-tagged text* was employed as input [15, 38, 65, 110, 141]. *Relation Embedding* was also incorporated as input in some approaches, particularly in graph-based architectures or knowledge-augmented strategies. Some models further extend the input space by including *Relation description* directly in prompts [19], or by encoding entities with additional probabilistic information, as in [2]. More complex forms of auxiliary input have also been proposed, such as symbolic rules [99], images, or structured web content [68].

## 7.3 Architecture

During our study, we identified 78 distinct architectural elements, reflecting the breadth of research proposals since 2019. Notably, more than 40 architectures were used only once. As illustrated in Figure 22, the introduction of Transformer-based models was followed by a dramatic surge in adoption, with 55 papers published in 2021 alone—accounting for nearly 50% of the models in our corpus. Nevertheless, the model families highlighted in [80] are still represented, although their prevalence shows a progressive decline across the studied period.

Overall, these architectures can be grouped into two broad families. (1) **Encoder-based models**: these approaches generally rely on a representation embedding obtained using a deep learning method, which is used as support to solve one subtask of relation extraction. In the papers studied, relation extraction with encoders relates to the
<figure><img src="image_20.png" /><figcaption>Fig. 21. Streamgraph of architecture families</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Fig. 20. Distribution of the input used by the proposed models</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Fig. 22. Type of transformer architecture used by the models</figcaption></figure>

<figure><img src="image_23.png" /></figure>

relation classification subtask, where entities are already identified in a text and models have to recognise the relations expressed between them. (2) **SeqToSeq models**: this family includes encoder-decoder and decoder only transformer architectures, and addresses relation extraction using an *End-to-End* paradigm, i.e. by directly generating a sequence of text representing the result of the extraction.

<figure><img src="image_24.png" /><figcaption>Fig. 24. Specialised embeddings created</figcaption></figure>

<figure><img src="image_25.png" /></figure>
**Embeddings**: 51 models in our corpus integrate or rely on embedding methods. Among the earlier approaches, we observed four papers that employed Word2vec [54, 81, 88, 154] and approximately 20 that used GloVe [1]; however, both methods were gradually abandoned after 2020. Graph embeddings, such as TransE, were used sparingly, appearing in only three papers [26, 116, 132], between 2019 and 2021. Since the introduction of BERT, most recent researches on RE have adopted BERT-related embeddings: 13 papers used the original BERT embeddings [1], while others relied on BERT variations such as RoBERTa (16 papers [2]), ALBERT [101, 127, 138, 158], or more specific ones: SciBERT [3], or CharacterBERT [148, 150].

Several models also attempted to specialize embeddings to make them more effective for the RE task. For instance, three studies explored Character-level embeddings, using as a pretrained model: ELMo [29], Word2Vec [81], or GloVe [119]. New representations were also investigated, generally on top of BERT or GloVe, including Relation Embedding also derived from TransE, as well as Entity-focused embeddings, Entity Type and Part-of-speech (POS) features.

**Recurrent Networks**: Recurrent networks were heavily employed before the advent of Transformer-based architectures. The simplest variant, RNN, appeared in two papers [65, 127], followed by the use of Bi-RNN in 2019 [150]. LSTM, which is more effective at modelling long-range dependencies, was adopted in 15 papers [1], while GRU was used in only three cases [67, 109, 110]. The bidirectional extension of LSTM (Bi-LSTM) proved particularly influential, being used in 26 papers [1], thus confirming the effectiveness of bidirectional designs, taking into account both side of the context during the encoding. More advanced variants also emerged, such as hierarchical Bi-LSTMs for lifelong RE [84]. Additionally, [113] also proposes the use of hyperbolic layers to better capture the specific typology of relational data.

**Convolutional Networks**: Given that graph data can be naturally represented as matrices, convolutional approaches were also widely adopted, with 12 papers designing the RE task using CNNs [1]. Variants such as Piecewise Convolutional Neural Networks (PCNNs) were also investigated to better focus on important input content [104, 115, 144], alongside the use of specialized architectures such as U-Net, as reported in one paper [152]. Logical reasoning modules also emerged, such as the use of R-GCN in the SIRE model [149].

**Attention Layers**: Attention mechanisms were explicitly incorporated in 10 papers [1], highlighting their growing role as standalone architectural components, even before they became dominant in transformer-based models. In the details, some proposals specialised the mechanism to better fit the input/output specificity, using Graph Attention [68], Hierarchical Attention [144], N-gram Attention [116] or Selective Attention strategy [6].

**Structured Networks**: Several models proposed specific structured architectures, including Span Graph Tokens [121], Gaussian Graph Generators (GDPNet) integrated within BERT-GT [159], and a Structured Perceptron for event extraction [82]. Other examples include prototype-based learning such as ProtoNet [89].

**Contrastive Networks**: Also discussed latter, contrastive approaches were explored to better handle negative examples, and generally imply specific training processes. For instance, Siamese Networks [113] and GAN-based models (e.g., CopyMTL [148]) were proposed to enhance robustness and discriminative power.

**Copy Mechanism**: Finally, several specialized network designs were introduced. For example, Pointer Networks were adopted several times [81, 88, 148, 150] to implement copy-paste mechanisms from input to output.

**End-To-End models**: Next to the embeddings approaches, End-to-End models constitute 20% of the models in our corpus, all of which relying on pretrained transformers. Among them, a significant number are Encoder-Decoder models, notably based on BART (11 papers [1]) and T5 [30, 32, 44, 69], as well as more specialized versions such as BioBART [44] and SciFive [30]. FLAN-T5 was also used in [58]. On the decoder side, there are initiatives using decoder-only pretrained models such as GPT-2 [6] and GLM [124], alongside emerging research using GPT-3.5. Although decoder-only models are today the most widely adopted design in large language models (LLMs), the collected papers make little use of them.
The models covered by our study and dealing with LLMs use them as synthetic data generators rather than as relation extractors.

## 7.4 Adaptation to the RE Task

**Usage of Negative Examples:** Half of the analysed models [自] use negative examples. They are either encoder-based models addressing RE as a classification task or seq2seq models updating the Loss function by taking into account these negative examples.

**Usage of synthetic data:** Four proposals integrate synthetic data into their framework. For example, [63] generate in a first step a summary containing the most important information of the input in order to extract, in a second step, the relation from this summary. [101] and **RelationPrompt** [24] use templates crafted for several relations to compose synthetic sentences used as support of the method proposed. Finally, **SynthIE** was trained thoroughly on synthetic text generated using triples coming from *Wikidata*.

**Training:** Figure 26 shows that *Fine-tuning* is employed in 70% of the model papers to address the RE task using the previously described architectures, allowing pretrained models or embeddings to be specialized. Alternatively, in 10% of the cases, researchers continue the *pretraining*, i.e. extend the training of a pretrained transformer model (PTM) using new types of textual content and, optionally, new pretraining objectives. Less common are initiatives training models *from scratch*, with only examples re-training from scratch the embeddings for encoder-based models. As noted in previous sections, models can be specifically trained to recognize negative examples using *Contrastive architectures* [自], which require dedicated training techniques. Additionally, *Reinforcement learning* is a valuable tool to better align the model's behaviour with user expectations. It is used in three papers of our corpus [32, 74, 150].

To assess the generalizability of models to unseen knowledge, researchers increasingly explore *Prompt-based* [自] and *Instruction-based Learning* [69], where input text is structured to define the task explicitly or to integrate more context. Another option to reduce the computational cost is to fine-tune the models using a reduced number of additional layers related to the prompt input. This strategy is also called *Prompt Tuning* [21, 73] and can be considered as a specific kind of *Parameter-Efficient Fine-Tuning (PEFT)* [153].

In this line, *Few-shot* [自] and *Zero-shot* [自] evaluation and training have emerged as important approaches. To accommodate model updates, *Continual learning* (also referred to as Lifelong learning or meta-learning) is applied in five papers [自], enabling models to adapt over time. More marginally, other techniques have been employed, including expectation-maximization optimization [99], self-training [49] or snowballing [40], and teacher-forcing in **DREEAM** approaches were conducted.

**Loss Update:** The choice of the loss function is a key element of the configuration used to build RE models. 70% of the models define a specific loss function to optimize the proposed architectures. This is particularly common for encoder-based models and occurs less frequently with end-to-end models. For encoder-only models, an example is provided by [6], while for encoder-decoder and decoder architectures, ClarET [160] demonstrates tailored loss function design to improve performance.

**Decoding Method:** End-To-End models generally allow the definition of a decoding strategy; many of them use beam search as the primary generation strategy, which is usually the default configuration of a PTM, as done in **REBEL** [50]. **GenIE** [57] introduces a constrained beam search combined with prefix trees, allowing constraints generation over the *Wikidata* schema. Additionally, **DORE** [44] uses token-ID vocabulary constraints during decoding to guide generation.
<figure><img src="image_26.png" /><figcaption>Fig. 26. Learning strategies distribution</figcaption></figure>

<figure><img src="image_27.png" /></figure>

## 7.5 Output

*Linearization:* Seq2seq models are capable of directly generating a sequence of tokens, the strategy chosen to represent a graph is generally referred to as *Linearization*. Several approaches have been proposed in the literature. The most frequent approaches are to represent a triple as a *list* of its subject, predicate and object, enclosed in parentheses, or to use tags in square brackets to delimit the subject, property and object of the triple. The definition of the delimiter tokens is done before the fine-tuning, and their representation are learnt during it. There are also approaches representing triples in a syntax close to that of first-order logic. Other approaches factorize the triples representation to avoid repetition of subjects and, in some cases, predicates. Additionally, a few approaches use more complex syntaxes like JSON.

<table><caption>Table 6. Linearisation details</caption><thead><tr><td>Linearisation</td><td>Factorised</td><td>Models</td></tr></thead><tbody><tr><td>(s,p,o)</td><td>0</td><td>[44, 142, 146]</td></tr><tr><td>&lt;s,p,o&gt;</td><td>0</td><td>[57, 116]</td></tr><tr><td>&lt;triplet&gt;s&lt;subj&gt;p&lt;obj&gt;o&lt;sub&gt;...</td><td>✓</td><td>[50]</td></tr><tr><td>[s]s[r]p[o]o[p]...</td><td>✓</td><td>[58]</td></tr><tr><td>p(s,o) p(s,o)</td><td>0</td><td>[30]</td></tr><tr><td>JsonData</td><td>✓</td><td>[98]</td></tr><tr><td>HeadEntity: s TailEntity: o Relation: p</td><td>0</td><td>[24]</td></tr><tr><td>(entityType: s, (r:o),...)</td><td>✓</td><td>[69]</td></tr></tbody></table>

*Datatypes of Output Values:* In 39% of the reviewed model papers (41) [自]) the data is extracted only as string values. In 37% of the reviewed model papers (39) [自]), the data is extracted only as URIs. The extraction is conducted using both literal values and URIs in only 20 models [自]. A few model papers explicitly represent dates (7) or typed numerical values (2).

## 7.6 Computational Cost

In the entire analysed corpus only 11 model papers clearly detail and discuss the computational cost of their experiments. As noted in the previous section, models may involve both pretraining and fine-tuning steps. The cost of these processes
<table><caption>Table 7. Estimated computational cost in GPU hours, based on the information available and detailed in the model papers. "?" refers to cases where data is not available or clearly stated in the papers.</caption><thead><tr><th rowspan="2">Year</th><th rowspan="2">Model</th><th rowspan="2">Type</th><th colspan="2">Finetuning</th><th colspan="2">Pretraining</th><th colspan="2">Total</th></tr><tr><th>min</th><th>max</th><th>min</th><th>max</th><th>min</th><th>max</th></tr></thead><tbody><tr><td>2020</td><td>LUKE</td><td>encoder-based</td><td>0,16</td><td>3,3</td><td>720</td><td>?</td><td>720</td><td>725</td></tr><tr><td>2020</td><td>BERT+MTB</td><td>encoder-based</td><td>48</td><td>?</td><td>96</td><td>?</td><td>50</td><td>150</td></tr><tr><td>2020</td><td>CorefBERT</td><td>encoder-based</td><td>?</td><td>?</td><td>288</td><td>2112</td><td>300</td><td>2120</td></tr><tr><td>2021</td><td>SIRE</td><td>encoder-based</td><td>?</td><td>?</td><td>18</td><td>24</td><td>20</td><td>30</td></tr><tr><td>2021</td><td>REBEL</td><td>encoder-decoder</td><td>0,27</td><td>5,6</td><td>27</td><td>200</td><td>0,27</td><td>210</td></tr><tr><td>2021</td><td>REDSandT</td><td>encoder-based</td><td>0,15</td><td>5</td><td>?</td><td>?</td><td>0,15</td><td>5</td></tr><tr><td>2021</td><td>BERT-GT</td><td>encoder-based</td><td>?</td><td>?</td><td>5</td><td>?</td><td>5</td><td>10</td></tr><tr><td>2022</td><td>GenIE</td><td>encoder-decoder</td><td>2,5</td><td>60</td><td>12</td><td>492</td><td>14,4</td><td>492</td></tr><tr><td>2022</td><td>ClarET</td><td>encoder-decoder</td><td>?</td><td>?</td><td>90</td><td>?</td><td>92</td><td>100</td></tr><tr><td>2023</td><td>SynthIE</td><td>encoder-decoder</td><td>?</td><td>?</td><td>160</td><td>240</td><td>162</td><td>300</td></tr><tr><td>2023</td><td>DREEAM</td><td>encoder-based</td><td>0,5</td><td>?</td><td>4</td><td>7</td><td>4,5</td><td>7,5</td></tr></tbody></table>

is reported in various units across the literature, including GPU days, GPU hours or GPU minutes, and must also account for the number of GPUs used. Comparing these results is challenging because recorded times depend not only on the dataset size but also on the GPU hardware employed.

We collected this information and estimated the minimum and maximum costs of each model, as summarised in Table 7. Pretraining is generally the most costly strategy compared to fine-tuning. Encoder-based architectures typically require smaller budgets, essentially because encoder-decoder models often demand more resources for pretraining, which is usually performed before fine-tuning on specific evaluation datasets. From the encoder perspective, **CorefBERT** provides an example of a model pretrained from scratch, which requires even more computational resources.

## 8 CITATION NETWORK

*Global Overview:* In our collection of 202 papers, 190 papers are identified by a DOI enabling to retrieve their citation network. The complete network was expanded to 6,253 papers by incorporating all the papers identified in the citation data of our corpus. The resulting graph has an average degree of 1.63 and a density close to zero, indicating that it is extremely sparse. This sparsity reflects a common pattern in citation networks, where most papers cite only a small subset of all the available works. A visualisation of the network is presented in Fig. 27¹³. Despite its sparsity, the graph has a diameter of 7, meaning that the longest shortest path between any two papers spans seven citation steps. This observation is consistent with the “small-world” property typically associated with citation networks, where citation chains connect otherwise distant works. The analysis of the network’s community structure yields a modularity score of 0.7, a relatively high value that indicates well-defined clusters of papers. Conversely, the clustering coefficient is low and close to zero, which can be explained by the fact that papers tend to cite prior work, yet two papers citing the same work have a relatively low probability of citing each other.

*Analysis by paper type:* A closer inspection highlights the central role and strong impact of survey papers in the network. Surveys summarize the state of the art and frequently serve as reference points for subsequent research. Notable examples include the comprehensive survey by [53], which provides a mathematically grounded overview of embedding methods and models for knowledge representation; the benchmarking initiative in the biomedical domain by [90]; and the influential work of [134] addressing event extraction. Other key contributions include [76] and [137], both of which focus on knowledge graph representation. Dataset papers also occupy central positions in the network, often anchoring specific communities of practice. Examples include the foundational CoNLL03 dataset [114], as well as

<footnote n="13">¹³Interactive version available online</footnote>
<figure><img src="image_28.png" /><figcaption>Fig. 27. Citation network, with model papers in green, dataset papers in blue, surveys in orange and papers out of the scope in grey. The color of an edge is the combination of the colors of the nodes it links.</figcaption></figure>

the hubs represented by CDR [62] and ADE [45]. Models proposed by the community are generally concentrated near the network's core, while more specialized or innovative approaches tend to be positioned towards the periphery.

*Communities analysis:* With the Louvain community detection algorithm, we identified over 40 clusters, highlighting the diverse research communities involved in the RE task, typically centred on specific datasets. The largest cluster is drawn by [53], which serves as a key foundation for encoder-based research. Another significant cluster encompasses news and event-related models, including FewRel, which has later been used to benchmark models trained on news datasets. We observed a notable cluster around the CoNLL03 dataset, frequently cited by many papers as one of the first datasets for Named Entity Recognition (NER) tasks. Additionally, there is substantial research activity surrounding the NYT-10 and NYT-H datasets, which paved the way for distant annotation techniques, along with updated versions like NYTHT10 and NYT11, exemplified by works like REBEL, TPLinker and RSAN. Fianlly, there is a cluster related to the N-Ary and WikiHop datasets, which tackle complex relations and have inspired various convolutional neural network approaches.
## 9 BENCHMARK

**Metrics:** In the literature, a variety of metrics were used to evaluate relation extraction models, depending on the task. Ranking metrics such as Hits@k, Mean Reciprocal Rank (MRR) and Mean Rank (MR) are commonly applied in knowledge graph completion tasks to assess the rank of correct predictions. Classification metrics, including classification Accuracy, Micro/Macro F1 and Ignored F1 (Ign.F1) measure the correctness of predicted relations, with some metrics ignoring irrelevant labels. Curve-based metrics like the Area Under the Curve (AUC) evaluate model performance across different decision thresholds. For sequence generation approaches, text generation metrics such as BLEU and ROUGE assess the quality of generated sequences by comparing them to reference triples or linearized outputs. But the most commonly used metric remains the micro F1, which is used in the following to compare and evaluate the performances of the analysed models of the corpus.

<figure><img src="image_29.png" /><figcaption>Fig. 28. Datasets usage and averaged micro F1 results</figcaption></figure>

**Dataset benchmarked:** The statistics gathered from PapersWithCode and previous surveys allow us to consolidate and analyze the results of 25 models evaluated on 29 datasets. Despite the importance of the macro F1 metric for assessing a model's ability to handle underrepresented relations, many studies report only micro-level results. Therefore we use the micro-F1 scores if available to compare all our results. Among the datasets, only 14 are used in more than two model papers, they are illustrated in Fig. 28. Considering a benchmark as saturated when at least one model achieves an F1 score higher than 0.9, we observe that **NYT-10** and **WebNLG** are saturated by six models, while **SemEval-2010 Task 8** and **Re-TACRED** are saturated by one model each. Beyond these, datasets such as **TACRED**, **DocRED**, **ACE2005**, **NYT11-HRL**, and **FewRel** remain challenging for current models.

**Performance Trends:** Our analysis highlights clear performance improvements on four widely studied datasets: TACRED, DocRED, WebNLG and NYT-10 (see Fig. 29, 30, 31, 32). Among them, the most striking progress was observed on WebNLG, where micro-F1 scores increased by 222% between 2017 and 2021. This leap can be attributed to the dataset's design: short texts containing few facts, which favour effective modelling. Comparable gains were recorded on NYT-10, with an improvement of 121% between 2017 and 2022. Early encoder-decoder systems, such as PNDEC [81], achieved F1 scores around 0.8, while later seq2seq architectures (e.g., DeepStruct [124], REBEL [50], UIE [69]) consistently reached or exceeded these levels. Notably, NYT-10 exhibited earlier improvements than other benchmarks, with a 39% relative gain as early as 2018, whereas most datasets only saw significant advances after 2020. Its other

Manuscript submitted to ACM
<figure><img src="image_30.png" /><figcaption>Fig. 29. MicroF1 on TACRED</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Fig. 30. MicroF1 on DOCRED</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>Fig. 31. MicroF1 on WEBNLG</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>Fig. 32. MicroF1 on NYT</figcaption></figure>

version, **NYT11**, has been less frequently benchmarked; however, BERT-based approaches still push F1 scores close to 0.9, compared to pre-2019 baselines of around 0.6.

In contrast, **TACRED** and **DocRED** showed only modest improvements. On **TACRED**, the best models achieved micro-F1 scores near 0.75, compared to pre-2019 baselines of about 0.6. **DeepStruct** achieved 0.76, while **REBEL** reported a notable 0.94 on the more robust Re-TACRED variant. **DocRED** remains particularly challenging: while **DREEAM** achieved the best F1 (0.67), most earlier systems hovered around 0.6. **REBEL** underperformed here, also around 0.6, underscoring the challenge of adapting models to long-document contexts.

Other benchmarks show similar trends. On **ACE2005**, F1 scores improved modestly from 0.60 to 0.65, while **CDR** rose from 0.60 to 0.75. **SemEval2010** tasks saw a sharper jump, from early CNN/SVM systems (0.80) to **KnowPrompt** [21] (0.90). In contrast, **SciERC** illustrates the persistent difficulty related to domain-specific RE, with performance plateauing below 0.60.

Few-shot datasets further highlight recent progress. Early prompt-based methods, such as **RelationPrompt** [24], achieved an F1 score of only 0.20 in 5–10 shot scenarios. In contrast, recent models, such as **DeepStruct**, nearly saturated the benchmark, achieving near-perfect extraction in both 5- and 10-shot settings.

# 10 TRENDS AND THE STORY SO FAR

Since its formalization in the early **1990s**, research on relation extraction (RE) has evolved through several major shifts. The first decade relied on manually crafted patterns and statistical methods. With the advent of Wikipedia in **2001**,
manually annotated corpora became available, establishing the first robust baselines, often based on newswire data. The emergence of Linked Open Data (2007–2014) further accelerated progress by enabling structured knowledge graphs at scale, which in turn supported the creation of large, distantly supervised datasets for training deep learning models. For much of this period, RE was framed as a pipeline task: entities were identified using NER systems and subsequently linked through classification models, increasingly supported by deep learning architectures such as LSTMs, recurrent networks, and GANs. A decisive boost came with the introduction of distributed word representations, starting with word2vec in 2013, which established embeddings as a foundation for modern NLP.

Our survey focuses on the more recent **Transformer era**, which has defined the last five years of RE research. This phase began with the “attention” mechanism and was consolidated in 2019 with BERT fine-tuning, applied in numerous configurations through task-specific representation layers. In parallel, the release of T-REX (2018) marked a milestone in dataset scale and methodology, and spurred new exploration of few-shot learning capabilities. Since 2020, RE research has expanded rapidly across models, datasets, and surveys. Encoder–decoder architectures such as T5 and BART have enable generative approaches that directly produce relational triples, saturating several benchmark datasets. Consequently, newer datasets have targeted more complex challenges, including multilinguality, fine-grained relations, and domain-specific extraction. In 2023, we entered in the **LLM-age**, since the release of ChatGPT 3.5 which introduced a new wave of research based on large decoder-only models (LLMs). These models allow longer input contexts and instruction-based interaction without task-specific fine-tuning. However, their limitations—particularly in reasoning, factuality, auditability, and interpretability—highlight the importance of ongoing work on **neurosymbolic** methods combining Knowledge Graphs and Language models.

## 11 CONCLUSION

This survey has shown that progress in relation extraction (RE) is closely tied to the quality of datasets and the ability of models to manage noisy input. Over time, datasets and models have iteratively evolved to address more complex relations, domains, and tasks. In the last five years, pre-trained Transformer architectures have driven remarkable improvements, enabling a shift from classification-based designs to generative models capable of directly producing structured relational outputs. While our review could not fully capture the most recent developments in large language models (LLMs), emerging approaches—including prompt-based, instruction-tuned, and chain-of-thought methods—are rapidly reshaping the field. At the same time, there is increasing interest in integrating LLMs with knowledge graphs [56, 71, 86, 87], with benchmarks such as LLM-KG-BENCH [36, 37] tracking these capabilities. We encourage readers to consult complementary surveys on knowledge integration in LLMs [131], hallucination reduction [4], reasoning in graph construction [161], and specialised perspectives such as domain-specific RE [128], few-shot learning [20], and multimodal contexts [22].

Despite these advances, several challenges remain. Many datasets, although derived from structured knowledge bases, oversimplify information: inputs are reduced to plain text and outputs are represented as triples that often fall short of Semantic Web standards. Research also remains fragmented across communities of practice. Encyclopedic datasets aim to cover entire knowledge graphs but struggle with overlapping or semantically close relations, while scientific and domain-specific corpora remain narrow in scope yet difficult for current models. Future RE systems will therefore need to remain **knowledge-grounded**, ensuring auditability and factual consistency, and increasingly **ontology-driven**, to enable principled specialisation and reuse of extracted data.

On the modelling side, transformers and LLMs have simplified task design and enabled richer relational extraction, but at significant computational cost: even adapting smaller models requires substantial GPU resources. As such, Manuscript submitted to ACM
lightweight and modular architectures are valuable, particularly for constrained tasks where efficiency is paramount. In knowledge-intensive applications, RE models can be seen not only as extractors but also as structured data generators and translators within complex agent-based systems. In such contexts, frugality and scalability will be key, reinforcing the relevance of exploring small language models and hybrid approaches [13].

## ACKNOWLEDGMENTS

This work was supported by the French government through the France 2030 investment plan managed by the National Research Agency (ANR), as part of the Initiative of Excellence Université Côte d'Azur (ANR-15-IDEX-01). Additional support came from French Government's France 2030 investment plan (ANR-22-CPJ2-0048-01), through 3IA Côte d'Azur (ANR-23-IACL-0001).

## REFERENCES

[1] Tong Zhu 0002, Haitao Wang 0019, Junjie Yu, Xiabing Zhou, Wenliang Chen, Wei Zhang 0027, and Min Zhang 0005. 2020. Towards Accurate and Consistent Evaluation: A Dataset for Distantly-Supervised Relation Extraction. In *Proceedings of the 28th International Conference on Computational Linguistics*, Vol. nan. nan, Barcelona, Spain (Online), 6436–6447. https://doi.org/10.18653/V1/2020.COLING-MAIN.566

[2] Bo Li 0099, Wei Ye 0004, Canming Huang, and Shikun Zhang. 2021. Multi-view Inference for Relation Extraction with Uncertain Knowledge. In *Proceedings of the AAAI Conference on Artificial Intelligence*, Vol. nan. AAAI, Online, 13234–13242. https://doi.org/10.1609/AAAI.V35I15.17563

[3] Mostafa Abdou, Cezar Sas, Rahul Aralikatte, Isabelle Augenstein, and Anders Søgaard. 2019. X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension. In *Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)*. ACL, Hong Kong, China, 265–274. https://doi.org/10.18653/v1/d19-6130

[4] Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. 2024. Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey. In *Proceedings of the 2024 Conference of the NAACL: Human Language Technologies (Volume 1: Long Papers)*, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). ACL, Mexico City, Mexico, 3947–3960. https://doi.org/10.18653/v1/2024.nacl-long.219

[5] Manzoor Ali, Muhammad Saleem, Diego Moussallem, Mohamed Ahmed Sherif, and Axel-Cyrille Ngonga Ngomo. 2023. RELD: A Knowledge Graph of Relation Extraction Datasets. In *The Semantic Web, Catia Pesquita, Ernesto Jimenez-Ruiz, Jamie McCusker, Daniel Faria, Mauro Dragoni, Anastasia Dimou, Raphael Troncy, and Sven Hertling* (Eds.), Vol. 13870. Springer Nature Switzerland, Berlin, Heidelberg, 337–353. https://doi.org/10.1007/978-3-031-33455-9_20

[6] Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction. In *Proceedings of the 57th Annual Meeting of the ACL* (2019-06-19), Vol. NA. ACL, Florence, Italy, NA. https://doi.org/10.18653/v1/p19-1134

[7] Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task. In *Proceedings of the 58th Annual Meeting of the ACL* (2020-04-30). ACL, Online, 1558–1569. https://doi.org/10.48550/arXiv.2004.14855

[8] Gabriel Amaral, Odinaldo Rodrigues, and Elena Simperl. 2022. WDV: A Broad Data Verbalisation Dataset Built from Wikidata. In *The Semantic Web – ISWC 2022 (Lecture Notes in Computer Science)*, Ulrike Sattler, Aidan Hogan, Maria Keet, Valentina Presutti, João Paulo A. Almeida, Hideaki Takeda, Pierre Monnin, Giuseppe Pirrò, and Claudia d'Amato (Eds.). Springer, Cham, 556–574. https://doi.org/10.1007/978-3-031-19433-7_32

[9] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In *The Semantic Web*, Karl Aberer, Key-Sun Choi, Natasha Noy, Dean Allemang, Kyung-Il Lee, Lyndon Nixon, Jennifer Golbeck, Peter Mika, Diana Maynard, Riichiro Mizoguchi, Guus Schreiber, and Philippe Cudré-Mauroux (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 722–735.

[10] Elisa Bassignana and Barbara Plank. 2022. CrossRE: A Cross-Domain Dataset for Relation Extraction. In *EMNLP 2022* (2022-10-17). ACL, Abu Dhabi, United Arab Emirates, 3592–3604. https://doi.org/10.18653/v1/2022.findings-emnlp.263

[11] Elisa Bassignana, Filip Ginter, Sampo Pyysalo, Rob van der Goot, and Barbara Plank. 2023. Multi-CrossRE A Multi-Lingual Multi-Domain Dataset for Relation Extraction. In *Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)* (2023-05). University of Tartu Library, Tõrshavn, Faroe Islands, 80–85. https://aclanthology.org/2023.nodalida-1.9

[12] Elisa Bassignana and Barbara Plank. 2022. What do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification. In *Proceedings of the 60th Annual Meeting of the ACL: Student Research Workshop* (2022-04-28). ACL, Dublin, Ireland, 67–83. https://doi.org/10.18653/v1/2022.acl-srw.7

[13] Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small Language Models are the Future of Agentic AI. arXiv:2506.02153 [cs.AI] https://arxiv.org/abs/2506.02153

[14] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In *Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD '08)*. Association for Computing Machinery, Vancouver, Canada, 1247–1250. https://doi.org/10.1145/1376616.1376746
[15] Bowen Yu, Zhen-Yu Zhang, Tingwen Liu, Bin Wang, Sujian Li, and Quangang Li. 2019. Beyond Word Attention: Using Segment Attention in Neural Relation Extraction. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (2019-08-10), Vol. NA. ACL, Online, NA. https://doi.org/10.24963/ijcai.2019/750

[16] Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software 80, 4 (2007), 571–583. https://doi.org/10.1016/j.jss.2006.07.009

[17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., Online, 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

[18] Pere-Lluís Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, and Roberto Navigli. 2023. REDS$^{\wedge}\{rm FM\}$: a Filtered and Multilingual Relation Extraction Dataset. In Proceedings of the 61st Annual Meeting of the ACL (Volume 1: Long Papers), Vol. Proceedings of the 61st Annual Meeting of the ACL (Volume 1: Long Papers). ACL, Toronto, Canada, 4326–4343. https://doi.org/10.18653/v1/2023.acl-long.237

[19] Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL, Online, 3470–3479. https://doi.org/10.18653/v1/2021.naacl-main.272

[20] Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Jeff Z. Pan, Yuan He, Wen Zhang, Ian Horrocks, and Hua zeng Chen. 2021. Zero-Shot and Few-Shot Learning With Knowledge Graphs: A Comprehensive Survey. Proc. IEEE 111 (2021), 653–685. https://apisemanticscholar.org/CorpusID:254247355

[21] Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction. In Proceedings of the Web Conference 2022 (2022-04-25). ACM, Virtual Event, Lyon, France, 2778–2788. https://doi.org/10.1145/3485447.3511998

[22] Yong Chen, Xinkai Ge, Shengli Yang, Linmei Hu, Jie Li, and Jinwen Zhang. 2023. A Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications. Mathematics 11, 8 (2023), nan. https://doi.org/10.3390/math11081815

[23] Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A Large-Scale Relation Extraction Dataset Toward Hard Cases in Practical Applications. In ACL-IJCNLP 2021. ACL, Online, 2819–2831. https://doi.org/10.18653/v1/2021.findings-acl.249

[24] Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022. RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction. In ACL 2022 (2022-03-17), Vol. NA. ACL, Dublin, Ireland, NA. https://doi.org/10.18653/v1/2022.findings-acl.5

[25] Yew Ken Chia, Lidong Bing, Sharifah Mahani Aljunied, Luo Si, and Soujanya Poria. 2022. A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. ACL, Abu Dhabi, United Arab Emirates, 10114–10133. https://doi.org/10.18653/v1/2022.emnlp-main.688

[26] Despina Christou and Grigorios Tsoumakas. 2021. Improving Distantly-Supervised Relation Extraction Through BERT-Based Label and Instance Embeddings. AI 9 (2021), 62574–62582. https://doi.org/10.1109/ACCESS.2021.3073428

[27] Orr Dave. 2015. google-research-datasets/relation-extraction-corpus. https://github.com/google-research-datasets/relation-extraction-corpus

[28] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). ACL, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

[29] Kalpit Dixit and Yaser Al-Onaizan. 2019. Span-Level Model for Relation Extraction. In Proceedings of the 57th Annual Meeting of the ACL (2019-07-01), Vol. NA. ACL, Florence, Italy, NA. https://doi.org/10.18653/v1/p19-1525

[30] Dmitriy Dligach, Steven Bethard, Timothy Miller, and Guergana Savova. 2022. Exploring Text Representations for Generative Temporal Relation Extraction. In Proceedings of the 4th Clinical Natural Language Processing Workshop (2022-07-01), Vol. NA. ACL, Seattle, WA, NA. https://doi.org/10.18653/v1/2022.clinicalnlp-1.12

[31] George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) Program Tasks, Data, and Evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Vol. Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04). ELRA, Lisbon, Portugal, nan. https://doi.org/10.35111/mwxc-vh88

[32] Pierre Dognin, Inkit Padhi, Igor Melnyk, and Payel Das. 2021. ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models. In Proceedings of the 2021 Conference on EMNLP Processing. ACL, Online and Punta Cana, Dominican Republic, 1084–1099. https://doi.org/10.18653/v1/2021.emnlp-main.83

[33] Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2015. Overview of Linguistic Resources for the TAC KBP 2015 Evaluations: Methodologies and Results. tac.nist, Online.

[34] Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Simperl, and Frederique Laforest. 2018. T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples. In LREC2018. European Language Resources Association (ELRA), Miyazaki, Manuscript submitted to ACM
Japan, nan. https://aclanthology.org/L18-1544

[35] Michael Färber and Achim Rettinger. 2018. Which Knowledge Graph Is Best for Me? arXiv:1809.11099 [cs.AI] https://arxiv.org/abs/1809.11099

[36] Johannes Frey, Lars-Peter Meyer, Natanael Arndt, Felix Brei, and Kirill Bulert. 2023. Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?. In *Proceedings of Workshop Deep Learning for Knowledge Graphs (DL4KG) @ ISWC23*. Springer Nature, Athens, Greece. https://doi.org/10.48550/ARXIV.2309.17122

[37] Johannes Frey, Lars-Peter Meyer, Felix Brei, Sabine Gründer-Fahrer, and Michael Martin. 2025. Assessing the Evolution of LLM capabilities for Knowledge Graph Engineering in 2023. In *The Semantic Web: ESWC 2024 Satellite Events*. Springer Nature Switzerland, Portoroz, Slovenia, 51–60. https://doi.org/10.1007/978-3-031-78952-6_5

[38] Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction. In *Proceedings of the 57th Annual Meeting of the ACL* (2019-07-01), Vol. NA. ACL, Florence, Italy, NA. https://doi.org/10.18653/v1/p19-1136

[39] Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Haifa Zargayouna, and Thierry Charnois. 2018. SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers. In *Proceedings of the 12th International Workshop on Semantic Evaluation* (2018-06), Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). ACL, New Orleans, Louisiana, 679–688. https://doi.org/10.18653/v1/S18-1111

[40] Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Neural Snowball for Few-Shot Relation Learning. In AAAI (2019-08-29). CoRR 34, 7772–7779. https://doi.org/10.1609/aaai.v34i05.6281

[41] Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. ACL, Hong Kong, China, 6250–6255. https://doi.org/10.18653/v1/d19-1649

[42] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating Training Corpora for NLG Micro-Planners. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*. ACL, Vancouver, Canada, 179–188. https://doi.org/10.18653/v1/p17-1017

[43] Goran Glavas̈, Jan ̈Snajder, Parisa Kordjamshidi, and Marie-Francine Moens. 2014. HiEve: A Corpus for Extracting Event Hierarchies from News Stories. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)*. European Language Resources Association (ELRA), Reykjavik, Iceland, 3678–3683. https://aclanthology.org/L14-1020/

[44] Qipeng Guo, Yuqing Yang, Hang Yan, Xipeng Qiu, and Zheng Zhang. 2022. DORE: Document Ordered Relation Extraction based on Generative Framework. In *EMNLP 2022* (2022-10-28). ACL, Abu Dhabi, United Arab Emirates, 3463–3474. https://doi.org/10.18653/v1/2022.findings-emnlp.253

[45] Harsha Gurulingappa, Abdul Mateen Bear, David Israel, Megumi Kameyalna, and Mabry Tyson. 1993. FASTUS: A System for Extracting Information from Text. In *Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993*. ACL, Columbus, Ohio, USA, nan. https://aclanthology.org/H93-1026/

[46] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*. ACL, Brussels, Belgium, 4803–4809. https://doi.org/10.18653/v1/d18-1514

[47] Ginny Hendricks, Dominika Tkaczyk, Jennifer Lin, and Patricia Feeney. 2020. Crossref: The sustainable source of community-owned scholarly metadata. *Quantitative Science Studies* 1, 1 (2020), 414–427. https://doi.org/10.1162/qss_a_00022

[48] Jerry R. Hobbs, Douglas Appelt, John Bear, David Israel, Megumi Kameyalna, and Mabry Tyson. 1993. FASTUS: A System for Extracting Information from Text. In *Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993*. ACL, Columbus, Ohio, USA, nan. https://aclanthology.org/H93-1026/

[49] Xuming Hu, Chenwei Zhang, Fukun Ma, Chenyao Liu, Lijie Wen, and Philip S. Yu. 2021. Semi-supervised Relation Extraction via Incremental Meta Self-Training. In *EMNLP 2021*. ACL, Punta Cana, Dominican Republic, 487–496. https://doi.org/10.18653/v1/2021.findings-emnlp.44

[50] Pere-Lluis Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation Extraction By End-to-end Language generation. In *EMNLP 2021* (2021-11). ACL, Punta Cana, Dominican Republic, 2370–2381. https://doi.org/10.18653/v1/2021.findings-emnlp.204

[51] Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. SciREX: A Challenge Dataset for Document-Level Information Extraction. In *Proceedings of the 58th Annual Meeting of the ACL* (2020-07). ACL, Online, 7506–7516. https://doi.org/10.18653/v1/2020.acl-main.670

[52] Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar. 2018. Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention. In *Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2017)*. arXiv, Online, 1031–1039. https://doi.org/10.48550/arXiv.1804.06987

[53] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2022. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications. *IEEE Transactions on Neural Networks and Learning Systems* 33, 2 (2022), 494–514. https://doi.org/10.1109/TNNLS.2021.3070843

[54] Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-Level N-ary Relation Extraction with Multiscale Representation Learning. In *Proceedings of the 2019 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long and Short Papers)* (2019-06-01), Vol. NA. ACL, Minneapolis, Minnesota, NA. https://doi.org/10.18653/v1/n19-1370

[55] Ming Jiang and Jana Diesner. 2019. A Constituency Parsing Tree based Method for Relation Extraction from Abstracts of Scholarly Publications. In *Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13)* (2019-11-01), Vol. NA. ACL, Hong Kong, NA. https://doi.org/10.18653/v1/d19-5323
[56] Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. 2024. Large Language Models on Graphs: A Comprehensive Survey. *IEEE Trans. on Knowl. and Data Eng.* 36, 12 (Dec. 2024), 8622–8642. https://doi.org/10.1109/TKDE.2024.3469578

[57] Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenIE: Generative Information Extraction. In *Proceedings of the 2022 Conference of the NAACL: Human Language Technologies*. ACL, Seattle, United States, 4626–4643. https://doi.org/10.18653/v1/2022.naacl-main.342

[58] Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction. In *Proceedings of the 2023 Conference on EMNLP Processing* (2023-03-07). ACL, Singapore, 1555–1574. https://doi.org/10.18653/v1/2023.emnlp-main.96

[59] Barbara Kitchenham, Rialette Pretorius, David Budgen, O. Pearl Brereton, Mark Turner, Mahmood Niazi, and Stephen Linkman. 2010. Systematic literature reviews in software engineering – A tertiary study. *Information and Software Technology* 52, 8 (2010), 792–805. https://doi.org/10.1016/j.infsof.2010.03.006

[60] Barbara Ann Kitchenham and Stuart Charters. 2007. *Guidelines for performing Systematic Literature Reviews in Software Engineering*. Technical Report EBSE 2007-001. Keele University and Durham University Joint Report. https://www.elsevier.com/%5F%5Fdata/promis%5Fmisc/525444systematicreviewsguide.pdf

[61] Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I. Oprea, and Olivier Taboureau. 2016. ChemProt-3.0: a global chemical biology diseases mapping. *Database (Oxford)* 2016 (2016), bav123. https://doi.org/10.1093/database/bav123

[62] Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. *Database* 2016 (05 2016), baw068. https://doi.org/10.1093/database/baw068 arXiv:https://academic.oup.com/database/article-pdf/doi/10.1093/database/baw068/8224483/baw068.pdf

[63] Sha Li, Heng Ji, and Jiawei Han. 2021. Document-Level Event Argument Extraction by Conditional Generation. In *Proceedings of the 2021 Conference of the NAACL: Human Language Technologies*. ACL, Online, 894–908. https://doi.org/10.18653/v1/2021.naacl-main.69

[64] Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2004. *Cognitive Computation Group*. Cognitive Computation Group. https://cogcomp.seas.upenn.edu/page/resource%5Fview/43

[65] Hongtao Lin, Jun Yan, Meng Qu, and Xiang Ren. 2019. Learning Dual Retrieval Module for Semi-supervised Relation Extraction. In *WWW ’19: WWW Conference* (2019-02-20). ACM, New York, NY, USA, 1073–1083. https://doi.org/10.1145/3308558.3313573

[66] Jia Wu Lingfeng Zhong, Qian Li, Hao Peng, and Xindong Wu. 2023. A Comprehensive Survey on Automatic Knowledge Graph Construction. *ACM Comput. Surv.* 56, 4, Article 94 (2023), 62 pages. https://doi.org/10.1145/3618295

[67] Sijia Liu, Li-Wei Wang, Vipin Chaudhary, and Hongfang Liu. 2019. Attention Neural Model for Temporal Relation Extraction. In *Proceedings of the 2nd Clinical Natural Language Processing Workshop*, Vol. NA. ACL, Minneapolis, Minnesota, USA, NA. https://doi.org/10.18653/v1/w19-1917

[68] Colin Lockard, Prashant Shiralkar, Xin Luna Dong, and Hannaneh Hajishirzi. 2020. ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages. In *Proceedings of the 58th Annual Meeting of the ACL*, Vol. nan. nan, Online, 8105–8117. https://doi.org/10.18653/V1/2020.ACL-MAIN.721

[69] Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified Structure Generation for Universal Information Extraction. In *Proceedings ACL 2022*. ACL, Dublin, Ireland, 5755–5772. https://doi.org/10.18653/v1/2022.acl-long.395

[70] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In *Proceedings of the 2018 Conference on EMNLP Processing* (2018-10), Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). ACL, Brussels, Belgium, 3219–3232. https://doi.org/10.18653/v1/D18-1360

[71] Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, and Haofen Wang. 2025. Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities. *arXiv preprint arXiv:2505.20099* (2025), nan.

[72] Youmi Ma, An Wang, and Naoaki Okazaki. 2023. DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction. In *Proceedings of the 17th Conference of the European Chapter of the ACL* (2023-02-17). ACL, Dubrovnik, Croatia, 1971–1983. https://doi.org/10.18653/v1/2023.eacl-main.145

[73] Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, and Jing Shao. 2022. Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction. In *Proceedings ACL 2022*. ACL, Dublin, Ireland, 6759–6774. https://doi.org/10.18653/v1/2022.acl-long.466

[74] Hieu Man, Nghia Trung Ngo, Linh Ngo Van 0001, and Thien Huu Nguyen. 2022. Selecting Optimal Context Sentences for Event-Event Relation Extraction., In Proceedings of the AAAI Conference on Artificial Intelligence. *Proceedings of the AAAI Conference on Artificial Intelligence* nan, 11058–11066. https://doi.org/10.1609/AAAI.V36I10.21354

[75] Diego Marcheggiani and Ivan Titov. 2016. Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations. *Transactions of the ACL* 4 (2016), 231–244. https://doi.org/10.1162/tacl_a_00095

[76] Jose L. Martinez-Rodriguez, Aidan Hogan, and Ivan Lopez-Arevalo. 2020. Information extraction meets the Semantic Web: A survey. *Semant. Web* 11, 2 (2020), 255–335. https://doi.org/10.3233/SW-180333

[77] Filipe Mesquita, Matteo Cannaviccio, Jordan Schmidek, Paramita Mirza, and Denilson Barbosa. 2019. KnowledgeNet: A Benchmark Dataset for Knowledge Base Population. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*. ACL, Hong Kong, China, 749–758. https://doi.org/10.18653/v1/d19-1069

[78] K. K. Mueen Ahmed and Bandar E. Al Dhubaib. 2011. Zotero: A bibliographic assistant to researcher. *J. Pharmacol Pharmacother* 2, 4 (2011), 304–305. https://doi.org/10.4103/0976-500X.85940
[79] Zara Nasar, Syed Waqar Jaffry, and Muhammad Kamran Malik. 2022. Named Entity Recognition and Relation Extraction: State-of-the-Art. *ACM Comput. Surv.* 54, 1 (2022), 20:1–20:39. https://doi.org/10.1145/3445965

[80] Tapas Nayak, Navonil Majumder, Pawan Goyal, and Soujanya Poria. 2021. Deep Neural Approaches to Relation Triplets Extraction: a Comprehensive Survey. *Cognitive Computation* 13, 5 (Aug. 2021), 1215–1232. https://doi.org/10.1007/s12559-021-09917-7

[81] Tapas Nayak and Hwee Tou Ng. 2020. Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction. In *Proceedings of the AAAI Conference on Artificial Intelligence* (2020-04-03), Vol. 34. AAAI, Online, 8528–8535. https://doi.org/10.1609/aaai.v34i05.6374

[82] Qiang Ning, Zhili Feng, and Dan Roth. 2019. A Structured Learning Approach to Temporal Relation Extraction. In *Proceedings of the 2017 Conference on EMNLP Processing* (2019-06-12), Vol. NA. ACL, Copenhagen, Denmark, NA. https://doi.org/10.18653/v1/d17-1108

[83] Qiang Ning, Hao Wu, and Dan Roth. 2018. A Multi-Axis Annotation Scheme for Event Temporal Relations. In *Proceedings of the 56th Annual Meeting of the ACL (Volume 1: Long Papers)*. ACL, Melbourne, Australia, 1318–1328. https://doi.org/10.18653/v1/P18-1122

[84] Abiola Obamuyide and Andreas Vlachos. 2019. Meta-Learning Improves Lifelong Relation Extraction. In *Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)* (2019-08-01), Vol. NA. ACL, Florence, Italy, NA. https://doi.org/10.18653/v1/w19-4326

[85] Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The GENIA corpus: an annotated research abstract corpus in molecular biology domain. In *Proceedings of the second international conference on Human Language Technology Research (HLT '02)*. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 82–86. https://doi.org/10.5555/1289189.1289260

[86] Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omelivanenko, Wen Zhang, Matteo Lissandrini, Russa Biswas, Gerard de Melo, Angela Bonifati, Edlira Vakaj, Mauro Dragoni, and Damien Graux. 2023. Large Language Models and Knowledge Graphs: Opportunities and Challenges. *Transactions on Graph Data and Knowledge* 1, 1 (2023), 2:1–2:38. https://doi.org/10.4230/TGDK.1.1.2

[87] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying Large Language Models and Knowledge Graphs: A Roadmap. *IEEE Transactions on Knowledge and Data Engineering* 36, 7 (2024), 3580–3599. https://doi.org/10.1109/TKDE.2024.3352100

[88] Yihe Pang, Jie Liu, Lizhen Liu, Zhengtao Yu, and Kai Zhang. 2019. A Deep Neural Network Model for Joint Entity and Relation Extraction. *IEEE Access* 7 (2019), 179143–179150. https://doi.org/10.1109/ACCESS.2019.2949086

[89] Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In *Proceedings of the 2020 Conference on EMNLP Processing (EMNLP)* (2020-10-05), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2020.emnlp-main.298

[90] Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In *Proceedings of the 18th BioNLP Workshop and Shared Task*, Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). ACL, Florence, Italy, 58–65. https://doi.org/10.18653/v1/W19-5006

[91] Silvio Peroni and David Shotton. 2020. OpenCitations, an infrastructure organization for open scholarship. *Quantitative Science Studies* 1, 1 (2020), 428–444. https://doi.org/10.1162/qss_a_00023

[92] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2020. KILT: a Benchmark for Knowledge Intensive Language Tasks. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2021.naacl-main.200

[93] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language Models as Knowledge Bases?. In *Proceedings of the 2019 Conference on EMNLP Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019*, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). ACL, Hong Kong, China, 2463–2473. https://doi.org/10.18653/V1/D19-1250

[94] Pustejovsky, James, Verhagen, Marc, Sauri, Roser, Littman, Jessica, Gaizauskas, Robert, Katz, Graham, Mani, Inderjeet, Knippen, Robert, and Setzer, Andrea. 2006. TimeBank 1.2. https://doi.org/10.35111/09B1-5N19

[95] Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie zhou. 2020. ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning. In *Proceedings of the 59th Annual Meeting of the ACL and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)* (2020-12-30), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2021.acl-long.260

[96] Simon Razniewski, Andrew Yates, Nora Kassner, and Gerhard Weikum. 2021. Language Models As or For Knowledge Bases. arXiv:2110.04888 [cs.CL] https://arxiv.org/abs/2110.04888

[97] Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Proceedings of the 2010th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part III (2010-09-20) (ECMLPKDD'10)*. Springer-Verlag, Berlin, Heidelberg, 148–163. https://doi.org/10.1007/978-3-642-15939-8_10

[98] Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, Owen Cornec, and Alfio Massimiliano Gliozzo. 2022. KnowGL: Knowledge Generation and Linking from Text. *Proceedings of the AAAI Conference on Artificial Intelligence* NA (2022), NA. https://doi.org/10.1609/aaai.v37i13.27084

[99] Dongyu Ru, Changzhi Sun, Jiangtao Feng, Lin Qiu, Hao Zhou 0012, Weinan Zhang 0001, Yong Yu 0001, and Lei Li 0005. 2021. Learning Logic Rules for Document-Level Relation Extraction. In *Proceedings of the 2021 Conference on EMNLP Processing*, Vol. nan. nan. Online and Punta Cana, Dominican Republic, 1239–1250. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.95
[100] Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021. Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes. *Transactions of the ACL* 9 (2021), 691–706. https://doi.org/10.1162/tacl_a_00392

[101] Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction. In *Proceedings of the 2021 Conference on EMNLP Processing*, Vol. nan. nan, Online and Punta Cana, Dominican Republic, 1199–1212. https://doi.org/10.18653/v1/2021.EMNLP-MAIN.92

[102] Taneeya Satyapanich, Francis Ferraro, and Tim Finin. 2020. CASIE: Extracting Cybersecurity Event Information from Text. *Proceedings of the AAAI Conference on Artificial Intelligence* 34, 05 (Apr. 2020), 8749–8757. https://doi.org/10.1609/aaai.v34i05.6401

[103] Phillip Schneider, Tim Schopf, Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022. A Decade of Knowledge Graphs in Natural Language Processing: A Survey. In *Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the ACL and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)* (2022-11). ACL, Online only, 601–614. https://doi.org/10.18653/v1/2022.aacl-main.46

[104] Yuming Shang. 2019. Are Noisy Sentences Useless for Distant Supervised Relation Extraction?, In AAAI (2019). *Proceedings of the AAAI Conference on Artificial Intelligence* nan, 8799–8806. https://doi.org/10.1609/AAAI.3V4I05.6407

[105] Daniil Sorokin and Iryna Gurevych. 2017. Context-Aware Representations for Knowledge Base Relation Extraction. In *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*. ACL, Copenhagen, Denmark, 1784–1789. https://doi.org/10.18653/v1/D17-1188

[106] George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-TACRED: Addressing Shortcomings of the TACRED Dataset. In *The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)* (2021-04-16). AAAI, Online, 13843–13850. https://doi.org/10.48550/arXiv.2104.08398

[107] Peng Su, Yifan Peng, and K. Vijay-Shanker. 2021. Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction. In *Proceedings of the 20th Workshop on Biomedical Language Processing* (2021-04-28), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2021.bionlp-1.1

[108] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In *Proceedings of the 16th International Conference on WWW (WWW '07)*. Association for Computing Machinery, Banff, Alberta, Canada, 697–706. https://doi.org/10.1145/1242572.1242667

[109] Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2001. Progressive Multi-task Learning with Controlled Information Flow for Joint Entity and Relation Extraction. *Proceedings of the AAAI Conference on Artificial Intelligence* nan, 15 (May 2021), 13851–13859. https://doi.org/10.1609/AAAI.V35I15.17632

[110] Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2020. Recurrent Interaction Network for Jointly Extracting Entities and Classifying Relations. In *Proceedings of the 2020 Conference on EMNLP Processing (EMNLP)* (2020-05-01), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2020.emnlp-main.304

[111] Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. 2019. A hierarchical framework for relation extraction with reinforcement learning. In *Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence* (2019-01-27) (AAAI’19/IAAI’19/EAAI’19). AAAI Press, Honolulu, Hawaii, USA, 7072–7079. https://doi.org/10.1609/aaai.v33i01.33017072

[112] Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022. Revisiting DocRED – Addressing the False Negative Problem in Relation Extraction. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*. ACL, Abu Dhabi, United Arab Emirates, 8472–8487. https://doi.org/10.48550/arxiv.2205.12696

[113] Xingwei Tan, Gabriele Pergola, and Yulan He. 2021. Extracting Event Temporal Relations via Hyperbolic Geometry. In *Proceedings of the 2021 Conference on EMNLP Processing* (2021-09-12), Vol. NA. ACL, Online and Punta Cana, Dominican Republic, NA. https://doi.org/10.18653/v1/2021.emnlp-main.636

[114] Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In *Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003*. ACL, Online, 142–147. https://doi.org/10.3115/1119176.1119195

[115] Thy Thy Tran, Phong Le, and Sophia Ananiadou. 2020. Revisiting Unsupervised Relation Extraction. In *Proceedings of the 58th Annual Meeting of the ACL*, Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2020.acl-main.669

[116] Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural Relation Extraction for Knowledge Base Enrichment. In *Proceedings of the 57th Annual Meeting of the ACL* (2019-07). ACL, Florence, Italy, 229–240. https://doi.org/10.18653/v1/p19-1023

[117] Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations. In *Second Joint Conference on Lexical and Computational Semantics (SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)* (2013-06), Suresh Manandhar and Deniz Yuret (Eds.). ACL, Atlanta, Georgia, USA, 1–9. https://aclanthology.org/S13-2001/

[118] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In *Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17)*. Curran Associates Inc., Long Beach, California, USA, 6000–6010.

[119] Severine Verlinden, Klim Zaporojets, Johannes Deleu, Thomas Demeester, and Chris Develder. 2021. Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolution. In *ACL-IJCNLP 2021*, Vol. nan. nan, Online, 1952–1957. https://doi.org/10.18653/v1/2021.FINDINGS-ACL.171
[120] Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM 57, 10 (Sept. 2014), 78–85. https://doi.org/10.1145/2629489

[121] David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, Relation, and Event Extraction with Contextualized Span Representations. In Proceedings of the 2019 Conference on EMNLP Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Vol. nan. nan, Hong Kong, China, 5783–5788. https://doi.org/10.18653/v1/D19-1585

[122] Walker, Christopher, Strassel, Stephanie, Medero, Julie, and Maeda, Kazuaki. 2006. ACE 2005 Multilingual Training Corpus. https://doi.org/10.3511/MWXC-VH88

[123] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (Eds.). ACL, Brussels, Belgium, 353–355. https://doi.org/10.18653/v1/W18-5446

[124] Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022. DeepStruct: Pretraining of Language Models for Structure Prediction. In ACL 2022 (2022-05-21), Vol. NA. ACL, Dublin, Ireland, NA. https://doi.org/10.18653/v1/2022.findings-acl.67

[125] Hongbo Wang, Weimin Xiong, YiFan Song, Dawei Zhu, Yu Xia, and Sujian Li. 2023. DocRED-FE: A Document-Level Fine-Grained Entity And Relation Extraction Dataset. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, Online, 1–5. https://doi.org/10.1109/ICASSP49357.2023.10095786

[126] Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence Embedding Alignment for Lifelong Relation Extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). ACL, Minneapolis, Minnesota, 796–806. https://doi.org/10.18653/v1/N19-1086

[127] Jue Wang and Wei Lu. 2020. Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders. In Proceedings of the 2020 Conference on EMNLP Processing. ACL, Online, 1706–1721. https://doi.org/10.18653/v1/2020.emnlp-main.133

[128] Jiahui Wang, Kun Yue, and Liang Duan. 2023. Models and Techniques for Domain Relation Extraction: A Survey. Journal of Data Science and Intelligent Systems 1, 2 (May 2023), 65–82. https://doi.org/10.47852/bonviewJDSIS3202973

[129] Luyu Wang, Yujia Li, Ozlem Aslan, and Oriol Vinyals. 2021. WikiGraphs: A Wikipedia Text - Knowledge Graph Paired Dataset. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15). ACL, Mexico City, Mexico, 67–82. https://doi.org/10.18653/v1/11.textgraphs-1.7

[130] Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie zhou. 2022. MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction. In Proceedings of the 2022 Conference on EMNLP Processing. ACL, Abu Dhabi, United Arab Emirates, 926–941. https://doi.org/10.18653/v1/2022.emnlp-main.60

[131] Yuqi Wang, Wei Wang, Qi Chen, Kaizhu Huang, Anh Nguyen, Suparna De, and Amir Hussain. 2023. Fusing external knowledge resources for natural language understanding techniques: A survey. Information Fusion 92 (2023), 190–204. https://doi.org/10.1016/j.inffus.2022.11.025

[132] Tongtong Wu, Xuekai Li, Yuan-Fang Li, Reza Haffari, Guilin Qi, Yujin Zhu, and Guoqiang Xu. 2021. Curriculum-Meta Learning for Order-Robust Continual Relation Extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. nan. nan, Online, 10363–10369. https://doi.org/10.1609/AAAI.35112.17241

[133] Ye Wu, Ruibang Luo, Henry C. M. Leung, Hing-Fung Ting, and Tak-Wah Lam. 2019. RENET: A Deep Learning Approach for Extracting Gene-Disease Associations from Literature. In Research in Computational Molecular Biology, Lenore J. Cowen (Ed.). Springer International Publishing, Cham, 272–284. https://doi.org/10.1007/978-3-030-17083-7_17

[134] Wei Xiang and Bang Wang. 2019. A Survey of Event Extraction From Text. IEEE Access 7 (2019), 173111–173137. https://doi.org/10.1109/ACCESS.2019.2956831

[135] Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han 0001. 2022. Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion. In ACL 2022, Vol. nan. nan, Dublin, Ireland, 257–268. https://doi.org/10.18653/v1/2022.FINDINGS-ACL.23

[136] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention. In Proceedings of the 2020 Conference on EMNLP Processing (EMNLP) (2020-10-02), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2020.emnlp-main.523

[137] Jihong Yan, Chengyu Wang, Wenliang Cheng, Ming Gao, and Aoying Zhou. 2018. A retrospective of knowledge graphs. Frontiers of Computer Science 12 (2018), 55–74. https://api semanticscholar.org/CorpusID:3395049

[138] Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. 2021. A Partition Filter Network for Joint Entity and Relation Extraction. In Proceedings of the 2021 Conference on EMNLP Processing (2021-08-27), Vol. NA. ACL, Online and Punta Cana, Dominican Republic, NA. https://doi.org/10.18653/v1/2021.emnlp-main.17

[139] Yuan YAO, Jiaju Du, Yankai Lin, Peng Li, Zhiyuan Liu, Jie zhou, and Maosong Sun. 2021. CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild. In Proceedings of the 2021 Conference on EMNLP Processing, Vol. NA. ACL, Online and Punta Cana, Dominican Republic, NA. https://doi.org/10.18653/v1/2021.emnlp-main.366

[140] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. In Proceedings of the 57th Annual Meeting of the ACL. ACL, Florence, Italy, 764–777. https://doi.org/10.18653/v1/p19-1074
[141] Deming Ye, Yankai Lin, Jiaju Du, Zheng-Hao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on EMNLP Processing (EMNLP) (2020-04-15), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2020.emnlp-main.582

[142] Hongbin Ye, Ningyu Zhang 0001, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang 0004, and Huajun Chen. 2020. Contrastive Triple Extraction with Generative Transformer. In *The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)*, Vol. nan. nan. Online, 14257-14265. https://doi.org/10.1609/AAAI.35116.17677

[143] Hongbin Ye, Ningyu Zhang, Hui Chen, and Huajun Chen. 2022. Generative Knowledge Graph Construction: A Review. In Proceedings of the 2022 Conference on EMNLP Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). ACL, Abu Dhabi, United Arab Emirates, 1-17. https://doi.org/10.18653/v1/2022.emnlp-main.1

[144] Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Distant Supervision Relation Extraction with Intra-Bag and Inter-Bag Attentions. In Proceedings of the 2019 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long and Short Papers) (2019-03-30), Vol. NA. ACL, Minneapolis, Minnesota, NA. https://doi.org/10.18653/v1/n19-1288

[145] Lucas Emanuel Silva e Oliveira Yohan Bonescki Gumiel, V. Claveau, N. Grabar, E. Paraiso, C. Moro, and D. Carvalho. 2021. Temporal Relation Extraction in Clinical Texts. ACM Comput. Surv. 54 (2021), 1 - 36. Issue nan. https://doi.org/10.1145/3462475

[146] Yue Yuan, Xiaofei Zhou, Shirui Pan, Qiannan Zhu, Zeliang Song, and Li Guo. 2020. A Relation-Specific Attention Network for Joint Entity and Relation Extraction. In Proceedings of the Twenty-Ninth IJCAI (2020-07-09), Vol. 4. ACL. Online, 4054-4060. https://doi.org/10.24963/ijcai.2020/561

[147] Klim Zaporojets, Johannes Deleu, Chris Develder, and Thomas Demeester. 2021. DWIE: An entity-centric dataset for multi-task document-level information extraction. Information Processing & Management 58, 4 (2021), 102563. https://doi.org/10.1016/j.ipm.2021.102563

[148] Daojian Zeng, Ranran Haoran Zhang, and Qianying Liu. 2019. CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning. In AAAI Technical Track: Natural Language Processing (2019-11-24). AAAI, Online, 4032-4038. https://doi.org/10.1609/aaai.v34i05.6495

[149] Shuang Zeng, Yuting Wu, and Baobao Chang. 2021. SIRE: Separate Intra- and Inter-sentential Reasoning for Document-level Relation Extraction. In ACL-IJCNLP 2021 (2021-06-03), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2021.findings-acl.47

[150] Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Learning the Extraction Order of Multiple Relational Facts in a Sentence with Reinforcement Learning. In Proceedings of the 2019 Conference on EMNLP Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019-11). ACL, Hong Kong, China, 367-377. https://doi.org/10.18653/v1/D19-1035

[151] Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism. In Proceedings of the 56th Annual Meeting of the ACL (Volume 1: Long Papers). ACL, Melbourne, Australia, 506-514. https://doi.org/10.18653/v1/P18-1047

[152] Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level Relation Extraction as Semantic Segmentation. In Proceedings of the Thirtieth IJCAI (2021-06-07). IJCAI, Online, 3999-4006. https://doi.org/10.24963/ijcai.2021/551

[153] Haodong Zhao, Ruifang He, Mengnan Xiao, and Jing Xu. 2023. Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition. In Proceedings of the 61st Annual Meeting of the ACL (Volume 1: Long Papers). ACL, Toronto, Canada, 6477-6492. https://doi.org/10.18653/v1/2023.acl-long.357

[154] Shan Zhao, Minghao Hu, Zhiping Cai, and Fang Liu. 2020. Modeling Dense Cross-Modal Interactions for Joint Entity-Relation Extraction. In IJCAI (2020-07-01), Vol. NA. IJCAI, Online, NA. https://doi.org/10.24963/ijcai.2020/558

[155] Tianyang Zhao, Zhao Yan, Yunbo Cao, and Zhoujun Li. 2020. Asking Effective and Diverse Questions: A Machine Reading Comprehension based Framework for Joint Entity-Relation Extraction. In Proceedings of the Twenty-Ninth IJCAI (2020-07). IJCAI, Yokohama, Japan, 3948-3954. https://doi.org/10.24963/ijcai.2020/546

[156] Xiaoyan Zhao, Yang Deng, Min Yang, Lingzhi Wang, Rui Zhang, Hong Cheng, Wai Lam, Ying Shen, and Ruifeng Xu. 2024. A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers. ACM Comput. Surv. 56, 11, Article 293 (July 2024), 39 pages. https://doi.org/10.1145/3674501

[157] Danna Zheng, Mirella Lapata, and Jeff Z. Pan. 2024. How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency. arXiv:2407.13578 [cs.CL] https://arxiv.org/abs/2407.13578

[158] Zexuan Zhong and Danqi Chen. 2020. A Frustratingly Easy Approach for Entity and Relation Extraction. In Proceedings of the 2021 Conference of the NAACL: Human Language Technologies (2020-10-24), Vol. NA. ACL, Online, NA. https://doi.org/10.18653/v1/2021.naACL-main.5

[159] Xueyang Zhou, Qiming Fu, Jianping Chen, Lanhui Liu, Yunzhe Wang, You Lu, and Hongjie Wu. 2023. Extracting biomedical relation from cross-sentence text using syntactic dependency graph attention network. Journal of Biomedical Informatics 144 (2023), 104445. https://doi.org/10.1016/j.jbi.2023.104445

[160] Yucheng Zhou, Tao Shen, Xiubo Geng, Guodong Long, and Daxin Jiang. 2022. ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification. In Proceedings ACL 2022. ACL, Dublin, Ireland, 2559-2575. https://doi.org/10.18653/v1/2022.acl-long.183

[161] Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2024. LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities. World Wide Web 27, 5 (Aug. 2024), 23 pages. https://doi.org/10.1007/s11280-024-01297-w