Linking Surface Facts to Large-Scale Knowledge Graphs
Authors: Gorjan Radevski
Kiril Gashteovski
Chia-Chien Hung
Carolin Lawrence
Goran Glavaš
Venue: EMNLP 2023 Main (2023)
Keywords: information extraction
fact linking
knowledge graphs
open information extraction
TL;DR: We propose a new task of KG fact linking (with data, benchmark and baseline models) that links surface-form (free-text) OIE facts to large scale Knowledge Graphs.

Abstract: Open Information Extraction (OIE) methods extract facts from natural language text in the form of ("subject"; "relation"; "object") triples. These facts are, however, merely surface forms, the ambiguity of which impedes their downstream usage; e.g., the surface phrase "Michael Jordan" may refer to either the former basketball player or the university professor. Knowledge Graphs (KGs), on the other hand, contain facts in a canonical (i.e., unambiguous) form, but their coverage is limited by a static schema (i.e., a fixed set of entities and predicates). To bridge this gap, we need the best of both worlds: (i) high coverage of free-text OIEs, and (ii) semantic precision (i.e., monosemy) of KGs. In order to achieve this goal, we propose a new benchmark with novel evaluation protocols that can, for example, measure fact linking performance on a granular triple slot level, while also measuring if a system has the ability to recognize that a surface form has no match in the existing KG. Our extensive evaluation of several baselines show that detection of out-of-KG entities and predicates is more difficult than accurate linking to existing ones, thus calling for more research efforts on this difficult task. We publicly release all resources (data, benchmark and code) on https://github.com/nec-research/fact-linking.

Linking Surface Facts to Large-Scale Knowledge Graphs
Gorjan Radevski 12, Kiril Gashteovski Chia-Chien Hung',
Carolin Lawrence', Goran Glavaš*
NEC Laboratories Europe, Heidelberg, Germany: "KU Leuven, Leuven, Belgium;
SCAIR, Ss. Cyril and Methodius University, Skopje, North Macedonia;
*CAIDAS, University of Würzburg, Würzburg, Germany
ffrsstname, lastname@necll.a ab. eu, "gorjan. podevski@kuleuveer be, de
gooann
Abstract
summarization (Ribeiro al., 2022), question an- swering (QA) (Wu al., 2022), event extraction (Dukić ct al., 2023), text clustering (Viswamathan et al., 2023) and video grounding (Nan et al., 2021). OIE triples, however, consist of surface-form enti- ties and relations, which are frequently ambiguous (e.g.. in l and (2, the entity mention "Michael Jordan" may refer to several entities, e.g. the bas- ketball player or the computer scientist). Resolv- ing such ambiguities, by linking the OIE triple slots to inventories of unambiguous concepts, im- proves their downstream utility, e.g., in QA (Sax- ena et al., 2020), automatic medical diagnosing (Li et al., 2022) or dialogue (Joko et al., 2021).
Open Information Extraction (OIE) methods extract facts from natural language text in the form of ("subject"; "relation"; "objecr") triples. These facts are, however, merely sur- face forms, the ambiguity of which impedes their downstream usage; e.g., the surface phrase " "Michael Jordan" " may refer either the for- mer basketball player or the university profes- sor. Knowledge Graphs (KGs), on the other hand, contain facts in canonical (i.e., unam- biguous) form, but their coverage is limited by a static schema (i.e., a fixed set of entities and predicates). To bridge this gap, we need the best of both worlds: (i) high coverage of free-text OIEs, and (ii) semantic precision (i.e., monosemy) of KGs. In order to achieve this goal, we propose a new benchmark with novel evaluation protocols that can, for example, mea- sure fact linking performance on a granular triple slot level, while also measuring if sys- tem has the ability to recognize that surface form has no match in the existing KG. Our ex- tensive evaluation of several baselines shows that detection of out-of KG entities and predi- cates is more di fffcultt than accurate linking to existing ones, thus calling for more research ef- forts on this difficult task. We publicly release all resources (data, benchmark and code)!
Knowledge Graphs (KGs), on the other hand, are inventories of canonical facts in the form of (subject; predicate: object)-triples, where each slot is unique (i.e., unambiguous) concept (Vrandečié, 2012). KGs are, however, limited by their own static and often hand-crafted schema (i.e., fixed set of entities and predicates). As a consequence, meth- ods that directly extract canonical KG triples from text (Trisedya et al., 2019; Josifoski et al., 2022), preemptively discard any information outside of the reference KG schema. Acknowledging this, Ye et al. (2023) recently proposed schema- addetalle KG construction, with the goal of extracting infor- mation for KG with an evolving schema. Ye al. (2023) conclude that OIE methods indeed extract meaningful new knowledge for such KGs, however, they point precisely to the ambiguity of the surface forms as the major obstacle.
1 Introduction
Open Information Extraction (OIE) methods ex- tract surface ("subject"; "relation"; " "object")- triples from natural language text in a schema-free free manner (Banko et al., 2007). For example, given the sentence "Michael Jordan, who grew up in Wilmington, played for Chicago Bulls", an OIE sys- tem should extract the triples (i.e., surface facts): I -("Michael Jordan": "played for"; "Chicago Bulls") ") and to =(" Michael Jordan": " "grew up in "illiington" ") The output of such systems used in a diverse range of downstream tasks, including
To combine the best of both worlds, we need to bridge the gap between the schema-fire (but am- biguous) surface facts extracted from text and the schema-fixed (but unambiguous) KG knowledge. However, existing benchmarks and models only partially address the problem. One line of work (Zhang et al., 2019; Jiang et al., 2021; Wood et al., 2021) assumes a setting, which is arguably unre- alistic, because the OIE entity slots are a priori

---

Open Information Extractions (OIE)
Knowledge Graphs (KG) represent canonical concepts
represent ambiguous surface forms
Transductive
mamiie spoig
Chicago
Inductive
loo (041-42)
Bulla [12128 103)
Polysemous
4
(Michall Jordan");
ppayed for
"Chicago Bulls"))
KG
(Pesa)
Other KG nodes
f= (==[[MMae
(MMccaall Jordan grew up In
Wilmington
Pigtiononn (Q66940C)
Jordan (C8308205)
(0165758)
Figure I: Open Information Extractions (OIEs) represent free-text surface-form triples, which may be ambiguous. Knowledge Graphs (KGs) represent canonical, unambiguous concepts, yet are limited by a hand-crafted schema. By linking OIEs to KG facts we bridge the gap between the schema-free (but ambiguous) surface facts extracted from text and the sshema-fixed (but unambiguous) KG knowledge. Our benchmark addresses all aspects of the OIE-to-KG linking. In transductive evaluation, the KG facts consist of entities seen during training, while in inductive evaluation the KG facts contain entities strietly outside of the training data. Polysemous OIEs match to multiple KG concepts (c.g.. "Michael Jordan"). Finally, some OIEs might refer to Out-of-KG entities or predicates.
linked to KG entities and thus only addresses rela- tion linking. A second line of research (Trisedya al., 2019; Cabot and Navigli, 2021; Josifoski et al., 2022; Sakor al., 2020; Elsahar al., 2018) entails canonical KG triples directly from text para- graphs, bypassing OIE, which makes such mod- els tied to the fixed KG schema. Critically, all these works conjecture that each OIE slot has corresponding KG entry. Gashteovski et al. (2019) showed that this is hardly ever the case in practice.
problem, which requires more research attention.
Fact Linking: Problem Statement
For a given surface-form OIE triple fi ("s"; "r";""""" "o"). the goal is to link each slot to a canonical concept in KG (if the correspond- ing concept exists in the KG): "s" - e1 E E; "n" -> P E P; "0" + E E, with & and P as the (fixed) sets of KG entities and predicates. Importuntly, our problem definition (and conse- quently evaluation) focuses on linking at the fact level, where each OIE slot is contextualized with the other two OIE slots.
Contributions and Findings. We move away from the unrealistic and incomplete assumptions of prior work and propose I a novel large-scale benchmark for OIE-to-KG linking; 2 multifaceted evaluation protocols that cover all aspects of linking OIE facts to KGs (for an overview, see Fig. 1); 3 several strong baselines, inspired by state-of-the-an entity linking (Wu et al., 2019) and cross-modal retrieval (Miech et al., 2021; Geigle et al., 2022) approaches.
Additionally and crucially, we want linking mod- els that can assign an empty set to OIE surface forms (e.g.. "8" - () when they refer to concepts not present in the KG (i.e., out-of-KCC KG concepts). To enable a realistic setup for linking free-text OIE triples to a KG, we build a benchmark which con- siders four different facets (see Fig. 1).
Through our experimental study, we found that the methods (i) perform well transductively but (ii) their performance deteriorates in an inductive evaluation. Further, we find that (iii) a dedicated OIE-to-KG fact re-ranking model improves the linking performance of both inductive and poly- semous OIEs, and that (iv) we obtain high perfor- mance by training models solely on a synthetic variant of our dataset (i.e., with the KG as the only human- auman--nnottted data). Lastly, investigate the largely underexplored issue of detecting Out-of- Knowledge-Graph extractions. We show that (v) is possible to detect Out-of-KC entities to an extent, however, the same does not hold for predicates: a task that our experiments identify as a difficult open
Transductive Fact Linking. In transductive link- ing, we measure how well the models link OIEs to KG facts consisting of entities and predicates seen during training (as components of training KG facts). Note that the testing KG facts (as whole triples) are not in the training data. Consider, for example, the extraction t in Fig. 1. In the trans- ductive linking task, the mentions "played for" and "Chicago Bulls" refer to the KG predicate (P54) and entity (Q128109 respectively; both seen by the model as part of other training KG facts. However, the whole triple (Q41421: P54: Q128109) to which
2 AAlernnttvvel. additional context can be included, c.g., the provenance from which the OIE surface fact is obtained.

---

& is linked was not part of the training data. Inductive Fact Linking. The inductive setup eval- uates the linking to KG facts that consist of entities that are not seen during the training of the models. In other words, this setup tests the generalization of fact linking models over entities. In Fig. given 12 as a test instance, the OIE entity " "Wilmington" " is inductive as it refers to a KG entity (Q659400) that is not part of any training KG fact.
Sentence-to-KG Fact Datasets. We build FaLB benchmarks for two such existing datasets: REBEL (Cabot and Navigli, 2021) and SynthIE (Josifoski et al., 2023). REBEL is built from Wikipedia ab- stracts, where the manually hyperlinked entities in each sentence are linked to Wikidata entities, thus making them golden. To match the sentence with KG fact, for each pair of KG entities ej and ej within the sentence, REBEL obtains all KG pred- icates P%: such that (ei, Pk: Ej) or (ej, Pk, eÿ) ex- ist. However, two Wikidata entity nodes connected with a predicate (thus constituting a KG fact) do not guarantee the predicate validity in the sentence. For that reason, Cabot and Navigli (2021) use a Natural Language Inference pre-trained RoBERTa (Liu et al., 2019) to filter the predicates not entailed by the Wikipedia sentence; see (Cabot and Navigli, 2021) for details. We also use SyuthE,, which is synthetically generated using a Large Language Model (LLM) (Brown et al., 2020). Given a set of KG facts, the LLM is prompted to generate a sentence that covers the KG facts.3
Polysemous Fact Linking, We focus OIEs for which the "s" and "o" slots ambiguous w.r.t. the KG, i... in isolation they refer to set of KG entities rather than a single entity. The mention "Michael Jordan" " from either i and l2 (Fig. I), in isolation, refers to both the basketball player (Q41421) and the computer scientist (Q3308205). Here, the other two OIE slots offer the disambigua- tion signal that is necessary for successful linking.
Out-of-KG Detection. We introduce a novel out- KG- detection task, in which the models are to recognize that an OIE triple component (i.e., "s", "r" or "o") cannot be linked because they do not have corresponding KG concept (e.g., the relation "grew up in" from the triple t2 in Fig. 1).
Reference KG. We use Wikidata (Vrandečić, 2012) as our reference KG, because REBEL and SynthIE align sentences to Wikidata facts. We lect the subgraph of Wikidata that contains all enti- ties which have a corresponding Wikipedia page, as per Wu et al. (2019). Following Lerer et al. (2019), we additionally filter out the most infrequent enti- ties and predicates, appearing less than times in the whole Wikidata dump. This results in large reference KG with 5, 794, 782 unique entities and 4, 153 unique predicates. We further create two smaller, dataset-specific reference KGs; one for each of the two datasets for which we apply FaLB: REBEL and SynthTE. These Brrchhaark-Rsttriccee KGs (BRKGs) contain only the Wikidata enti- ties and predicates referenced by at least one OIE triple extracted from their respective datasets. The BRKG for REBEL contains 625, 125 entities and 565 predicates, while the one for SynthlE contains 702, 334 entities and 758 predicates.
3 FaLB: Fact Linking Benchmark
We set up an automatic data processing pipeline to derive an OIE-to-KG fact linking benchmark, which supports all four evaluation facets from $2. We refer to both the process (i.e., pipeline) and the resulting benchmark as FaLB. FaLB's input is a dataset with (gold) alignments between natural language sentences and KG facts entailed by the sentence; i.e., each data instance is (sentence, KG fact) pair. Consequently, the creation of FaL. en- tails five design decisions: selection of (i) sentence- to-KG fact dataset(s), and (ii) a reference KG; (iii) generating OIE triples, (iv) high-precision OIE-KG fact alignments, and (v) a data augmentation strat- egy to increase the diversity of the data. Below is example instance of the FaLB dataset:
Example Data Instance from FaLB
Generating OIE Triples. We use four state-of- the-art OIE methods to obtain a set of OIE triples for each of the sentences in the dataset. To increase diversity, we use two of-te-of-thcaasl art rase-based OIE methods: MinIE (Gashteovski et al., 2017) and StanfordOIE (Angeli et al., 2015); and two
Sentence: "M. J., who was born Brooklyn, played for the Bulls."
OIE surface factst =[" J"; "played for"; "the Bulls") ") and
t2 =("M. J.;; "was born in"; "Brooklyn") ")
KG canonical fact identifiers: (Q41421;544 P54; Q128109);
P19; Q18419)
KG text facts: (Michuel Jordan; member of sports team; Chicago
Bulls); (Michacl Jordan; place of birth; Brooklyn)
KG entity aliases: (Q41421: Air Jordan, Michuel Jeffrey Jordan,
'Josifoski et al. (2023) manually evaluated that SynthlF: is of higher quality than REBEL (sce Appendix D for details).
His Aimess); (Q128109: Bulls, The Bulls)...

---

state-of-the-art neural OIE models: MillE (Kotnis et al., 2022c) and MuIti (Ro et al., 2020).
OIE-to-KG Fact Linking Models
Our goal is to link surface-form (*8"; "r"; "0") OIE triples, to canonical Knowledge Graph facts (e1;;; e2), where e1 62 E E, and p E P. Each KG entity or predicate is represented as its surface- form KG label (e.g., "Michael Jordan"), and its KG-provider description (e.g.. "American basket- ball player and businessman"), We henceforth refer to the entities and predicates as KG entries. Intu- itively, since both data streams- the OIE triples and the KG facts-are in natural language, we opt to obtain their representations (i.c., embeddings) with a pre-trained language model. We decouple the OlE-to-KG linking in two steps: pre-ranking and re-ranking (see $4.1 and $4.2, respectively). as is done commonly in entity retrieval (Wu et al., 2019) and image-text matching (Geigle et al., 2022; Li et al., 2021). See Fig. 2 for an overview and Ap- pendix C for implementation details.
High-precision OIE-KG Fact Alignments. Next, we need to match the extracted OIE triples against the set of KG facts associated with the sentences. Crucially, this automatic step needs to have high-precision in order to create high quality benchmark. As per the distant supervision assumption of Mintz et al. (2009), we create alignment t <> f between any OIE triple I and any KG fact f that exactly match (i.e., have identical text form) in subject and object, assuming that the relation of is aligned with the predicate of f. When multiple OIE triples (excluding exact duplicates) 4, 12.... 4 (e.g.. extracted with different OIE systems) match with the same KG fact J. we obtain all & alignments: t1 + f, ..., tx + f. Finally, training alignments <+ f that exist in the test portion. To verify that this strategy indeed has high precision, we randomly sample 100 test instances, and conduct a human-study with two expert annotators. The annotators found 97% correct pairings (inter-annotator agreement of 99%; Kohen's 's kappa of 0.80); see Appendix F for details. We therefore confirm the reliability of this design choice to produce high quality data.
remove all
4.1 Pre-ranking OIE Slots to KG Entries
We denote this model as Ol It aims to generate OIE slot embeddings and KG entry em- beddings, such that an OIE slot embedding yields higher similarity with its aligned KG entry's 's em- bedding compared the other KG entries. There- fore, during training, we contrast the positive pairs against a set of negatives, thereby training the model to generate embeddings for a matching OIE slot and KG entry that lie close in the latent space. The motivation for such formulation is two-fold: (i) the number of entities is large, and could practically grow further, therefore computing the softmax over all KG entities during training is prohibitive; (ii) there may be unseen KG entries that we encounter during inference, therefore posing the problem as a standard classification inevitably leads to the model ignoring them. We use RoBERTa (Liu et al., 2019) to encode the OIE slots and the KG entries.
Data Augmentation to Increase Diversity. Lack of example diversity-most linked OIE entity men- tions exactly match the text of the corresponding KG entities a common problem in existing link- ing benchmarks (Cabot and Navigli, 2021). This stems from strict data curation procedures that aim for high alignment precision, representing a mis- match with practice, where often the entity men- tions do not exactly match the canonical KG entity denotation. To train and evaluate fact linking meth- ods on more complex linking examples (c.g.. link- ing from initials and abbreviations), we augment the data using the additional information available in the reference KG. For each aligmment <> f, we fetch the Wikidata aliases (i.e., non non-canonical deno- tations) for the subject and object entities of f. We then create additional pairs t' <+> ff with augmented OIE triples t' for all possible alias combinations. For example, for the original OIE triple ("Michael Jordan"; ; "played for"; "Chicago Bulls") "), this pro- cess results in augmented triples such as ( ("Air Jor- dan"; "played for"; "Chicago Bulls") ) ("M.J"; "or"" "The Bulls") ") etc.
OIE Embeddings. We first add special tokens to indicate the start of each OIE slot: <SUBJ> for "subject", <REL> for "relation", <QBJ> for "ob- ject". Hence, t1 = ("M. Jordan"; "grew up in"; "Wilmington") ") is represented as "<SUBJ> M. Jor- dan <REL> grew up in <OBJ> Wilmington". We then tokenize the OIE representation, denoted as i,
*We perform the linking in roughly O[EE) [P] (E)) time complexity, where [E] and (P] are the number of KG entities and predicates respectively. Directly encoding whole KG facts would have prohibitive complexity: In the limit, we could have [3] x [P] x [8] KG facts.

---

OIE Slots Pre-Ranker
OIE Fact Re- Re-Ranker
Cosine Similariyy
Classiller
shaned
RoBERTa
RoBERTa
RoBERTa
Michael Jordan
Jordan place of birthn "Bookkyn
place of birtnn
Brooklyn
pisco
barth
Michael
Jandan
Negalivee
Graph
Motanl
Hard Negatives
Figure 2: Left: OIR
Tunker which performs pre-ranking of the OIE slots to KG entries independently (i.e., no context between them is considered). Trained with negatives sampled from the whole KG; Right: Fact which attends between the whole OIE triple and KG fact to output their similarity; Trained with hard- sard-negatives.
and obtain RoBERTa token embeddings. Finally, we pool only the special tokens' embeddings sub- sequently linearly projected in the desired latent space: o; Linear (RoBERTa(f)), where is the slot embedding, and E R the OTE slot index.
where is the temperature. Note that during train- ing, we only sample negative KG entry embeddings for each OIE slot, but not the other way around. Sampling only in in-batch negatives presents an is- sue as the training data represents only a limited subset of the whole KG (i.e., only the KG entries with paired OIE). During inference, however, we contrast each OIE slot against the whole KG to find the KG entry with which it exhibits the highest similarity. Therefore, for each OIE slot, addi- tionally sample e negative entities and p negative predicates at random from the whole KG (e.g., we would sample ΣN=1+p R negative predicates).
KG Embeddings. We represent both the entities and predicates as their label followed by their de- scription (if available the KG); e.g., the entity (e) Michael Jordan is represented as "Michael Jor- dan <DESC> American basketball player and busi- nessman.."" ... and the predicate (p) place of birth is represented as "place of birth <DESC> most specific known birth location of person..." "" The <DESC> special token indicates the start of the description. We then tokenize the representation (è for entity, á for predicate), and obtain embeddings using the same RoBERTa model. We pool the <CLS> token embedding and linearly project in the OIE rep- resentation space as: kj- Lincar( RoBBRRTa(;)). where j (c, (), k is the KG entry embedding, and is the tokenized KG entry representation.
4.2 Re-ranking OIEs to KG Facts
In certain scenarios (c.g., in the case of polysemous OIEs), matching whole OIEs with whole KG facts (i.e., not decoupled per OIE slot) could resolve the ambiguity and thus improve performance. To that end, for each OIE slot, we re-rank the OlE:Pe fanker top- most probable KG links. We denote this model as Fact We perform self-attention between the OIE and the KG fact (both provided as input, separated by <FACT> special token) with single RoBERTa transformer, and return their similarity as: & or ((Linear (RoBERTa(2)) where é is the concatenated OIE and KG fact representation, and Src the sigmoid (a) normalized similarity.
Linking OIES to KG Facts. Given OIE slot embedding 0; and KG entry embedding ky, obtain their dot- dot-prroucct as: Spre = of kj, where O₂ and ký are norm-scaled (i.e., Spre represents the OlEpre cosine similarity). During inference, we link an OIE to KG fact by selecting the most similar KG entry for each of the OIE slots.
Factre ranker Training. We train by sampling matching OIE ++ KG fact pairs as positives, and negatives, where we replace some KG fact slots (subject, predicate, object) with incorrect ones, gen- erated as follows: We first obtain embeddings for each KG entry using the OlEP top-k most similar candidates w.r.t. other KG entries. We then corrupt the ground truth KG fact by randomly sampling only from the top-k (hard)
ranker Training. We train the model using standard contrastive loss: we sample N-1 in-batch negative KG entries for each positive OIE slot <>> KG entry pair, where N is the batch size. As per standard practice (van den Oord al., 2018), train the model using temperature-scalee InfoNCE cotk/T
PIE
Tanker and find its
contrastive loss; L -log coTe/s 4 /T

---

negative candidates. Lastly, with 50% probabil- ity, we randomly mask (i.e., replace with <mask> token) the description of the KG fact entries.
off-the-shelf zero-shot models fail. 6 Additionally, there is significant performance drop on the in- ductive and polysemous split compared to the trans- ductive split, suggesting that the models are neither robust w.r.t. entities unseen during training, nor can cope with polysemous entities. Expectedly, leveraging extra context (via the sentence from where the OIE is obtained) aids the linking pro- cess in the inductive and polysemous split, it helps generalization and disambiguation. Similarly, the Fact brings significant performance gain especially prominent for linking complete facts. Fi- nally, we observe a significant performance impact of the KG size: across all splits, OlEito-KG linking is more challenging on large-scale KG compared to the smaller Benchmark- KG.
5 Experiments and Discussions
We measure accuracy to evaluate both OIE slot linking to KGs ($5.1), and Out-of-KG detection of OIE slots ($5.2). To measure OIE fact linking, we score hit if all OIE slots are linked correctly. The error bars represent the standard error of the mean.
5.1 Linking OIEs to Knowledge Graphs
Setup. We explore the extent to which we can link OIE slots to a large-scale KG (Wikidata). We address two main research questions concerning the OIE-to-KG fact linking task: (i) To what ex- tent do methods generalize to different KG entity facets? We consider transductive, inductive, pol- ysemous entities (see $2 for detailed definition); (ii) What is the performance impact the KG size? We test two reference KG sizes: Benchmark Re- stricted KG (~650k entities, ~00.6k predicates) and Large KG (~5..9M entities, -4k predicates).
Training on Synthetic Data Improves Perfor- mance. We explore the extent to which we can learn fact linking models using synthetic data. Syn- thIE (Josifoski et al., 2023) is a dataset that features natural language sentences paired with KG facts, where the sentences are obtained using LLM. Namely, given set of KG facts, Josifoski et al. (2023) prompt the LLM to generate sentence which mentions (i.e., entails) all of the KG facts. Notably, if we can link OIE slots to KG facts by training on such synthetic dataset, then the KG re- mains the only numan-annotater component for learning the OlE-to-KG fact linking task.
Methods. We use the following methods to ob- tain results for the OIE slot linking task: (i) RAN- DOM: for each OIE slot, sample random KG entry; (ii) FREQUENCY: based on the training data statistics, we link each OIE slot to the most fre- quent KG entry (entity or predicate) in the training set; (iii) SIMCSE: we use a pretrained SimCSE model (Gao et al., 2021), where we represent each OIE slot and each KG entry (optionally, its descrip- tion as well - if available in the KG) in natural language format, and obtain their embeddings. We finally obtain the cosine similarity between each OIE slot and KG entry to perform the linking; (iv) O : We train a pre-ranking model as per the setup described in $4.1; (v) + Context: We append the context sentence to the OIES; (vi) + Fact
To measure what extent we can link OIEs to KG facts using only synthetic data, we train OTE
Taaker models on both REBEL and SynthIE, and report results (in Table 2) on inductive testing splits w.r.t. each dataset." 7 We observe that models trained on Synthll are overall better OIE-to-KG fact link- than models trained on REBEL (i.e., higher macro accuracy across datasets). This indicates that learning to link OIEs to KGs is possible using only synthetic data, thus the only aaman-annotated requirement remains to be a reference KG.
We additionally re-rank the top-k (k = 3) pre- ranked OIE slot links per the setup in $4.2.
A blation Study: Importance of Entity Alias Aug- mentation. We observe that in current datasets most surface form entities (in the natural language sentences) appear only with their "canonical" la-
Results. In Table we report the OlE-to-KG link- ing performance for each OIE slot, as well link- ing on fact level. Overall, across the data splits, we observe that all unsupervised baselines perform poorly compared to models proposed in this work; indicating that OIE linking is not trivial hence
'Note that simply linking the OIE relation to the most frequent KG predicate yields high accuracy, due to the KG predicates imbalance in REBEL.
"To preserve the inductive property. SynthlE's inductive test split contains only entities found in SynthlF that are not present in the REBEL train data.
*We structure the input as: "OIE <SENT> Sentence", where <ENT>> is a special separator token.

---

Method
Split Type
Subject
Relation
Object
Fact
Subject
Relation
Object
Fact
Benchmark-
Knowledge Gruph
Large Knowledge Graph
Random
Transductive
0.0 0.0
uo
+
0.0 +
0.0
+
0.0
Frequeney
Transductive
0.0
53.3 =
0.0
0.0
53.3
0.1
0.0
SimCSE
Transductive
5.4
0.0
0.1
0.0
2.1
0.0
0.5
0.0
OlF
Sanker
Transductive
86.8
93.5 +
95.7 0.0
79,1 +
78.7
93.5 +.1
93.1 =01
70.7 0.2
Context
Transductive
84.9
92.2 +
94.8 #.1
77.7
77.8
92.2 0.1
92.2
70.2 0.1
Frequeney
Inductive
0.0
0.0
0.0 0.00
0.1
0.0 0.0
0.0
SimCSE
Inductive
1.3
0.4
6.8
0.0 0.00
8.1 0.0
0.0
3.2
0.0
Our
banker
Inductive
+..9
69.8
59.5
34,9 0.5
62.0 +
69.8
48.2 + 0.5
25,4 0.4
Context
Inductive
74.5
73,8
62.3 + 0.5
38.9
64.5
73.8
50.8 0.5
29.2 0:5
Fact Finker
Inductive
76.2 +: 0.4
67,6 + 0,5
60.8 6.5
40.6 0.5
64,8 0.5
66.5 + 0.5
54,8 0.5
32,9 0.5
Frequency
Polysemous
0.0
68.1 +
11.7 6.3
0.0
0.0
68.1 + 0.4
11.7 = 0.3
0.0 + 0.
SimCSE
Polyscmous
0.1
0.0 0.0
0.3 0.1
0.0 0.0
0.1
0.0 0.0
0.0 +:
0.0 4 0.0
OIE Banker
Polysemous
68.8 1.8.100 0.4
93.0 1 0.2
94.6 6.2
4.1
58.2 1
93.0 0.2
92.4-100
51.7 0.1
Context
Polysemous
0.3
93,3 :: 0.2
95.8 0.2
71.6
69.5
:: 0.4
93,3 :: 0,2
94.0
63.5 #:4
Faceio
Taaker
Polyscmous
75.3 0.4
90.6 a 0.2
95.3 0.2
69.5
0.4
74.1 = 0.4
90.6 d: 0.2
95.6 =: 0.2
69.3 d: 0.4
Table I: OIE slot and fact linking accuracy with models trained and evaluated on REBEL, Evaluation on a smaller Benchmark Restricted KG, and a Large KG. The error bars indicate standard error of the mean.
Subject
Relation
Object
Training Testing
Subject
Relation
Object
Fact
REBEL REDEL
REBEL
62.0
69.8
48.2 1055
25.4 1..4
Augmentation
SymhIE
53.6 04
57.2 044
44,9 =.44
Models trained evaluated REBEL
Macro Score
57.8
63.5
46.6
21.4
90.7
0.1
87.1 4:02
74.0
SymhIE
REBEL
69.7 0.3
63.2 03
41.6 0.3
22.4 #.2
61.6
6.2
64.8
41.5
2
SymthIE
SynbhE
64.1 = 06
81.2 0.
57.6
0.6
34.5 0.6
Macro Score
76.2
78.4
64.3
49.6
Macro Score
66.9
72.2
49.6
28.5
x
89.0
92.8 t.1
85.1
7244
/
79.0 0.2
92.2 0.1
89.0
1:10
67.9 0.2
Table 2: Experiments human-created (REBEL) and synthetically generated (SynthIE) datasets. OIE linking Large KG variant inductive splits w.r.t. each dataset (see Appendix 1 for details).
Macro Score
84.0
92,5
87.1
70.15
Models trained evaluated Syn/h7E
x
91.8 0.2
86.3 + u.2
90.7 00
72.8 =
61.2 0.2
70.1 +:
40.7 :: 02
22.6
Macro Score
78.2
65.7
47.7
91.1
88.9 022
90.5
74.3 0.3
8 Since the OIEs represent surface-form facts, such lack of diversity prevents the models from learning more complex linking patterns. To over- come this, we perform entity alias augmentation in FaLB by adding the surface form aliases of the entities entities-vviilablle available in Wikidata and manually curated-and and ablate its impact on the OIE linking task. Besides OIE"Ie Tanker models trained on REBEL and SynthlE with entity alias augmentation, we train additional models without the alias augmented samples. We report results in Table on inductive REBEL and SynthlE testing data, which does and does not feature entity aliases.
80.1 0.2
89.9 0.1
86.3 + 02
64.8 0.2
Macro Score
85.6
89.4
88.4 69.6
Table 3: Impact of the FaLB entity alias augmentation step on REBEL and SynthlE. Inference is done on the full test set. The model is OIE! Danker
and SynthIE. We further observe that this step hurts the linking of specific OIE entity mentions only moderately, suggesting that OIE linking methods could be trained to be robust w.r.t. entity synsets. Finally, across all OIE slots and fact linking, the macro scores are significantly in favor of the model trained with entity aliases, on both datasets.
Expectedly, we observe that training with entity aliases allows us to link such OIE entity mentions more successfully than training without them. This was reflected models trained on both REBEL
Importance of Fact-reranking. We ablate the number of KG facts we rerank (k) with Factfe and report results in Fig. 3. We observe the highest fact linking accuracy when per- forming reranking using the top-2 highest scoring KG entries for each OTE slot, although performance is within standard deviation for & = 2, 3, 4. If reranking = facts, effectively, the Fact
Ablation Study:
*REBEL is constructed from Wikipedia abstracts, where the references use the canonical form name; SynthlE provides the KG fact (in text format) as is to the LLM, so naturally, the sentence generated does not feature the entity aliases. There- fore, Michael Jordan's 's synonyms such as M.J., Air Jordan and His A Aimess, rarcly appear in the data.

---

Impact Kreranked facts
accuracy over the two scenarios for each OIE slot.
Methods. We create three methods on top of Old ranker trained on REBEL: (i) CONFIDENCE@ 1- BASED HEURISTIC: We get the cosine similari- ties of the top-5 predictions from OIE funker com- pule the softmax, and threshold the top-1 con- fidence; (ii) ENTROPY-BASED HEURISTIC: We also obtain the top-5 cosine similarities, how- ever, after softmax normalization compute their entropy. Finally, we threshold the entropy to obtain the prediction;" (iii) QUERY-KFY- KFY- VALUE CROSS- ATTENTION: We train a lightweight query-key- value cross-attention module on top of the frozen OlE funker embeddings. Given query OIE slot em- bedding, the model attends over the KG entry em- beddings representing the keys and values, and outputs a probability indicating the presence of the OIE slot in the KG (See Appendix E.1 for details).
Fact Linking Accuracy
40.5
40.0
39.5
39.0
38.5
1
Value of K the Fact-Reranker
Figure 3: Ablating the impact of K in the Face
Tanker
constructs 2 OIE-KG fact pairs, reranks the list, and returns the highest scoring KG fact.
5.2 Detecting Out-of- Knowledge Graph OIEs
Setup. Prior works which study the OIE KG linking problem (Zhang al., 2019; Wood al., 2021; Jiang al., 2021) make the assumption that all OIE slots that need to be linked are present in the KG, which rarely the case in practice. Here, we study (i) whether OIE slot linking methods can be converted to Out-of-KE KG detectors; (ii) the per- formance impact of the KG size the Out-of-KG detection task; and (iii) to what extent it more difficult to recognize the presence or absence of aliased OIE entity mentions.
Results. We report the Out-of-Ke detection per- formance Table First, we observe that none of the models we evaluate are able to recognize whether an OIE relation has a corresponding KG predicate, indicating that models do not cope with zero-shot relations. Intuitively, the number of enti- ties is orders of magnitude more than the relations, thus the models learn features which generalize unseen data. On the other hand, the number of lations is limited (~-00 during training) and there- fore, the models overfit on this limited set. Second, similar to our observations on the OIE linking task, detection of Out-of-KG slots is significantly more difficult on data which features entity aliases, and even more difficult on larger KGs. Lastly, the best performing model is entropy- masse heuristic top of the OlE Oa output scores. Overall, results indicate that Out-of-Ko of- detection remains an open research problem.
Datasets. We create the Out-of-KG test split by imposing constraints such that we mimic an out-of- distribution setting. The constraints are: (i) both the KG entities and predicates referred by OIEs are unseen during training (i.e.. not in the KG); (ii) the testing OIE-to-KG pairs do not come from the training data distribution: we use models trained on REBEL, but evaluate on SynthlE (which use to create the Out-of-KC split). We measure Out-of-KG detection performance on the original OlEs, and OIEs with alias augmented entity slots.
Related Work
Prior work (Zhang et al., 2019: Wood et al.. 2021; Jiang et al., 2021) on the ReVerb45k dataset (Vashishth et al., 2018) fact linking as inductive and polysemous for the en- tities, but perform inductive inference for the rela- tions. They also assume the OIE entities are linked to the KG entities a priori. In contrast, FaLB quires linking all OIE slots KG entries, with multiple evaluation facets (including the out-of-KG
Evaluation Protocol. We evaluate to what extent we can detect if an OIE slot refers to a concept outside of the KG. For each OIE slot, we add its corresponding concept (KG entity predicate) to the KG, and score a hit if the model outputs positive score for that slot. Conversely, for each OTE slot, we remove its corresponding entity or predicate from the KG, and score a hit if the model outputs a negative score. We report the averaged
FFrr cach method, we find the optimal threshold on hold- out set which we build on top of the REBEL validation set.

---

Method
Subject
Relation
Object
Faet
Subject
Relation
Object
Fact
Bonchmark Restrieted Knowledge Graph
Rensseark-Resrreend
Large Knowledge Graph
Testing data with entity alias ougmentation
Randomm 50.0 + 1.
50.0
50.0
12.5 +.
50.0
=00=113 1.
50.0 13
50.0
12.5 =
Confidence 1-based Heuristic
63.8
21.5
62.0
Entropy- based Heuristic 67.7
49.0
68.6
24
63.1
49.0
64
23,0
Query Key. Value Cross Attention
62.4 + 1.2
48.8 1.0
63.8 1.2
+7.2++02
55.3 1.
49.2
56.8 1.2
16.6 0.2
Testing data without entity alias auguentation
Random 50.0 + 2.8
50.0 2.8
50.0 2.28
12.5 + 18
50.0
50.0 13
50.0 1:3
12.5
Confidence 1-based Heuristic 71.2
72.1 2:23
M02
71.2
4:5
Entropy-hased Heuristic 77.5
49,0
78.4
2.2
29,9
73.3
49.1
72.8
Query-Key-Value Cross-Altention
70.7 2.5 49.6
70.9
25
25.9 1
59.8 2:27
49.4 277
60.8 125
19.7
Table 4: Detection accuracy of top of SymthlE. All models trained on REBEL with entity alias augmentation.
Graph entities and predicates on the Out-of-KG split built on
Dataset
Golden
Manually validated rels.
Multifaceted
Out-of-KG
Inductive
Transcuctive
Polysemous
OIF Finities
OIE Relations
entities
FaLB (REBEL)
936,655
159,597
FaLB (SynthIE)
1,049,922
147.056
ReVerb4
28.796
21.925
Table 5: Comparison of OIE-to-Ko datasets: FaLB Verbb5k As transductivity and inductivity are defined
only w.r.t. what the model has observed during training, we leave these entries blank, because Re ReVerb45k has only validation and testing dataset.
setup). In Re Verb45k (Vashishth et al., 2018) the links from the OIE entities to the KG entities are not golden (i.e., human labelled), but rather auto- matically obtained with outdated entity linker (Lin et al., 2012); thus, the poor performance of the entity linker caps the performance of the fact linking models. addition, the number of bench- mark KG predicates is unrealistically small (only 250) compared to modern KGs (e.g., Wikidata). In this work, we mitigate these issues and build benchmark (i) with golden links to KG entities that also (ii) reflects the size of modern KGs.
Verb45k (Vashishth et al., 2018), which, as indi- cated above, has several drawbacks. For detailed comparisons with and FaLB, see Table5.
Conclusion
We shed light on the OIE to KG linking problem, allowing us to fuse the surface-form and opened- ended knowledge found in OlEs, with the canonical real-world KG KGcaats. We introduced novel multi- faceted benchmark which fixes prior work deficien- cies, and proposed a of task-specific baselines. Our experiments uncover that (i) linking inductive or polysemous OTEs large KGs challenging; (ii) we can learn OIE linking methods using only synthetic data; and (iii) detecting whether OIEs are Out-of-KG is an open research problem.
Furthermore, all prior work to date has relied on the strict assumption that all OIE surface form slots have corresponding reference KG entity predicate (Wu et al., 2019; Jiang al., 2021; Zhang al., 2019; Josifoski et al., 2022). This is obvi- ously a false assumption, as any text corpus "in the wild" contains entities and predicates not present in even the largest of KGs (Gashteovski et al., 2020a). See App. A for detailed related work discussion. Finally, most existing publicly available datasets do not address the problem of OIE-to-KG link- ing. Popular datasets like T-REx (Elsahar et al., 2018) and REBEL (Cabot and Navigli, 2021)-if considered without modification-addees only Text- to-KG alignment, thus lack the OIE component. Therefore, these datasets are not directly compa- rable to FaLB. To the best of our knowledge, the only publicly-availate OIE-to-Ke KG dataset is Re-
A Acknowledgements
We thank Dina Trajkovska for the help with the figures, and Mike Zhang for the fruitful discussions and feedback at the initial stages of the project.

---

Limitations
the Association for Computaaional Linguisties (ACL). page 2296 2296-2308.
Notably, the set of models we explore ignore the KG structure to obtain KG entry embeddings. Leveraging the underlying graph structure should, in theory, yield representations which generalize better zero-shot samples (e.g., as is the case with detecting out-of-KG relations). Such KG entry em- beddings could be even trained offline (i.e., as a separate step) with standard KG embedding meth- ods (Bordes et al., 2013).
Tom B. Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurlPS). 33:1877 1877-1901.
Last but not least, all data, resources, and mod- cls used in this work are specific to the English language. Notice however, our approach can be readily extended languages other than English, while Wikipedia and Wikidata have versions in other languages- - which we leave for future work.
Pere- -luís Huguet Cabot and Roberto Navigli, 2021. REBEL: Relation Extraction By End-to-end Lan- guage generation. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2370-2381.
Ethical Impact
We are not aware of any direct ethical impact gener- ated by our work. However, in general, care should be taken when applying our technology to sensitive use cases in high risk domains, such as healthcare.
Luciano Del Corro and Rainer Gemulla. 2013. ClausIB: Clause-based Open Information Extraction. In Pro- ceedings of the International Conference on World Wide Web (WWW), page 355-366.
Daniel Daza, Michael Cochez, and Paul T. Groth. 2021. Inductive Entity Representations from Text via Link Prediction. Proceedings of the Web Conference (WWW), page 798-808.
References
Gabor Angeli. Melvin Johnson, and Christopher D. Manning. 2015. Leveraging Linguistic Structure For Open Domain Information Extraction. In Annual Meeting of the Association for Computational Lin- guistics (ACL), page 344-354.
David Dukić, Kiril Gashteovski. Goran Glavaš, and Jan Snajder. 2023. Leveraging Open Information Extraction for Improving Few-Shot Trigger Detection Domain Transfer. arXiv preprint arXiv:2305. 14163.
Michele Banko, Michael J. Cafarella, Stephen Soder- land, Matthew Broadhead, and Oren Etzioni. 2007. Open Information Extraction from the Web. In Com- munications of the ACM.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018 T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018).
Farimah Bayat, Nikita Bhutani, and H. V. Jagadish. 2022. CompactlE: Compact Facts in Open Informa- tion Extraction. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 900-910.
David Friede and Kiril Gashteovski. 2022. Context- Aware Entity Linking for Knowledge Graphs. US Patent App. 17/392,319.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi- relational Data. Advances in Neural Information Processing Systems (NeurIPS), 26.
Niklas Friedrich, Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Caroline V. Lawrence, Mathias Niepert, and Goran Glavavs. 2022. AnnlE: An An- notation Platform for Constructing Complete Open Information Extraction Benchmark. In Anmual Meet- ing of the Association for Computational Linguistics (ACL). System Demonstrations. page 44-60.
Samuel Broscheit, Kiril Gashteovski, and Martin Achen- bach. 2017. OpenIE for Slot Filling at TAC KBP 2017 System Description. Text Analysis Confer- ence (TAC).
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Conference on Empirical Methods in Natural Language Processing (EMNLP), page 6894-6910.
Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, and Rainer Gemulla. 2020. Can We Predict New Facts with Open Knowledge Graph Embeddings? Bench- mark for Open Link Prediction. In Annual Meeting of

---

Kiril Gashteovski, Rainer Gemulla, and Luciano Del Corro. 2017. MinlE: Minimizing Facts in Open In- formation Extraction. In Conference Empirical Methods in Natural Language Processing (EMNLP),
Zhengbao Jiang, Jialong Han, Bunyamin Sisman, and Xin Dong. 2021. CoRI: Collective Relation Integra- tion with Data Augmentation for Open Information Extraction. In Anmual Meeting of the Association for Computational Linguistics (ACL), page 4706-4716.
page 2630-2640.
Kiril Gashteovski, Rainer Gemulla, Bhushan Kotnis, Sven Hertling, and Christian Meilicke. 2020a. On Aligning OpenIE Extractions with Knowledge Bases: Case Study. Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP@@ EMNLP).
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7(3):335-544
Hideaki Joko, Faegheh Hasibi, Krisztian Balog, and Ar- jen P de Vries. 2021. Conversational Entity Linking: Problem Definition and Datasets. In Proceedings of the International ACM SIGIR Conference on Re- search and Development in Information Retrieval (SIGIR), pages 2390-2399
Kiril Gashteovski, Rainer Gemulla. Bhushan Kotnis,
Sven Hertling, and Christian Meilicke. 2020b. On Aligning OpenIE. Extractions with Knowledge Bases: Case Study. In Workshop on Evaluation and Com- parison of NLP Systems (Eval4NLP@ EMNLP), page 143-154.
Martin Josifaski, Nicola De Cao, Maxime Peyrard, and Robert West. 2022. GenlE: Generative Information Extraction. In North American Chapter of the Associ- ation for Computational Linguistics (NAACL), pages 4626 4643.
Kiril Gashteovski, Sebastian Wanner, Sven Hertling, Samuel Broscheit, and Rainer Gemulla. 2019. OPIEC: An Open Information Extraction Corpus. In Conference on Automatic Knowledge Base Con- sstruction (AKBC).
Martin Josifoski, Marija Sakota, Maxime Peyrard. and Robert West, 2023. Exploiting Asymmetry for Synthetic Training Data Generation: SynthlE and the Case of Information Extraction. ArXiv, abs/2303.04132.
Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Car- olin Lawrence, Mathias Niepert, and Goran Glavaš. 2022a. BenchIE: A Framework for Fact-Based Open Information Extraction Evaluation. In Annual Meeting of the Association for Computa- rional Linguistics (ACL), pages 4472-4490.
Bhushan Kotnis, Kiril Gashteovski, J. Gastinger, Giuseppe Serra, Francesco Alesiani, Timo Sztyler, Ammar Shaker, Na Gong, Carolin (Haas) Lawrence, and Zhao Xu. 2022a. Human-Centrie Research for NLP: Towards Definition and Guiding Questions. ArXiv, abs/2207,0444 0044477.
Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Car- olin Lawrence, Mathias Niepert, and Goran Glavaš. 2022b. BenchIE: A framework for multi-faceted fact-based open information extraction evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume
Bhushan Kotnis, Kiril Gashteovski, and Carolin Lawrence. 2022b. Open Information Extraction from Low Resource Languages. US Patent App. 17/342,575.
1: Long Papers), pages 4472-4490, Dublin, Ireland. Association for Computational Linguisties.
Bhushan Kotnis, Kiril Gashteovski, and Carolin Lawrence. 2023. Keyword Based Open Informa- tion Extraction for Fact-Relevant Knowledge Graph Creation and Link Prediction. 17/734.129.
Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulié, and Iryna Gurevych. 2022. Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Im- proved Cross-Modal Retrieval. Transactions of the Association for Computational Linguistics (TACL), 10:503-521.
US Patent App.
Bhushan Kotnis, Kiril Gashteovski, Daniel Rubio, Am- mar Shaker, Vanesa Redrriguez-eemrras, Makoto Takamoto, Mathias Niepert, and Carolin Lawrence. 2022c. MILIE: Modular & Iterative Multilingual Open Information Extraction. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6939-6950.
Patrick Hohenecker, Frank Martin Mtumbuka, Vid Ko- cijan, and Thomas Lukasiewicz. 2020. Systematic Comparison of Neural Architectures and Training Approaches for Open Information Extraction. In Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), page 3554-8565.
Anne Lauscher, Yide Song, and Kinil Gashteovski. 2019. MinSelE: -eettered Open Information Ex- traction. ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 386-387.
Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes. Pengfei Liu, Gra- ham Neubig, and Chunting Zhou. 2023. Multi- Dimensional Evaluation of Text Summarization with In-Context Learning, In Annual Meeting of the Asso- ciation for Computational Linguistics (ACL). Find- ings., page 8487-8495.
Adam Lerer, Ledell Wu. Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019. scale Graph Embedding System. Proceedings of Machine Learning and Systems, 1:120-131.
A Large-

---

Irene Li. Jessica Pan, Jeremy Goldwasser, Neha Verma, Wai Pan Wong. Muhammed Yavuz Nuzum- lali, Benjamin Rosand, Yixin Li, Matthew Zhang, David Chang. Richard Andrew Taylor, Harlan M. Krumholz, and Dragomir R. Radev. 2022. Neural Natural Language Processing for Unstructured Data in Electronic Health Records: Review. Computer Science Review, 46: 100511.
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf- sky. 2009. Distant Supervision for Relation Extrac- tion without Labeled Data. In Proceedings of the Joint Conference of the Annual Meeting of the ACL and the International Joint Conference on Natural Langugge Processing of the AFNLP (ACI/IJCNLP), pages 1003-1011.
Guoshun Nan, Rui Qiao, Yao Xiao. Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu. 2021. Interventional Video Grounding with Dual Contrastive Learning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 2765-2775.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distilla- tion. Advances in Neural Information Processing Systems (NeurIPS), 34:9694-9700 9705.
Federico Nanni, Jingyi Zhang, Ferdinand Betz, and Kiril Gashteovski. 2019. EAL: A toolkit and dataset for entity-asped linking. In ACM/IEEE Joins Confer- ence on Digital Libraries (JCDL), pages 430-431. IEEE.
Thomas Lin, Mausam, and Oren Etzioni. 2012. Entity Linking at Web Scale. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construc- tion and Web-scale Knowledge Extraction (AKBC- WEKEX TEKEXxNAACL-HLII pages 84-88.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. 7ach DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lever. 2017. Automatic Differentiation in PyTorch.
Xueling Lin, Haoyang Li, Hao Xin, Zijian Li, and Lei Chen. 2020. KBPcarl: A Knowledge Base Popu- lation System Supported by Joint Entity and Rela- tion Linking. Proceedings of the VLDB Endowment, 1335):903
Miao Peng. Ben Liu, Qianqian Xie, Wenjie Xu. Hua Wang, and Min Peng. 2022. SMiLE: Schema- augmented Multi-level Contrastive Learning for Knowledge Graph Link Prediction. In Findings of the Association for Computational Linguistics: Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4194-4206.
Pengfei Liu, Jinlan Fu, Yanghua Xiao, Weizhe Yuan, Shuaichen Chang. Junqi Dai, Yixin Liu, Zihuiwen Ye. Zi-Yi Dou, and Graham Neubig. 2021. Ex- plainaBoard: An Explainable Leaderboard for NLP. In Annual Meeting of the Association for Compu- tational Linguistics (ACL). System Demonstrations, page 280-289.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. 2020. KILT: a Benchmark for Knowledge Intensive Language Tasks. In North American Chapter of the Association for Computa- tional Linguistics (NAACL), page 2523-2544.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv preprint 1x7.v:197.11192.
Mausam, Michael Schmitz. Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open Language Leam- ing for Information Extraction. In Conference on Empirical Methods in Natural Language Processing (EMNLP), page 523-534.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Interna- rional Conference Machine Learning (ICML), pages 8748 1763, PMLR.
Mary L. McHugh. 2012.
Interrater Reliability: the Kappa Statistic. Biochemia Medica, 2263)227-
Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stucken- schmidt. 2018. Fine-Grained Evaluation of Rule- and Embedding- -aaadd Systems for Knowledge Graph Completion. In International Semantic Web Confer- ence (ISWC), page 3-20.
Leonardo F. R. Ribeiro, Mengwen Liu, Iryna Gurevych, Markus Dreyer, and Mohit Bansal. 2022. FactGraph: Evaluating Factuality in Summarization with Seman- tic Graph Representations.
Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (NAACL- (LLT, pages 3238-33 3253..
Antoine Miech, Jean- Baptistt Alayrac, Ivan Laptev. Josef Sivic, and Andrew Zisserman. 2021. Think- ing Fast and Slow: Efficient Text-to-Viste Retrieval with Transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 9826-9836.
Marco Tulio Ribeiro, Tongshuang Sherry Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accu- racy: Behavioral Testing of NLP Models with Check- List. In Annual Meeting of the Association for Com- putational Linguistics (ACL), page 4902-4912.

---

Wiem Ben Rim, Carolin (Haas) Lawrence. Kiril Gash- teovski, Mathias Niepert, and Naoaki Okazaki. 2021. Behavioral Testing of Knowledge Graph Embedding Models for Link Prediction. In Conference on Auto- mated Knowledge Base Construction (AKBC).
Vijay Viswanathan, Kiril Gashteovski. Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2023. Large Language Models Enable Few-Shot Clustering, arXiv preprint arXiv:3307.00524
Denny Vrandečić. 2012. Wikidata: A new platform for collaborative data collection. In Proceedings of the International Conference on World Wide Web (WWW), pages 1063-1064.
Youngbin Ro, Yukyung Lee, and Pilsung Kang. 2020. OultiPOIE: Multilingual Open Information Extrac- tion Based on Multi-Head Attention with BERT. In Findings of the Association for Computational Lin- guistics: Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 1107- 1117.
Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Mod- cls. In Association for Computational Linguistics (ACL), pages 4281-4294
Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria- Esther Vidal. 2020. Falcon 2.0: An Entity and Rela- tion Linking Tool over Wikidata. In Proceedings of the ACM International Conference on Information & Knowledge Management (CIKM), pages 3141-3148.
Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLEER: A Unified Model for Knowledge Em- bedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics (TACL), 9:176-194.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint
Haris Widjaja, Kiril Gashteovski, Wiem Ben Rim, Pengfei Liu, Christopher Malon, Daniel Ruffinelli, Carolin (Haas) Lawrence, and Graham Neubig. 2022. KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models. In Conference on Empirical Methods in Nat- ural Language Processing (EMNLP). System Demon- strations., page 338-350.
Sascha Saralajew, Ammar Shaker, Zhao Xu, Kiril Gash- teovski, Bhushan Kotnis, Wiem Ben-Rim, Jürgen Quittek, and Carolin Lawrence. 2022. A Human- Centric Assessment Framework for AL. ArXiv, abs/2205. 12749.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Em- beddings. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 4507.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020. Transformers: State-of-the- Art Natural Language Processing. In Conference on Empirical Methods in Natural Language Processing (EMNLP), System Demonstrations, page 38-45.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018 Supervised Open Informa- tion Extraction. In Proceedings of the Conference of the North American Chapter of the Association for Computationnll Linguistics: Human Language Technologies (NAACL-HLT), pages 885-895.
Ian D. Wood, Mark Johnson, and Stephen Wan. 2021, Integrating Lexical Information into Entity Neigh- bourhood Representations for Relation Prediction. In North American Chapter of the Association for Com- putational Linguistics (NAACL), page 3429-3436.
Dianbo Sui, Chenhao Wang, Yubo Chen, Kang Liu, Jun Zhao, and Wei Bi. 2021. Set Generation Networks for End-to-End Knowledge Base Population. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9650-99 56609
Chengmin WU, Enrui Hu, Ke Zhan, Lan Luo, Xinyu Zhang, Hao Jiang, Qirui Wang, Zhao Cao, Fan Yu, and Lei Chen. 2022. Triple-Fact Retriever: An Ex- plainable Reasoning Retrieval Model for Multi-hop QA Problem. In International Conference Data Engineering (ICDE), pages 1206-1218 IEEE
Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural Relation Extraction for Knowledge Base Enrichment. In Anmual Meet- ing of the Association for Computational Linguistics (ACL), pages 229-240.
Ledell Yu Wu, Fabio Petroni, Martin Josifoski, Sebas- tian Riedel, and Luke Zettlemoyer. 2019. Scalable Zero-shot Entity Linking with Dense Entity Retrieval. In Conference on Empirical Methods in Natural Lan- guage Processing (EMNIP), page 6397-6407.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. ArXiv, abs/1887 0011180703447 .03748.
Shikhar Vashishth. Prince Jain, and Partha Pratim Taluk- dar. 2018. CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information. Pro- ceedings of the World Wide Web Conference (WWW), page
Honghin Ye, Honghao Gui, Xin Xu, Huajun Chen, and Ningyu Zhang, 2023. achema-adappable Knowledge Graph Construction. arXiv preprint arXiv:2305.08703.

---

Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neu- big. 2021. Towards More Vine-grained and Reliable NLP Performance Prediction. In Proceedings of the Conference the European Chapter of the Associ- ation for Computational Linguistics (EACL), page 3703-3714.
Dongxu Zhang, Subhabrata Mukherjee, Colin Lockard, Xin Dong, and Andrew McCallum, 2019. OpenKI: Integrating Open Information Extraction and Knowl- edge Bases with Relation Inference. In North Amer- ican Chapter of the Association for Computational Linguistics (NAACL).
Markus Zopf and Kiril Gashteovski. 2023. Fine- Grained Concept Identification for Open Informa- tion Knowledge Graph Population. US Patent App. 17/572,685.

---

A Detailed Discussion on Related Work
alignment (Friede and Gashteovski, 2022). There- fore, the task of OIE fact linking to KGs is of great importance, as OIE facts can provide knowledge that is outside of the KG schema, as well as knowl- edge that aligns with the KG schema (Gashteovski et al., 2020b). Several methods (Zhang et al., 2019; Jiang et al., 2021; Wood et al., 2021) address the problem of linking OIEs to KGs. These methods, however, make the assumption that the subject and object OIE slots (which link to KG entities) are linked a priori. These methods can link inductive OIE relations (outside of the training data), how- ever, the assumption that the OIE entity slots are linked beforehand renders these methods not ap- plicable for our task of linking (free-text) surface facts-also known as OlEs-to canonical large- scale KGs.
In $2, we go over the problem statement, and in $6 we discussed how the different facets of our benchmark relate to prior and closely related work. Here, we provide a broader discussion where we group the related work based on the problem they address, and provide more detailed discussion w.r.t. the differences with our work.
Open Information Extraction (OIE). OIE methods extract structured surface-form factual information from natural language text data, in the form of ("subject"; "relation"; "object")- triples (Banko et al., 2007). Such systems are typi- cally either rule-based (Mausam et al., 2012: Corro
and Gemulla, 2013; Angeli et al., 2015; Gash-
teovski et al., 2017; Lauscher et al.. 2019; Zopf and Gashteovski, 2023) or bhaedd (Stanovsky et al., 2018: Hohenecker et al., 2020; Kotnis et al., 2022c,b; Bayat et al., 2022). Recent research showed that neural OIE systems still fall behind rule-based systems (Gashteovski et al., 2022a;; Friedrich et al., 2022). To use the best of both worlds, in this work we use two rule-based systems: MinlE (Gashteovski et al., 2017) and Stanford OIE (Angeli et al., 2015); and two neural systems: MI- LIE (Kotnis et al., 2022c) and OultiPOIE (Ro et al.. 2020).
Knowledge Graph Link Prediction. These methods (Daza et al., 2021; Wang al., 2021, 2022: Peng al., 2022) address the problem of Knowledge Graph population by predicting the missing facts in the KG given only the current set of facts. They assign higher scores to valid KG facts (composed of entities and predicates currently in the graph), which need to be added in the graph, and lower scores to erroneous facts (i.e., wrong fact proposals). Even though, during inference, these methods deal with entities and predicates which were unseen during training, an implausible assumption made that information is already ex- tracted and canonicalized as a KG facts, which is never the case in practice.
Closed Information Extraction. Given input text, ClosedIE methods extract set of (subject; relation; object)-triples where each triple can be expressed within the predefined schema- fixed sets of entities and predicates-of the reference KG (Josifoski et al., 2022; Trisedya et al., 2019; Sui et al., 202 1; Josifoski et al., 2023). To ensure that each triple belongs to the KG, these methods prune candidates outside of the KG schema. Because of the pruning, these methods are unable to generate triples which contain entities or predicates that are not already the KG. In turn, the task we address- OIE linking to KGs-allows us to both link OIE to existing KG candidates, as well as detect novel candidates which are outside of the KG schema.
Multifaceted Evaluation. NLP and KG tasks are typically evaluated on held-out test set, by us- ing evaluation frameworks that assign performance scores on single value; e.g., accuracy (Petroni et al., 2020). In recent years, researchers have observed that such evaluation protocols are some- what limited (Jain al., 2023; Ye et al., 2021; Liu et al., 2021), because they do not expose the par- ticular types of problems that the models might have. Hence, with such evaluation protocols, the tested models are more difficult to diagnose when they make errors (Ribeiro et al., 2020). Following prior work multifaccted OIE (Gashteovski et al., 2022b) and multifaceted KG evaluation (Meilicke et al.. 2018: Rim al., 2021; Widjaja et al., 2022). we propose Fal.B: a multifaceted evaluation frame- work for fact linking. FaLB allows fin-grained evaluation that helps users to pinpoint the source
Open Information Extraction and Knowledge Graphs. The context of OIE facts is used for many tasks for knowledge graphs, such as knowl- edge graph population (Broscheit al., 2017; Lin et al., 2020; Zopf and Gashteovski, 2023), (open) link prediction (Broscheit et al., 2020; Kotnis et al., 2023), entity linking (Nanni et al., 2019) and entity

---

Josifoski et al. (2023) perform human evalua- tion to verify whether their synthetically generated dataset (of natural language sentences pair with KG facts) is of higher quality than REBEL. They obtain 44 data samples from REBEL., and generate sen- tence using a Large Language Model (LLM) given only the KG triplets from the sample. Finally, they verify the triplet-set-to-text compatibility for both REBEL and SynthIE. Importantly, human evalu- ators find that the LLM generated sentences (i.e.. the synthetic ones) are more compatible with the set of the KG facts (that is, the validity of the KG facts is higher in SynthlE compared to REBEL). of error (e.g., on which slot an examined model makes an error), which makes the benchmark more interpretable and useful for subsequent diagnostics of the models. In addition, our benchmark eval- uates the performance of the models in different scenarios: inductive, transductive, polysemous, and out-of-KG. With this, the evaluation framework is more human-centric, in sense that it can help users identify the model that they want for their needs instead of relying on single-score metrics (Kotnis et al., 2022a; Saralajew et al., 2022).
them are obtained using a set of heuristics. This leads to imbalanced data, where most predicates oc- cur only few times, and others occur orders of mag- nitude more. Consequently, such issue is reflected in the OIE-to-KG fact linking data that we obtain. To address this problem of REBEL, Josifoski et al. (2023) proposed SynthlE: synthetically generated dataset which deals with the imbalance.
B Benchmark Datasets Statistics
In Table 6 we report statistics of the benchmark datasets we create on top of REBEL and SynthlE, using the Wikidata Knowledge Graph.
C Implementation Details
We train all models for 10 epochs using Adam with a learning rate of 5e-5 and weight decay of le-
3. We use RoBERTa (Liu et al., 2019) model, dis- tilled following the procedure of Sanh et al. (2019). The model consists of 6 layers, hidden size of 768, and has 12 self-attention heads. To reduce the computational complexity and memory demands, we further linearly project the embeddings obtained from the RoBERTa model to a 200-dimensional la- tent space. When training the OfE, models, we initialize the temperature T to 0.07 as per Radford et al. (2021). We train OlE models with nega- tives that are sampled from the whole KG, where we sample 128 negative KG entities and 64 nega- tive KG predicates. When training the Fact we sample negatives such that we corrupt the slots of the KG fact by replacing them with incorrect ones. Instead of choosing the negatives at random, for each KG entry, we find its 10p-10 most similar KG entries, and sample negatives from this subset. This ensures that the Fact model learns how refine the predictions of the OlE We im- plement everything using Py (Paszke al., 2017), while we use HuggingFace transformers (Wolf et al., 2020) for the RoBERTa implementa- tion. Lastly, we use Faiss (Johnson et al., 2019) to enable fast linking to Knowledge Graphs.
On the other hand, the issue we observe with SynthIE, is that due the way the data provided to LLM, the surface-form entities remain in their canonical text-form (i.c., their canonical denotation in the KG), and thus contain little variation. This diverges from the data that we find "in the wild". Critically, when people refer to entities in free-form natural language, they commonly use synonyms. aliases, abbreviations, nicknames, etc. For exam- ple, the former basketball player "Michael Jordan" " could be referred as "Air Jordan" " and "M.J.".
To cope with this issue, we leverage the entity alias augmentation step in FaLB. By increasing the diversity of the OIE entity surface form, we are able to obtain high quality OIE-to-Ko KG fact linking synthetic dataset, thus the only auman-annotated component remains to be the KG.
D.1 Inductive Splits between Datasets
Importantly, the inductive evaluation is testing OIE linking to KG facts that consist of entities which were not part of the models' training data. In $5.1 we perform experiments by training models on REBEL and SynthlE, and then evaluate how well they perform on inductive data splits w.r.t. each of the datasets. Namely, to obtain an inductive REBEL testing split w.r.t. SynthIE, we find all test- ing samples from REBEL, which contain KG enti-
D Discussion on the Quality of REBEL and SynthIE
While REBEL's is entities are golden (i.e., obtained as the hyperlinks from Wikipedia abstracts which link to Wikipedia pages), the predicates between
More precisely, in the paragraph titled "Results from Training on Synthetic Data"

---

Backbone Dataset
Split Type
Total Samples
Unique Entities
Unique Predicates
# Unique Facts
REBEL
Training
5,638,244
572.020
555
613,047
SynthEE
Training
7,749,603
685,959
757
766,032
REBEL
Full Validation
421,547
48,103
303
41,092
SymthI
Full Validation
48,347
6,827
466
4,766
REBEL
Full Testing split
427.961
48,807
41,767
SymthIE
Full Testing split
240,300
28,660
635
22,661
REBEL
Testing Transductivity
241,995
14,730
192
13,496
REBEL
Testing Inductivity
10,300
4,423
185
2,297
SynthIE
Testing Inductivity
6,592
3,372
1,775
REBEL
Testing Polysemy
15,339
3,504
117
3,290
SynthIE
Testing Out-of-KG
1,604
443
106
264
Table REBEL and SynthlE: overview of the number of samples in each dataset, number unique KG entities, number of unique KG predicates, number of unique KG facts. Reported for all data splits of REBEL and SynthlE.
Dataset Type
Samples
Eniiiies
Predicates
# Facts
(i.e., model performs well on the linking task, but performs poorly on the Out-of-KG task) when detecting out-of-KG relations.
REBEL wott. Synhhi
26,937
8,781
181
5,304
SynthE w.r. REBEL
17,931
6,318
488
3,499
Table Overview of the total number of samples, num- ber of unique KG entities, number of unique KG pred- icates, number of unique KG facts, and the. Reported for inductive splits w.r.t. each dataset.
Therefore, to ensure that the Out-of-KG data is indeed out-of-distribution (as would be the case in practice), we impose the following constraints: (i) We select OTE-to-KG pairs from SynthIE, while the "backbone" model we build on top of is trained on REBEL; (ii) All entities and predicates- which are part of the KG facts from the testing data-are not in the KG at the time of training the OlF, Tanker
ties that are not part of any SynthlE training sam- ples. In turn, to obtain an inductive SynthlE. testing split w.r.t REBEL, we find SynthlE testing sam- ples which contain KG entities that are not part of any training samples from REBEL. We report statistics of the inductive splits w.r.t. each dataset in Table 7,
To perform the evaluation, for each OIE slot we either leave its corresponding KG entry outside of the KG, and score a hit if the models predict a neg- ative score for that slot; or, we perform imputation of the Out-of-KG entries (thus, they are now part of the KG), and score a hit if the models predict a positive score for that slot. Finally, we report the averaged accuracy over the two scenarios for each OIE slot. Note that, in this case both micro- and macro- aneraaeverage yield the same number, because we have the same number samples for each scenario.
E
Details on the Out-of-Knowledge Graph Detection Task
In $5.2 we evaluate the ability of the models to detect whether an OIE slot (surface-form entity, or surface-form relation) is present in the Knowledge Graph. Intuitively, this task is more difficult than the OIE linking task, as the models need general- ize beyond the training data distribution to perform well on this task. Namely, when linking OIEs to a KG, all prior work (Zhang et al., 2019; 10; Jiang et al., 2021; Wood et al., 2021) makes the conjecture that the testing data (in the open- (pen-world) is independent and identically distributed (i.i.d.) w.r.t. the training data, which is invalid assumption in certain sec- narios. Notably, the Out-of-KG OIEs lie outside of the training data distribution. Therefore, model that performs well on the OIE linking task does not warrant high performance on the Out-of-KG detection task. We observed that this is the case
E.1 Out-of-Knowledge Graph Detection Models
All models that we use for the Out-of-Ket detection task in $5.2 are built on top of a OTE fanker which is trained on REBEL. For all models, to obtain an out-of-KG indicator- (1), or False (0)- we threshold the output of the models. For each model, we determine the optimal threshold (the confidence, or the entropy) on a hold-out validation dataset which build top of REBEL.
CONFIDENCE @ 1-BASED HEURISTIC: We ob- tain the KG links for each the OIE slots using

---

the OlL manerr The linking characterized by the cosine similarity between the embeddings of each OIE slot and the KG entries. We then com- pute the softmax of the top-5 highest cosine sim- ilarities, and finally threshold the confidence@ to obtain a prediction; such that confidence T. indicates an out-of-KG instance, and con- fidence > Te indicates instance inside the KG, where T: the confidence threshold. We use Te [0.235; 0.260; 0.235] for detecting out-of-
each data point contained information about the provenance sentence, a KG fact that is contained in the sentence, and a corresponding OIE surface fact that was extracted from the sentence.
Two expert annotators annotated each data point independently of whether the KG fact matches the OIE extraction semantically (see two labelled examples in Table 8).
KG subjects, relations and objects respectively.
3.
We considered a data point as "correct" only if the two annotators agreed that the KG fact semantically matches the information in the OIE triple.
ENTROPY-BASED HEURISTIC: Similar to the confidence @ b-based method, we obtain the cosine similarities with the OlE However, instead of using the top-1 probability, we obtain the en- tropy of the top-5 predictions. We finally threshold the entropy obtain prediction, such that an entropy T. indicates out-of-KG instance, and an entropy < Te indicates an instance inside the KG, where Te is the entropy threshold. We use Te = [1.60; 1.58; 1.60] for detecting out-of-KG subjects, relations and objects respectively.
4. We computed accuracy, inter-annotator agree- ment and Kohen's kappa score.
We found that 97% of the data points are con- sidered "correcr" by both annotators. We also observed that the inter-annotator agreement high: the annotators agreed in 99% of the cases, with high Kohen's kappa score (McHugh, 2012) of 0.80. Please refer to the supplementary material for the subset of samples which was provided to the expert annotators.
QUERY-KEY-VALU VALUE CROSS-ATTENTION:
Using the Oll:Pre embeddings. we train lightweight query-key-value cross-attention module on top with weights that are initialized with the identity matrix- at the start of training the O Trnkerr embeddings are used as is. Given a query OIE slot embedding, the model attends over the KG entry embeddings representing the keys and values, and outputs sigmoid normalized score (i.e., probability) indicating the presence confidence of the OIE slot in the KG. During training. to obtain in-batch negatives, for each OIE slot we drop the KG entry counterpart with 50% probability. We obtain additional negatives by sampling KG entries from the whole KG, which do not match any of the OIE slots in the batch. We train the model using the binary cross-entropy loss, such that, if the corresponding KG entry for OIE slot is in the sampled KG subset, the model predicts a positive score averaged over the KG graph subset, and negative otherwise. We use uniform threshold of Ta = 0.3 for all three slots.
F
Data Quality
To assess FaLB's data quality, we performed man- ual human evaluation. In particular, did the following steps:
1. We randomly selected 100 data points, where

---

KG fact (IDs)
KG fact (names)
OIE Triple
Sentence: "Hekinoglu Ali Pasha Mosque built between 1734-1735 in the Fatth district of Istanbul by Hekimoğlu All Pusha, who was born in Istanbul in 1689."
(Q1584693; P19; Q406) (Hekimoglu Ali Pasha; place of birth; Istanbul)
("H. Ali Pasha", "was born in", "Istanbul")
Label: correct
Sentence: "Pierre Hétu (April 22, 1936 in Montreul December 3, 1998 in Montreal) was conducter and pianist." ("Pierre Hétu", "April in ", "Montreul")
(Q3385492, P19, Q340) (Pierre Hétu; place of birth; Montreal) Label: incorrect
Table 8: Example annotations for FaLB data. The annotator
the input sentence; (2) the KG fact with the
original IDs (thus, the user can further check the meaning of the entity and predicate); (3) the surface text of the KG fact; (4) OIE triple, extracted from the input sentence. The first matching is labelled as "correct", because the KG fact semantically matches the OIE triple. The second matching is labelled "incorrect". due to the incorrect OIE relation "April in".


Reviews:
[{"review_id": "02xR4U0RMm", "reviewer": "Reviewer_kopq", "summary": "", "questions": "1. What is the usefulness of reporting the slot linking performance? If the end task is to extract KG facts from text, does it matter if the model can correctly link one of the slots while incorrectly linking on the other slots?\n\n2. Why does SimCSE perform so poorly? It is mentioned in the paper that the OIE triples and KG facts were converted to sentences. How was this conversion done? Was  any LLM used for this?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for the extensive review and insightful comments!\n\n> No clear comparison between the proposed dataset and existing large-scale datasets (e.g., T-Rex, ReVerb45K, etc.)\n\nWe thank the reviewer for the suggestion. We do make a comparison between these datasets and ours in the paper, though perhaps this is not as clearly articulated, as the reviewer suggests. Explicitly, our benchmark is superior to ReVerb45K in the following aspects:\n\n1. In ReVerb45k, the links of the OIE entities to KG entities are *not golden* (i.e., human validated), but rather automatically obtained with a very outdated entity linker [1]. Therefore, as we state in the paper (L178 -- L183): the ``poor performance of the entity linker caps the fact linking models' performance'' on ReVerb45k.\n\n2. The number of unique OIE entities & relations in ReVerb45k is drastically smaller than our benchmark, making our dataset a much more challenging benchmark for fact linking models. \n\n|           | # OIE Entities | # OIE relations |\n|-----------|----------------|-----------------|\n| ReVerb45k | 28k            | 22k             |\n| Ours      | 936k           | 160k            |\n|           |                |                 |\n\n\n3. ReVerb45k provides a single-facet of OIE evaluation: in which different types/sources of errors are conflated. In contrast, our benchmark decouples these and allows for evaluation of mutually very different OIE-to-KG linking scenarios: Transductive, Inductive, Polysemous, (and Out-of-KG, not supported by ReVerb45k).\n\n4. ReVerb45k is based a confidence threshold of an automatic aligner, while our data is manually validated by two independent annotators -- inter-annotator agreement of 99%; Kohen’s kappa of 0.80 (L291); \n\n5. In ReVerb45k there is no Out-of-KG testing data, which makes it inadequate to test a scenario of immense practical value: recognizing OIE extractions that correspond to facts consisting of entities and/or relations that are not in the KG at all. \n\nIn a similaer fashion, T-Rex differs from our benchmark in the following:\n\n1. T-Rex does not support neither the inductive evaluation, nor the Out-of-KG evaluation, both critical for the applicability of fact linking models and systems in real-world applications (where entities and/or relations for which the predictions need to be made were not seen in model training or are not present in the KG at all). In other words, the fact linking performance on T-Rex does not really measure models'  usefulness \"in the wild\".   \n\n2. The dataset is primarily constructed for text-to-KG generation, while our dataset is meant to link OIEs to large-scale KGs.\n\nWe will articulate these critical differences w.r.t. to both datasets more clearly in the final version of the paper, perhaps in a tabular overview.\n\n> Lack of strong baselines: the method is overwhelmingly better than the baselines. There are existing works (e.g., KBPearl, CESI) that could serve as a strong baseline.\n\nWe agree that KBPearl would be a nice baseline to compare against. Unfortunately, the KBPearl code is not publicly available and, we therefore could not compare it on our benchmark. As for CESI, it solves a different problem, which makes it incompatible to our study. In particular, CESI deals with entity and relation *canonicalization*, and not *linking*. That means that if there are entity mentions like \"Joe Biden\", \"Joseph Biden\" and \"46th President of USA\", CESI should be able to group them into an entity synset. CESI, however, is not able to link these entity synsets to a unique KG entity. In the final version of the paper, we will include these references and explain these differences clearly.\n\n> Usefulness of reporting the OIE slot linking performance? The end task is extracting KG facts from text, does it matter if the model can correctly link one of the slots while incorrectly linking on the other slots?\n\nThe goal is to pinpoint the source of error, which makes the benchmark more interpretable and useful for subsequent diagnostics. For example, Gashteovski et al. 2022 [2] showed that OIE systems struggle with extracting objects. Many other NLP benchmarks aim at such fine-grained measurements as well [3, 4, 5, 6], because they provide insights on models' failures. Relying on single-score metrics (e.g., only fact-level accuracy) limits the analysis of the sources of models' errors. \n\n> Poor performance of SimCSE?\n\nSimCSE is an unsupervised sentence embedding model. On the other hand, the OIE-to-KG fact linking task requires methods to link an OIE mention (entity or relationship) with a large-scale KG that contains 5M entities and 4k predicates (e.g., in Wikidata). Because of that, models not explicitly trained for this task (including SimCSE) are suboptimal. \n\n### References\n\n[1] Lin et al., 2012. Entity Linking at Web Scale. https://aclanthology.org/W12-3016/ \n\n[2] Gashteovski et al., 2022. BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation. https://aclanthology.org/2022.acl-long.307/\n\n[3] Hupkes et al., 2022. State-of-the-art generalisation research in NLP: A taxonomy and review https://arxiv.org/pdf/2210.03050.pdf\n\n[4] Liu et al., 2021. ExplainaBoard: An Explainable Leaderboard for NLP. https://aclanthology.org/2021.acl-demo.34/\n\n[5] Ye et al., 2021. Towards More Fine-grained and Reliable NLP Performance Prediction. https://aclanthology.org/2021.eacl-main.324/\n\n[6] Fu et al., 2020. Interpretable Multi-dataset Evaluation for Named Entity Recognition. https://aclanthology.org/2020.emnlp-main.489/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed data set can be used as a new benchmark for KG fact extraction from text.\n2. The dataset curation process considers a wide variety of settings which can serve a test bed for existing and future models for this task.\n3. The paper proposes a solution that works well on the proposed data set.\n", "reasons_to_reject": "1. No clear comparison is made between the proposed data set and other existing large scale datasets (e.g., T-Rex, ReVerb45K, etc.). Such comparisons would make it clear why the community should adopt this data set instead of existing ones.\n2. Lack of strong baselines. The proposed method is overwhelmingly better than any of the baselines. There are existing works of fact extraction from text (e.g., KBPearl, CESI) that could serve as a strong baseline.\n", "questions_to_authors": "1. What is the usefulness of reporting the slot linking performance? If the end task is to extract KG facts from text, does it matter if the model can correctly link one of the slots while incorrectly linking on the other slots?\n\n2. Why does SimCSE perform so poorly? It is mentioned in the paper that the OIE triples and KG facts were converted to sentences. How was this conversion done? Was  any LLM used for this?\n", "paper_topic_and_main_contributions": "The paper curates a new data set for the task of extracting canonicalized facts from text. The proposed method uses existing Open IE techniques to extract triples from sentences and maps them to a target knowledge graph (Wikidata). The data curation process includes transductive (entities and relations are seen) and inductive settings (some of entities and relations are unseen) as well as  polysemous and out-of-KG entities. The paper also proposes a model that maps OIE triples to KG facts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* L250-L261 can be presented as a table."}, "extra_text": {"Missing_References": "1. KBPearl: A Knowledge Base Population System Supported by Joint Entity and Relation Linking, Lin et al., 2020\n2. CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information, Vashisth et al., 2019\n"}}, {"review_id": "9sRDQSbxJS", "reviewer": "Reviewer_PQB3", "summary": "", "questions": "What is the main novelty of the framework(fig. 2)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Limited novelty of the paper\n\nand\n\n>Main novelty of the framework (Fig. 2)?\n\nThe main contributions (i.e., novelty) of our work is in the following: (1) proposed task -- fact linking, including four mutually complementary evaluation facets (transductive, inductive, polysemous, Out-of-KG) that allow for a comprehensive evaluation and profiling of OIE-to-KG linking models, (2) the proposed and created benchmark that allows to actually evaluate the usefulness of linking models for real-world applications targeting KGs augmentation. The implemented baseline model(s) do not represent the central contribution of this work, i.e., the goal of the model itself is not be be novel, but to represent a solid baseline on the new task(s)/benchmark: *we propose a novel benchmark dataset*, not a novel model. The strong (baseline) models we propose for this task are based on the pre-rank & re-rank paradigm, which are commonly used in image-text matching and retrieval.\n\n>Experimental results in Table 1 are somewhat unbelievable.\n\nCould the reviewer be more specific? What in particular is \"unbelievable\"? The results in Table 1 report the OIE-to-KG facts linking accuracy for different aspects of our benchmark and for a series of models. The experimental setup, a discussion of the results, and their implication are detailed in Section 5.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper gave a valuable dataset for OIE-to-KG linking. It would also be beneficial for those who considering the OIE-to-KG linking problem.", "reasons_to_reject": "The novelty is somewhat limited, making the model is a combination of existing works.", "questions_to_authors": "What is the main novelty of the framework(fig. 2)?", "paper_topic_and_main_contributions": "To bridge the gap between the schema-free surface facts extracted from text and schema-fixed KG knowledge., the authors move away from unrealistic and incomplete assumptions of prior work and proposed a large-scale dataset for OIE-to-KG linking, multifaceted evaluation protocols that cover all aspects of linking OIE facts to KGs, and OIE-to-KG fact linking models. Extensive computational experiments on the dataset with the several evaluations showed the effectiveness of the framework.\n\nStrengths:\n1.Sufficient technical details, showing the transformation process;\n2.Extensive computational experiments results showed the effectiveness of the framework;\n3.The paper is very well written with clarity, easy-to-follow logic, and adequate discussions.\n\n\nWeaknesses:\n1.The novelty is somewhat limited, making the model is a combination of existing works;\n2.The experimental results in Table 1 are somewhat unbelievable.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn’t find any missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "The structure of this paper is complete and English writing skills used in the paper is good. I didn’t find any obvious writing problems in the paper."}, "extra_text": {}}, {"review_id": "qok8Wd2eIb", "reviewer": "Reviewer_WyCA", "summary": "", "questions": "Question A (line 078): Why is the setting where entity slots are a priori linked to KG entities unrealistic? \nThis could come from a pipeline system. The second system is just responsible for relation linking.\n\nQuestion B: Is this setup of just linking triples without the context sentence realistic?\n\nQuestion C (Section 3, High-precision OIE-KG Fact Alignments): The text suggests that if you match the subject and object of a triple with a subject and object of any KG fact then you consider it as a match. It means that you use all the predicates in KG between those entities. Aren't you using the already existing alignments mentioned in the Section 2 - Sentence-to-KG-dataset?\n\nQuestion D: (Section 3, Data Augmentation to Increase Diversity): Do you also modify the original sentences with the given aliases? If not, then the future methods can easily learn shortcuts - if the triple subject/object doesn't occur in the source sentence then it's definitely in KG (because its alias was taken from KG).\n\nQuestion E: (Section 3, Data Augmentation to Increase Diversity): What about pronouns or common nouns referring to the entity? The setup without them seems to be a bit unrealistic.\n\nQuestion F (Section 4.1 Linking OIEs to KG Facts): In the embeddings you encode separately the information about subject, object and predicate. How do you deal with the contextual information then? If not, then for Michael Jordan slot entity \"Michael Jordan basketball player\" will be retrieved and there will be no room for disambiguation procedure. \n\nQuestion G (Line 469): Does it mean you rerank 3^3 fact candidates (three entries per slot).\n\nQuestion H: You report accuracy. Would precision/recall/f1 be more suitable to this task?\n\nQuestion I: regarding SynthIE - If LLM is able to generate a sentence based on the fact, could it generate a fact based on a sentence? Should LLMs be considered as a baseline?\n\nQuestion J (Section 5.1 Ablation study) \"We observe that training with entity aliases allows us to link such OIE entity mentions more successfully than training without them\" - where can it be observed? The scores on the non augmented test set get worse for subjects/objects.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for an extensive and insightful review! \n\nWe are, however, of the impression that the bulk of the reviewer's concerns (which we address with additional experiments and results reported below) relate to the baseline models which we evaluated and the fact that they do not leverage the sentential context of the OIE extraction. While the reviewer makes a very valid point -- which is why we additionally evaluated a baseline model that does leverage the sentence context -- we would like to emphasize that the baseline models are really not the main contribution of our work. Rather, it is the task (fact linking) and the comprehensive (and large) benchmark for evaluating linking models according to several facets that, crucially, reflect the usefulness of such models in real-world KG completion/augmentation applications (i.e., in particular the inductive and out-of-KG evaluation facets).   \n\nBelow we address the invididual points raised by the reviewer one by one. \n\n> An artificial setup where only triple slots (subj, relation, object) are used for linking without the use of the sentence in the context; Weak baselines (that don't use the context sentence).\n\nWe would not describe this as an artificial setting. Indeed, the source sentence can be used to aid the linking, and, if such extra context is relevant to the task (may or may not be), it should lead to improved performance. Based on your suggestion, we train a model where we the source sentence (i.e., the sentence from where the OIE is obtained) is provided to the OIE pre-ranker model to generate OIE embeddings conditioned on the source sentence. Initially (in the experiments reported in the paper), the OIE is structured in the following way (L356 -- L367):\n\n<SUBJ> Michael Jordan <REL> grew up in <OBJ> Wilmington\n\nand is provided to RoBERTa to obtain OIE slot embeddings. Now, we add the source sentence together with another separator special token (<SENT>) in the following way:\n\n<SUBJ> Michael Jordan <REL> grew up in <OBJ> Wilmington <SENT> Michael Jordan is a famous basketball player who grew up in Wilmington.\n\nThen, as previously, we obtain OIE slot embeddings using the RoBERTa model. We report results of this sentence-augmented baseline below on our three (REBEL-based) OIE-to-KG fact linking splits: Transductive, Inductive, and Polysemous; and on the two versions of the the Knowledge Graph: Benchmark Restricted Knowledge Graph  (650k entities, 700 predicates) and Large Knowledge Graph (5M entities, 4k predicates). \n\n(OIE): Indicates just using the OIE to obtain the slot embeddings (as reported initially in the submitted paper)\n(OIE + Sent.): Indicates using the sentence context in addition to the OIE to obtain the slot embeddings\n\n|  | Benchmark Restricted Knowledge Graph | Large Knowledge Graph |\n|---|---|---|\n| Transductive (OIE) | 79.1 ± 0.1 | 70.7 ± 0.2 |\n| Transductive (OIE + Sent.) | 76.3 ± 0.2 | 68.2 ± 0.21 |\n| Inductive (OIE) | 34.9 ± 0.5 | 25.4 ± 0.4 |\n| Inductive (OIE + Sent.) | 37.4 ± 0.5 | 27.14 ± 0.44 |\n| Polysemous (OIE) | 62.5 ± 0.4 | 51.7 ± 0.4 |\n| Polysemous (OIE + Sent.) | 70.0 ± 0.37 | 62.5 ± 0.4 |\n\nFrom the reported results we can conclude that, leveraging the sentence context helps linking inductive and especially polysemous entity mentions, but not in the transductive task. In those cases, the context provided by the source sentence aids the linking performance, as it, expectedly, helps generalization and disambiguation. In the case of transductive OIEs, the sentence context hurts the performance as these OIE mentions have already been observed during training -- the sentential context here presumably \"noisifies\" the representations for these entities observed in training. We will add this baseline (OIE + Sent.) in Table 1. \n\nWe would, however, like to emphasize once mode that the baseline models are not really the central contribution of our work: it is the multi-faceted evaluation benchmark for fact linking (the contribution that the reviewer is appreciative of, as far as we can tell from the review listed reasons for acceptance).  \n\n> Data augmentation (paraphrasing / using different aliases for entities) applied on the triple level and not on the sentence level, which wouldn't allow to use source sentence (because the model could use the mismatch information between the surface form in the triple and the sentence)\n\nWe do not think that this is correct. Namely, in the experimental results reported above, we perform the entity alias augmentation on both the triple level and the corresponding sentence added to the context (after the special <SENT> token). From an implementation standpoint, we locate both entities in the source sentence (via regex matching), and simply replace them with their aliases in both the OIE triple and the source sentence. \n\n> Question A: Why is the setting where entity slots are a priori linked to KG entities unrealistic? This could come from a pipeline system. The second system is just responsible for relation linking.\n\nThis is realistic for a model, it is not realistic (or perhaps a more suitable word here is \"appropriate\") as a starting point for creating a fact-linking benchmark, if the entity linking has been carried out automatically by an (imperfect) entity linking model (and there exist no large-scale manually entity-linked datasets). Erroneously linked entities leads to erroneous facts -- evaluating fact linking models  on such facts is then misleading and not really indicative of models' true performance. (see also reply to to R1 regarding the ReVerb45K dataset).     \n\n> Question B: Is this setup of just linking triples without the context sentence realistic?\n\nThe main contribution of the paper is the comprehensive multi-faceted benchmark that allows one to evaluate both models that leverage only the information in the OIE triples (as we did in the original submission) as well as those that additionally leverage the sentence from which the OIE triple is extracted (as we did in the additional experiment reported above). As the results above show, having sentential context is not always beneficial, it depends on the task -- it does not help, for example, in transductive linking, but is beneficial in inductive and especially polysemous linking. \n\nArguably, there are many different ways in which one could leverage/incorporate the sentence information in the fact-linking model: it was not our aim to extensively evaluate a wide range of plausible models but rather provide a most comprehensive benchmark to date (one that tests the linking models for things that existing benchmarks do not, but which are critical in real-world application of those models) so and offer it to the research community to catalyze efforts towards more robust fact linking models.\n\n> Question C (Section 3, High-precision OIE-KG Fact Alignments): The text suggests that if you match the subject and object of a triple with a subject and object of any KG fact then you consider it as a match. It means that you use all the predicates in KG between those entities. Aren't you using the already existing alignments mentioned in the Section 2 - Sentence-to-KG-dataset?\n\nWe need to do high-precision OIE-to-KG, not sentence-to-KG (i.e., we need to know exactly to which KG entities the OIE slots correspond to; yet we start from a dataset in which the KG entities are assigned to sentences). This is why it is crucial to ensure that the OIE triples are of high quality.\n\n> Question D: (Section 3, Data Augmentation to Increase Diversity): Do you also modify the original sentences with the given aliases? If not, then the future methods can easily learn shortcuts - if the triple subject/object doesn't occur in the source sentence then it's definitely in KG (because its alias was taken from KG).\n\nYes we do, as we stated above: we perform the entity alias augmentation on both the triple level and the (context) sentence level.\n\n> Question E: (Section 3, Data Augmentation to Increase Diversity): What about pronouns or common nouns referring to the entity? The setup without them seems to be a bit unrealistic.\n\nIf the pronouns and common nouns are present in the data, then we include them. That being said, we do include common nouns, because they appear as Wikidata aliases (e.g., our links are also for concepts such as \"train\", not just named entities like names of people). Pronouns are not present here, because they are not part of the manually curated data (Wikidata). If we artificially add pronouns and common nouns, we introduce bias and reduce the quality of our data.\n\n> Question F (Section 4.1 Linking OIEs to KG Facts): In the embeddings you encode separately the information about subject, object, and predicate. How do you deal with the contextual information then? If not, then for Michael Jordan slot entity \"Michael Jordan basketball player\" will be retrieved and there will be no room for a disambiguation procedure.\n\nWe obtain a separate embedding for each OIE slot, where each OIE slot embedding (subj, relation or object) is contextualized by the other. In other words, the entire triple is the input to the RoBERTa encoder, so tokens of each slot are contextualized with the other two slots (this is detailed in L356-367). \n\n> Question G (Line 469): Does it mean you rerank 3^3 fact candidates (three entries per slot).\n\nYes. We retrieve the top-3 candidates for each OIE slot, and construct all possible OIE-KG fact combinations (3^3).\n\n> Question H: Would precision/recall/f1 be more suitable to this task instead of accuracy?\n\nWe follow standard practice of entity linking methods [1] and report accuracy.\n\n> Question I: In SynthIE, If an LLM is able to generate a sentence based on the fact, could it generate a fact based on a sentence? Should LLMs be considered as a baseline?\n\nWe do not see a straightforward way to use an LLM for OIE-to-KG fact linking (or sentence-to-KG fact linking), as the LLM would need to be able to access to the entire KG: this amount of data by far exceeds what even the largest LLMs could store in their context. In fact, the main premise of SynthIE [2] is that LLMs can generate synthetic data even for tasks that cannot be solved directly by the LLM. They show that, for problems with structured outputs, an LLM can be prompted to perform the task in the opposite direction (i.e., to generate plausible text for the target structure).\n\n> Question J (Section 5.1 Ablation study) \"We observe that training with entity aliases allows us to link such OIE entity mentions more successfully than training without them\" - where can it be observed? The scores on the non augmented test set get worse for subjects/objects.\n\nWe apologize for our ambiguous wording. We here refer to the results from Table 3 where we only evaluate on entity aliases (numbers on REBEL, same trend can be observed on SynthIE):\n\n| Training aliases | Testing aliases | Subject | Relation | Object | Fact |\n|---|---|---|---|---|---|\n| ✗ | ✓ | 61.6 ± 0.2 | 64.8 ± 0.2 | 41.5 ± 0.2 | 25.2 ± 0.2 |\n| ✓ | ✓ | 79.0 ± 0.2 | 92.2 ± 0.1 | 89.0 ± 0.1 | 67.9 ± 0.2 |\n\nIndeed, training with entity aliases (row 2) allows us to link such (aliased) OIE entity mentions more successfully than if we perform training without entity aliases (row 1).\n\n### References\n\n[1] We et al., 2019. Scalable Zero-shot Entity Linking with Dense Entity Retrieval\n\n[2] Josifovski et al., 2023. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A novel benchmark for OIE-to-KG linking that covers multiple aspects of OIE to KG linking, as well as Out-of-KG detection.\n - Evaluation with multiple baselines.\n - Clear presentation.", "reasons_to_reject": "- An artificial setup where only triple slots (subj, obj, pred) are used for linking without the use of the sentence in the context.\n - Weak baselines (that don't use the context sentence)\n - A data augmentation (paraphrasing / using different aliases for entities) that is applied on the triple level and not on the sentence level, which wouldn't allow to use source sentence (because the model could use the mismatch information between the surface form in the triple and the sentence)", "questions_to_authors": "Question A (line 078): Why is the setting where entity slots are a priori linked to KG entities unrealistic? \nThis could come from a pipeline system. The second system is just responsible for relation linking.\n\nQuestion B: Is this setup of just linking triples without the context sentence realistic?\n\nQuestion C (Section 3, High-precision OIE-KG Fact Alignments): The text suggests that if you match the subject and object of a triple with a subject and object of any KG fact then you consider it as a match. It means that you use all the predicates in KG between those entities. Aren't you using the already existing alignments mentioned in the Section 2 - Sentence-to-KG-dataset?\n\nQuestion D: (Section 3, Data Augmentation to Increase Diversity): Do you also modify the original sentences with the given aliases? If not, then the future methods can easily learn shortcuts - if the triple subject/object doesn't occur in the source sentence then it's definitely in KG (because its alias was taken from KG).\n\nQuestion E: (Section 3, Data Augmentation to Increase Diversity): What about pronouns or common nouns referring to the entity? The setup without them seems to be a bit unrealistic.\n\nQuestion F (Section 4.1 Linking OIEs to KG Facts): In the embeddings you encode separately the information about subject, object and predicate. How do you deal with the contextual information then? If not, then for Michael Jordan slot entity \"Michael Jordan basketball player\" will be retrieved and there will be no room for disambiguation procedure. \n\nQuestion G (Line 469): Does it mean you rerank 3^3 fact candidates (three entries per slot).\n\nQuestion H: You report accuracy. Would precision/recall/f1 be more suitable to this task?\n\nQuestion I: regarding SynthIE - If LLM is able to generate a sentence based on the fact, could it generate a fact based on a sentence? Should LLMs be considered as a baseline?\n\nQuestion J (Section 5.1 Ablation study) \"We observe that training with entity aliases allows us to link such OIE entity mentions more successfully than training without them\" - where can it be observed? The scores on the non augmented test set get worse for subjects/objects.", "paper_topic_and_main_contributions": "The authors introduce a new large-scale benchmark for OIE-to-KG linking. The benchmark assesses the performance of fact linking at a granular triple slot level. It also measures whether a system can recognize that a surfaceform does not have a match in the existing knowledge graph.\n\nThe benchmark defines multifaceted evaluation protocols and proposes several baselines. The evaluation focuses on:\n- Transductive fact linking (linking entities and predicates seen during training)\n- Inductive fact linking (test entities are not seen during training; measuring the generalization of fact linking models over entities)\n - Polysemous fact linking (entities are ambiguous)\n - Out-of-KG detection\n\nThe authors also show that, with the current baseline, it is possible to detect out-of-KG entities, but it is harder to detect out-of-KG relations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new benchmark (dataset and evaluations) for OIE fact linking. The main focus of the benchmark is the different aspects of OIE linking (seen vs unseen vs polysemous entities), as well as out-of-KG detection.

All three reviewers find the proposed benchmark as valuable resource and, as Reviewer kopq points out, the dataset curation process itself is an interesting contribution. The lack of extensive comparison to other existing datasets was brought up as an issue, but this has been addressed by the authors and should be included in the revised paper. 

All three viewers pointed to the lack of strong baselines as an issue, with Reviewer WyCA explicitly calling out the lack of models that used the context sentence as input. While I agree with the authors that the benchmark itself and not the baselines are the main contribution of the paper, I think that using the strongest baselines possible would better demonstrate the usefulness of the resource. In response to Reviewer WyCA, the authors have included an additional baseline (that uses the context sentence) and this - in part - addresses the reviewers' main concern. The issues regarding the novelty (presumably of the baselines) raised by reviewer PQB3 were not backed up with further clarifications and were not considered further.
Decision: Accept-Main