SlowGuess/ACL_2020_OCR huggingface dataset:
Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, Rainer Gemulla
Data and Web Science Group
University of Mannheim, Germany
broscheit@informatik.uni-mannheim.de,
{k.gashteovski,ywang,rgemulla}@uni-mannheim.de,
Open Information Extraction systems extract ("subject text", "relation text", "object text") triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task, i.e., predicting test facts by completing ("subject text", "relation text", ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OLPBENCH. We performed experiments with a prototypical knowledge graph embedding model for open link prediction. While the task is very challenging, our results suggest that it is possible to predict genuinely new facts, which can not be trivially explained.
A knowledge graph (KG) (Hayes-Roth, 1983) is a set of (subject, relation, object)-triples, where the subject and object correspond to vertices, and relations to labeled edges. In curated KGs, each triple is fully disambiguated against a fixed vocabulary of entities<sup>1</sup> and relations.
An application for KGs, for example, is the problem of drug discovery based on bio-medical knowledge (Mohamed et al., 2019). The construction of a curated bio-medical KG, which is required for
Figure 1: Entities and relations in curated knowledge graphs vs. open knowledge graphs.
such an approach, is challenging and constrained by the available amount of human effort and domain expertise. Many tools that could assist humans in KG construction (e.g., an entity linker) need a KG to begin with. Moreover, current methods for KG construction often rely on the rich structure of Wikipedia, such as links and infoboxes, which are not available for every domain. Therefore, we ask if it is possible to make predictions about, for example, new drug applications from raw text without the intermediate step of KG construction.
Open information extraction systems (OIE) (Etzioni et al., 2011) automatically extract ("subject text", "relation text", "object text")-triples from unstructured data such as text. We can view OIE data as an open knowledge graph (OKG) (Galarraga et al., 2014), in which vertices correspond to mentions of entities and edges to open relations (see Fig. 1). Our overarching interest is whether and how we can reason over an OKG without any canonicalization and without any supervision on its latent factual knowledge. The focus of this study are the challenges of benchmarking the inference abilities of models in such a setup.
A common task that requires reasoning over a
<sup>1</sup>For brevity, "entities" denotes both entities (e.g. Prince) and concepts (e.g. musician) throughout the paper.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2296-2308 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Comparing evaluation of link prediction and open link prediction.
KG is link prediction (LP). The goal of LP is to predict missing facts in a KG. In general, LP is defined as answering questions such as (NBC, headquartered In, ?) or (?, headquartered In, New York City); see Fig. 2a. In OKGs, we define open link prediction (OLP) as follows: Given an OKG and a question consisting of an entity mention and an open relation, predict mentions as answers. A predicted mention is correct if it is a mention of the correct answer entity. For example, given the question ("NBC-TV", "has office in", ?), correct answers include "NYC" and "New York"; see Fig. 2b).
To evaluate LP performance, the LP model is trained on known facts and evaluated to predict unknown facts, i.e., facts not seen during training. A simple but problematic way to transfer this approach to OKGs is to sample a set of evaluation triples from the OKG and to use the remaining part of the OKG for training. To see why this approach is problematic, consider the test triple ("NBC-TV", "has office in", "New York") and suppose that the triple ("NBC", "has headquarters in", "NYC") is also part of the OKG. The latter triple essentially leaks the test fact. If we do not remove such facts from the training data, a successful models only paraphrases known facts but does not perform reasoning, i.e., does not predict genuinely new facts.
Furthermore, we also want to quantify if there are other trivial explanations for the prediction of an evaluation fact. For example, how much can be predicted with simple popularity statistics, i.e., only the mention, e.g. ("NBC-TV", ?), or only the relation, e.g. ("has office in", ?). Such non-relational information also does not require reasoning over the graph.
To experimentally explore whether it is possible to predict new facts, we focus on knowledge graph embedding (KGE) models (Nickel et al., 2016), which have been applied successfully to LP in KGs. Such models can be easily extended to handle the surface forms of mentions and open relations.
Our contributions are as follows: We propose the OLP task, an OLP evaluation protocol, and a method to create an OLP benchmark dataset. Using the latter method, we created a large OLP benchmark called OLPBENCH, which was derived from the state-of-the-art OIE corpus OPIEC (Gashteovski et al., 2019). OLPBENCH contains 30M open triples, 1M distinct open relations and 2.5M distinct mentions of approximately 800K entities. We investigate the effect of paraphrasing and non-relational information on the performance of a prototypical KGE model for OLP. We also investigate the influence of entity knowledge during model selection with different types of validation data. For training KGE models on such large datasets, we describe an efficient training method.
In our experiments, we found the OLP task and OLPBENCH to be very challenging. Still, the KGE model we considered was able to predict genuinely new facts. We also show that paraphrasing and non-relational information can indeed dilute performance evaluation, but can be remedied by appropriate dataset construction and experimental settings.
2 Open Knowledge Graphs
OKGs can be constructed in a fully automatic way. They are open in that they do not require a vocabulary of entities and relations. For this reason, they can capture more information than curated KGs. For example, different entity mentions can refer to different versions of an entity at different points of time, e.g., "Senator Barack Obama" and "President Barack Obama". Similarly, relations may be of varying specificity: headquartered in may be expressed directly by open relations such as "be based in" or "operate from" but may also be implied by "relocated their offices to". In contrast to KGs, OKGs contain rich conceptual knowledge. For example, the triple ("a class action lawsuit", "is brought by", "shareholders") does not directly encode entity knowledge, although it does provide information about entities that link to "a class action lawsuit" or "shareholders".
OKGs tend to be noisier and the factual knowledge is less certain than in a KG, however. They
2297
Figure 3: Mention-ranking protocol: Example for computing the filtered rank for a test question.
can not directly replace KGs. OKGs have mostly been used as a weak augmentation to KGs, e.g., to infer new unseen entities or to aid link prediction (see App. A for a comprehensive discussion of related work). Much of prior work that solely leverages OKGs without a reference KG—and therein is closest to our work—focused on canonicalization and left inference as a follow-up step (Cohen et al., 2000, inter alia). In contrast, we propose to evaluate inference in OKGs with OLP directly.
3 Open Link Prediction
The open link prediction task is based on the link prediction task for KGs (Nickel et al., 2016), which we describe first. Let  $\mathcal{E}$  be a set of entities,  $\mathcal{R}$  be a set of relations, and  $\mathcal{T} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$  be a knowledge graph. Consider questions of the form  $q_{h} = (?, k, j)$  or  $q_{t} = (i, k, ?)$ , where  $i, j \in \mathcal{E}$  is a head and tail entity, respectively, and  $k \in \mathcal{R}$  is a relation. The link prediction problem is to provide answers that are correct but not yet present in  $\mathcal{T}$ .
In OKGs, only mentions of entities and open relations are observed. We model each entity mention and each open relation as a non-empty sequence of tokens from some vocabulary  $\mathcal{V}$  (e.g., a set of words). Denote by  $\mathcal{M} = \mathcal{V}^+$  the set of all such sequences and observe that  $\mathcal{M}$  is unbounded. An open knowledge graph  $\mathcal{T} \subset \mathcal{M} \times \mathcal{M} \times \mathcal{M}$  consists of triples of form  $(i, k, j)$ , where  $i, j \in \mathcal{M}$  are head and tail entity mentions, resp., and  $k \in \mathcal{M}$  is an open relation. Note that we overload notation for readability:  $i, j$ , and  $k$  refer to entity mentions and open relations in OKGs, but to disambiguated entities and relations in KGs. The intended meaning
will always be clear from the context. We denote by  $\mathcal{M}(\mathcal{E})$  and  $\mathcal{M}(\mathcal{R})$  the sets of entity and relations present in  $\mathcal{T}$ , respectively. The open link prediction task is to predict new and correct answers to questions  $(i, k, ?)$  or  $(?, k, j)$ . Answers are taken from  $\mathcal{M}(\mathcal{E})$ , whereas questions may refer to arbitrary mentions of entities and open relations from  $\mathcal{M}$ . For example, for the question ("NBC-TV", "has office in", ?), we expect an answer from the set of mentions {"New York", "NYC", ...} of the entity NewYorkCity. Informally, an answer  $(i, k, j)$  is correct if there is a correct triple  $(e_1, r, e_2)$ , where  $e_1$  and  $e_2$  are entities and  $r$  is a relation, such that  $i, j$ , and  $k$  are mentions of  $e_1, e_2$ , and  $r$ , respectively.
3.1 Evaluation protocol
To describe our proposed evaluation protocol, we first revisit the most commonly used methodology to evaluate link prediction methods for KGs, i.e., the entity-ranking protocol (Bordes et al., 2013). Then, we discuss its adaptation to OLP, which we call the mention-ranking protocol (see Fig. 3).
KGs and entity ranking. For each triple  $z = (i, k, j)$  in the evaluation data, a link prediction model ranks the answers for two questions,  $q_{t}(z) = (i, k, ?)$  and  $q_{h}(z) = (?, k, j)$ . The model is evaluated based on the ranks of the correct entities  $j$  and  $i$ ; this setting is called raw. When true answers for  $q_{t}(z)$  and  $q_{h}(z)$  other than  $j$  and  $i$  are filtered from the rankings, then the setting is called filtered.
OKGs and mention ranking. In OLP, the model predicts a ranked list of mentions. But questions might have multiple equivalent true answers,
i.e., answers that refer to the same entity but use different mentions. Our evaluation metrics are based on the highest rank of a correct answer mention in the ranking. For the filtered setting, the mentions of known answer entities other than the evaluated entity are filtered from the ranking. This mention-ranking protocol thus uses knowledge of alternative mentions of the entity in the evaluation triple to obtain a suitable ranking. The mention-ranking protocol therefore requires (i) ground truth annotations for the entity mentions in the head and tail of the evaluation data, and (ii) a comprehensive set of mentions for these entities.
4 Creating the Open Link Prediction Benchmark OLPBENCH
An OLP benchmark should enable us to evaluate a model's capability to predict genuinely new facts, i.e., facts can not be trivially derived. Due to the nature of OKGs, paraphrasing of facts may leak facts from validation and test data into training, making the prediction of such evaluation facts trivial. Nevertheless, the creation of training and validation data should require as little human effort as possible so that the methodology can be readily applied to new domains. Our mention-ranking protocol uses knowledge about entities for disambiguation (of the evaluation data, not the training data), however, which requires human effort to create. We investigate experimentally to what extent this entity knowledge is necessary for model selection and, in turn, how much manual effort is required to create a suitable validation dataset.
In the following, we describe the source dataset of OLPBENCH and discuss how we addressed the points above to create evaluation and training data.
4.1 Source Dataset
OLPBENCH is based on OPIEC (Gashteovski et al., 2019), a recently published dataset of OIE triples that were extracted from the text of English Wikipedia with the state-of-the-art OIE system MinIE (Gashteovski et al., 2017). We used a subset of 30M distinct triples, comprised of 2.5M entity mentions and 1M open relations. In 1.25M of these triples, the subject and the object contained a Wikipedia link. Fig. 4 shows how a Wikipedia link is used to disambiguate a triple's subject and object mentions. Tab. 1 shows an excerpt from the unlinked and linked triples. For the evaluation protocol, we collected a dictionary, where each entity
Figure 4: Example for a (subject, relation, object) triple extracted from Wikipedia. With a Wikipedia hyperlink, a mention is disambiguated to its entity. Inversely this yields a mapping from an entity to all its mentions.
is mapped to all possible mentions. See App. B for more details about the dataset creation.
4.2 Evaluation Data
From the source dataset, we created validation and test data with the following requirements:
Data quality. The evaluation data should be challenging, and noise should be limited as much as possible. We chose a pragmatic and easy heuristic: we did not consider short relations with less than three tokens as candidates for sampling evaluation data. This decision was based on the following observations: (i) Due to the OPIEC's extractions, short relations—e.g. ("kerry s. walters", "is", "professor emeritus")—are often subsumed by longer relations—e.g. ("kerry s. walters", "is professor emeritus of", "philosophy")—, which would always lead to leakage from the longer relation to the shorter relation. (ii) Longer relations are less likely to be easily captured by simple patterns that are already successfully used by KG construction methods, e.g. ("elizabeth of hungary", "is the last member of", "the house of árpád"). We conjecture that long relations are more interesting for evaluation to measure progress in reasoning with OKG data. (iii) The automatically extracted entity annotations were slightly noisier for short relations; e.g., ("marc anthony", "is" "singer") had the object entity annotation SinFrenos.
Human effort for data creation. The mention-ranking protocol uses knowledge about entities for disambiguation. We want to experimentally quantify the influence of this entity knowledge on model selection, i.e., whether entity knowledge is necessary to find a good model. If so, human expertise is necessary to create the validation data. While our goal is to require almost no human domain expertise to learn a good model, the size of validation data is much smaller than the size of the training data. Therefore, this effort—if helpful—may be
subject	relation	object	subject mentions	object mentions
Unlinked	conway	has	plot
henry s. conway	is	field marshal
conway tearle	has	members
highway 319	begins outside	conway
bloomsbury	bought	conway publishing
mike conway	is teammate of	toyota
w. conway gordon	served as adc to	gen. p. maitland
w. conway gordon	entered	the service
Linked	willam conway	distinguished himself during	civil war	willam conway	civil war
terry venables	is manager of	fc barcelona	conway	american civil war
terry venables	fc barcelona
f.c. barcelona
futbol club barcelona
cf barcelona
barcelona
background music	is composed by	hikaru nanase	the background music	masumi ito
background music	hikaru nanase
Table 1: Example from the unlinked and linked data in OLPBENCH. For the unlinked data, we show the first of 3443 triples from the unlinked data containing the token "conway". For the linked data, we show the triples and also the alternative mentions for their entities. The first linked triple is about William_Conway_(U.S._Navy).
feasible. To investigate this, we perform model selection performed with three different validation datasets that require increasingly more human effort to create: VALID-ALL (no effort), VALID-MENTION (some effort) and VALID-LINKED (most amount of human effort).
TEST and VALID-LINKED data. Sample 10K triples with relations that have at least three tokens from the 1.25M linked triples. In these triples, the subject and object mentions have an annotation for their entity, which allows the mention-ranking protocol to identify alternative mentions of the respective entities.
VALID-MENTION data. Proceed as in VALID-LINKED but discard the entity links. During validation, no access to alternative mentions is possible so that the mention-ranking protocol cannot be used. Nevertheless, the data has the same distribution as the test data. Such validation data may be generated automatically using a named entity recognizer, if one is available for the target domain.
VALID-ALL data. Sample 10K triples with relations that have at least three tokens from the entire 30M unlinked and linked triples. This yields mostly triples from the unlinked portion. These triples may also include common nouns such as "a nice house" or "the country". Entity links are discarded, i.e., the mention-ranking protocol cannot be used for validation.
4.3 Training Data
To evaluate LP models for KGs, evaluation facts are generated by sampling from the KG. Given an evaluation triple  $(i,k,j)$ , the simplest action to avoid leakage from the training data is to remove only this evaluation triple from training. For KGs, it was observed this simple approach is not satisfactory in that evaluation answers may still leak and thus can be trivially inferred (Toutanova et al., 2015; Dettmers et al., 2018). For example, an evaluation triple  $(a,siblingOf,b)$  can be trivially answered with the training triple  $(b,siblingOf,a)$ .
In OKGs, paraphrases of relations pose additional sources of leakage. For example, the relations "is in" and "located in" may contain many of the same entity pairs. For evaluation triple  $(i,k,j)$ , such leakage can be prevented by removing any other relation between  $i$  and  $j$  from the training data. However, individual tokens in the arguments or relations may also cause leakage. For example, information about test triple ("NBC-TV", "has office in", "NYC") is leaked by triples such as ("NBC Television", "has NYC offices in", "Rockefeller Plaza") even though it has different arguments. Fig. 5 visualizes this example.
We use three levels of leakage removal from training: SIMPLE, BASIC, and THOROUGH. To match evaluation triple  $(i,k,j)$  with training triples, we ignored word order and stopwords.
Figure 5: Examples of test fact leakage into training data; comparing link prediction and open link prediction. The example test triples are (NBC, headquarteredIn, NewYorkCity) and ("NBC-TV", "is in", "NYC"), respectively. Link Prediction: (1) the sampled test triple (2) any link between the test triple's arguments could leak the test fact; Open Link Prediction: (1) the sampled open test triple, (2) consider any link between any mention of the open triple's arguments, (3) consider test fact leakage from the tokens in the open triple's arguments or relation. Underlined tokens are the source of leakage.
SIMPLE removal. Only the triple  $(i,k,j)$  is removed. Triples with alternative mentions for  $i$  or  $j$  are kept.
BASIC removal.  $(i,k,j)$  as well as  $(j,k,i)$  are removed from the training data. Triples with with alternative mentions of  $i$  and  $j$  are also removed.
THOROUGH removal. Additionally to BASIC removal, we also remove triples from training matched by the following patterns. The patterns are explained with the example ("J. Smith", "is defender of", "Liverpool"):
(a)  $(i, *, j)$  and  $(j, *, i)$ . E.g., matches ("J. Smith", "is player of", "Liverpool").
(b)  $(i,k + j,*)$  and  $(*,k + i,j)$ .<sup>2</sup> E.g., matches ("J. Smith", "is Liverpool's defender on", "Saturday").
(c)  $(i + k + j, *, *)$  and  $(*, *, i + k + j)$ . E.g., matches ("Liverpool defender J. Smith", "kicked", "the ball").
For OLPBENCH, THOROUGH removed 196,717 more triples from the OKG than BASIC. Note that this yields three different training data sets.
5 Open Knowledge Graph Embeddings
KG embedding (KGE) models have been successfully applied for LP in KGs, and they can be easily extended to handle surface forms, i.e., mentions and open relations. We briefly describe KGE models and their extension.
Knowledge Graph Embedding (KGE) model. A KGE model (Nickel et al., 2016) associates an embedding with each entity and each relation. The embeddings are dense vector representations that are learned with an LP objective. They are used to compute a KGE model-specific score  $s(i, k, j)$  for a triple  $(i, k, j)$ ; the goal is to predict high scores for true triples and low scores for wrong triples.
KGE model with composition. For our experiments, we considered composition functions to create entity and relation representations from the tokens of the surface form. Such an approach has been used, for example, by Toutanova et al. (2015) to produce open relation embedding via a CNN. A model that reads the tokens of mentions and open relations can, in principle, handle any mention and open relation as long as the tokens have been observed during training.
We use a general model architecture that combines a relational model and a composition func
2Other permutations of this pattern did not occur in our data.
Figure 6: KGE model with composition. The tokens in triple  $(i,k,j)$  are first embedded individually and then composed into mention or relation embeddings. Finally, a KGE model  $\mathcal{R}\mathcal{M}$  is used to compute the triple's score.
tion, see Fig. 6. Formally, let  $\mathcal{V}(\mathcal{E})^{+}$  be the set of non-empty token sequences over the token vocabulary  $\mathcal{V}(\mathcal{E})$  of entity mentions. We denote by  $d,o\in \mathbb{N}_{+}$  the size of the embeddings of entities and relations. We first embed each entity mention into a continuous vector space via an entity mention embedding function  $f:\mathcal{V}(\mathcal{E})^{+}\to \mathbb{R}^{d}$ . Similarly, each open relation is embedded into a continuous vector space via a relation embedding function  $g:\mathcal{V}(\mathcal{R})^{+}\to \mathbb{R}^{o}$ . The embeddings are then fed into a relational scoring function  $\mathcal{RM}:\mathbb{R}^d\times \mathbb{R}^o\times \mathbb{R}^d\to \mathbb{R}$ . Given a triple  $(i,k,j)$ , where  $i,j\in \mathcal{V}(\mathcal{E})^{+}$  and  $k\in \mathcal{V}(\mathcal{R})^{+}$ , our model computes the final score as  $s(i,k,j) = \mathcal{RM}(f(i),g(k),f(j))$ .
In our experimental study, we investigated whether a simple prototypical OLP model can predict genuinely new facts or if many successful predictions can be trivially explained by leakage or nonrelational information. Our goal was to study the effectiveness and necessity of the mention-ranking protocol and leakage removal, and how much human effort is necessary to create suitable validation data. Finally, we inspected data and model quality.
We first describe the models and their training, then the performance metrics, and finally the evaluation. In our experimental results, model performance dropped by  $\approx 25\%$  with THOROUGH leakage removal so that leakage due to paraphrasing is indeed a concern. We also implemented two diagnostic models that use non-relational infor