SlowGuess/ACL_2020_OCR huggingface dataset:
mation (only parts of a triple) to predict answers. These models reached  $\approx 20 - 25\%$  of the prototypical model's performance, which indicates that relational modelling is important. In our quality and error analysis, we found that at least  $74\%$  of the prediction errors were not due to noisy data. A majority of incorrectly predicted entity mentions have a type similar to the one of the true entity.
6.1 Models and Training
Prototypical model. We use COMPLEX (Trouillon et al., 2016) as relational model, which is an efficient bilinear model and has shown state-of-the-art results. For the composition functions  $f$  and  $g$ , we used an LSTM (Hochreiter and Schmidhuber, 1997) with one layer and the hidden size equivalent to the token embedding size. We call this model COMPLEX-LSTM. $^3$
Diagnostic models. To expose potential biases in the data, we employ two diagnostic models to discover how many questions can simply be answered without looking at the whole question, i.e., by exploiting non-relational information. Given question  $(i,k,?)$ , the model PREDICT-WITH-REL considers  $(r,?)$  for scoring. E.g., for question ("Jamie Carragher", "is defender of", ?), we actually ask ("is defender of", ?). This is likely to work reasonably for relations that are specific about the potential answer entities; e.g., predicting popular football clubs for ("is defender of", ?). The model uses scoring functions  $s_t: \mathbb{R}^o \times \mathbb{R}^d \to \mathbb{R}$  and  $s_h: \mathbb{R}^d \times \mathbb{R}^o \to \mathbb{R}$  for questions  $(i,k,?)$  and  $(?,k,j)$  respectively:
$$
s _ {t} (k, e) = g (k) ^ {T} f (j), \quad s _ {h} (i, k) = f (i) ^ {T} g (k)
$$
Likewise, the PREDICT-WITH-ENT model ignores the relation by computing a score for pair  $(i,j)$ . We use  $s_e(i,j) = f(i)^T f(j)$
Training. See App. C for details about the hyperparameters, training and model selection.
Performance metrics. For evaluating a model's predictions, we use the ranking metrics mean reciprocal rank (MRR) and HITS@k. MRR is sensitive to the top-3 ranks with rapidly decaying reward,
3In a preliminary study, we investigated COMPLEX, ANALOGY, DISTMULT and RESCAL as relational models. COMPLEX was the most efficient and best performing model. For composition functions, we also investigated unigram pooling, bi-gram pooling with CNNs, self-attention and LSTMs. Here LSTMs worked well consistently. See App. E for additional results.
Leakage
Removal	Model	Model
Selection	MRR	HITS@1	HITS@10	HITS@50
SIMPLE	PRED-WITH-ENT	LINKED	0.0	0.0	0.0	0.0
PRED-WITH-REL	LINKED	1.5	0.8	2.6	5.4
COMPLEX-LSTM	LINKED	6.5	3.8	11.6	20.7
BASIC	PRED-WITH-ENT	LINKED	0.0	0.0	0.0	0.0
PRED-WITH-REL	LINKED	1.0	0.5	1.6	3.6
COMPLEX-LSTM	LINKED	4.8	2.6	8.9	17.6
THOROUGH	PRED-WITH-ENT	LINKED	0.0	0.0	0.0	0.0
PRED-WITH-REL	LINKED	1.0	0.6	1.5	3.3
COMPLEX-LSTM	LINKED	3.9	2.1	7.0	14.6
COMPLEX-LSTM	ALL	2.7	1.5	4.7	9.1
COMPLEX-LSTM	MENTION	3.8	2.1	7.1	14.1
Table 2: Test results. Comparing COMPLEX-LSTM, PREDICT-WITH-ENT and PREDICT-WITH-REL with all removal settings. Model selection on VALID-LINKED for all settings except in THOROUGH, where we also show VALID-MENTION and VALID-LINKED. Results in percent.
while HITS@k equally rewards correct answers in the top-  $k$  ranks. See App. D for a more formal definition of MRR and HITS@k. The ranks are based on mention ranking for VALID-LINKED and TEST and on entity-ranking (treating distinct mentions as distinct entities) for VALID-ALL and VALID-MENTION.
Influence of leakage. In Tab. 2, we observed that BASIC leakage removal of evaluation data lowers the performance of all models considerably in contrast to the SIMPLE leakage removal. With the THOROUGH leakage removal, performance drops further; e.g., HITS@50 performance dropped by  $\approx 25\%$  from SIMPLE. This confirms our conjecture that leakage can trivially explain some successful predictions. Most predictions, however, cannot be explained by paraphrasing leakage.
Influence of non-relational information. In Tab. 2, we see that PREDICT-WITH-ENT, which essentially learns popularity statistics between entity mentions, has no success on the evaluation data. However, PREDICT-WITH-REL reaches  $\approx$ $20 - 25\%$  of HITS@50 performance of COMPLEX-LSTM by simply predicting popular mentions for a relation, even in the THOROUGH setting.
Effectiveness of mention-ranking. Tab. 3 shows validation results for the three types of validation data for COMPLEX-LSTM and THOR-ough removal. The evaluation protocol has access
to alternative mentions only in VALID-LINKED, but not in VALID-ALL and VALID-MENTION. Clearly, using VALID-LINKED results in higher metrics when models associate different mentions to an answer entity.
Influence of model selection. The THOROUGH block of Tab. 2 shows the results for model selection based on VALID-ALL, VALID-MENTION or VALID-LINKED. In VALID-ALL, many triples contain common nouns instead of entity mentions, while in VALID-MENTION or VALID-LINKED triples have entity mentions in both arguments. Model selection based on VALID-ALL clearly picked a weaker model than model selection based on VALID-LINKED, i.e., it led to a drop of  $\approx 35\%$  of HITS@50 performance. However, there is no improvement when we pick a model based on VALID-LINKED versus VALID-MENTION. Thus, computing the MRR using alternative entity mentions did not improve model selection, even though—as Tab. 3 shows—the mention-ranking protocol gives more credit when alternative mentions are ranked higher. Our results suggest that it may suffice to use validation data that contains entity mentions but avoid costly entity disambiguation.
Overall performance. In Tab. 2 we observed that performance numbers seem generally low. For comparison, the HITS@10 of COMPLEX on FB15k-237—a standard evaluation dataset for LP in curated KGs—lies between  $45\%$  and  $55\%$ . We conjecture that this drop may be due to: (i) The
Leakage
Removal	Model	Model
Selection	MRR	HITS@1	HITS@10	HITS@50
THOROUGH	COMPLEX-LSTM	ALL	2.9	1.8	5.0	8.9
COMPLEX-LSTM	MENTION	3.6	2.0	6.5	13.0
COMPLEX-LSTM	LINKED	4.2	2.3	7.5	14.9
Table 3: Validation results. Comparing the performances of COMPLEX-LSTM for different validation datasets.
Types of prediction errors
correct sense / wrong entity	68.0 %
wrong sense	13.5 %
noise	18.5 %
Types of data errors
triple has error	12.0 %
mention is generic	14.0 %
Table 4: Error assessment of 100 sampled HITS@50 (filtered) prediction errors from VALID-LINKED.
level of uncertainty and noise in the training data, i.e., uninformative or even misleading triples in OKGs (Gashteovski et al., 2019). (ii) Our evaluation data is mostly from the more challenging long tail. (iii) OKGs might be fragmented, thus inhibiting information flow. Also, note that the removal of evaluation data from training removes evidence for the evaluated long-tail entities. (iv) Naturally, in LP, we do not know all the true answers to questions. Thus, the filtered rank might still contain many true predictions. In OLP, we expect this effect to be even stronger, i.e., the filtered ranking metrics are lower than in the KG setting. Still, like in KG evaluation, with a large enough test set, the metrics allow for model comparisons.
Model and data errors. We inspected predictions for VALID-LINKED from COMPLEX-LSTM trained on THOROUGH. We sampled 100 prediction errors, i.e., triples for which no correct predicted mention appeared in the filtered top-50 rank. We classified prediction errors by inspecting the top-3 ranks and judged their consistency. We classified triple quality judging the whole triple. We counted an error as correct sense / wrong entity, when the top-ranked mentions are semantically sensible, i.e. for ("Irving Azoff", "was head of", ?) the correct answer would be "MCA Records", but the model predicted other record companies. We counted an error as wrong sense when—for the
same example—the model mostly consistently predicted other companies or music bands, but not other record companies. If the predictions are inconsistent, we counted the error as noise.
An additional quality assessment is the number of wrong triples caused by extraction errors in OPIEC, e.g., ("Finland", "is the western part of", "the balkan peninsula"), ("William Macaskill", "is vice-president of", "giving"), or errors in alternative mentions. We also looked for generic mentions in the evaluation data. Such mentions contain mostly conceptual knowledge like in ("computer science", "had backgrounds in", "mathematics"). Other generic triples, like ("Patrick E.", "joined the team in", "the season"), have conceptual meaning, but miss context to disambiguate "the season".
The results in Tab. 4 suggest that the low performance in the experiments is not due to noisy evaluation data.  $74\%$  of the examined prediction errors on VALID-LINKED contained correct, nongeneric facts. The shown model errors raise the question of whether there is enough evidence in the data to make better predictions.
We proposed the OLP task and a method to create an OLP benchmark. We created the large OLP benchmark OLPBENCH, which will be made publicly available<sup>4</sup>. We investigated the effect of leakage of evaluation facts, non-relational information, and entity-knowledge during model selection using a prototypical open link prediction model. Our results indicate that most predicted true facts are genuinely new.
The first author would like to gratefully thank the NVIDIA Corporation for the donation of a TITAN Xp GPU that was used in this research.
4https://www.uni-mannheim.de/dws/ research/resources/olpbench/
Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787-2795.
William W. Cohen, Henry Kautz, and David McAllester. 2000. Hardening soft information sources. In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '00, pages 255-259, New York, NY, USA. ACM.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1811-1818.
John C. Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159.
Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam. 2011. Open information extraction: The second generation. In *IJCAI* 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16-22, 2011, pages 3-10.
Luis Galárraga, Geremy Heitz, Kevin Murphy, and Fabian M. Suchanek. 2014. Canonicalizing open knowledge bases. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pages 1679-1688.
Luis Galárraga, Geremy Heitz, Kevin Murphy, and Fabian M. Suchanek. 2014. Canonicalizing open knowledge bases. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 1679-1688, New York, NY, USA. ACM.
Kiril Gashteovski, Rainer Gemulla, and Luciano Del Corro. 2017. Minie: Minimizing facts in open information extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2630-2640.
Kiril Gashteovski, Sebastian Wanner, Sven Hertling, Samuel Broscheit, and Rainer Gemulla. 2019.
OPIEC: an open information extraction corpus. CoRR, abs/1904.12324.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pages 249-256.
Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. 2017. Knowledge transfer for out-of-knowledge-base entities: A graph neural network approach. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1802-1808.
Frederick Hayes-Roth. 1983. Building expert systems, volume 1 of Advanced book program. Addison-Wesley.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Sameh K. Mohamed, Aayah Nounu, and Vít Nováček. 2019. Drug target discovery using knowledge graph embeddings. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC '19, page 11-18, New York, NY, USA. Association for Computing Machinery.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33.
Fabio Petroni, Luciano Del Corro, and Rainer Gemulla. 2015. CORE: context-aware open relation extraction with factorization machines. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1763-1773.
Jay Pujara, Hui Miao, Lise Getoor, and William Cohen. 2013. Knowledge graph identification. In *The Semantic Web - ISWC* 2013, pages 542-557, Berlin, Heidelberg. Springer Berlin Heidelberg.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, Atlanta, Georgia. Association for Computational Linguistics.
Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In Proceedings of the
Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1957-1964.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Ho-fung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1499-1509.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2071-2080.
Shikhar Vashishth, Prince Jain, and Partha Talukdar. 2018. Cesi: Canonicalizing open knowledge bases using embeddings and side information. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pages 1317-1327, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
Patrick Verga, Arvind Neelakantan, and Andrew McCallum. 2017. Generalizing to unseen entities and entity pairs with row-less universal schema. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 613-622.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL, 6:287-302.
Tien-Hsuan Wu, Zhiyong Wu, Ben Kao, and Pengcheng Yin. 2018. Towards practical open knowledge base canonicalization. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18, pages 883-892, New York, NY, USA. ACM.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369-2380.
The following studies investigated KGs and OKGs in various ways, either by deriving KGs from OKGs or by using them jointly to improve inference in KGs.
Unseen entities. Shi and Weninger (2018) introduced open-world knowledge base completion (OWKBC), which assumes a curated KG as basis. The goal is to obtain new triples with unseen entities and known relations from the KG. Shi and Weninger (2018) proposes a link prediction model that allows questions involving unseen entities. Their model leverages the KG, relevant text fragments, word embeddings as well as an entity resolution module. Other approaches use structural information from the KG itself. Hamaguchi et al. (2017) assigns an embedding to an unseen entity based on embeddings of its neighboring entities and relations, whereas Verga et al. (2017) encodes an unseen entity pair by averaging the embeddings of the relations that link to it.
OpenIE-enhanced KGs. Universal schema models (Riedel et al., 2013) augment an existing KG with open relations between KG entities. Petroni et al. (2015) build upon Riedel et al.'s work by considering context information to improve the results further. Toutanova et al. (2015) embed open relations based on their tokens and dependency relations to augment the KG. In our work, we explore LP for OKGs, which differs in that only mentions are observed for both entities and relations. Neither a KG nor a vocabulary of entities is available during training and prediction.
Canonicalizing open knowledge. Cohen et al. (2000); Pujara et al. (2013); Vashishth et al. (2018); Wu et al. (2018); Galarraga et al. (2014) are the closest in spirit to this study, as they also want to make OKGs accessible without using a reference knowledge base. Cohen et al. (2000) calls open information a soft database, while Pujara et al. (2013) calls it an extraction graph from which a latent KG has to be identified. Common to all those approaches is that their ultimate target is to create a symbolic database with disambiguated entities and distinct relations. Thus they canonicalize the entities and the relations. In contrast, we are not canonicalizing the OKG but reason directly on the OKG. Galarraga et al. (2014) directly evaluates the induction of entity clusters, while we evaluate this
jointly in the context of LP.
Reading comprehension QA and language modelling. Two recently published reading comprehension question answering datasets—QAngaroo (Welbl et al., 2018) and HotPotQA (Yang et al., 2018)—evaluate multi-hop reasoning over facts in a collection of paragraphs. In contrast to these approaches, OLP models reason over the whole graph, and the main goal is to investigate the learning of relational knowledge despite ambiguity and noise. We consider those two directions as complementary to each other. Also, in their task setup, they do not stipulate a concept of relations between entities, i.e., the relations are assumed to be a latent/inherent property of the text in which the entities occur. This is true as well for language models trained on raw text. It has been shown that such language models can answer questions in a zero-shot setting (Radford et al., 2019). The authors of the latter study inspected the training data to estimate the number of near duplicates to their test data and could show that their model seemed to be able to generalize, i.e., to reason about knowledge in the training data.
TAC KBP Slot Filling. The TAC KBP Slot Filling challenge datasets provide a text corpus paired with canonicalized multi-hop questions. There are similarities to our work in terms of building knowledge from scratch and answering questions. The main difference is that our goal is to investigate the learning of knowledge without supervision on canonicalization and that we use link prediction questions to quantify model performance. If models in OLP show convincing progress, they could and should be applied to TAC KBP.
B Dataset creation
The process of deriving the dataset from OPIEC was as follows. Initially, the dataset contained over 340M non-distinct triples, which are enriched with metadata such as source sentence, linguistic annotations, confidence scores about the correctness of the extractions and the Wikipedia links in the triple's subject or object. Triples of the following types are not useful for our purpose and are removed: (i) having a confidence score  $< 0.3$ ,
(ii) having personal or possessive pronouns, wh-determiner, adverbs or determiners in one of their arguments, (iii) having a relation from an implicit appositive clause extraction, which we found to be very noisy, and (iv) having a mention or a relation that is longer than 10 tokens. This left 80M non-distinct triples. Next, we lowercased the remaining 60M distinct triples and collect an entity-mentions map from all triples that have an annotated entity. We collected token counts and created a mention token vocabulary with the top 200K most frequent tokens, and a relation token vocabulary with the top 50K most frequent tokens. This was done to ensure that each token is seen at least  $\approx 50$  times. Finally, we kept only the triples whose tokens were contained in these vocabularies, i.e., the final 30M distinct triples.
C.1 Multi-Label Binary Classification Batch-Negative Example Loss
Recent studies (Dettmers et al., 2018) obtained state-of-the-art results using multi-label binary classification over the full entity vocabulary. Let the cardinality of the OKG's mention set be  $N = |\mathcal{T}_h \cup \mathcal{T}_t|$ . A training instance is either a prefix  $(i, k)$  with label  $y^{ik} \in \{0, 1\}^N$  given by
$$
y _ {c} ^ {i k} = \left\{ \begin{array}{l l} 1 & \text {i f} (i, k, c) \in \mathcal {T} \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \quad \text {f o r} c \in \{1,.., N \}
$$
or, likewise, a suffix  $(k,j)$  and  $y^{kj}\in \{0,1\} ^N$
Computing such a loss over the whole entity mention vocabulary is infeasible because (a) our entity mention vocabulary is very large and (b) we have to recompute the entity mention embeddings after each parameter update for each batch. To improve memory efficiency and speed, we devise a strategy to create negative examples dubbed batch negative examples. This method simplifies the batch construction by using only the entities in the batch as negative examples. Formally, after sampling the prefix and suffix instances for a batch  $b$ , we collect all true answers in a set  $\hat{B}_b$ , such that the label vectors  $y^{ik}$  and  $y^{kj}$  in batch  $b$  is defined over  $\hat{B}_b$  and the loss in batch  $b$  is computed by
$$
\begin{array}{l} L ^ {i k} = \frac {1}{| B _ {b} |} \sum_ {c \in \hat {B} _ {b}} - [ y _ {c} ^ {i k} \cdot \log \sigma (s (i, k, c)) \\ \left. + \left(1 - y _ {c} ^ {i k}\right) \cdot \log \left(1 - \sigma \left(s (i, k, c)\right)\right) \right] \\ \end{array}
$$
5The triples can be non-distinct, i.e., duplicates, when they have been extracted from different sentences.
<sup>6</sup>The confidence score is computed by a classifier that determines the probability of the triple having an extraction error. Refer to OPIEC's publication for further description.