Year: 2020

Abstract:
This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pretrained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

heading: 1 INTRODUCTION
text: Knowledge graphs (KGs) are an important resource for both humans and machines. Factual knowledge in KGs is injected into AI applications to imitate important skills possessed by humans, e.g., reasoning and understanding. KG construction is mainly supervised, requiring humans to handwrite every fact, such as Freebase (Bollacker et al., 2008) and Wikidata. KGs can also be constructed in a semi-supervised way, in which a semi-automatic extractor is used to obtain the facts from web corpora (e.g., NELL (Carlson et al., 2010) and Knowledge Vault (Dong et al., 2014)). Humans however still need to interact with the extractor to improve the quality of the discovered facts. Therefore, human supervision, which is often expensive, is required in constructing KGs.
Recent progress in language models (LMs), such as BERT (Devlin et al., 2018) and GPT-2/3 (Radford et al., 2019; Brown et al., 2020), has led to superior results even outperforming humans in a wide range of tasks, e.g., sentence classification (Wang et al., 2018), question answering (Brown et al., 2020). Pre-trained LMs are also capable to write poetry, music, and code, while such tasks often require we human to spend a significant amount of time in learning the relevant knowledge to work well. In fact, these pre-trained LMs automatically acquire factual knowledge from large-scale corpora (e.g., BookCorpus (Zhu et al., 2015), Common Crawl (Brown et al., 2020)) via pre-training. The learned knowledge in pre-trained LMs is the key to the current success. We therefore consider the following question: instead of using the manually created knowledge, can we use the knowledge stored in pre-trained LMs to construct KGs?
In this paper, we design an unsupervised approach called MAMA that successfully recovers the factual knowledge stored in LMs to build KGs from scratch. MAMA constructs a KG with a single forward pass of a pre-trained LM (without fine-tuning) over a textual corpus. As illustrated in Figure 1, MAMA has two stages: Match and Map. Match stage generates a set of candidate facts by matching the facts in the textual corpus with the knowledge in the pre-trained LM. General or world knowledge from large-scale corpora is embedded in the LM, thus candidate facts in the target corpus are often covered by the knowledge in the LM. The candidate facts are matched through an efficient beam search in the attention weight matrices of the pre-trained LM without fine-tuning. Map stage produces an open KG via mapping the matched candidate facts from Match stage to both fixed KG schema and open schema. If the schema of candidate facts exists in the KG schema, we map the candidate facts directly to the fixed KG schema. Otherwise, we reserve the unmapped candidate
1
Under review as a conference paper at ICLR 2021
with a single forward pass of the pre-trained language model (LM) (without fine-tuning) over the corpus. Given the input: a textual corpus containing passages and sentences, e.g., English Wikipedia, and a pre-trained LM, e.g., BERT, GPT-2/3, MAMA (1) generates a set of candidate facts via matching the knowledge in the pretrained LM with facts in the textual corpus, e.g., a candidate fact (Dylan, is, songwriter) from the sentence “Dylan is a songwriter.”, and (2) produces an open KG by mapping the matched candidate facts to both an existing KG schema, e.g., (Bob Dylan.Q392, occupation.P106, Songwriter.Q753110) in Wikidata schema, and an open schema, e.g., (Bob Dylan.Q392, sign, Albert Grossman.Q708584).
facts in the open schema. This results in a new type of KG, open KG, with a mixture of mapped facts in fixed KG schema and unmapped facts in the open schema.
Our contributions are as follows:
1. We show how to construct KGs from pre-trained LMs. The KGs are constructed with a single forward pass of the pre-trained LMs without fine-tuning over the textual corpora. This helps researchers explicitly understand what the language models learn, bridging the deep LM and KG communities through enhanced model transparency.
2. We propose an unsupervised two-stage approach, MAMA, to first match the candidate facts in the corpora with the knowledge stored in LMs, then map the matched candidate facts to both fixed and open schema to produce a KG.
3. We generate a new type of KG, namely open KG, consists of mapped facts in the fixed KG schema of existing KGs (Wikidata and TAC KBP) annotated by humans; and unmapped facts in the open schema that are new in the reference KG schema. The reach of this result is broad and has downstream utility for knowledge graph construction, deep neural network interpretation, and information extraction.

heading: 2 MAMA
text: We introduce an unsupervised end-to-end approach Match and Map (MAMA) as illustrated in Figure 1 to construct open knowledge graphs (KGs) from language models (LMs). MAMA constructs the KGs with a single forward pass of the pre-trained LMs (without fine-tuning) over the corpora. The two stages of MAMA are: Match generates a set of candidate facts from a textual corpus. LMs contain global or world knowledge learned from large-scale corpora, which often does not perfectly match the knowledge in the target corpus. The goal of this stage is to match the knowledge stored in pre-trained LMs with facts in the corpus. Each fact is represented as a triplet (head, relation, tail) 1, in short, (h, r, t), and passed to Map stage. Match procedure is detailed in Sec. 2.1. Map produces an open KG using the matched candidate facts from Match stage. The constructed open KG has two portions: (a) mapped candidate facts that are in a fixed KG schema, e.g., (Dylan, is, songwriter) is mapped to (Bob Dylan.Q392, occupation.P106, Songwriter.Q753110) according to Wikidata schema; and (b) unmapped candidate facts that are in an open schema, e.g., a candidate fact (Dylan, signed, Albert Grossman) is partially mapped to (Bob Dylan.Q392, sign, Albert Grossman.Q708584) in the open schema. This stage is described in Sec. 2.2.
2
Under review as a conference paper at ICLR 2021
the best matched candidate fact (Dylan, is, songwriter) from the sentence “Dylan is a songwriter.” The lower portion shows the corresponding step-by-step process. Given a head-tail pair (Dylan, songwriter), at each step, the search chooses one of the actions, i.e., START, YIELD, STOP to produce an intermediate candidate fact. The search starts by adding the head “Dylan” as an initial candidate (step 0). The matching degree of the candidate is initialized as 0. Next, a new candidate is yielded if the candidate has not reached the tail “songwriter” (step 1 and step 2), by appending the next largest attended token (with the largest score from the attention matrix (b) of the sentence) to the end of the current candidate, and the corresponding matching degrees are increased by the associated attention scores (0.3 and 0.4) to 0.3 (0+0.3) and 0.7 (0.3+0.4) respectively. Otherwise, the search stops, and the candidate fact with the best matching degree is returned for the head-tail pair (step 3). The attention matrix (b) is from the forward pass of the LM without fine-tuning over the sentence. “x” marks the tokens to prevent searching backward.

heading: 2.1 MATCH
text: We frame the matching procedure as a search problem. To obtain the best matched candidate facts of an input sentence, the candidates with the top matching degrees are returned from a search process. The matching degree is derived from the search in the attention weight matrices of the pre-trained LM, since the attention weight matrices are one of the main containers of the knowledge in the pre-trained LM. The attention weight matrices are simply from the forward pass of the LM without fine-tuning over the sentence.

heading: 2.1.1 BEAM SEARCH
text: We design a simple-yet-effective beam search to find the best matched candidate facts. For every head-tail pair (h, t) in a sentence, the search maintains the k-best matched candidate facts of the pair. Let’s first consider the search from left to right with beam size equals to 1. An example search process is shown in Figure 2. Given a head-tail pair (Dylan, songwriter), at each step, the search performs one of the following actions: START the search from the head. The head h is added as an initial candidate into the beam. For
simplicity, we use START(h) to denote the action, which returns a candidate (h,. In Figure 2(a), at step 0, the head “Dylan” is added as (Dylan, into the beam. The matching degree is initialized to 0.
YIELD a new intermediate candidate in the beam if the current candidate has not reached the tail. The next largest attended token (with the largest score from the attention matrix) is appended to the end of the current candidate to yield the new candidate. The corresponding matching degrees are increased by the associated attention scores. At step 1 (orange arrow in Figure 2(a)), “is” is appended to the current candidate to yield (Dylan, is, , since “is” has the largest attention score with “Dylan” in the attention matrix. The attention score is 0.3 as highlighted in orange in Figure 2(b). The matching degree becomes 0.3 (i.e. 0+0.3). The multi-head attention is reduced to a single head so that every two tokens of the sentence are associated with one attention weight. We experiment with different reduction
1 We use the term “head” and “tail” to denote head and tail’s “entities” or “entity mentions” for simplicity.
3
Under review as a conference paper at ICLR 2021
Algorithm 1 Beam search for matching candidate facts. Input: Head-tail pair (h, t), sentence s, attention matrix As, action manager O = {START,YIELD,STOP},
beam size k Output: Candidate facts T(h,t) 1: T(h,t) {START(h)} . Start by adding the head as a candidate in the beam 2: while 9c 2 T(h,t)[O(c) = YIELD] do 3: eT(h,t) ; . Initialize a new beam 4: for each c 2 T(h,t) do 5: if O(c) = YIELD then 6: eT(h,t) eT(h,t) [ {YIELD(c, s,As)} . Yield a new candidate if not reached the tail 7: else 8: eT(h,t) eT(h,t) [ {STOP(c, t)} . Stop then produce a valid fact if reached the tail 9: end if
10: end for 11: T(h,t) TOP(k, eT(h,t)) . Maintain k-best candidates in the beam 12: end while 13: return T(h,t)
setups in Sec. A.3. “x” marks the tokens (prior to the current token) that are not considered in the search to prevent searching backward. Step 2 similarly takes YIELD action to produce (Dylan, is songwriter, . The matching degree is now 0.7 (i.e. 0.3+0.4). We use YIELD(c, s,As) to denote the action, where c is a current candidate, s represents the sentence, and As is the attention matrix from the forward pass of the pre-trained LM over s, which yields a new candidate.
STOP the search step if the candidate has reached the tail, then add the candidate as a valid candidate fact into the beam. As beam size equals to 1, (Dylan, is, songwriter) is the only returned candidate fact for the given pair. The final matching degree of the candidate is 0.7. We denote this step using STOP(c, t), which returns a valid fact.
The details of the proposed beam search are in Algorithm 1. The inputs of the search algorithm are a head-tail pair (h, t), a sentence s, an attention matrix As of s. Both h and t are identified by the noun chunk in s. As is the attention matrix associated with s from the forward pass of LM without fine-tuning. The search gets started by adding the head h as the initial candidate in the beam (line 1). While there are still new candidates waiting to be yielded (line 2), the search continues, and the top k candidates sorted by the matching degrees are maintained (line 3-11) in the beam. In practice, we implement an action manager O to decide which action to take at each step. Given a candidate c in the beam, O(c) = START always happens at the beginning of the search. If c has not reached the tail t yet, O(c) = YIELD. Otherwise, O(c) = STOP. We convert the subwords to the corresponding full words. We also notice some facts are in reverse order in the sentence, e.g., “· · · said Jason Forcier , a vice president at battery maker A123 Systems Inc.” for facts of relation “org:top members employees”, thus enable bidirectionality by running the algorithm in both directions (left to right and right to left). The beam search is implemented by the breadth-first search, which is efficient as the time complexity is O(k · d), where d is the maximum depth of the search tree.

heading: 2.1.2 FILTER
text: Although the basic functionality provided by beam search is sufficient for finding useful candidate facts, we have found a few constraints useful. Given a candidate fact (h, r, t) from beam search result T(h,t), it remains as a fact if satisfying all the following constraints. Constraint #1 The matching degree of (h, r, t) is above a threshold. We compare the matching degrees corpus-wide to only reserve the facts that are matched better with the knowledge in LMs. For example, MAMA extracts a fact (Rolling Stone, wrote, pop song) from “Rolling Stone wrote: “No other pop song has so thoroughly challenged artistic conventions””, which is not an accurate fact based on the sentence. We observe the associated matching degree is below a proper threshold, while the matching degrees of high-quality facts from the same documents, e.g., (Dylan, is, songwriter), or confident facts from the other documents are beyond the threshold.
4
Under review as a conference paper at ICLR 2021
Constraint #2 The distinct frequency of r is above a threshold. To avoid facts to be over-specified, e.g., (Dylan, signed to Sam Peckinpah’s film, Pat Garrett and Billy the Kid), we require r should take many distinct head-tail pairs in the corpus. Constraint #3 Relation r is a contiguous sequence in the sentence. We can avoid r that has no meaningful interpretation (Fader et al., 2011), e.g., (Rolling Stone, wrote challenged, conventions) from the above sentence.

heading: 2.2 MAP
text: The objective of Map stage is to generate an open KG. The open KG contains (a) mapped facts in a KG schema (Sec. 2.2.1), e.g., Wikidata schema, if the schema of the candidate facts is within the existing KG schema; and (b) unmapped facts from (a) in an open schema (Sec. 2.2.2).

heading: 2.2.1 MAPPED FACTS IN KG SCHEMA
text: The goal is to map a candidate fact (h, r, t) to a fact (hk, rk, tk) in the KG schema. The reason for mapping to an existing KG schema is to make use of the high-quality schema designed by experts (to avoid duplicated efforts of building from scratch) and enable evaluating the candidate facts with oracle KG facts contributed by human volunteers. We first map both entities h, t to hk, tk, then map the relation r to rk in the reference KG schema. Additional details are presented in Sec. A.1. Entity linking to KG schema We adapt an unsupervised entity linker based on a mention-to-entity dictionary (Spitkovsky & Chang, 2012) to link the entities for scalability consideration. Besides, contextual information is crucial to link the entities correctly, we use the word embedding of the context to disambiguate the entities, which means we only link the entities with high contextual similarities based on the word embedding. We adopt the entity linker to map h, t to hk, tk. Relation mapping with KG schema We largely follow the relation mapping method proposed by Angeli et al. (2015) to construct an offline relation map between KG relation and relation phrases of the candidate facts. The basic idea is that the more often linked head-tail pairs (i.e., entities are with type information from the entity linking step) co-occur between the candidate facts and KG facts, the more likely the corresponding relations are mapped to each other. In addition, we normalize each relation phrase of the candidate facts by lemmatization, and removing inflection, auxiliary verbs, adjectives, adverbs. After the relation mapping is constructed, one author manually checks whether the top 15 relation phrases are true mappings for each KG relation. We only reserve the true ones in the final relation mapping. This process takes approximately one day. Later, r is mapped to rk with an efficient look-up operation in the relation mapping.

heading: 2.2.2 UNMAPPED FACTS IN OPEN SCHEMA
text: An unmapped candidate fact (h, r, t) means at least one of h, r, and t is not mapped to the KG schema based on the method in Sec. 2.2.1. There are two types of unmapped candidate facts: Partially unmapped facts represent at least one of h, r, and t are mapped to the KG schema. It can be h or t mapped to hk or tk based on the entity linker in Sec. 2.2.1. It can also be r that mapped to rk using the relation mapping in Sec. 2.2.1. This actually results in unmapped facts that are in a mixture of the KG schema and the open schema. As the overall schema of the unmapped facts is not in the KG schema, we use open schema to denote such unmapped facts in the rest of the paper for simplicity. An example is (Dylan, signed, Albert Grossman) in Figure 1, where both head and tail are linked to Wikidata schema based on the entity linker in Sec. 2.2.1, but the relation cannot be mapped since there is no relation mapping from “signed” to a KG relation in Wikidata schema. Completely unmapped facts indicate all h, r, and t are not mapped to the KG schema. This means neither the entity linker nor the relation mapping is able to map h, r, and t to hk, rk, tk respectively. The resulting unmapped candidate facts stay in the open schema, e.g., a candidate fact (Jacob, was, A Registered Mennonite) stays the same in the open schema from a sentence “Jacob was a registered Mennonite in Amsterdam.”.
The resulting open KG is a new type of KG that mixes the fixed KG schema with the flexible open schema, suggesting new directions for the next generation of KGs. The open KG not only contains existing knowledge in the reference KGs, but also extends the fixed KG schema with an additional open schema to improve the coverage of the KGs, that benefits the downstream KG based applications, e.g., QA and commonsense reasoning (Wang et al., 2019; Brown et al., 2020).
5
Under review as a conference paper at ICLR 2021
Slot Filling 2013 challenge. # of oracle facts for TAC KBP is the number of oracle facts in the 2013 task. # of documents for TAC KBP is the number of the documents in the 2013 task. # of oracle facts for Wikidata is the total number of oracle facts in Wikidata. # of documents for Wikidata is the size of English Wikipedia.

heading: 3 EXPERIMENTS
text: How well can language models generate knowledge graphs? We experimentally explore how well can MAMA answer the question in the section. To measure the ability of LMs in generating KGs, we directly measure the quality of resulting open KGs. The open KG contains two types of facts: mapped facts in the fixed KG schema; and unmapped facts in the open schema. We first quantitatively evaluate MAMA by comparing the mapped facts to oracle KGs annotated by humans in Sec. 3.1, then conduct an in-depth analysis of the unmapped facts in Sec. 3.2.

heading: 3.1 RESULTS ON MAPPED FACTS
text: We first study the quality of the mapped facts. As the candidate facts have been mapped to the schema of oracle KGs, we are able to quantitively compare the candidate facts with the oracle facts in the reference KGs.

heading: 3.1.1 DATASETS
text: We compare the mapped facts from MAMA with the facts in two KGs: TAC KBP TAC Knowledge Base Population (KBP) Slot Filling is a task to search a document collection to fill in the tail/object entity for predefined relations (slots) for a given head/subject entity in a reference KG. We experiment with the reference KG in the 2013 challenge. We use the document collection and oracle facts of the 2013 task. The statistic of the dataset is shown in Table 1. Wikidata We use popular Wikidata as another KG. We use all the oracle facts in Wikidata. We use the English Wikipedia as the text corpus, since a large amount of facts in Wikidata is from English Wikipedia. The statistic is in Table 1.
To evaluate the mapped facts, we first use Match stage of MAMA to run over the corresponding documents to generate the candidate facts. Then Map stage is leveraged to map the candidate facts to the schema of TAC KBP and Wikidata respectively. The parameter settings, such as beam size in Algorithm 1, are shared across TAC KBP and Wikidata based on the parameter study in Sec. A.3.

heading: 3.1.2 TAC KBP
text: To verify the ability to produce correct facts, we compare candidate facts from MAMA to the outputs of two open information systems, which also produce triplets in the form of (h, r, t). After collecting the triplets from the corresponding system, we use the same Map procedure with MAMA (Sec. 2.2.1) to map the triplets to the corresponding KG schema. Stanford OpenIE leverages POS tag and dependency parser, and generates self-contained clauses from long sentences to extract the triplets, which is the best open information extraction system (Angeli et al., 2015) on TAC KBP (Surdeanu, 2013). OpenIE 5.1
2 is one of the state-of-the-art open information extraction systems, which is the successor to Ollie (Schmitz et al., 2012), and it improves extractions from noun relations, numerical sentences, and conjunctive sentences depending on the linguistic patterns.
We use two families of pre-trained LMs with MAMA. We use BERTBASE and BERTLARGE from Devlin et al. (2018) with MAMA, namely MAMA-BERTBASE and MAMA-BERTLARGE. Besides, GPT-2s from Radford et al. (2019) are used, i.e., MAMA-GPT-2, MAMA-GPT-2MEDIUM, MAMA-GPT-2LARGE, and MAMA-GPT-2XL.
2 https://github.com/dair-iitd/OpenIE-standalone
6
Under review as a conference paper at ICLR 2021
of the pre-trained LM.
Table 2 shows the results on TAC KBP. We use the official scorer of TAC KBP Slot Filling 2013 to evaluate precision, recall, and F1 on TAC KBP 3. MAMA constructs improved KGs compared to open IE. From the results, we find that all our methods achieve competitive precision, which is greater than 60%, given the unsupervised nature of MAMA. All the proposed methods outperform the two open IE systems. This shows that MAMA is able to produce high-quality knowledge directly from pre-trained LMs by a single forward pass without human supervision. The results show the effectiveness of MAMA in generating candidate facts from Match stage, and producing high-quality KGs through Map stage. We also find that MAMA-GPT-2XL performs the best. MAMA-GPT-2XL outperforms the previous state-of-the-art Stanford OpenIE by over 2.6% in F1. This shows the proposed end-to-end MAMA is able to recover the knowledge stored in pre-trained LMs without relying on any extra linguistic features, such as POS tag and dependency parser used in open IE systems. The main reason leading to the moderate results of OpenIE 5.1 is that the system generates objects of the triplets with extraneous words, which hurt the performance in slot filling tasks. Even though the proposed methods all outperform the two open IE systems in the recall, however improving recall is clearly the future direction to further improve the performance of MAMA. We find that the main cause of the moderate recall is the incorrect entities caused by spaCy noun chunk as summarized in Sec. A.2. Larger/deeper LMs produce KGs of higher quality. BERTLARGE outperforms BERTBASE since the doubling parameter size. GPT-2s share similar trends, where we observe performance increases when the model size increases. This complies with our intuition on more knowledge is stored in deeper and larger models. Such increases in performance seem subtle on TAC KBP, we find this might due to the relatively small number of oracle facts by noticing a more significant improvement on Wikidata in Sec. 3.1.3. We plan to further improve the results with larger pre-trained LMs, e.g., GPT-3 (Brown et al., 2020), Megatron-LM (Shoeybi et al., 2019). BERT LMs outperform GPT-2 LMs under similar model sizes. More specifically, BERTBASE performs better than MAMA-GPT-2 in F1, and MAMA-BERTLARGE outperforms MAMA-GPT2MEDIUM in F1. BERTBASE and MAMA-GPT-2 are similar in size, while MAMA-BERTLARGE and MAMA-GPT-2MEDIUM are similar in model size as well. This is mainly because that the recall of BERT LMs is higher than that of corresponding GPT-2 LMs. The result indicates that the Cloze-style loss function (i.e., masked language model) of BERT is more effective and flexible in recovering more knowledge than the autoregressive LM objective. We also notice that the precision of GPT-2 LMs is higher than that of according BERT LMs. The reason is that the autoregressive LM objective captures more accurate knowledge than Cloze-style loss does by not introducing extra noise (e.g., masked tokens) during pre-training.

heading: 3.1.3 WIKIDATA
text: We select our best BERT based method MAMA-BERTLARGE, and GPT-2 based method MAMAGPT-2XL on TAC KBP to compare with Stanford OpenIE (the best open IE system on TAC KBP) for scalability experiments on Wikidata. We follow the same definition as the slot filling task to calculate precision, recall, and F1 on Wikidata. Table 3 summarizes the results. MAMA is scalable to larger corpora. Similar to the trends on TAC KBP, MAMA-GPT-2XL performs the best in precision, recall, and F1. The results show the effectiveness of MAMA in generating candidate facts and high-quality KGs. We also find that MAMA-GPT-2XL outperforms MAMA-BERTLARGE by over 1% in F1. This shows that the larger model (GPT-2XL has 5x more
3 There are 2,383 correct oracle facts based on the “manual runs” assessment in TAC KBP.
7
Under review as a conference paper at ICLR 2021
of the pre-trained LM.
parameters compared to BERTLARGE) contains more knowledge, and MAMA is able to restore the knowledge. When larger or deeper models (e.g., GPT-3) are used with MAMA, we can expect more gains of the KG quality. Thanks to the efficient nature of MAMA, which relies only on the forward pass of the LMs without fine-tuning, the results suggest that MAMA is scalable to large KGs. Larger corpora embed more complete KGs. In particular, MAMA-GPT-2XL outperforms Stanford OpenIE by 5.6% in F1. MAMA-BERTLARGE outperforms Stanford OpenIE by approximately 4.4% in F1. Both F1 gains are larger compared to that on TAC KBP. This is because that the LMs contain world knowledge from pre-training corpora, e.g. Wikipedia and Common Crawl. The larger the textual corpora are, the more knowledge our method is able to recover and match to the knowledge stored in LMs. The finding is particularly important, since we are now able to construct larger KGs of high quality from scratch when larger datasets are used, such as WebText2 and Common Crawl (Raffel et al., 2019; Brown et al., 2020). Similar to the observations on TAC KBP, the precision is higher compared to recall. Wikidata is not fully built from Wikipedia, MAMA could improve the recall by running on those larger corpora to collect more facts.

heading: 3.2 ANALYSIS OF UNMAPPED FACTS
text: The open KG constructed by MAMA is a new type of KG combining the fixed KG schema with the flexible open schema. We turn to study the quality of the candidate facts that are not mapped to the above reference KG schema, but are in the open schema generated by MAMA. We manually judge such unmapped facts generated by our best method MAMA-GPT-2XL from 100 sampled documents in Wikidata and TAC KBP respectively. The quality of unmapped facts is verified by human annotators. We find 35.3% of the unmapped facts are true on Wikidata. We find 83.2% of those true facts are partially unmapped facts as defined in Sec. 2.2.2, e.g., (Bob Dylan.Q392, tour with, the Grateful Dead.Q212533), whose relation is not within the schema of Wikidata, while both head and tail are in the schema. The remaining true facts are completely unmapped facts (Sec. 2.2.2), e.g., a candidate fact (Jacob, was, A Registered Mennonite) stays the same in the open schema. Accurate entity detection is desired. We also notice 45.5% of the untrue unmapped facts on Wikidata are due to the incorrect entities detected by the spaCy. Incorrect or missing entity linking (to either head or tail) in Sec. 2.2.1 causes additional 9.1% errors in the unmapped facts. 4.5% of the untrue unmapped facts are caused by the missing relation mapping in Sec. 2.2.1. The rest errors made by MAMA-GPT-2XL are incorrect relation phrases, such as uninformative relation phrases, e.g., (Dylan, made, his breakthrough), which is similar to the errors made by open IE systems (Fader et al., 2011). Both entity linking and relation mapping of Map stage rely heavily on the accuracy of entity detection from the spaCy noun chunk. We conclude that the main root cause of the untrue unmapped facts is due to the errors made by the spaCy noun chunk.
We observe similar trends on TAC KBP. We plan to leverage crowdsourcing platforms, e.g., Mechanical Turk, to conduct quantitative evaluations over the unmapped facts to better understand the strengths and shortage of MAMA. We plan to identify more accurate entities by relying on attention weights in LMs (Clark et al., 2019; Hewitt & Manning, 2019) instead of using extra resources. We will also investigate stronger entity linkers (Kolitsas et al., 2018) and learn a more robust relation mapping through weak or distant supervision (Mintz et al., 2009; Ratner et al., 2017). We will investigate more sophisticated approaches, such as graph neural networks (Kipf & Welling, 2016), to generate more accurate relation phrases from the attention weight matrices by considering structural information.

heading: 4 RELATED WORK
text: Knowledge graph construction can be generally categorized into two groups, 1) supervised approaches. Wikidata, Freebase (Bollacker et al., 2008), YAGO (Suchanek et al., 2007), YAGO2 (Hof-
8
Under review as a conference paper at ICLR 2021
fart et al., 2013), DBpedia (Auer et al., 2007) are built based on human supervision from Wikipedia infoboxes and other structured data sources; 2) semi-supervised approaches. Open information extraction systems, e.g., OLLIE (Schmitz et al., 2012), Reverb (Fader et al., 2011), Stanford OpenIE (Angeli et al., 2015), and OpenIE 5.1 2 aim to leverage carefully-designed patterns based on linguistic features (e.g., dependencies and POS tags), to extract triplets from web corpora for open schema KG. Besides, NELL (Carlson et al., 2010), DeepDive (Niu et al., 2012), Knowledge Vault (Dong et al., 2014) extract information based on a fixed schema or ontology, where humans help improve the accuracy of the extractions. Probase (Wu et al., 2012) produces taxonomies instead of rich typed relations in general KGs. MAMA instead uses learned knowledge stored in pre-trained LMs without human supervision to construct an open KG, which is a mixture of fixed schema and open schema. Different from commonsense knowledge construction using Transformers (Davison et al., 2019; Bosselut et al., 2019), the proposed method is unsupervised and end-to-end, and constructs general-purpose KGs instead of commonsense knowledge.
Language models, e.g., BERT (Devlin et al., 2018), GPT (Radford et al., 2018), GPT-2/3 (Radford et al., 2019; Brown et al., 2020), ELMo (Peters et al., 2018), Transformer-XL (Dai et al., 2019), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and MegatronLM (Shoeybi et al., 2019) contain factual knowledge obtained via pre-training on large-scale corpora such as Wikipedia and BookCorpus (Zhu et al., 2015). Studies have leveraged the pre-trained LMs as virtual KGs, and show reasonable performance in QA tasks (Dhingra et al., 2020; Guu et al., 2020), and language modeling (Khandelwal et al., 2019). LMs are further enhanced by KGs (Peters et al., 2019) to improve knowledge-driven tasks. While the existing work utilizes knowledge in an implicit way, the main difference is that our approach explicitly extracts knowledgeable facts from the LMs. Compare to the joint training with knowledge base to improve shallow word embedding (Wang et al., 2014), we show that the knowledge is already stored in the deep LMs. We plan to incorporate domain knowledge into language models to construct domain-specific KGs. The main difference between LAMA (Petroni et al., 2019; 2020) and MAMA is mainly two-fold: (1) LAMA aims to complete Cloze-style statement, e.g., given “Dylan is a ”, LAMA predicts which words/phrases should fill the blank “ ”, which has no direct connection to KGs. There are several fundamental limitations when adapting LAMA to construct KGs, e.g., additional queries must be constructed first, and the answers for the queries must be linked to KGs. MAMA aims to solve a reasoning problem, e.g., given a passage, MAMA directly matches the fact in the form of a triplet (Dylan, is, songwriter) at the first step, then maps the fact to produce a KG. (2) The benchmark datasets used with MAMA are larger compared to the LAMA benchmark, e.g., Wikidata is 3 orders of magnitude larger compared to the largest dataset in the LAMA benchmark.
Neural network interpretation here specifically refers to pre-trained deep language model analysis. There has been a lot of work to understand what the neural networks learn (Linzen et al., 2016; Adi et al., 2016; Tenney et al., 2019). With regards to analyzing Transformer (Vaswani et al., 2017) based language models (e.g., BERT and GPT-3), substantial recent work focuses on both visualizing and analyzing the attention (Vig, 2019; Jain & Wallace, 2019; Clark et al., 2019; Michel et al., 2019; Vig et al., 2020; Ramsauer et al., 2020; Hendrycks et al., 2020). Instead of analyzing or visualizing, we use LMs to generate structured KGs to directly recover what LMs learn from the corpora.

heading: 5 CONCLUSION
text: We show that the knowledge graphs can be constructed by a single forward pass of the language models over textual corpora. We propose a two-stage unsupervised approach MAMA to first match the facts in the corpus with the internal knowledge of the language model, and then map the matched facts to produce a knowledge graph. We demonstrate the quality of the resultant open knowledge graphs by comparing to two knowledge graphs (Wikidata and TAC KBP). The open knowledge graph also features new facts in the open schema, which could have broad implications for knowledge graphs and their downstream applications. The results also suggest that larger language models store richer knowledge than existing knowledge graphs, and generating on even larger high-quality text corpora could continue improving knowledge graphs. Additionally, the knowledge graphs generated by our approach can help researchers to look into what the language models learn, so our interpretable knowledge graphs establish a bridge between the deep learning and knowledge graph communities.
9
Under review as a conference paper at ICLR 2021

heading: A ADDITIONAL DETAILS AND ANALYSIS OF MAMA
text: A.1 METHOD DETAILS
Map stage details To evaluate the mapped facts, we first use Match stage of MAMA to run over the corresponding documents to generate the candidate facts. For Map stage on TAC KBP, we link to the oracle annotation of the entities or spans in the TAC KBP corpus. On Wikidata, the entity linking method described in Sec. 2.2.1 is first leveraged to link entities in the candidate facts to Wikipedia anchors. We build an enhanced mention-to-entity dictionary based on Spitkovsky & Chang (2012). In particular, we add new Wikipedia anchors to the dictionary which results in 26 million entries comparing to 21 million entries in Spitkovsky & Chang (2012). Then a Wikipedia anchor to the Wikidata item dictionary is constructed and used to further link the entities to Wikidata. If the head or tail is a pronoun, we further use neuralcoref 4 for coreference resolution. We use GloVe (Pennington et al., 2014) embedding for disambiguation. The relation mapping is constructed offline for TAC KBP and Wikidata respectively using the method in Sec. 2.2.1. Besides the automatic relation mapping method proposed in Angeli et al. (2015), we manually check whether the top relation phrases are true as described in Sec. 2.2.1. For relation mapping, we randomly sampled a hold-out dataset including 2,000 documents from the TAC KBP corpus and English Wikipedia for the relation mapping construction on TAC KBP and Wikidata respectively. For oracle facts in Wikidata, we only preserve those facts describing relations between entities that could be linked to corresponding Wikipedia anchors. We rule out facts of attributes about entities and facts of auxiliary relations (such as topic’s main category.P901) and finally results in 27,368,562 oracle facts.
Implementation details For Wikidata, at Match stage, we randomly split the English Wikipedia data into 20 partitions, and map the data partitions to 20 distributed servers to run. Each server is configured with four Tesla K80 12Gs. We set the max sequence length to 256, and batch size as 32 for MAMA-BERTLARGE and 4 for MAMA-GPT-2XL. We use implementations of pre-trained LMs in Transformers package 5. We use spaCy sentencizer 6 to segment the documents into sentences. MAMA-BERTLARGE takes approximately 48 hours, and MAMA-GPT-2XL costs around 96 hours. The resulting candidate facts of Match stage from the 20 servers are then reduced a data server, where a MongoDB database is maintained to store the oracle Wikidata and entity linking results to enable the efficient Map stage. To produce the open KGs, Map stage takes around 18 hours. The setup is similar to TAC KBP. Match stage is done within 48 hours for all the settings. The batch sizes of MAMA-BERTBASE, MAMA-GPT-2, MAMA-GPT-2MEDIUM, MAMA-GPT-2LARGE are 64, 32, 16, 8 respectively.
Parameter settings The parameter settings are shared across TAC KBP and Wikidata. All the choices are based on the parameter study in Sec. A.3. The beam size of Algorithm 1 is set to 6. The matching degree threshold of Constraint #1 (Sec. 2.1.2) is set to 0.005, and the number of distinct head-tail pairs of Constraint #2 (Sec. 2.1.2) is set to 10. To generate the attention weight matrix As of a sentence, we reduce the weights of every attention head in the last layer of pre-trained LMs using the mean operator.
A.2 ERROR ANALYSIS
There is still significant room to improve MAMA. To further understand the shortage of MAMA, we conduct an error analysis of the errors in precision (i.e., incorrect facts returned by MAMA) of Table 2 and Table 3. We choose our best method MAMA-GPT-2XL for the study. We sample 100 documents from the Wikidata dataset, and manually check the reasons for the errors. We find 33.1% of the errors are caused by incorrect entities, while the relation phrases are correct. The errors are due to the incorrect noun chunk detected by the spaCy 7. 18.3% of the errors are due to the missing relation mapping created in Sec. 2.2.1. Note that we find approximately 23.8% of the errors are actually correct facts that are new in the reference KGs. e.g., (Bob Dylan.Q392, residence.P551, Nashville.Q23197) (in Figure 16) is not an existing fact in Wikidata, but it is a correct mapped fact based on our annotation. The rest errors made by MAMA-GPT-2XL are incorrect relation phrases,
4 https://github.com/huggingface/neuralcoref 5 https://github.com/huggingface/transformers 6 https://spacy.io/api/sentencizer 7 https://spacy.io/usage/linguistic-features/#noun-chunks
13
Under review as a conference paper at ICLR 2021
such as uninformative relation phrases. We find similar errors are made by MAMA-GPT-2XL on TAC KBP. Similar to Sec. 3.2, enhancing the entity detection, entity linker, relation mapping, and relation generation are helpful. We also plan to leverage lifelong learning (Carlson et al., 2010) to add true facts to the reference KGs to improve the evaluation.
A.3 PARAMETER STUDY
We study the effects of the parameters using MAMA-BERTBASE on TAC KBP. We randomly sample 20% of the oracle query entities as a hold-out dataset to tune the parameters, and use the best parameter setting achieved for both TAC KBP and Wikidata experiments. When studying the effect of a certain parameter, we keep the remaining parameters as default described in Sec. A.1. We use F1 to measure the effects.
Effects of beam size Figure 3(a) illustrates the effects of various beam sizes in Algorithm 1. We find that in general, the larger the beam size is, the better F1 the setting achieves. This is because that MAMA is able to reserve more potentially correct facts when more candidates are allowed in the Match stage. However, F1 improvement gradually becomes subtle, while the computation costs increase more significantly. For sake of the efficiency, we do not explore larger beam sizes. We set the beam size as 6.
Effects of search constraints Figure 3(b) compares the effect of different thresholds of the matching degree of Constraint #1 in Sec. 2.1.2. We set the threshold as 0.005 since it achieves the best result. Note that the summed attention score is normalized by the length of the fact to penalize the cumbersome facts. The matching degree threshold is effective, which is mainly because of the knowledge contained in the self-attention matrix. The score in the attention matrix is representing the chance of the facts to be the true facts based on the stored knowledge. Figure 3(c) shows the impact of the number of distinct head-tail pairs in identifying common relations of Constraint #2 in Sec. 2.1.2. The best result is achieved when it equals 10. This shows that while MAMA mostly identifies frequent relations, it is also able to capture some rare relations for the open schema.
Effects of attention weights Figure 3(d) shows the comparison between attention weights of the last layer and the mean of all layers. The attention weights of the last layer perform better. This is due to the attention weights in lower layers are low-level linguistic knowledge according to (Clark et al., 2019; Ramsauer et al., 2020), which are less relevant to the factual knowledge for the KG construction. Figure 3(e) compares the impact of different attention reduction, i.e., mean, max, over the attention heads of the last layer. We find the “mean” perform better. The reason is that the token often intensively attends to several specific tokens in the sequence (Michel et al., 2019), and the “mean” operator is more sensitive to such information.
14
Under review as a conference paper at ICLR 2021

heading: B SAMPLES FROM MAMA ON TAC KBP
text: B.1 MAPPED FACTS
We randomly sample 100 documents from TAC KBP corpus, then randomly sample sentences from those documents. The uncurated candidate facts and the corresponding mapped facts of the sampled sentences based on our best methods MAMA-BERTLARGE and MAMA-GPT-2XL are shown in Figure 4 and Figure 5 respectively. We also randomly sample several sentences in which MAMABERTLARGE differs from MAMA-GPT-2XL in the resulting facts for comparison, which is illustrated in Figure 6. In each table, “ID” represents the document ID of a sampled sentence in TAC KBP corpus. “Sentence” indicates the sampled sentence. “Candidate facts to mapped facts” column contains the candidate facts (on the left side of “!”) and their corresponding mapped facts (on the right side of “!”).
B.2 UNMAPPED FACTS
We randomly sample 100 documents from TAC KBP corpus. From those documents, we show unmapped facts from the sampled sentences from those documents. We manually check the correctness of the unmapped facts according to Sec. 3.2, and show the correct ones. The original candidate facts with the corresponding unmapped facts of the sampled sentences generated by MAMABERTLARGE and MAMA-GPT-2XL are shown in Figure 7 and Figure 8. A further comparison of the unmapped candidate facts is illustrated in Figure 9. In each table, “ID” represents the document ID of a sampled sentence in TAC KBP corpus. “Sentence” indicates the sampled sentence. “Candidate facts to unmapped facts” column contains the candidate facts (on the left side of “!”) and their corresponding unmapped facts (on the right side of “!”).

heading: C SAMPLES FROM MAMA ON WIKIDATA
text: C.1 MAPPED FACTS
Similar to TAC KBP, we randomly sample 100 documents from the Wikidata corpus (i.e., English Wikipedia), then randomly sample sentences from those documents. The uncurated candidate facts and the corresponding mapped facts of the sampled sentences based on our best methods MAMABERTLARGE and MAMA-GPT-2XL are shown in Figure 10 and Figure 11 respectively. We also randomly sample several sentences in which MAMA-BERTLARGE differs from MAMA-GPT-2XL in the resulting facts for comparison, which is illustrated in Figure 12. In each table, “ID” represents the Wikipedia page’s title of a sampled sentence. “Sentence” indicates the sampled sentence. “Candidate facts to mapped facts” column contains the candidate facts (on the left side of “!”) and their corresponding mapped facts (on the right side of “!”).
C.2 UNMAPPED FACTS
Similar to TAC KBP, we randomly sample 100 documents from the Wikidata corpus. From those documents, we show unmapped facts from several sampled sentences from those documents. We manually check the correctness of the unmapped facts according to Sec. 3.2, and show the correct ones. The original candidate facts with the corresponding unmapped facts of the sampled sentences generated by MAMA-BERTLARGE and MAMA-GPT-2XL are shown in Figure 13 and Figure 14. A further comparison of the unmapped candidate facts is illustrated in Figure 15. In each table, “ID” represents the Wikipedia page’s title of a sampled sentence. “Sentence” indicates the sampled sentence. “Candidate facts to unmapped facts” column contains the candidate facts (on the left side of “!”) and their corresponding unmapped facts (on the right side of “!”).

heading: D ADDITIONAL OPEN KG SUBGRAPHS FROM MAMA ON WIKIDATA
text: We sample several documents from the Wikidata corpus. We visualize the mapped facts and unmapped facts from those documents as examples of subgraphs in the resulting open KGs. We show the snapshots of the subgraphs generated by MAMA-BERTLARGE from Figure 16 to Figure 24. We
15
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to mapped facts
text: similarly illustrate the snapshots of the subgraphs constructed by MAMA-GPT-2XL from Figure 25 to Figure 32. In each figure, the blue node and arrow represent the mapped facts in the Wikidata schema, while the yellow node and arrow denote the unmapped facts in the open schema. We additionally visualize the correct facts that are new in Wikidata according to Sec. A.2 in yellow.
16
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to mapped facts
text: 17
Under review as a conference paper at ICLR 2021
ID Sentence Candidate facts to mapped facts BERTLARGE GPT-2XL SF13 ENG 005 The men , who live in the Chicago area , were identified as David Coleman Headley
, 49 , a US citizen , and Tahawwur Hussain Rana , 48 , Rana , a native of Pakistan and citizen of Canada .
(Rana, , a native of, Canada) ! (Tahawwur Hussain Rana, per:origin, Canada) -
SF13 ENG 006 Khan died at his music centre near San Francisco in the United States after a prolonged kidney ailment , the news agency said . (Khan, died at his, The United States) ! (Ali Akbar Khan, per:country of death, United States) - SF13 ENG 006 By his early 20s he was music director of All-India Radio in Lucknow, broadcasting as a solo artist and composing for the radio’s orchestra. - (He, of, All-India Radio) ! (Ali Akbar Khan, per:employee or member of, AllIndia Radio) SF13 ENG 006 Defying his father , Khan moved to Bombay and began writing scores for films , including Chetan Anand ’ s ” Aandhiyan ” ( 1952 ) , Satyajit Ray ’ s ” Devi ” ( 1960 ) and Tapan Sinha ’ s ” Hungry Stones ” ( 1960 ) . (Khan, moved, Bombay) ! (Ali Akbar Khan, per:cities of residence, Bombay) - SF13 ENG 007 In 1966 he graduated from the faculty of theology at Al-Azhar, the 10th century university that has trained the majority of Sunni Muslim clerics from Africa and Asia. - (He, graduated from, Al-Azhar) ! (Mohammed Sayed Tantawi, per:schools attended, Al-Azhar) SF13 ENG 009 He became AIBA president in 2006 by defeating longtime incumbent Anwar Chowdhry of Pakistan, who was later barred for alleged financial corruption. - (Longtime Incumbent Anwar Chowdhry, by, Aiba President) ! (Anwar Chowdhry, per:employee or member of, AIBA) SF13 ENG 011 Marshall could get a lengthy prison term if convicted of grand larceny. - (Marshall, of, Grand Larceny) ! (Anthony Marshall, per:charges, larceny) SF13 ENG 012 Gadahn , also known as Azzam al - Amriki , was dressed in white robes and wearing
a white turban as he called for attacks on what he described as ” high - value targets . ”
(Gadahn, known as, Azzam Al-Amriki) ! (Adam Gadahn, per:alternate names, Azzam al-Amriki) -
SF13 ENG 012 Gadahn grew up on a farm in California and converted to Islam before moving to Pakistan in 1998 and reportedly attending an al - Qaida training camp . (Gadahn, to, Pakistan) ! (Adam Gadahn, per:countries of residence, Pakistan) - SF13 ENG 012 He was born in Oregon on Sept. 1, 1978 to a Jewish family. - (He, born in, Sept.) ! (Adam Gadahn, per:date of birth, 1978-09-01) SF13 ENG 015 Besides her sister , Philip , survivors include a son , Graham ; three daughters ,
Sidney , Gillian and Alexia ; and three grandchildren . (Her Sister, son ,, Graham) ! (Lucille Clifton, per:children, Graham) -
SF13 ENG 015 In addition to her daughter , of Columbia , other survivors include two daughters , Sidney Clifton of Los Angeles and Gillian Clifton - Monell of Fort Washington , Md . , ; a son , Graham Clifton of Wichita ; a sister ; and three grandchildren . (Her Daughter, ,, Sidney Clifton) ! (Lucille Clifton, per:children, Sidney Clifton) - SF13 ENG 017 Haig , 85 , died early Saturday in a Baltimore , Maryland , hospital . (Haig, died early, A Baltimore, Maryland, Hospital) ! (Alexander Haig, per:stateorprovince of death, Maryland) - SF13 ENG 019 McGregor was born in Iowa , but grew up in Colorado where his father , Brian , was a coaching legend at Arvada West . (Mcgregor, but grew up, Brian) ! (Keli McGregor, per:parents, Brian) - SF13 ENG 025 Kissel was born in Adrian, Michigan, but her family had also lived in Minneapolis. - (Kissel, in, Minneapolis) ! (Nancy Kissel, per:cities of residence, Minneapolis) SF13 ENG 025 Kissel, 46, of Adrian, Michigan, was convicted of drugging then bashing her hus-
band Robert to death in a luxury Hong Kong apartment and sentenced to life in prison in September 2005.
- (Kissel, of, Michigan) ! (Nancy Kissel, per:statesorprovinces of residence, Michigan)
SF13 ENG 026 Neal had lung cancer and died at her home in Edgartown , Massachusetts on Martha ’ s Vineyard , said longtime friend Bud Albers of Knoxville . (Neal, died at her home in, Edgartown) ! (Patricia Neal, per:city of death, Edgartown) - SF13 ENG 026 She grew up in Knoxville, Tenn., where Neal was turned onto acting at 10 after seeing an impassioned speech about ”demon rum” at the local temperance union speaking contest. - (She, in, Tenn.) ! (Patricia Neal, per:statesorprovinces of residence, Tennessee) SF13 ENG 027 Al-Hakim, who died Wednesday of lung cancer in Tehran, was a symbol for many of the re-emergence of Iraq’s Shiite political majority after decades of oppression under Saddam Hussein’s Sunni-led regime. - (Al-Hakim, who died, Tehran) ! (Abdul Aziz Al-Hakim, per:city of death, Tehran) SF13 ENG 028 TAMPA Heart disease , exacerbated by cocaine use , killed celebrated TV pitchman Billy Mays , according to the Hillsborough County medical examiner ’ s final autopsy report released Friday . (Celebrated Tv Pitchman Billy Mays, killed, Tampa) ! (Billy Mays, per:cause of death, TAMPA Heart disease) - SF13 ENG 030 Family spokesman Dan Ewald said Anderson died from complications from dementia . (Anderson, died from, Dementia) ! (Sparky Anderson, per:cause of death, dementia) - SF13 ENG 031 He served in the Coast Guard during World War II and had a rare leading role in the low - budget thriller ” Strangler of the Swamp ” ( 1946 ) . (He, in, The Coast Guard) ! (Blake Edwards, per:employee or member of, Coast Guard) - SF13 ENG 031 His agent Lou Pitt confirmed he had died , while the ” Entertainment Tonight ” news show and website reported that he died in Santa Monica on Wednesday night from complications of pneumonia , with Andrews at his side . (He, died, Complications) ! (Blake Edwards, per:cause of death, complications) - SF13 ENG 033 In a career that spanned seven decades, Ginzburg authored several groundbreaking studies in various fields – such as quantum theory, astrophysics, radio-astronomy and diffusion of cosmic radiation in the Earth’s atmosphere – that were of ”Nobel Prize caliber,” said Gennady Mesyats, the director of the Lebedev Physics Institute in Moscow, where Ginzburg worked. - (Ginzburg, in, The Lebedev Physics Institute) ! (Vitaly Ginzburg, per:employee or member of, Lebedev Physics Institute) SF13 ENG 035 Kaczynska , who died aged 59 , married Kaczynski in 1978 . (Kaczynska, married, Kaczynski) ! (Maria Kaczynska, per:spouse, Kaczynski) - SF13 ENG 037 Prime Minister Francois Fillon called Chabrol a ” great director , producer and
screenwriter ( who ) was one of the grand figures of the ’ Nouvelle vague , ’ which revolutionized the style and techniques of cinema by looking at real experience , true life , that which is indiscreet and subtle . ”
(Afp Chabrol, was, director) ! (Claude Chabrol, per:title, film director); (Chabrol, director, Producer) ! (Claude Chabrol, per:title, producer) -
SF13 ENG 038 Cunningham’s survivors include a brother, Jack Cunningham of Centralia. - (Cunningham’S Survivors, brother, Jack Cunningham) ! (Merce Cunningham, per:siblings, Jack Cunningham) SF13 ENG 038 Mercier Philip Cunningham was born in Centralia, Wash., on April 16, 1919. - (Mercier Philip Cunningham, was born in, April) ! (Merce Cunningham, per:date of birth, 1919-04-16) SF13 ENG 041 A year after enrolling in New York University on a track scholarship , Hewitt dropped out . (Hewitt, ,, New York University) ! (Don Hewitt, per:schools attended, New York University) - SF13 ENG 041 By the time Hewitt was 6, his family was living in New Rochelle, N.Y. - (Hewitt, in, New Rochelle) ! (Don Hewitt, per:cities of residence, New Rochelle) SF13 ENG 043 Greenwich also worked as an arranger and singer , a role that saw her working with
artists including Frank Sinatra and Ella Fitzgerald . (Greenwich, worked as an, Singer) ! (Ellie Greenwich, per:title, singer) -
SF13 ENG 044 Dunne was born in 1925 in Hartford, Connecticut, to a wealthy Roman Catholic family and grew up in some of the same social circles as the Kennedys. - (Dunne, was born in 1925 in, Connecticut) ! (Dominick Dunne, per:stateorprovince of birth, Connecticut) SF13 ENG 044 In 1957 , Dunne moved to Los Angeles to work on the CBS showcase ” Playhouse 90 . ” (Dunne, moved to, Los Angeles) ! (Dominick Dunne, per:cities of residence, Los Angeles) - SF13 ENG 044 Dunne found his greatest prominence as a celebrity journalist while covering the 1995 murder trial of football star and actor O.J. Simpson, who had been accused of killing his ex-wife Nicole Brown Simpson and her friend Ronald Goldman. - (Dunne, as, A Celebrity Journalist) ! (Dominick Dunne, per:title, celebrity journalist) SF13 ENG 045 Goldstein died in New York on Aug . 28 . (Goldstein, died, Aug.) ! (Adam Goldstein, per:date of death, XXXX-08-28) - SF13 ENG 046 After acting in movies and television , Meredith receded into a quiet life in Santa Fe
, writing , painting , golfing and acting in a stage production of ” The Odd Couple . ”
(Meredith, in, Santa Fe) ! (Don Meredith, per:cities of residence, Santa Fe) -
SF13 ENG 046 In high school and at Southern Methodist University , where , already known as Dandy Don ( a nickname bestowed on him by his brother ) , Meredith became an all - American . (Dandy Don, where ,, Southern Methodist University) ! (Don Meredith, per:schools attended, Southern Methodist University) - SF13 ENG 046 He spent much of his life backing away from the nickname Dandy Don , particularly during his secluded later decades in New Mexico . (He, the nickname, Dandy Don) ! (Don Meredith, per:alternate names, Dandy Don) - SF13 ENG 048 The group, known variously as the Five or the Whites (for the color of most of their buildings) or the New York School, consisted of Gwathmey, Michael Graves, Eisenman, John Hejduk and Richard Meier. - (Gwathmey, of, The New York School) ! (Charles Gwathmey, per:employee or member of, New York School) SF13 ENG 048 Meier , who said he had known Gwathmey for 50 years , has particularly fond memories from the time when Gwathmey was first courting his second wife , Bette - Ann Damson , and they all picked corn for dinner in a field adjacent to a barn Meier was renting on the East End of Long Island . (Gwathmey, his second wife ,, Bette-Ann Damson) ! (Charles Gwathmey, per:spouse, Bette-Ann Damson) - SF13 ENG 057 By 2013, automakers will have dozens of plug-in electric hybrid vehicles and fully electric vehicles, said Jason Forcier, a vice president at battery maker A123 Systems Inc. - (A123 Systems Inc., a vice president, Jason Forcier) ! (A123 Systems Inc., org:top members employees, Jason Forcier) SF13 ENG 058 Access Industries , a privately held company founded in 1986 by Len Blavatnik , has a diverse portfolio of investments in industry , real estate , media and telecommunications . (Access Industries, founded in 1986 by, Len Blavatnik) ! (Access Industries, org:founded by, Len Blavatnik) - SF13 ENG 069 &quot;The indications are positive,&quot; said Vincent Cogliano, director of the Monographs program at IARC, which decides on carcinogen classifications. - (Iarc, , director of the, Vincent Cogliano) ! (International Agency for Research on Cancer, org:top members employees, Vincent Cogliano) SF13 ENG 082 ”There hasn’t been a concerted push to open doors for Muslim orphans because the expectation would be that those efforts would fall flat,” said Chuck Johnson, chief executive of the National Council for Adoption, a policy group in Alexandria, Va. - (Adoption, in, Alexandria) ! (National Council for Adoption, org:city of headquarters, Alexandria) SF13 ENG 084 ” The scenario itself is secret , ” said Eileen McMenamin , vice president of communications for the Bipartisan Policy Center ( BPC ) , which is hosting the event dubbed ” Cyber ShockWave . ” (The Bipartisan Policy Center, , vice president of, Eileen Mcmenamin) ! (Bipartisan Policy Center, org:top members employees, Eileen McMenamin) - SF13 ENG 088 Bank Julius Baer Co., based in Basel, Switzerland, sued because WikiLeaks posted accountholder information from its Cayman outpost amid allegations of money laundering and tax evasion. - (Bank Julius Baer Co., based, Basel) ! (Bank Julius Baer, org:city of headquarters, Basel) SF13 ENG 094 The other manufacturers are two US companies, L-3 Communications and Rapiscan Systems, a unit of OSI Systems, and British rival Smiths Detection. - (Rapiscan, ,, Two Us Companies) ! (Rapiscan Systems, org:country of headquarters, US) SF13 ENG 096 Freedom was founded in the 1930s by R.C. Hoiles and is still majority owned by the Hoiles family. - (Freedom, was founded in the, R.C. Hoiles) ! (Freedom Communications, org:founded by, R.C. Hoiles) SF13 ENG 098 Shares of Millipore have more than doubled since Martin D . Madaus took over as chief executive in January 2005 . (Millipore, have, Martin D.) ! (Millipore Corp., org:top members employees, Martin D. Madaus) -
Figure 6: Mapped facts: MAMA-BERTLARGE vs. MAMA-GPT-2XL on TAC KBP.
18
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to unmapped facts
text: SF13 ENG 001 Bashardost was born in 1965 in the southern Ghanzi province and his family migrated to Iran and then to Pakistan after successive coup and factional fighting in Afghanistan . (Bashardost, in, The Southern Ghanzi Province) ! (Ramazan Bashardost, per:stateorprovince of birth, The Southern Ghanzi Province) SF13 ENG 012 Gadahn , also known as Azzam the American , was born in 1978 . (Gadahn, ,, Azzam) ! (Adam Gadahn, per:alternate names, Azzam) SF13 ENG 015 Besides her sister , Philip , survivors include a son , Graham ; three daughters , Sidney , Gillian and Alexia ; and three
grandchildren . (Her Sister, ,, Philip) ! (Lucille Clifton, per:siblings, Philip)
SF13 ENG 015 The poet laureate of Maryland from 1979 to 1985 , Clifton was a writer in residence at Coppin State College , now Coppin State University , a historically black college in Baltimore . (Clifton, in, Coppin State College) ! (Lucille Clifton, per:school attended, Coppin State College) SF13 ENG 015 Her father , Samuel , was a steelworker ; her mother , Thelma , worked in a laundry . (Her Father, ,, Samuel) ! (Lucille Clifton, per:parents, Samuel) SF13 ENG 015 Lucille Clifton , a distinguished American poet whose work trained lenses wide and narrow on the experience of being
black and female in the 20th century , exploring vast subjects like the indignities of history and intimate ones like the indignities of the body , died on Saturday in Baltimore .
(Lucille Clifton, ,, poet) ! (Lucille Clifton, per:title, poet)
SF13 ENG 015 The precise cause of death had not been determined , her sister , Elaine Philip , told The Associated Press on Sunday . (Her Sister, ,, Elaine Philip) ! (Lucille Clifton, per:siblings, Elaine Philip) SF13 ENG 037 CLAUDE CHABROL , PIONEER FRENCH FILMMAKER , DIES AT 80 Joseph Berger contributed reporting from
New York , and Maia de la Baume contributed from Paris . (Claude Chabrol, ,, Pioneer French Filmmaker) ! (Claude Chabrol, per:title, Pioneer French Filmmaker)
SF13 ENG 037 Chabrol ’ s survivors also include his third wife , Aurore Pajot , who acted as his script supervisor on nearly all of his movies from 1968 on and whom he married in 1981 ; and Pajot ’ s daughter , Cecile Maistre , who was an assistant director on his films and wrote the script with him for ” The Girl Cut in Two ” ( 2007 ) . (His Third Wife, ,, Aurore Pajot) ! (Claude Chabrol, per:spouse, Aurore Pajot) SF13 ENG 044 Dominick Dunne , a novelist and journalist who chronicled true - crime tales of the rich and infamous , including O . J . Simpson and Claus von Bulow , and in turn became a celebrity in his own right , died of bladder cancer Aug . 26 at his home in New York City . (Dominick Dunne, ,, A Novelist) ! (Dominick Dunne, per:title, A Novelist) SF13 ENG 044 Dunne and his wife , Ellen Griffin Dunne , known as Lenny , were married in 1954 . (His Wife, ,, Ellen Griffin Dunne) ! (Dominick Dunne, per:spouse, Ellen Griffin Dunne) SF13 ENG 044 Dunne was born in 1925 in Hartford , Connecticut , to a wealthy Roman Catholic family and grew up in some of the
same social circles as the Kennedys . (Dunne, ,, Connecticut) ! (Dominick Dunne, per:stateorprovince of birth, Connecticut)
SF13 ENG 049 A professor emeritus at Yale University , Mandelbrot was born in Poland but as a child moved with his family to France where he was educated . (Mandelbrot, ,, Yale University) ! (Benoit Mandelbrot, per:employee or member of, Yale University) SF13 ENG 058 Access Industries , a privately held company founded in 1986 by Len Blavatnik , has a diverse portfolio of investments in industry , real estate , media and telecommunications . (Access Industries, ,, A Privately Held Company) ! (Access Industries, properties, A Privately Held Company) SF13 ENG 079 Water and its links to development , peace and conflict were key words in the annual sessions , Anders Berntell , executive director of Stockholm International Water Institute ( SIWI ) , said in his opening address . (Stockholm International, ,, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell)
Figure 7: Unmapped facts: MAMA-BERTLARGE on TAC KBP.

heading: ID Sentence Candidate facts to unmapped facts
text: SF13 ENG 009 He took office in 2006 by defeating longtime incumbent Anwar Chowdhry from Pakistan, who was later barred for alleged financial corruption. (Longtime Incumbent Anwar Chowdhry, from, Pakistan) ! (Anwar Chowdhry, per:countries of residence, Pakistan) SF13 ENG 012 The message also featured several appearances of Adam Gadahn, also known as Azzam al-Amriki, an American who grew up in Riverside County and converted to Islam and joined al-Qaida. (Adam Gadahn, also, Azzam Al-Amriki) ! (Adam Gadahn, per:alternate names, Azzam Al-Amriki); (Adam Gadahn, ,, Azzam Al-Amriki) ! (Adam Gadahn, ,, Azzam Al-Amriki) SF13 ENG 014 Mohammed Oudeh, a former math teacher who became the mastermind of the deadly attack on Israeli athletes at the 1972 Munich Olympics, died Friday in Damascus. (Mohammed Oudeh, ,, teacher) ! (Mohammed Oudeh, per:title, teacher) SF13 ENG 014 He settled in the West Bank town of Ramallah, but in 1999, after a trip to Jordan, he was barred by Israel from returning. (He, in, The West Bank Town) ! (Mohammed Oudeh, per:cities of residence, The West Bank Town) SF13 ENG 014 Mohammed Oudeh, a former math teacher who became the mastermind of the deadly attack on Israeli athletes at the
1972 Munich Olympics, died Friday in Damascus. (Mohammed Oudeh, ,, A Former Math Teacher) ! (Mohammed Oudeh, per:title, A Former Math Teacher)
SF13 ENG 014 In later years, as a graying member of the Palestinian old guard, Oudeh, most commonly known by his guerrilla name, Abu Daoud, showed no remorse for the botched hostage taking and killings of 11 members of the Israeli Olympic team that shook the world. (Oudeh, ,, Abu Daoud) ! (Mohammed Oudeh, per:alternate names, Abu Daoud) SF13 ENG 015 After her husband died in 1984, Clifton taught at the University of California at Santa Cruz before returning to Maryland in 1989. (Clifton, of, Santa Cruz) ! (Lucille Clifton, per:cities of residence, Santa Cruz) SF13 ENG 015 Lucille Clifton, a National Book Award-winning poet and Pulitzer finalist, has died. (Lucille Clifton, ,, poet) ! (Lucille Clifton, per:title, poet) SF13 ENG 015 When she was a girl, Lucille Clifton sat on her mother’s lap and listened to her recite poetry. (Lucille Clifton, ,, A Girl) ! (Lucille Clifton, gender, A Girl) SF13 ENG 015 In 1967, they moved to Baltimore, and Clifton worked in Washington for the old U.S. Office of Education. (They, to, Baltimore) ! (Lucille Clifton, per:cities of residence, Baltimore) SF13 ENG 015 Clifton’s sister, Elaine Philip, said the former poet laureate of Maryland passed away Saturday morning at Johns Hop-
kins University Hospital in Baltimore. (Clifton’S Sister, ,, Elaine Philip) ! (Lucille Clifton, per:siblings, Elaine Philip)
SF13 ENG 018 Harvey Pekar, the writer of comics whose autobiographical comic book series ”American Splendor” chronicled his life as a filing clerk, record collector, freelance jazz critic and one of life’s all-around misfits, was found dead July 12 at his home near Cleveland. (Harvey Pekar, ,, writer) ! (Harvey Pekar, per:title, writer) SF13 ENG 018 Pekar attended what became Case Western Reserve University, served in the Navy in the late 1950s and worked a series of menial jobs before taking a 30-year jobas a filing clerk at a VA hospital in Cleveland. (Pekar, what became, Case Western Reserve University) ! (Harvey Pekar, per:schools attended, Case Western Reserve University) SF13 ENG 025 Kissel, 46, of Adrian, Michigan, was convicted of drugging then bashing her husband Robert to death in a luxury Hong Kong apartment and sentenced to life in prison in September 2005. (Kissel, ,, Adrian) ! (Nancy Kissel, per:cities of residence, Adrian); (Kissel, ,, Michigan) ! (Nancy Kissel, per:statesorprovinces of residence, Michigan) SF13 ENG 038 Mercier Philip Cunningham was born in Centralia, Wash., on April 16, 1919. (Mercier Philip Cunningham, was born in, Centralia) ! (Merce Cunningham, was born in, Centralia); (Mercier Philip Cunningham, in, Wash.) ! (Merce Cunningham, per:stateorprovince of birth, Wash.); (Mercier Philip Cunningham, in, April) ! (Merce Cunningham, per:date of birth, 1919-04-16) SF13 ENG 038 Cunningham’s survivors include a brother, Jack Cunningham of Centralia. (Cunningham’S Survivors, ,, Jack Cunningham) ! (Merce Cunningham, per:siblings, Jack Cunningham) SF13 ENG 041 By the time Hewitt was 6, his family was living in New Rochelle, N.Y. (Hewitt, in, New Rochelle) ! (Don Hewitt, per:cities of residence, New Rochelle) SF13 ENG 041 He is survived by his third wife, former television news correspondent Marilyn Berger; his sons, Steven and Jeffrey; his
daughter, Lisa Cassara; his stepdaughter, Jilian Childers Hewitt, whom Hewitt adopted; and three grandchildren. (Hewitt, ,, Jilian Childers Hewitt) ! (Don Hewitt, per:children, Jilian Childers Hewitt)
SF13 ENG 044 Dunne was born in 1925 in Hartford, Connecticut, to a wealthy Roman Catholic family and grew up in some of the same social circles as the Kennedys. (Dunne, was born in 1925 in, Connecticut) ! (Dominick Dunne, per:stateorprovince of birth, Connecticut) SF13 ENG 044 Dunne was part of a famous family that also included his brother, novelist and screenwriter John Gregory Dunne; his brother’s wife, author Joan Didion; and his son, Griffin. (His Brother, ,, Novelist) ! (His brother, per:title, Novelist) SF13 ENG 044 Dunne and his wife, Ellen Griffin Dunne, known as Lenny, were married in 1954. (Dunne, and, Ellen Griffin Dunne) ! (Dominick Dunne, per:spouse, Ellen Griffin Dunne) SF13 ENG 047 The women were accompanied by Reyna Luisa Tamayo, the mother of political prisoner Orlando Zapata, who died at
age 42 in a hunger strike February 23 to protest prison conditions. (Political Prisoner Orlando Zapata, ,, Reyna Luisa Tamayo) ! (Orlando Zapata, per:children, Reyna Luisa Tamayo)
SF13 ENG 061 InterContinental Hotels Group, owners of Holiday Inn Worldwide, has been ordered by a court to pay $25 million in damages to a franchisee for fraud. (Intercontinental Hotels Group, ,, Holiday Inn Worldwide) ! (InterContinental Hotels Group, org:parents, Holiday Inn Worldwide) SF13 ENG 079 Water and its links to development, peace and conflict were key words in the annual sessions, Anders Berntell, executive director of Stockholm International Water Institute (SIWI), said in his opening address.
(Stockholm International, director, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell); (Stockholm International, , executive director, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell)
SF13 ENG 090 ECO member countries include Afghanistan, Azerbaijan, Iran, Kazakhstan, Kyrgyzstan, Pakistan, Tajikistan, Turkey, Turkmenistan and Uzbekistan. (Eco Member Countries, ,, Azerbaijan) ! (Economic Cooperation Organization, org:members, Azerbaijan) SF13 ENG 096 A phone message left Sunday at Freedom’s Irvine, California, headquarters was not immediately returned. (Freedom’S Irvine, ,, California)! (Freedom Communications, org:stateorprovince of headquarters, California) SF13 ENG 097 STX Finland is part of the international STX Europe Group, with shipyards in Brazil, Norway, France, Romania and
Vietnam. (Stx Finland, of, The International Stx Europe Group)! (STX Finland, org:subsidaries, The International Stx Europe Group)
Figure 8: Unmapped facts: MAMA-GPT-2XL on TAC KBP.
19
Under review as a conference paper at ICLR 2021
ID Sentence Candidate facts to mapped facts BERTLARGE GPT-2XL SF13 ENG 001 Bashardost was born in 1965 in the southern Ghanzi province and his family mi-
grated to Iran and then to Pakistan after successive coup and factional fighting in Afghanistan .
(Bashardost, in, The Southern Ghanzi Province) ! (Ramazan Bashardost, per:stateorprovince of birth, The Southern Ghanzi Province) -
SF13 ENG 009 He took office in 2006 by defeating longtime incumbent Anwar Chowdhry from Pakistan, who was later barred for alleged financial corruption. (Longtime Incumbent Anwar Chowdhry, from, Pakistan) ! (Anwar Chowdhry, per:origin, Pakistan) (Longtime Incumbent Anwar Chowdhry, from, Pakistan) ! (Anwar Chowdhry, per:countries of residence, Pakistan) SF13 ENG 012 Gadahn , also known as Azzam the American , was born in 1978 . (Gadahn, ,, Azzam) ! (Adam Gadahn, per:alternate names, Azzam) (Gadahn, also known as, Azzam) ! (Adam Gadahn, per:alternate names, Azzam) SF13 ENG 012 The message also featured several appearances of Adam Gadahn, also known as
Azzam al-Amriki, an American who grew up in Riverside County and converted to Islam and joined al-Qaida.
(Adam Gadahn, also known as, Azzam Al-Amriki) ! (Adam Gadahn, per:alternate names, Azzam Al-Amriki) (Adam Gadahn, also, Azzam Al-Amriki) ! (Adam Gadahn, per:alternate names, Azzam Al-Amriki); (Adam Gadahn, ,, Azzam Al-Amriki) ! (Adam Gadahn, per:alternate names, Azzam Al-Amriki)
SF13 ENG 014 Mohammed Oudeh, a former math teacher who became the mastermind of the deadly attack on Israeli athletes at the 1972 Munich Olympics, died Friday in Damascus. (Mohammed Oudeh, ,, teacher) ! (Mohammed Oudeh, per:title, teacher) - SF13 ENG 014 He settled in the West Bank town of Ramallah, but in 1999, after a trip to Jordan, he was barred by Israel from returning. (He, in, The West Bank Town) ! (Mohammed Oudeh, per:cities of residence, The West Bank Town) (He, settled in, The West Bank Town) ! (Mohammed Oudeh, per:cities of residence, The West Bank Town) SF13 ENG 014 Mohammed Oudeh, a former math teacher who became the mastermind of the deadly attack on Israeli athletes at the 1972 Munich Olympics, died Friday in Damascus. (Mohammed Oudeh, ,, A Former Math Teacher) ! (Mohammed Oudeh, per:title, A Former Math Teacher) (Mohammed Oudeh, ,, teacher) ! (Mohammed Oudeh, per:title, teacher) SF13 ENG 014 In later years, as a graying member of the Palestinian old guard, Oudeh, most commonly known by his guerrilla name, Abu Daoud, showed no remorse for the botched hostage taking and killings of 11 members of the Israeli Olympic team that shook the world. (Oudeh, ,, Abu Daoud) ! (Mohammed Oudeh, per:alternate names, Abu Daoud) - SF13 ENG 015 The precise cause of death had not been determined , her sister , Elaine Philip , told The Associated Press on Sunday . - (Her Sister, ,, Elaine Philip) ! (Lucille Clifton, per:siblings, Elaine Philip) SF13 ENG 015 After her husband died in 1984, Clifton taught at the University of California at Santa Cruz before returning to Maryland in 1989. (Clifton, of, Santa Cruz) ! (Lucille Clifton, per:cities of residence, Santa Cruz) (Clifton, taught at, the University) ! (Lucille Clifton, per:employee or member of, the University of California) SF13 ENG 015 Lucille Clifton, a National Book Award-winning poet and Pulitzer finalist, has died. - (Lucille Clifton, ,, poet) ! (Lucille Clifton, per:title, poet) SF13 ENG 015 When she was a girl, Lucille Clifton sat on her mother’s lap and listened to her
recite poetry. - (Lucille Clifton, ,, A Girl) ! (Lucille Clifton, gender, A Girl)
SF13 ENG 015 In 1967, they moved to Baltimore, and Clifton worked in Washington for the old U.S. Office of Education. (Clifton, worked in, old U.S. Office) ! (Lucille Clifton, per:employee or member of, U.S. Office of Education) (They, to, Baltimore) ! (Lucille Clifton, per:cities of residence, Baltimore) SF13 ENG 015 Clifton’s sister, Elaine Philip, said the former poet laureate of Maryland passed away Saturday morning at Johns Hopkins University Hospital in Baltimore. - (Clifton’S Sister, ,, Elaine Philip) ! (Lucille Clifton, per:siblings, Elaine Philip) SF13 ENG 018 Harvey Pekar, the writer of comics whose autobiographical comic book series ”American Splendor” chronicled his life as a filing clerk, record collector, freelance jazz critic and one of life’s all-around misfits, was found dead July 12 at his home near Cleveland. - (Harvey Pekar, ,, writer) ! (Harvey Pekar, per:title, writer) SF13 ENG 018 Pekar attended what became Case Western Reserve University, served in the Navy in the late 1950s and worked a series of menial jobs before taking a 30-year jobas a filing clerk at a VA hospital in Cleveland. - (Pekar, what became, Case Western Reserve University) ! (Harvey Pekar, per:schools attended, Case Western Reserve University) SF13 ENG 025 Kissel, 46, of Adrian, Michigan, was convicted of drugging then bashing her husband Robert to death in a luxury Hong Kong apartment and sentenced to life in prison in September 2005. - (Kissel, ,, Adrian) ! (Nancy Kissel, per:cities of residence, Adrian); (Kissel, ,, Michigan) ! (Nancy Kissel, per:statesorprovinces of residence, Michigan) SF13 ENG 037 CLAUDE CHABROL , PIONEER FRENCH FILMMAKER , DIES AT 80 Joseph Berger contributed reporting from New York , and Maia de la Baume contributed from Paris . (Claude Chabrol, ,, Pioneer French Filmmaker) ! (Claude Chabrol, per:title, Pioneer French Filmmaker) - SF13 ENG 037 Chabrol ’ s survivors also include his third wife , Aurore Pajot , who acted as his script supervisor on nearly all of his movies from 1968 on and whom he married in 1981 ; and Pajot ’ s daughter , Cecile Maistre , who was an assistant director on his films and wrote the script with him for ” The Girl Cut in Two ” ( 2007 ) . (His Third Wife, ,, Aurore Pajot) ! (Claude Chabrol, per:spouse, Aurore Pajot) (Chabrol’s survivors, third wife, Aurore Pajot) ! (Claude Chabrol, per:spouse, Aurore Pajot) SF13 ENG 038 Mercier Philip Cunningham was born in Centralia, Wash., on April 16, 1919. - (Mercier Philip Cunningham, was born in, Centralia) ! (Merce Cunningham, was born in, Centralia); (Mercier Philip Cunningham, in, Wash.) ! (Merce Cunningham, per:stateorprovince of birth, Wash.); (Mercier Philip Cunningham, in, April) ! (Merce Cunningham, per:date of birth, 1919-04-16) SF13 ENG 038 Cunningham’s survivors include a brother, Jack Cunningham of Centralia. - (Cunningham’S Survivors, ,, Jack Cunningham) ! (Merce Cunningham, per:siblings, Jack Cunningham) SF13 ENG 041 By the time Hewitt was 6, his family was living in New Rochelle, N.Y. - (Hewitt, in, New Rochelle) ! (Don Hewitt, per:cities of residence, New Rochelle) SF13 ENG 041 He is survived by his third wife, former television news correspondent Marilyn
Berger; his sons, Steven and Jeffrey; his daughter, Lisa Cassara; his stepdaughter, Jilian Childers Hewitt, whom Hewitt adopted; and three grandchildren.
- (Hewitt, ,, Jilian Childers Hewitt) ! (Don Hewitt, per:children, Jilian Childers Hewitt)
SF13 ENG 044 Dominick Dunne , a novelist and journalist who chronicled true - crime tales of the rich and infamous , including O . J . Simpson and Claus von Bulow , and in turn became a celebrity in his own right , died of bladder cancer Aug . 26 at his home in New York City . (Dominick Dunne, ,, A Novelist) ! (Dominick Dunne, per:title, A Novelist) (Dunne, ,, novelist) ! (Dominick Dunne, per:title, novelist) SF13 ENG 044 Dunne and his wife , Ellen Griffin Dunne , known as Lenny , were married in 1954 . (His Wife, ,, Ellen Griffin Dunne) ! (Dominick Dunne, per:spouse, Ellen Griffin Dunne) (Dunne, and, Ellen Griffin Dunne) ! (Dominick Dunne, per:spouse, Ellen Griffin Dunne) SF13 ENG 044 Dunne was born in 1925 in Hartford, Connecticut, to a wealthy Roman Catholic family and grew up in some of the same social circles as the Kennedys. (Dunne, was born in 1925 in, Connecticut) ! (Dominick Dunne, per:stateorprovince of birth, Connecticut) (Dunne, was born in 1925 in, Connecticut) ! (Dominick Dunne, per:stateorprovince of birth, Connecticut) SF13 ENG 044 Dunne was part of a famous family that also included his brother, novelist and screenwriter John Gregory Dunne; his brother’s wife, author Joan Didion; and his son, Griffin. - (His Brother, ,, Novelist) ! (His brother, per:title, Novelist) SF13 ENG 047 The women were accompanied by Reyna Luisa Tamayo, the mother of political prisoner Orlando Zapata, who died at age 42 in a hunger strike February 23 to protest prison conditions. (Political Prisoner Orlando Zapata, ,, Reyna Luisa Tamayo) ! (Orlando Zapata, per:children, Reyna Luisa Tamayo) (Political Prisoner Orlando Zapata, ,, Reyna Luisa Tamayo) ! (Orlando Zapata, per:children, Reyna Luisa Tamayo) SF13 ENG 049 A professor emeritus at Yale University , Mandelbrot was born in Poland but as a child moved with his family to France where he was educated . (Mandelbrot, ,, Yale University) ! (Benoit Mandelbrot, per:employee or member of, Yale University) (Mandelbrot, was born in, Poland) ! (Benoit Mandelbrot, per:country of birth, Poland) SF13 ENG 058 Access Industries , a privately held company founded in 1986 by Len Blavatnik , has a diverse portfolio of investments in industry , real estate , media and telecommunications . (Access Industries, ,, A Privately Held Company) ! (Access Industries, properties, A Privately Held Company) (Access Industries, founded in 1986 by, Len Blavatnik) ! (Access Industries, org:founded by, Len Blavatnik) SF13 ENG 061 InterContinental Hotels Group, owners of Holiday Inn Worldwide, has been ordered by a court to pay $25 million in damages to a franchisee for fraud. (Intercontinental Hotels Group, ,, Holiday Inn Worldwide) ! (InterContinental Hotels Group, org:parents, Holiday Inn Worldwide) (Intercontinental Hotels Group, ,, Holiday Inn Worldwide) ! (InterContinental Hotels Group, org:parents, Holiday Inn Worldwide) SF13 ENG 079 Water and its links to development, peace and conflict were key words in the annual sessions, Anders Berntell, executive director of Stockholm International Water Institute (SIWI), said in his opening address. (Stockholm International, director, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell) - SF13 ENG 079 Water and its links to development, peace and conflict were key words in the annual sessions, Anders Berntell, executive director of Stockholm International Water Institute (SIWI), said in his opening address. (Stockholm International, , executive director, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell) (Stockholm International, , executive director, Anders Berntell) ! (Stockholm International Water Institute, per:top members or employees, Anders Berntell) SF13 ENG 090 ECO member countries include Afghanistan, Azerbaijan, Iran, Kazakhstan, Kyrgyzstan, Pakistan, Tajikistan, Turkey, Turkmenistan and Uzbekistan. (Eco Member Countries, include, Azerbaijan) ! (Economic Cooperation Organization, org:members, Azerbaijan) (Eco Member Countries, ,, Azerbaijan) ! (Economic Cooperation Organization, org:members, Azerbaijan) SF13 ENG 096 Freedom was founded in the 1930s by R.C. Hoiles and is still majority owned by the Hoiles family. (Freedom, was founded in, The 1930S)! (Freedom Communications, org:date founded, 1930-XX-XX) (Freedom, was founded in, The 1930S)! (Freedom Communications, org:date founded, 1930-XX-XX) SF13 ENG 096 A phone message left Sunday at Freedom’s Irvine, California, headquarters was not immediately returned. - (Freedom’S Irvine, ,, California)! (Freedom Communications, org:stateorprovince of headquarters, California) SF13 ENG 097 STX Finland is part of the international STX Europe Group, with shipyards in Brazil, Norway, France, Romania and Vietnam. (Stx Finland, of, The International Stx Europe Group)! (STX Finland, org:subsidaries, The International Stx Europe Group) (Stx Finland, of, The International Stx Europe Group)! (STX Finland, org:subsidaries, The International Stx Europe Group)
Figure 9: Unmapped facts: MAMA-BERTLARGE vs. MAMA-GPT-2XL on TAC KBP.
20
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to mapped facts
text: Appropriate Rural Technology Institute Appropriate Rural Technology Institute Appropriate Rural Technology Institute is an Indian non - governmental organization founded by a group of about 20 scientists and technologists to develop innovative and environmentally friendly rural technologies based on modern scientific knowledge in 1996 . (Appropriate Rural Technology Institute, is, An Indian Non-Governmental Organization) ! (appropriate rural technology institute.Q16245764, country.P17, india.Q668) Ayatollah (Record Producer) Ayatollah Lamont Dorrell , known as Ayatollah , is a hip - hop record producer from Queens , New York who has produced for predominantly New York - based rappers including Mos Def , Talib Kweli , R . A . The Rugged Man , Tragedy Khadafi , Wordsworth , Vast Aire , Afu - Ra , Guru , M . O . P . , Inspectah Deck , Cormega , Ghostface Killah as well as many others . (Ayatollah, is, A Hip-Hop Record Producer) ! (ayatollah (record producer).Q4830984, occupation.P106, record producer.Q183945) Benjamin Darbelet Benjamin Darbelet Benjamin Darbelet is a French judoka . (Benjamin Darbelet, is, A French Judoka) ! (benjamin darbelet.Q2456207, country of citizenship.P27, france.Q142); (Benjamin Darbelet, is, A French Judoka) ! (benjamin darbelet.Q2456207, sport.P641, judo.Q11420) Bharat Khawas He plays for club Tribhuvan Army in Martyr ’ s Memorial A - Division League as a striker . (He, plays, Club Tribhuvan Army) ! (bharat khawas.Q4901128, member of sports team.P54, nepal army club.Q6994327); (He, plays, Club Tribhuvan Army) ! (bharat khawas.Q4901128, sport.P641, association football.Q2736) Bill White (Shortstop) White was born on May 1 , 1860 in Bridgeport , Ohio , and he began his professional baseball career in 1883 with the Pottsville Antharcites of the International Association . (White, in, Bridgeport) ! (bill white (shortstop).Q4911357, place of birth.P19, bridgeport, ohio.Q2062947) Bob Dylan.Q392 Dylan married Sara Lownds , who had worked as a model and a secretary to Drew Associates , on November 22 , 1965 . (Dylan, married, Sara Lownds) ! (Bob Dylan.Q392, spouse.P26, Sara Lownds.Q457433) Bob Dylan.Q392 In 1991 , Dylan received a Grammy Lifetime Achievement Award from American actor Jack Nicholson . (Dylan, received, Grammy Lifetime Achievement Award) ! (Bob Dylan.Q392, award received.P166, Grammy Lifetime Achievement Award.Q935843) Boog Powell He also played for the Cleveland Indians and the Los Angeles Dodgers . (He, also played for the, Cleveland Indians) ! (boog powell.Q4942810, member of sports team.P54, cleveland indians.Q642553) Boog Powell He played in Major League Baseball as a first baseman and left fielder from through , most notably as a member of the Baltimore Orioles dynasty that won four American League pennants and two World Series championships between 1966 and 1971 . (He, played, A First Baseman) ! (boog powell.Q4942810, position played on team / speciality.P413, first baseman.Q1326154) Bouchraya Hammoudi Bayoun He studied Economics at the University of Havana , Cuba . (He, studied Economics, Havana) ! (bouchraya hammoudi bayoun.Q4949324, educated at.P69, university of havana.Q837320) Cezaro Rossetti Of Italian - Swiss derivation , he was born in Glasgow and lived in Britain . (He, was, Glasgow) ! (cezaro rossetti.Q2947206, place of birth.P19, glasgow.Q4093) Chan Sau Ying Chan Sau Ying Chan Sau Ying is a retired athlete from Hong Kong who specialised in the 100 metres hurdles . (Chan Sau Ying, is, A Retired Athlete) ! (chan sau ying.Q21858345, sport.P641, sport of athletics.Q542) Christian Mcgrath He was born in the Bronx in 1972 and became interested in art and illustration as a small child . (He, was born in, The Bronx) ! (christian mcgrath.Q5109897, place of birth.P19, the bronx.Q18426) Cliff Michelmore He was a regular presenter on BBC1 ’ s ” Holiday ” programme from 1969 to 1986 , and presented other shows for
BBC TV , ITV and BBC Radio . (He, was, A Regular Presenter) ! (cliff michelmore.Q5132704, occupation.P106, television presenter.Q947873)
Crooked I Crooked I Dominick Antron Wickliffe , better known by his stage name Crooked I or Kxng Crooked , is an American rapper from Long Beach , California . (Kxng Crooked, ,, An American Rapper) !(crooked i.Q253695, country of citizenship.P27, united states.Q30) Denise Johns She graduated in 2002 , and moved to Atlanta , playing amateur beach volleyball on the east coast of America . (She, , playing, Amateur Beach Volleyball) ! (denise johns.Q5257690, sport.P641, beach volleyball.Q4543) Diana Rast She also competed at the 1996 Summer Olympics . (She, competed at, The 1996 Summer Olympics) ! (diana rast.Q2491787, participant of.P1344, 1996 summer olympics.Q8531) Eduard Friedrich Poeppig Eduard Friedrich Poeppig Eduard Friedrich Poeppig was a German botanist , zoologist and explorer . (Eduard Friedrich, was, A German Botanist) ! (eduard friedrich poeppig.Q61501, field of work.P101, botany.Q441) Edward L. Romero After the Vietnam war , he became a county chairman of Democratic Party . (A County Chairman, of, Democratic Party) ! (edward l. romero.Q5344025, member of political party.P102, democratic party (united states).Q29552) Endre Tilli Endre Tilli Endre Tilli was a Hungarian fencer . (Endre Tilli, was, A Hungarian Fencer) ! (endre tilli.Q1642140, country of citizenship.P27, hungary.Q28); (Endre Tilli, was, A Hungarian Fencer) ! (endre tilli.Q1642140, sport.P641, fencing.Q12100) Enrico Porro Porro competed at the 1908 Summer Olympics in London where he won the gold medal in Greco - Roman wrestling ,
the lightweight class . (Porro, competed, The 1908 Summer Olympics) ! (enrico porro.Q1343905, participant of.P1344, 1908 summer olympics.Q8111)
Erfan Zeneli Erfan Zeneli Erfan Zeneli is a Finnish footballer of Kosovar Albanian descent who currently plays for HJK Helsinki as a striker or left winger . (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, country of citizenship.P27, finland.Q33); (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, sport.P641, association football.Q2736); (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, country for sport.P1532, finland.Q33) Erminio Favalli He then moved to Mantova before to join Palermo , where he spent seven seasons as a ” rosanero ” mainstay . (He, then moved to, Mantova) ! (erminio favalli.Q1405871, member of sports team.P54, mantova 1911 s.s.d..Q430993) George Dowty Dowty was born in Pershore , Worcestershire in 1901 . (Dowty, was born in, Pershore) ! (george dowty.Q1507239, place of birth.P19, pershore.Q767858) Harald Fairhair In the Saga of Harald Fairhair in ” Heimskringla ” , which is the most elaborate although not the oldest or most
reliable source to the life of Harald , it is written that Harald succeeded , on the death of his father Halfdan the Black Gudrödarson , to the sovereignty of several small , and somewhat scattered kingdoms in Vestfold , which had come into his father ’ s hands through conquest and inheritance .
(Harald Fairhair, of his father, Halfdan the Black) ! (harald fairhair.Q206123, father.P22, halfdan the black.Q504932)
Hector Dyer Born in Los Angeles , California , Hector Dyer enrolled at the Stanford University and won the IC4A championships in in 1930 . (Hector Dyer, enrolled at, The Stanford University) ! (hector dyer.Q1364518, educated at.P69, stanford university.Q41506) Henny Magnussen Thanks to the support she received , she was able to practise in all the courts of Denmark from 1909 , becoming the first woman to do so . (The First Woman, ,, Denmark) ! (henny magnussen.Q56577333, country of citizenship.P27, denmark.Q35) Hina Pervaiz Butt Hina Pervaiz Butt Hina Pervaiz Butt is a Pakistani politician who was a Member of the Provincial Assembly of the Punjab , since May 2013 . (Hina Pervaiz Butt, is, A Pakistani Politician) ! (hina pervaiz butt.Q47918210, occupation.P106, politician.Q82955) Isabella Of Valois, Duchess Of Bourbon She was the wife of Peter I , Duke of Bourbon . (She, was the wife of, Peter I) ! (isabella of valois, duchess of bourbon.Q2363495, spouse.P26, peter i, duke of bourbon.Q444668) Jahir Butrón Then from 1995 , he played for Peruvian Second Division side Guardia Republicana . (He, played for, Peruvian Second Division Side Guardia Republicana) ! (jahir butrón.Q6123259, member of sports team.P54, guardia republicana.Q5614012) Jahir Butrón Jahir Butrón Jahir Butrón Gotuzzo is a Peruvian footballer who plays as a center back . (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) ! (jahir butrón.Q6123259, country of citizenship.P27, peru.Q419); (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) ! (jahir butrón.Q6123259, sport.P641, association football.Q2736) Joan Of France, Duchess Of Berry She was the second daughter of King Louis XI of France and of his second wife Charlotte of Savoy ; her surviving siblings were King Charles VIII of France and Anne of France . (She, was the second daughter, King Louis Xi) ! (joan of france, duchess of berry.Q236220, father.P22, louis xi of france.Q8058) John Ogilvie (Footballer) He played for Hibs in their most successful era renowned for their front five known as The Famous Five . (He, played for, Hibs) ! (john ogilvie (footballer).Q6251147, member of sports team.P54, hibernian f.c..Q192597) Johnedel Cardel He moved to Sta . (He, to, Sta) ! (johnedel cardel.Q56599981, member of sports team.P54, sta. lucia realtors.Q3547751) Joseph W. Nega Joseph W . Nega Joseph W . Nega is a Judge of the United States Tax Court . (Joseph W. Nega, is, A Judge) ! (joseph w. nega.Q16733242, occupation.P106, judge.Q16533) José Alejandro Aguilar López José Alejandro Aguilar López José Alejandro Aguilar López is a Mexican politician from the National Action Party . (José Alejandro Aguilar López, is, A Mexican Politician) ! (josé alejandro aguilar lópez.Q18385891,
country of citizenship.P27, mexico.Q96); (José Alejandro Aguilar López, is, A Mexican Politician) ! (josé alejandro aguilar lópez.Q18385891, occupation.P106, politician.Q82955); (José Alejandro Aguilar López, Mexican politician, The National Action Party) ! (josé alejandro aguilar lópez.Q18385891, member of political party.P102, national action party (mexico).Q851087)
Juan Bautista Azopardo Juan Bautista Azopardo was born in Senglea , Malta , the son of Rosina and Salvatore Azopardo . (Juan Bautista Azopardo, ,, Malta) ! (juan bautista azopardo.Q3805864, country of citizenship.P27, malta.Q233); (Juan Bautista Azopardo, was born, Senglea) ! (juan bautista azopardo.Q3805864, place of birth.P19, senglea.Q846593) Kevin Kerr (Scottish Footballer) He signed for AGOVV Apeldoorn of Netherlands ’ Eerste Divisie in January 2012 after his contract in Bielefeld had expired the previous summer . (He, signed for, Agovv Apeldoorn) ! (kevin kerr (scottish footballer).Q326761, member of sports team.P54, agovv apeldoorn.Q292618) Lachezar Kotev Lachezar Kotev Lachezar Kotev is a Bulgarian footballer who plays as a midfielder for Vitosha Bistritsa . (Lachezar Kotev, is, A Bulgarian Footballer) ! (lachezar kotev.Q49560773, sport.P641, association football.Q2736); (Lachezar Kotev, is, A Midfielder) ! (lachezar kotev.Q49560773, position played on team / speciality.P413, midfielder.Q193592) Marc Zeno In 1988 , Zeno was drafted in the 7th round by the Pittsburgh Steelers . (Zeno, was drafted in the, The Pittsburgh Steelers) ! (marc zeno.Q16196353, member of sports team.P54, pittsburgh steelers.Q191477) Marcel Dheere Marcel Dheere Marcel Albert ” Ching ” Dheere was a Canadian professional ice hockey forward who played 11 games in the National Hockey League for the Montreal Canadiens .
(Marcel Albert ”Ching, in, A Canadian Professional Ice Hockey) ! (marcel dheere.Q6756083, country of citizenship.P27, canada.Q16); (The National Hockey League, for, The Montreal Canadiens) ! (marcel dheere.Q6756083, member of sports team.P54, montreal canadiens.Q188143); (A Canadian Professional Ice Hockey, played, The National Hockey League) ! (marcel dheere.Q6756083, league.P118, national hockey league.Q1215892); (Marcel Albert ”Ching, played, A Canadian Professional Ice Hockey) ! (marcel dheere.Q6756083, sport.P641, ice hockey.Q41466)
Mark Schultz (Wrestler) Mark Schultz was born in 1960 in Palo Alto , California to Dorothy Jean St . Germain and Philip Gary Schultz . (Mark Schultz, was born in 1960 in, Dorothy Jean St. Germain) ! (mark schultz (wrestler).Q3487332, place of birth.P19, palo alto, california.Q47265) Michael Jones (Footballer) Michael Jones Michael Jones is an English footballer , who plays for Cefn Druids . (Michael Jones, is, An English Footballer) ! (michael jones (footballer).Q6831707, sport.P641, association football.Q2736) Mike Bellotti He is an alumnus of the Delta Sigma Phi fraternity . (He, is an alumnus of, The Delta Sigma Phi Fraternity) ! (mike bellotti.Q14950800, educated at.P69, california state university, east bay.Q1026916) Olga Semenova Tyan-Shanskaya She learned to play chess at the age of 18 ; four years later she won the Leningrad city women ’ s chess championship . (She, to play, Chess) ! (olga semenova tyan-shanskaya.Q2392240, sport.P641, chess.Q718) Owen Paterson Owen Paterson Owen William Paterson is a British Conservative Party politician who was the Secretary of State for
Environment , Food and Rural Affairs from 2012 to 2014 . (Owen Paterson, is, A British Conservative Party Politician) ! (owen paterson.Q197894, occupation.P106, politician.Q82955)
Pan Gongsheng Pan Gongsheng Pan Gongsheng is a Chinese economist , banker , reformist and bureaucrat . (Pan Gongsheng, is, A Chinese Economist) ! (pan gongsheng.Q9309080, country of citizenship.P27, china.Q148) Paul Carrington (American Football) Paul Carrington Paul Carrington is a former American football defensive end . (Paul Carrington, is, A Former American Football Defensive End) ! (paul carrington (american football).Q9057011, sport.P641, american football.Q41323) Peter Jebsen Peter Jebsen Peter Jebsen was a Norwegian businessperson and politician . (Peter Jebsen, was, A Norwegian Businessperson) ! (peter jebsen.Q1774966, country of citizenship.P27, nor-
way.Q20); (Peter Jebsen, was, A Norwegian Businessperson) ! (peter jebsen.Q1774966, occupation.P106, businessperson.Q43845); (Peter Jebsen, was, Politician) ! (peter jebsen.Q1774966, occupation.P106, politician.Q82955)
Pina Piovani Pina Piovani Pina Piovani was an Italian stage and film actress . (Pina Piovani, was, An Italian Stage And Film Actress) ! (pina piovani.Q3388583, country of citizenship.P27, italy.Q38) Robert G. Cole Rapidly advancing through the ranks at Fort Benning as the parachute infantry battalions were expanded to regiments , he was a lieutenant colonel commanding the 3rd Battalion of the 502nd Parachute Infantry Regiment on June 6 , 1944 , the date of his unit ’ s first combat jump . (He, was, A Lieutenant Colonel) ! (robert g. cole.Q969944, military rank.P410, lieutenant colonel.Q493898) Robert Holt Carpenter Robert Holt Carpenter Robert Holt Carpenter was a New Zealand bookbinder , local politician , bookseller and character . (Robert Holt Carpenter, was, A New Zealand Bookbinder) ! (robert holt carpenter.Q7345593, country of citizenship.P27, new zealand.Q664) Robert Van Boxel He joined amateur side FC Lisse in summer 2015 from Sparta . (He, joined, Amateur Side Fc Lisse) ! (robert van boxel.Q2017870, member of sports team.P54, fc lisse.Q978325) Rolf Maurer Rolf Maurer Rolf Maurer was a Swiss road racing cyclist who competed professionally between 1960 and 1969 . (Rolf Maurer, was, A Swiss Road Racing Cyclist) ! (rolf maurer.Q870132, country of citizenship.P27,
switzerland.Q39); (Rolf Maurer, was, A Swiss Road Racing Cyclist) ! (rolf maurer.Q870132, sport.P641, road bicycle racing.Q3609)
Ron Maudsley Maudsley was born in Lostock Gralam , Cheshire , and educated at Malvern College and Birmingham University , then served in the Royal Army Service Corps during World War II . (Maudsley, was born in, Lostock Gralam) ! (ron maudsley.Q16009653, place of birth.P19, lostock gralam.Q4640202) Safa Al-Safi Safa al - Safi Safa al - Din Mohammed al - Safi is an Iraqi politician and former Justice Minister who is currently Minister of State for the Council of Representatives . (Mohammed Al-Safi, is, An Iraqi Politician) ! (safa al-safi.Q7398343, occupation.P106, politician.Q82955) Salvador Luria In 1959 , he became chair of Microbiology at the Massachusetts Institute of Technology . (He, became chair of, Microbiology) ! (salvador luria.Q205667, field of work.P101, microbiology.Q7193) Sanele Vavae Tuilagi He now plays for US Carcassonne . (He, now plays, Us Carcassonne) ! (sanele vavae tuilagi.Q3472139, member of sports team.P54, us carcassonne.Q3550526) Seth Plum Seth Plum Seth Lewis Plum was an English international footballer who played as a wing half . (Seth Lewis Plum, was, An English International Footballer) ! (seth plum.Q7456581, sport.P641, associa-
tion football.Q2736); (Seth Lewis Plum, was, An English International Footballer) ! (seth plum.Q7456581, member of sports team.P54, england national football team.Q47762)
Sofija Korkutytė Sofija Korkutytė Sofija Korkutytė was a Lithuanian rower who won three European titles in the eights event in 1963 , 1965 and 1967 ; she finished second in 1964 and 1966 . (Sofija Korkutytė, was, A Lithuanian Rower) ! (sofija korkutytė.Q16014909, sport.P641, rowing (sport).Q159354) Thomas Rogne He made his first start for Celtic at home against Dundee United in a 1 – 0 victory on 20 February 2010 , he played very well and impressed the coach Tony Mowbray . (He, at, Celtic) ! (thomas rogne.Q609801, member of sports team.P54, celtic f.c..Q19593) Uri Avnery Avnery was born in Beckum , near Münster in Westphalia , as Helmut Ostermann , the youngest of four children , to a well - established German Jewish family , his father being a private banker in the town . (Westphalia, in, Beckum) ! (uri avnery.Q325679, place of birth.P19, beckum, germany.Q2707) Ursula Lehr She was a member of the Christian Democratic Union . (She, of, The Christian Democratic Union) ! (ursula lehr.Q290708, member of political party.P102, christian democratic union of germany.Q49762) Valdı́via (Brazilian Footballer) Valdı́via Wanderson Ferreira de Oliveira , commonly known as Valdı́via , is a Brazilian footballer who plays as an attacking midfielder for Avaı́ , on loan from Internacional . (A Brazilian Footballer, who plays as, An Attacking Midfielder) ! (valdı́via (brazilian footballer).Q18176477, position played on team / speciality.P413, midfielder.Q193592) Victor-François, 2Nd Duc De Broglie His victory over Prince Ferdinand at Bergen won him the rank of marshal of France from the French King Louis XV and the title of Prince of the Holy Roman Empire ” Reichsfürst ” from Holy Roman Emperor Francis I . In 1759 , he won the Battle of Bergen and followed that with the capture the city of Minden , later fighting at the Battle of Minden under the command of Contades , whom he would succeed in command . (His victory, won him the, Marshal) ! (victor-françois, 2nd duc de broglie.Q695929, award received.P166, marshals of france.Q474485)
Figure 10: Mapped facts: MAMA-BERTLARGE on Wikidata.
21
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to mapped facts
text: Appropriate Rural Technology Institute Appropriate Rural Technology Institute Appropriate Rural Technology Institute is an Indian non-governmental organization founded by a group of about 20 scientists and technologists to develop innovative and environmentally friendly rural technologies based on modern scientific knowledge in 1996. (Appropriate Rural Technology Institute, is, An Indian Non-Governmental Organization) ! (appropriate rural technology institute.Q16245764, country.P17, india.Q668) Ayatollah (Record Producer) Ayatollah Lamont Dorrell, known as Ayatollah, is a hip-hop record producer from Queens, New York who has produced for predominantly New York-based rappers including Mos Def, Talib Kweli, R.A. The Rugged Man, Tragedy Khadafi, Wordsworth, Vast Aire, Afu-Ra, Guru, M.O.P., Inspectah Deck, Cormega, Ghostface Killah as well as many others. (Ayatollah, is, A Hip-Hop Record Producer) ! (ayatollah (record producer).Q4830984, occupation.P106, record producer.Q183945) Benjamin Darbelet Benjamin Darbelet Benjamin Darbelet is a French judoka. (Benjamin Darbelet, is, A French Judoka) ! (benjamin darbelet.Q2456207, country of citizenship.P27, france.Q142); (Benjamin Darbelet, is, A French Judoka) ! (benjamin darbelet.Q2456207, sport.P641, judo.Q11420) Bharat Khawas He plays for club Tribhuvan Army in Martyr’s Memorial A-Division League as a striker. (He, plays for, Club Tribhuvan Army) ! (bharat khawas.Q4901128, member of sports team.P54, nepal army club.Q6994327); (He, plays for, Club Tribhuvan Army) ! (bharat khawas.Q4901128, sport.P641, association football.Q2736) Bill White (Shortstop) White was born on May 1, 1860 in Bridgeport, Ohio, and he began his professional baseball career in 1883 with the Pottsville Antharcites of the International Association. (White, in, Bridgeport) ! (bill white (shortstop).Q4911357, place of birth.P19, bridgeport, ohio.Q2062947) Bob Dylan.Q392 Dylan married Sara Lownds , who had worked as a model and a secretary to Drew Associates , on November 22 , 1965 . (Dylan, married, Sara Lownds) ! (Bob Dylan.Q392, spouse.P26, Sara Lownds.Q457433) Bob Dylan.Q392 In 1991 , Dylan received a Grammy Lifetime Achievement Award from American actor Jack Nicholson . (Dylan, received, A Grammy Lifetime Achievement Award) ! (Bob Dylan.Q392, award received.P166, Grammy Lifetime Achievement Award.Q935843) Boog Powell He also played for the Cleveland Indians and the Los Angeles Dodgers. (He, also played for, The Cleveland Indians) ! (boog powell.Q4942810, member of sports team.P54, cleveland indians.Q642553) Boog Powell He played in Major League Baseball as a first baseman and left fielder from through , most notably as a member of the Baltimore Orioles dynasty that won four American League pennants and two World Series championships between 1966 and 1971. (He, played in, A First Baseman) ! (boog powell.Q4942810, position played on team / speciality.P413, first baseman.Q1326154) Bouchraya Hammoudi Bayoun He studied Economics at the University of Havana, Cuba. (He, studied, University of Havana) ! (bouchraya hammoudi bayoun.Q4949324, educated at.P69, university of havana.Q837320) Cezaro Rossetti Of Italian-Swiss derivation, he was born in Glasgow and lived in Britain. (He, was born in, Glasgow) ! (cezaro rossetti.Q2947206, place of birth.P19, glasgow.Q4093) Chan Sau Ying Chan Sau Ying Chan Sau Ying is a retired athlete from Hong Kong who specialised in the 100 metres hurdles. (Sau Ying, is, A Retired Athlete) ! (chan sau ying.Q21858345, sport.P641, sport of athletics.Q542) Christian Mcgrath He was born in the Bronx in 1972 and became interested in art and illustration as a small child. (He, was born in, The Bronx) ! (christian mcgrath.Q5109897, place of birth.P19, the bronx.Q18426) Cliff Michelmore He was a regular presenter on BBC1’s ”Holiday” programme from 1969 to 1986, and presented other shows for BBC
TV, ITV and BBC Radio. (He, was, A Regular Presenter) ! (cliff michelmore.Q5132704, occupation.P106, television presenter.Q947873)
Crooked I Crooked I Dominick Antron Wickliffe , better known by his stage name Crooked I or Kxng Crooked, is an American rapper from Long Beach, California. (Crooked I, is, An American Rapper) ! (crooked i.Q253695, country of citizenship.P27, united states.Q30) Denise Johns She graduated in 2002, and moved to Atlanta, playing amateur beach volleyball on the east coast of America. (She, playing, Amateur Beach Volleyball) ! (denise johns.Q5257690, sport.P641, beach volleyball.Q4543) Diana M. Fennell Fennell was born in Emporia, Virginia and attended the public schools there. (Fennell, was born in, Emporia) ! (diana m. fennell.Q28037279, place of birth.P19, emporia, virginia.Q177678) Diana Rast She also competed at the 1996 Summer Olympics. (She, also competed at, The 1996 Summer Olympics) ! (diana rast.Q2491787, participant of.P1344, 1996 summer olympics.Q8531) Eduard Friedrich Poeppig Eduard Friedrich Poeppig Eduard Friedrich Poeppig was a German botanist, zoologist and explorer. (Eduard Friedrich, was, A German Botanist) ! (eduard friedrich poeppig.Q61501, field of work.P101, botany.Q441) Edward L. Romero After the Vietnam war, he became a county chairman of Democratic Party. (He, of, Democratic Party) ! (edward l. romero.Q5344025, member of political party.P102, democratic party (united states).Q29552) Endre Tilli Endre Tilli Endre Tilli was a Hungarian fencer. (Endre Tilli, was, A Hungarian Fencer) ! (endre tilli.Q1642140, country of citizenship.P27, hungary.Q28); (Endre Tilli, was, A Hungarian Fencer) ! (endre tilli.Q1642140, sport.P641, fencing.Q12100) Enrico Porro Porro competed at the 1908 Summer Olympics in London where he won the gold medal in Greco-Roman wrestling,
the lightweight class. (Porro, competed at, The 1908 Summer Olympics) ! (enrico porro.Q1343905, participant of.P1344, 1908 summer olympics.Q8111)
Erfan Zeneli Erfan Zeneli Erfan Zeneli is a Finnish footballer of Kosovar Albanian descent who currently plays for HJK Helsinki as a striker or left winger. (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, country of citizenship.P27, finland.Q33); (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, sport.P641, association football.Q2736); (Erfan Zeneli, is, A Finnish Footballer) ! (erfan zeneli.Q2053367, country for sport.P1532, finland.Q33) Erminio Favalli He then moved to Mantova before to join Palermo, where he spent seven seasons as a ”rosanero” mainstay. (He, then moved to, Mantova) ! (erminio favalli.Q1405871, member of sports team.P54, mantova 1911 s.s.d..Q430993) George Dowty Dowty was born in Pershore, Worcestershire in 1901. (Dowty, was born in, Pershore) ! (george dowty.Q1507239, place of birth.P19, pershore.Q767858) Harald Fairhair In the Saga of Harald Fairhair in ”Heimskringla”, which is the most elaborate although not the oldest or most reliable
source to the life of Harald, it is written that Harald succeeded, on the death of his father Halfdan the Black Gudrödarson, to the sovereignty of several small, and somewhat scattered kingdoms in Vestfold, which had come into his father’s hands through conquest and inheritance.
(Harald, of his father, Halfdan the Black) ! (harald fairhair.Q206123, father.P22, halfdan the black.Q504932)
Hector Dyer Born in Los Angeles, California, Hector Dyer enrolled at the Stanford University and won the IC4A championships in in 1930. (Hector Dyer, at, The Stanford University) ! (hector dyer.Q1364518, educated at.P69, stanford university.Q41506) Henny Magnussen Thanks to the support she received, she was able to practise in all the courts of Denmark from 1909, becoming the first woman to do so. (She, , she, Denmark) ! (henny magnussen.Q56577333, country of citizenship.P27, denmark.Q35) Hina Pervaiz Butt Hina Pervaiz Butt Hina Pervaiz Butt is a Pakistani politician who was a Member of the Provincial Assembly of the Punjab, since May 2013. (Hina Pervaiz Butt, is, A Pakistani Politician) ! (hina pervaiz butt.Q47918210, occupation.P106, politician.Q82955) Isabella Of Valois, Duchess Of Bourbon She was the wife of Peter I, Duke of Bourbon. (She, was the wife of, Peter I) ! (isabella of valois, duchess of bourbon.Q2363495, spouse.P26, peter i, duke of bourbon.Q444668) Jahir Butrón Then from 1995, he played for Peruvian Second Division side Guardia Republicana. (He, played for, Peruvian Second Division Side Guardia Republicana) ! (jahir butrón.Q6123259, member of sports team.P54, guardia republicana.Q5614012) Jahir Butrón Jahir Butrón Jahir Butrón Gotuzzo is a Peruvian footballer who plays as a center back. (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) ! (jahir butrón.Q6123259, country of citizenship.P27, peru.Q419); (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) ! (jahir butrón.Q6123259, sport.P641, association football.Q2736) Janina Altman In 1950, Hescheles emigrated to Israel, where she eventually earned a doctorate in chemistry at Technion . (She, earned a doctorate in, Technion) ! (janina altman.Q22574794, educated at.P69, technion – israel institute of technology.Q333705) Joan Of France, Duchess Of Berry She was the second daughter of King Louis XI of France and of his second wife Charlotte of Savoy; her surviving siblings were King Charles VIII of France and Anne of France. (She, was the second daughter of, King Louis Xi) ! (joan of france, duchess of berry.Q236220, father.P22, louis xi of france.Q8058) John Ogilvie (Footballer) He played for Hibs in their most successful era renowned for their front five known as The Famous Five. (He, played for, Hibs) ! (john ogilvie (footballer).Q6251147, member of sports team.P54, hibernian f.c..Q192597) Johnedel Cardel He moved to Sta. (He, moved to, Sta) ! (johnedel cardel.Q56599981, member of sports team.P54, sta. lucia realtors.Q3547751) Joseph W. Nega Joseph W. Nega Joseph W. Nega is a Judge of the United States Tax Court. (Joseph W. Nega, is, A Judge) ! (joseph w. nega.Q16733242, occupation.P106, judge.Q16533) José Alejandro Aguilar López José Alejandro Aguilar López José Alejandro Aguilar López is a Mexican politician from the National Action Party. (José Alejandro Aguilar López, is a Mexican politician, The National Action Party) !
(josé alejandro aguilar lópez.Q18385891, member of political party.P102, national action party (mexico).Q851087); (José Alejandro Aguilar López, is, A Mexican Politician) ! (josé alejandro aguilar lópez.Q18385891, country of citizenship.P27, mexico.Q96); (José Alejandro Aguilar López, is, A Mexican Politician) ! (josé alejandro aguilar lópez.Q18385891, occupation.P106, politician.Q82955)
Juan Bautista Azopardo Juan Bautista Azopardo was born in Senglea, Malta, the son of Rosina and Salvatore Azopardo. (Juan Bautista Azopardo, was born in, Senglea) ! (juan bautista azopardo.Q3805864, place of birth.P19, senglea.Q846593); (Juan Bautista Azopardo, was born in, Malta) ! (juan bautista azopardo.Q3805864, country of citizenship.P27, malta.Q233) Kevin Kerr (Scottish Footballer) He signed for AGOVV Apeldoorn of Netherlands’ Eerste Divisie in January 2012 after his contract in Bielefeld had expired the previous summer. (He, signed for, Agovv Apeldoorn) ! (kevin kerr (scottish footballer).Q326761, member of sports team.P54, agovv apeldoorn.Q292618) Lachezar Kotev Lachezar Kotev Lachezar Kotev is a Bulgarian footballer who plays as a midfielder for Vitosha Bistritsa. (Lachezar Kotev, is, A Bulgarian Footballer) ! (lachezar kotev.Q49560773, sport.P641, association football.Q2736) Marc Zeno In 1988, Zeno was drafted in the 7th round by the Pittsburgh Steelers. (Zeno, by, The Pittsburgh Steelers) ! (marc zeno.Q16196353, member of sports team.P54, pittsburgh steelers.Q191477) Marcel Dheere Marcel Dheere Marcel Albert ”Ching” Dheere was a Canadian professional ice hockey forward who played 11 games
in the National Hockey League for the Montreal Canadiens. (Marcel Albert ”Ching, was, A Canadian Professional Ice Hockey) ! (marcel dheere.Q6756083, country of citizenship.P27, canada.Q16); (Marcel Albert ”Ching, was, A Canadian Professional Ice Hockey) ! (marcel dheere.Q6756083, sport.P641, ice hockey.Q41466); (Marcel Albert ”Ching, for, The Montreal Canadiens) ! (marcel dheere.Q6756083, member of sports team.P54, montreal canadiens.Q188143)
Mark Schultz (Wrestler) Mark Schultz was born in 1960 in Palo Alto, California to Dorothy Jean St. Germain and Philip Gary Schultz. (Mark Schultz, was born in 1960 in, Palo Alto) ! (mark schultz (wrestler).Q3487332, place of birth.P19, palo alto, california.Q47265) Michael Jones (Footballer) Michael Jones Michael Jones is an English footballer, who plays for Cefn Druids. (Michael Jones, is, An English Footballer) ! (michael jones (footballer).Q6831707, sport.P641, association football.Q2736) Olga Semenova Tyan-Shanskaya She learned to play chess at the age of 18; four years later she won the Leningrad city women’s chess championship. (She, learned, Chess) ! (olga semenova tyan-shanskaya.Q2392240, sport.P641, chess.Q718) Owen Paterson Owen Paterson Owen William Paterson is a British Conservative Party politician who was the Secretary of State for
Environment, Food and Rural Affairs from 2012 to 2014. (Owen William Paterson, is, A British Conservative Party Politician) ! (owen paterson.Q197894, occupation.P106, politician.Q82955)
Pan Gongsheng Pan Gongsheng Pan Gongsheng is a Chinese economist, banker, reformist and bureaucrat. (Pan Gongsheng, is, A Chinese Economist) ! (pan gongsheng.Q9309080, country of citizenship.P27, china.Q148) Paul Carrington (American Football) Paul Carrington Paul Carrington is a former American football defensive end. (Paul Carrington, is, A Former American Football Defensive End) ! (paul carrington (american football).Q9057011, sport.P641, american football.Q41323) Peter Jebsen Peter Jebsen Peter Jebsen was a Norwegian businessperson and politician. (Peter Jebsen, was, A Norwegian Businessperson) ! (peter jebsen.Q1774966, country of citizenship.P27, nor-
way.Q20); (Peter Jebsen, was, A Norwegian Businessperson) ! (peter jebsen.Q1774966, occupation.P106, businessperson.Q43845)
Pina Piovani Pina Piovani Pina Piovani was an Italian stage and film actress. (Pina Piovani, was, An Italian Stage And Film Actress) ! (pina piovani.Q3388583, country of citizenship.P27, italy.Q38) Robert G. Cole Rapidly advancing through the ranks at Fort Benning as the parachute infantry battalions were expanded to regiments, he was a lieutenant colonel commanding the 3rd Battalion of the 502nd Parachute Infantry Regiment on June 6, 1944, the date of his unit’s first combat jump. (He, was, A Lieutenant Colonel) ! (robert g. cole.Q969944, military rank.P410, lieutenant colonel.Q493898) Robert Holt Carpenter Robert Holt Carpenter Robert Holt Carpenter was a New Zealand bookbinder, local politician, bookseller and character. (Robert Holt Carpenter, was, A New Zealand Bookbinder) ! (robert holt carpenter.Q7345593, country of citizenship.P27, new zealand.Q664) Robert Van Boxel He joined amateur side FC Lisse in summer 2015 from Sparta. (He, joined, Amateur Side Fc Lisse) ! (robert van boxel.Q2017870, member of sports team.P54, fc lisse.Q978325) Rolf Maurer Rolf Maurer Rolf Maurer was a Swiss road racing cyclist who competed professionally between 1960 and 1969. (Rolf Maurer, was, A Swiss Road Racing Cyclist) ! (rolf maurer.Q870132, country of citizenship.P27,
switzerland.Q39); (Rolf Maurer, was, A Swiss Road Racing Cyclist) ! (rolf maurer.Q870132, sport.P641, road bicycle racing.Q3609)
Ron Maudsley Maudsley was born in Lostock Gralam, Cheshire, and educated at Malvern College and Birmingham University, then served in the Royal Army Service Corps during World War II. (Maudsley, was born in, Lostock Gralam) ! (ron maudsley.Q16009653, place of birth.P19, lostock gralam.Q4640202) Safa Al-Safi Safa al-Safi Safa al-Din Mohammed al-Safi is an Iraqi politician and former Justice Minister who is currently Minister of State for the Council of Representatives. (Mohammed Al-Safi, is, An Iraqi Politician) ! (safa al-safi.Q7398343, occupation.P106, politician.Q82955) Salvador Luria In 1959, he became chair of Microbiology at the Massachusetts Institute of Technology . (He, became chair of, Microbiology) ! (salvador luria.Q205667, field of work.P101, microbiology.Q7193) Sanele Vavae Tuilagi He now plays for US Carcassonne. (He, now plays for, Us Carcassonne) ! (sanele vavae tuilagi.Q3472139, member of sports team.P54, us carcassonne.Q3550526) Seth Plum Seth Plum Seth Lewis Plum was an English international footballer who played as a wing half. (Seth Lewis Plum, was, An English International Footballer) ! (seth plum.Q7456581, sport.P641, associa-
tion football.Q2736); (Seth Lewis Plum, was, An English International Footballer) ! (seth plum.Q7456581, member of sports team.P54, england national football team.Q47762)
Sofija Korkutytė Sofija Korkutytė Sofija Korkutytė was a Lithuanian rower who won three European titles in the eights event in 1963, 1965 and 1967; she finished second in 1964 and 1966. (Sofija Korkutytė, was, A Lithuanian Rower) ! (sofija korkutytė.Q16014909, sport.P641, rowing (sport).Q159354) Thomas Rogne He made his first start for Celtic at home against Dundee United in a 1–0 victory on 20 February 2010, he played very well and impressed the coach Tony Mowbray. (He, made his first start for, Celtic) ! (thomas rogne.Q609801, member of sports team.P54, celtic f.c..Q19593) Uri Avnery Avnery was born in Beckum, near Münster in Westphalia, as Helmut Ostermann, the youngest of four children, to a well-established German Jewish family, his father being a private banker in the town. (Avnery, born in, Beckum) ! (uri avnery.Q325679, place of birth.P19, beckum, germany.Q2707) Ursula Lehr She was a member of the Christian Democratic Union . (She, was a member of, The Christian Democratic Union) ! (ursula lehr.Q290708, member of political party.P102, christian democratic union of germany.Q49762) Valdı́via (Brazilian Footballer) Valdı́via Wanderson Ferreira de Oliveira , commonly known as Valdı́via, is a Brazilian footballer who plays as an attacking midfielder for Avaı́, on loan from Internacional. (A Brazilian Footballer, plays as, An Attacking Midfielder) ! (valdı́via (brazilian footballer).Q18176477, position played on team / speciality.P413, midfielder.Q193592) Victor-François, 2Nd Duc De Broglie His victory over Prince Ferdinand at Bergen won him the rank of marshal of France from the French King Louis XV and the title of Prince of the Holy Roman Empire ”Reichsfürst” from Holy Roman Emperor Francis I. In 1759, he won the Battle of Bergen and followed that with the capture the city of Minden, later fighting at the Battle of Minden under the command of Contades, whom he would succeed in command. (His victory, won him the, Marshal) ! (victor-françois, 2nd duc de broglie.Q695929, award received.P166, marshals of france.Q474485)
Figure 11: Mapped facts: MAMA-GPT-2XL on Wikidata.
22
Under review as a conference paper at ICLR 2021
ID Sentence Candidate facts to mapped facts BERTLARGE GPT-2XL Andy Hilbert In the same season he was claimed off waivers by the Pittsburgh Penguins on March
9, 2006. - (He, by, The Pittsburgh Penguins) ! (andy hilbert.Q526158, mem-
ber of sports team.P54, pittsburgh penguins.Q193643) Bob Dylan.Q392 Dylan married his backup singer Carolyn Dennis on June 4, 1986. - (Dylan, married, Carolyn Dennis) ! (Bob Dylan.Q392, spouse.P26, Carolyn Dennis.Q5045345) Bones Hillman During the hiatus of Midnight Oil , Hillman returned to New Zealand , working as
a studio and live musician with Dave Dobbyn and recorded the album ” Available Light ” .
(Hillman, returned, New Zealand) ! (bones hillman.Q2910026, country of citizenship.P27, new zealand.Q664) -
Brent Hinds Hinds continues to concentrate on Mastodon, with the majority of his time spent touring or in the studio. - (His Time, with, Mastodon) ! (brent hinds.Q909746, member of.P463, mastodon (band).Q548844) Butch Vig Their debut album, ”Garbage”, was an unexpected smash, selling over 4 million copies and certified double platinum in the UK, United States and Australia. - (Over 4 Million Copies, in, United States) ! (butch vig.Q451084, country of citizenship.P27, united states.Q30) C-Kan In addition to being known as a rapper, C-Kan is also known for supporting the free use of marijuana in México. - (Marijuana, in, México) ! (c-kan.Q27734073, country of citizenship.P27, mexico.Q96) Charles Gratiot Sr. Gratiot died of a stroke in St . Louis . (Gratiot, died, St. Louis) ! (charles gratiot sr..Q2959257, place of death.P20, st. louis.Q38022) - Cheryl Chan Cheryl Chan Cheryl Chan Wei Ling , PBM , is a Singaporean politician . (Cheryl Chan Wei Ling, is, A Singaporean Politician) ! (cheryl chan.Q22003605, occupation.P106, politician.Q82955) - Cyba Audi Audi is an experienced moderator and presenter at local, regional, and international business events. - (Audi, is, Presenter) ! (cyba audi.Q18921358, occupation.P106, television presenter.Q947873) Douglas Lenat He is a Fellow of the AAAS , AAAI , and Cognitive Science Society , and an editor of the J . Automated Reasoning , J . Learning Sciences , and J . Applied Ontology . (He, is a Fellow, Aaai) ! (douglas lenat.Q559334, member of.P463, association for the advancement of artificial intelligence.Q2739680) - Douglas Lenat While attending the University of Pennsylvania, Lenat supported himself through programming, notably designing and developing a natural language interface to a U.S. Navy data base question–answering system serving as an early online shipboard operations manual used on US aircraft carriers. - (Lenat, ,, The University of Pennsylvania) ! (douglas lenat.Q559334, educated at.P69, university of pennsylvania.Q49117) Fred Stewart (Football Manager) Despite managing for 39 years he only ever took charge of two clubs, Stockport County and Cardiff City, and he holds the record for longest serving manager in the history of both clubs. - (He, charge of, Stockport County) ! (fred stewart (football manager).Q5496345, member of sports team.P54, stockport county f.c..Q18526) Félix Tanco Born Félix Manuel de Jesús Tanco y Bosmeniel , in Bogotá , Colombia , he arrived in Cuba at a very young age . (Bogotá, ,, Cuba) ! (félix tanco.Q5511649, country of citizenship.P27, cuba.Q241) - Gary Ablett Jr. He also won his second Brownlow Medal , becoming the first Gold Coast player to win the award and the 14th player in VFL / AFL history to win it twice .
(He, also won, His Second Brownlow Medal) ! (gary ablett jr..Q3098509, award received.P166, leigh matthews trophy.Q6519632); (He, also won, His Second Brownlow Medal) ! (gary ablett jr..Q3098509, award received.P166, brownlow medal.Q3853498) -
Geeto Mongol After meeting his would - be tag team partner , Bepo Mongol , Tattrie returned to the WWWF as Geto Mongol and The Mongols were brought to the United States in 1968 . (The Mongols, were, The United States) ! (geeto mongol.Q11324907, country of citizenship.P27, united states.Q30) - Georgi Plekhanov Georgi Plekhanov Georgi Valentinovich Plekhanov was a Russian revolutionary , philosopher and a Marxist theoretician . (Georgi Valentinovich Plekhanov, was, A Marxist Theoretician) ! (georgi plekhanov.Q182905, occupation.P106, philosopher.Q4964182) - Győző Drozdy Győző Drozdy Győző Drozdy was a Hungarian teacher , journalist , and politician . (Győző Drozdy, was, Politician) ! (győző drozdy.Q910001, occupation.P106, politician.Q82955) - Horace Rawlins Rawlins died on 22 January 1935 in a nursing home in Reading , Berkshire . (Rawlins, died, Reading) ! (horace rawlins.Q4436711, place of death.P20, reading, berkshire.Q161491) - Horace Rawlins He played in a professional event at Stanmore Golf Club in June 1894 but finished well out of the prizes. - (He, played in, Stanmore Golf Club) ! (horace rawlins.Q4436711, sport.P641, golf.Q5377) Hugo Winterhalter Winterhalter remained with RCA Victor until 1963, at which time he moved to Kapp; that same year, he also penned the main title theme for the film, ”Diamond Head”. - (Winterhalter, to, Kapp) ! (hugo winterhalter.Q1635872, record label.P264, kapp records.Q1728690) Ioannis Talianos He competed in the men ’ s 4 × 100 metres relay at the 1924 Summer Olympics . (He, competed in the, The 1924 Summer Olympics) ! (ioannis talianos.Q35692847, participant of.P1344, 1924 summer olympics.Q8132) - Joan Of France, Duchess Of Berry The nuns of the Order of the Annunciation of the Blessed Virgin Mary still maintain their way of life in monasteries in France, Belgium, Costa Rica and Poland. - (Their Way, in, France) ! (joan of france, duchess of berry.Q236220, country of citizenship.P27, france.Q142) John Maynard Keynes John Maynard Keynes John Maynard Keynes , 1st Baron Keynes , was a British economist , whose ideas fundamentally changed the theory and practice of macroeconomics and the economic policies of governments . (John Maynard Keynes, was, A British Economist) ! (john maynard keynes.Q9317, occupation.P106, economist.Q188094) - Jonathan Culler Currently , he is Class of 1916 Professor of English and Comparative Literature at Cornell University . (He, is, English) ! (jonathan culler.Q933332, languages spoken, written or signed.P1412, english language.Q1860) - Jordi Cañas Pérez In the regional elections of Catalonia in November 2010 , he was chosen in the primary as number three for Ciutadans ’ candidacy in the province of Barcelona . (The Province, of, Barcelona) ! (jordi cañas pérez.Q557693, member of political party.P102, citizens (spanish political party).Q1393123) - Jules Deligny He competed in the freestyle lightweight event at the 1920 Summer Olympics . (He, competed in the, The 1920 Summer Olympics) ! (jules deligny.Q16207961, participant of.P1344, 1920 summer olympics.Q8128) - Liz Mcgregor Liz McGregor Liz McGregor is a South African author and a journalist who worked for leading South African newspapers such as the ” Sunday Times ” and the ” Rand Daily Mail ” . (Liz Mcgregor, is, A Journalist) ! (liz mcgregor.Q23541069, occupation.P106, journalist.Q1930187) - Louise Shropshire The granddaughter of slaves, Louise Shropshire was born Louise Jarrett on February 15, 1913 in Coffee County, Alabama.
- (Louise Shropshire, was, Coffee County, Alabama) ! (louise shropshire.Q15430857, place of birth.P19, coffee county, alabama.Q485660)
Macit Gürdal He competed in the men ’ s tournament at the 1952 Summer Olympics . (He, competed in the, The 1952 Summer Olympics) ! (macit gürdal.Q57313835, participant of.P1344, 1952 summer olympics.Q8407) - Markus Pröll At the beginning of his career , Pröll became a target of some ridicule because his name ” Pröll ” literally means something like ” lout ” in German . (”Pröll, ” in, German) ! (markus pröll.Q704630, country of citizenship.P27, germany.Q183) - Monde Hadebe Their Round Four match against the saw Hadebe score the first senior try of his career in a 27 – 10 victory , and the Sharks XV again finished top of the Southern Section log . (His Career, ,, The Sharks Xv) ! (monde hadebe.Q6898677, member of sports team.P54, sharks (currie cup).Q744636) - Neville Southall He moved on to Everton for £150,000 in 1981 and established himself as the club’s first-choice goalkeeper by the 1983–84 season. - (He, moved on to, Everton) ! (neville southall.Q436650, member of sports team.P54, everton f.c..Q5794) Olga Semenova Tyan-Shanskaya She died in Leningrad . (She, died, Leningrad) ! (olga semenova tyan-shanskaya.Q2392240, place of death.P20, saint petersburg.Q656) - Parvathy Ratheesh Parvathy Ratheesh made her film acting debut in 2015 similar to that of her younger brother , Padmaraj Ratheesh who also made his acting debut in the same year through ” Fireman ” . (Her Film, that of her younger brother, Padmaraj Ratheesh) ! (parvathy ratheesh.Q19895785, sibling.P3373, padmaraj ratheesh.Q19895782) - Peter Dalla Riva In 1993, he was inducted into the Canadian Football Hall of Fame. - (He, was inducted into the, Fame) ! (peter dalla riva.Q7173528, award received.P166, canadian football hall of fame.Q3517653) Pierre Duhem Unlike many former historians , who denigrated the Middle Ages, he endeavored to show that the Roman Catholic Church had helped foster Western science in one of its most fruitful periods. - (The Roman Catholic Church, had, Foster Western Science) ! (pierre duhem.Q314172, field of work.P101, philosophy of science.Q59115) Rafael Arcadio Bernal Supelano He then served as bishop of the Roman Catholic Diocese of Arauca, Colombia, from 1990 to 2003 and as bishop of the Roman Catholic Diocese of Libano–Honda, Colombia, from 2003 to 2004. - (The Roman Catholic Diocese, ,, Colombia) ! (rafael arcadio bernal supelano.Q2126904, country of citizenship.P27, colombia.Q739) Rajeshwar Dayal The Government of India awarded him the second highest civilian award of the Padma Vibhushan, in 1969. - (Him, award, The Padma Vibhushan) ! (rajeshwar dayal.Q15448539, award received.P166, padma vibhushan.Q672392) Ray Johnson (American Football) Ray Johnson Raymond Robert Johnson was an American football defensive back who played three seasons in the National Football League with the Cleveland Rams and Chicago Cardinals . (The National Football League, back who played, Raymond Robert Johnson) ! (ray johnson (american football).Q21077889, sport.P641, american football.Q41323) - Sean Hanish Sean Hanish Sean Hanish is an American film writer , producer and director best known for ” Saint Judy ” , ” Return to Zero ” and ” Sister Cities ” .
(Sean Hanish, is, An American Film Writer) ! (sean hanish.Q19867622, country of citizenship.P27, united states.Q30); (Sean Hanish, is, An American Film Writer) ! (sean hanish.Q19867622, occupation.P106, screenwriter.Q28389) -
Seth Plum Born in Edmonton, Plum played professionally for Charlton Athletic, Chelsea and Southend United. - (Plum, played professionally for, Charlton Athletic) ! (seth plum.Q7456581, member of sports team.P54, charlton athletic f.c..Q19462) Seth Plum Plum received his only cap for England at age 23 while playing for Charlton Athletic , starting and playing the full 90 minutes in a 4 – 1 win over France on 10 May 1923 . (His Only Cap, playing for, Charlton Athletic) ! (seth plum.Q7456581, member of sports team.P54, charlton athletic f.c..Q19462) - Stanislav Nedkov A two - time national champion in freestyle wrestling , he has also had first - place finishes in international tournaments in Russia , Moldova , Turkey , and Bulgaria . (Turkey, , and, Bulgaria) ! (stanislav nedkov.Q4844375, country of citizenship.P27, bulgaria.Q219) - Stiliyan Petrov In addition he is Bulgaria’s all-time most-capped player with 105 appearances for the side. - (He, is, Bulgaria’S All-Time Most-Capped Player) ! (stiliyan petrov.Q166263, country of citizenship.P27, bulgaria.Q219) Stiliyan Petrov On 14 January 2010 , it was announced that Petrov had come second in Bulgaria ’ s Player of the Year . (Petrov, in, Bulgaria’S Player) ! (stiliyan petrov.Q166263, country of citizenship.P27, bulgaria.Q219) - Thomas A. Dunn He graduated from DePaul University College of Law in 1971. - (He, graduated from, Depaul University College) ! (thomas a. dunn.Q33082990, educated at.P69, depaul university college of law.Q5244034) Thomas Scott (Commentator) In 1803 , Scott left the Lock Hospital to become Rector of Aston Sandford in Buckinghamshire where he remained until his death in 1821 .
(His Death, he remained until, Aston Sandford) ! (thomas scott (commentator).Q7793831, place of death.P20, aston sandford.Q3088093) -
Thor Heyerdahl Thor Heyerdahl Thor Heyerdahl was a Norwegian adventurer and ethnographer with a background in zoology, botany and geography. - (Thor Heyerdahl, was, A Norwegian Adventurer) ! (thor heyerdahl.Q133622, country of citizenship.P27, norway.Q20) Venkatesh Kulkarni His first novel , ” Naked in Deccan ” , won the 1984 American Book Award of the Before Columbus Foundation and was listed among the top ten novels of the decade by the ” Chicago Tribune ” . (His First Novel, , won, The 1984 American Book Award) ! (venkatesh kulkarni.Q7920091, award received.P166, american book awards.Q463606) - Vitor Castro De Souza Vitor Castro de Souza Vı́tor Castro de Souza , or simply Vitor Castro, is a Brazilian striker.
- (Vı́tor Castro De Souza, is, A Brazilian Striker) ! (vitor castro de souza.Q5604876, position played on team / speciality.P413, forward (association football).Q280658)
Vyacheslav Tsaryov He made his debut in the Soviet Top League in 1990 for FC Dynamo Moscow . (He, for, Fc Dynamo Moscow) ! (vyacheslav tsaryov.Q4503307, member of sports team.P54, fc dynamo moscow.Q17497) - Zare Markovski Zare Markovski as player played in KK Rabotnički and MZT Skopje . (Zare Markovski, played in, Kk Rabotnički) ! (zare markovski.Q4023877, member of sports team.P54, kk rabotnički.Q2603345) -
Figure 12: Mapped facts: MAMA-BERTLARGE vs. MAMA-GPT-2XL on Wikidata.
23
Under review as a conference paper at ICLR 2021

heading: ID Sentence Candidate facts to unmapped facts
text: Nick Aldis He was also a co - presenter of ” Britain ’ s Strongest Man ” on Challenge TV in the United Kingdom . (Challenge Tv, in, The United Kingdom) ! (Challenge TV, in, united kingdom.Q145) Douglas Bader At the age of two , Bader joined his parents in India for a year . (His Parents, in, India) ! (douglas bader.Q348780, in, india.Q668) Douglas Bader He remained in France after the war , where , having attained the rank of major , he died in 1922 of complications from
those wounds in a hospital in Saint - Omer , the same area where Bader would bail out and be captured in 1941 . (He, in, France) ! (douglas bader.Q348780, in, france.Q142)
Douglas Bader On 2 July 1941 he was awarded the bar to his DSO. (He, was awarded the bar, his DSO) ! (douglas bader.Q348780, was awarded the bar, distinguished service order.Q615838) Helen Storrow His father , also named James Jackson Storrow , was a prominent attorney , whose clients included Alexander Graham Bell and the government of Venezuela ; his mother , Ann Maria Perry , was the grandchild of naval hero Commodore Oliver Hazard Perry , and a distant cousin of President Thomas Jefferson . (His Father, was, A Prominent Attorney) ! (His Father, occupation.P106, lawyer.Q40348) Helen Storrow Helen Storrow Helen Osborne Storrow was a prominent American philanthropist , early Girl Scout leader , and chair of the World Committee of the World Association of Girl Guides and Girl Scouts for eight years . (Helen Osborne Storrow, was, A Prominent American Philanthropist) ! (helen storrow.Q5703224, place of birth.P19, united states.Q30) Jacob Van Ruisdael His cousin Jacob was a registered Mennonite in Amsterdam . (His Cousin, was, A Registered Mennonite) ! (His Cousin, was, A Registered Mennonite) Jacob Van Ruisdael He appears to have been strongly influenced by other contemporary local Haarlem landscapists , most notably Cornelis
Vroom and Allaert van Everdingen . (He, have been strongly influenced by, landscape painting) ! (jacob van ruisdael.Q213612, have been strongly influenced by, landscape art.Q191163)
Liaquat Ali Khan Liaquat Ali Khan was educated at the Aligarh Muslim University in India , and then at Oxford University in the United Kingdom . (Liaquat Ali Khan, in, India) ! (liaquat ali khan.Q295713, in, india.Q668) Pauline Baynes Baynes began her education at a convent school . (Her Education, at, A Convent School) ! (pauline baynes.Q101951, at, catholic school.Q1138671) Pauline Baynes Her most vivid recollection of their New Year ’ s Eve lunch at Magdalen College was of his gleefully picking nuts out
of a bowl of Brussels sprouts . (Their New Year’S Eve Lunch, at, Magdalen College) ! (pauline baynes.Q101951, at, magdalen college, oxford.Q81162)
Pauline Baynes By the time that she left , she had already formed the ambition of becoming an illustrator . (She, formed the ambition of becoming, An Illustrator) ! (pauline baynes.Q101951, formed the ambition of becoming, illustrator.Q644687) Thor Heyerdahl He was an atheist . (He, was, An Atheist) ! (thor heyerdahl.Q133622, was, An Atheist) Thor Heyerdahl He is buried in the garden of the family home in Colla Micheri . (He, is buried, Colla Micheri) ! (thor heyerdahl.Q133622, is buried, colla micheri.Q3647771) Neville Southall He moved on to Cheshire County League club Winsford United at the age of 20 . (He, to, Cheshire County League Club Winsford United) ! (neville southall.Q436650, to, winsford united f.c..Q5213266) Neville Southall He later became a player - coach at York City , Rhyl , Shrewsbury Town and Dagenham & Redbridge . (He, became, A Player-Coach) ! (neville southall.Q436650, occupation.P106, A Player-Coach) John Maynard Keynes After the war , Winston Churchill attempted to check the rise of Keynesian policy - making in the United Kingdom and
used rhetoric critical of the mixed economy in his 1945 election campaign . (Keynesian Policy-Making, in, The United Kingdom) ! (john maynard keynes.Q9317, in, united kingdom.P145)
John Maynard Keynes Apart from Great Britain , Keynes ’ s attention was primarily focused on the United States . (Keynes’S Attention, was, The United States) ! (john maynard keynes.Q9317, was, united states.Q30) John Maynard Keynes John Maynard Keynes John Maynard Keynes , 1st Baron Keynes , was a British economist , whose ideas fundamentally
changed the theory and practice of macroeconomics and the economic policies of governments . (John Maynard Keynes, was, A British Economist) ! (john maynard keynes.Q9317, was, united kingdom.P145)
Ernst Haeckel Ernst Haeckel Ernst Heinrich Philipp August Haeckel was a German zoologist , naturalist , philosopher , physician , professor , marine biologist , and artist who discovered , described and named thousands of new species , mapped a genealogical tree relating all life forms , and coined many terms in biology , including ” ecology ” , ” phylum ” , ” phylogeny ” , and ” Protista . ” (August Haeckel, was, A German Zoologist) ! (ernst haeckel.Q48246, was, germany.Q183) Ernst Haeckel Haeckel was awarded the title of Excellency by Kaiser Wilhelm II in 1907 and the Linnean Society of London ’ s prestigious Darwin - Wallace Medal in 1908 . (Haeckel, was awarded the, Excellency) ! (ernst haeckel.Q48246, award received.P166, Excellency) Ernst Haeckel He was also a pacifist until the First World War , when he wrote propaganda in favor of the war . (He, was, A Pacifist) ! (ernst haeckel.Q48246, was, A Pacifist)
Figure 13: Unmapped facts: MAMA-BERTLARGE on Wikidata.

heading: ID Sentence Candidate facts to unmapped facts
text: Douglas Bader He left St Edward’s in early 1928, aged 18. (He, left, St Edward) ! (douglas bader.Q348780, left, st edward’s school, oxford.Q128260) Douglas Bader Group Captain Sir Douglas Robert Steuart Bader, was a Royal Air Force flying ace during the Second World War. (Douglas Bader Group Captain Sir Douglas Robert Steuart Bader, was, A Royal Air Force Flying Ace) ! (douglas bader.Q348780, was, flying ace.Q222982) Douglas Bader On 2 July 1941 he was awarded the bar to his DSO. (He, was awarded the bar to, his DSO) ! (douglas bader.Q348780, was awarded the bar to, distinguished service order.Q615838) Helen Storrow His father, also named James Jackson Storrow, was a prominent attorney, whose clients included Alexander Graham
Bell and the government of Venezuela; his mother, Ann Maria Perry, was the grandchild of naval hero Commodore Oliver Hazard Perry, and a distant cousin of President Thomas Jefferson.
(His Father, was, A Prominent Attorney) ! (His Father, occupation.P106, lawyer.Q40348)
Helen Storrow Helen Osborne Storrow was a prominent American philanthropist, early Girl Scout leader, and chair of the World Committee of the World Association of Girl Guides and Girl Scouts for eight years. (Helen Osborne Storrow, was, A Prominent American Philanthropist) ! (helen storrow.Q5703224, was, united states.Q30) Jacob Van Ruisdael Ruisdael lived and worked in Amsterdam for the rest of his life. (Ruisdael, lived, Amsterdam) ! (jacob van ruisdael.Q213612, lived, amsterdam.Q727); (Ruisdael, lived and worked in, Amsterdam) ! (jacob van ruisdael.Q213612, lived and worked in, amsterdam.Q727) Liaquat Ali Khan Liaquat Ali Khan was educated at the Aligarh Muslim University in India, and then at Oxford University in the United Kingdom. (Liaquat Ali Khan, was educated at, The Aligarh Muslim University) ! (liaquat ali khan.Q295713, was educated at, aligarh muslim university.Q196544) Liaquat Ali Khan After the death of his father in 1919, Ali Khan, with British Government awarding the grants and scholarship, went to England, attending Oxford University’s Exeter College to pursue his higher education. (His Higher Education, pursue, Oxford University’S Exeter College) ! (liaquat ali khan.Q295713, pursue, university of oxford.Q34433) Liaquat Ali Khan In 1940, Khan was made the deputy leader of the Muslim League Parliamentary party. (Khan, was made the deputy leader of, The Muslim League Parliamentary Party) ! (liaquat ali khan.Q295713, deputy leader, all-india muslim league.Q223898) Pauline Baynes When Baynes’s father retired, he left India and returned to England, settling with Baynes’s mother in a house close to Baynes’s own near Farnham in southwest Surrey. (He, left, India) ! (pauline baynes.Q101951, left, india.Q668) Pauline Baynes Baynes began her education at a convent school. (Her Education, at, A Convent School) ! (pauline baynes.Q101951, at, catholic school.Q1138671) Pauline Baynes Her most vivid recollection of their New Year’s Eve lunch at Magdalen College was of his gleefully picking nuts out of
a bowl of Brussels sprouts. (Their New Year’S Eve Lunch, at, Magdalen College) ! (pauline baynes.Q101951, at, magdalen college, oxford.Q81162)
Pauline Baynes She was not a diligent student, frittering away her time on c̈offee and parties,̈ and she left the Slade without a qualification, But she did achieve the distinction, one shared with her sister, of exhibiting at the Royal Academy of Arts in 1939. In 1940, a year into World War II, both Baynes sisters joined the Women’s Voluntary Service. (She, without a qualification, The Slade) ! (pauline baynes.Q101951, without a qualification, slade school of fine art.Q1399299) Thor Heyerdahl His theories rarely won any scientific acceptance, whereas Heyerdahl himself rejected all scientific criticism and concentrated on publishing his theories in popular books aimed at the general public. (His Theories, at, The General Public) ! (thor heyerdahl.Q133622, at, general public) Thor Heyerdahl He was an atheist. (He, was, An Atheist) ! (thor heyerdahl.Q133622, was, An Atheist) Neville Southall As a teenager, Southall had unsuccessful trials at Wrexham, Crewe Alexandra and Bolton Wanderers. (Southall, at, Wrexham) ! (neville southall.Q436650, at, wrexham a.f.c..Q18529); (Southall, at, Crewe Alexandra) ! (neville southall.Q436650, at, crewe alexandra f.c..Q19587) John Maynard Keynes John Maynard Keynes, 1st Baron Keynes, was a British economist, whose ideas fundamentally changed the theory and
practice of macroeconomics and the economic policies of governments. (John Maynard Keynes, was, A British Economist) ! (john maynard keynes.Q9317, was, united kingdom.Q145)
John Maynard Keynes In January 1889 at the age of five and a half, Keynes started at the kindergarten of the Perse School for Girls for five mornings a week. (Keynes, at, The Perse School) ! (john maynard keynes.Q9317, at, the perse school.Q7756751) John Maynard Keynes In January 1892, at eight and a half, he started as a day pupil at St Faith’s preparatory school. (He, at, St Faith’S Preparatory School) ! (john maynard keynes.Q9317, at, st faith’s school.Q7593056) John Maynard Keynes Keynes was a lifelong member of the Liberal Party, which until the 1920s had been one of the two main political parties
in the United Kingdom, and as late as 1916 had often been the dominant power in government. (Keynes, was a lifelong member of, The liberal party) ! (john maynard keynes.Q9317, was a lifelong member of, liberal party (uk).Q622441)
Ernst Haeckel Haeckel was awarded the title of Excellency by Kaiser Wilhelm II in 1907 and the Linnean Society of London’s prestigious Darwin-Wallace Medal in 1908. (Haeckel, was awarded the, Excellency) ! (ernst haeckel.Q48246, award received.P166, Excellency) Ernst Haeckel He was also a pacifist until the First World War, when he wrote propaganda in favor of the war. (He, was, A Pacifist) ! (ernst haeckel.Q48246, was, A Pacifist) Ernst Haeckel He was also a social Darwinist who believed that ”survival of the fittest” was a natural law, and that struggle led to
improvement of the race. (He, was, A Social Darwinist) ! (ernst haeckel.Q48246, was, A Social Darwinist)
Ernst Haeckel Ernst Haeckel was born on 16 February 1834, in Potsdam. (Haeckel, was born on 17 February 1834, in, Potsdam) ! (ernst haeckel.Q48246, was born on 17 February 1834, in, potsdam.Q1711)
Figure 14: Unmapped facts: MAMA-GPT-2XL on Wikidata.
24
Under review as a conference paper at ICLR 2021
ID Sentence Candidate facts to mapped facts BERTLARGE GPT-2XL Nick Aldis He was also a co - presenter of ” Britain ’ s Strongest Man ” on Challenge TV in
the United Kingdom . (Challenge Tv, in, The United Kingdom) ! (Challenge TV, in, united kingdom.Q145) -
Douglas Bader Group Captain Sir Douglas Robert Steuart Bader, was a Royal Air Force flying ace during the Second World War. - (Douglas Bader Group Captain Sir Douglas Robert Steuart Bader, was, A Royal Air Force Flying Ace) ! (douglas bader.Q348780, was, flying ace.Q222982) Douglas Bader At the age of two , Bader joined his parents in India for a year . (His Parents, in, India) ! (douglas bader.Q348780, in, india.Q668) - Douglas Bader On 2 July 1941 he was awarded the bar to his DSO. (He, was awarded the bar, his DSO) ! (douglas bader.Q348780, was awarded the
bar, distinguished service order.Q615838) (He, was awarded the bar to, his DSO) ! (douglas bader.Q348780, was awarded the bar to, distinguished service order.Q615838)
Helen Storrow His father , also named James Jackson Storrow , was a prominent attorney , whose clients included Alexander Graham Bell and the government of Venezuela ; his mother , Ann Maria Perry , was the grandchild of naval hero Commodore Oliver Hazard Perry , and a distant cousin of President Thomas Jefferson . (His Father, was, A Prominent Attorney) ! (His Father, occupation.P106, lawyer.Q40348) (His Father, was, A Prominent Attorney) ! (His Father, occupation.P106, lawyer.Q40348) Helen Storrow Helen Osborne Storrow was a prominent American philanthropist, early Girl Scout leader, and chair of the World Committee of the World Association of Girl Guides and Girl Scouts for eight years. (Helen Osborne Storrow, was, A Prominent American Philanthropist) ! (helen storrow.Q5703224, was, united states.Q30) (Helen Osborne Storrow, was, A Prominent American Philanthropist) ! (helen storrow.Q5703224, was, united states.Q30) Jacob Van Ruisdael Ruisdael lived and worked in Amsterdam for the rest of his life. - (Ruisdael, lived, Amsterdam) ! (jacob van ruisdael.Q213612, lived, amsterdam.Q727); (Ruisdael, lived and worked in, Amsterdam) ! (jacob van ruisdael.Q213612, lived and worked in, amsterdam.Q727) Jacob Van Ruisdael His cousin Jacob was a registered Mennonite in Amsterdam . (His Cousin, was, A Registered Mennonite) ! (His Cousin, was, A Registered Mennonite) - Jacob Van Ruisdael He appears to have been strongly influenced by other contemporary local Haarlem landscapists , most notably Cornelis Vroom and Allaert van Everdingen .
(He, have been strongly influenced by, landscape painting) ! (jacob van ruisdael.Q213612, have been strongly influenced by, landscape art.Q191163) -
Liaquat Ali Khan Liaquat Ali Khan was educated at the Aligarh Muslim University in India, and then at Oxford University in the United Kingdom. (Liaquat Ali Khan, in, India) ! (liaquat ali khan.Q295713, in, india.Q668) (Liaquat Ali Khan, was educated at, The Aligarh Muslim University) ! (liaquat ali khan.Q295713, was educated at, aligarh muslim university.Q196544) Liaquat Ali Khan After the death of his father in 1919, Ali Khan, with British Government awarding the grants and scholarship, went to England, attending Oxford University’s Exeter College to pursue his higher education. - (His Higher Education, pursue, Oxford University’S Exeter College) ! (liaquat ali khan.Q295713, pursue, university of oxford.Q34433) Liaquat Ali Khan In 1940, Khan was made the deputy leader of the Muslim League Parliamentary party. - (Khan, was made the deputy leader of, The Muslim League Parliamentary Party) ! (liaquat ali khan.Q295713, deputy leader, all-india muslim league.Q223898) Pauline Baynes When Baynes’s father retired, he left India and returned to England, settling with Baynes’s mother in a house close to Baynes’s own near Farnham in southwest Surrey. - (He, left, India) ! (pauline baynes.Q101951, left, india.Q668) Pauline Baynes Baynes began her education at a convent school. (Her Education, at, A Convent School) ! (pauline baynes.Q101951, at, catholic school.Q1138671) (Her Education, at, A Convent School) ! (pauline baynes.Q101951, at, catholic school.Q1138671) Pauline Baynes Her most vivid recollection of their New Year’s Eve lunch at Magdalen College was of his gleefully picking nuts out of a bowl of Brussels sprouts. (Their New Year’S Eve Lunch, at, Magdalen College) ! (pauline baynes.Q101951, at, magdalen college, oxford.Q81162) (Their New Year’S Eve Lunch, at, Magdalen College) ! (pauline baynes.Q101951, at, magdalen college, oxford.Q81162) Pauline Baynes She was not a diligent student, frittering away her time on c̈offee and parties,̈ and she left the Slade without a qualification, But she did achieve the distinction, one shared with her sister, of exhibiting at the Royal Academy of Arts in 1939, In 1940, a year into World War II, both Baynes sisters joined the Women’s Voluntary Service. - (She, without a qualification, The Slade) ! (pauline baynes.Q101951, without a qualification, slade school of fine art.Q1399299) Pauline Baynes By the time that she left , she had already formed the ambition of becoming an illustrator . (She, formed the ambition of becoming, An Illustrator) ! (pauline baynes.Q101951, formed the ambition of becoming, illustrator.Q644687) - Thor Heyerdahl His theories rarely won any scientific acceptance, whereas Heyerdahl himself rejected all scientific criticism and concentrated on publishing his theories in popular books aimed at the general public. - (His Theories, at, The General Public) ! (thor heyerdahl.Q133622, at, general public) Thor Heyerdahl He was an atheist . (He, was, An Atheist) ! (thor heyerdahl.Q133622, was, An Atheist) (He, was, An Atheist) ! (thor heyerdahl.Q133622, was, An Atheist) Thor Heyerdahl He is buried in the garden of the family home in Colla Micheri . (He, is buried, Colla Micheri) ! (thor heyerdahl.Q133622, is buried,
colla micheri.Q3647771) -
Neville Southall As a teenager, Southall had unsuccessful trials at Wrexham, Crewe Alexandra and Bolton Wanderers. - (Southall, at, Wrexham) ! (neville southall.Q436650, at, wrexham a.f.c..Q18529); (Southall, at, Crewe Alexandra) ! (neville southall.Q436650, at, crewe alexandra f.c..Q19587) Neville Southall He moved on to Cheshire County League club Winsford United at the age of 20 . (He, to, Cheshire County League Club Winsford United) ! (neville southall.Q436650, to, winsford united f.c..Q5213266) - Neville Southall He later became a player - coach at York City , Rhyl , Shrewsbury Town and Dagenham & Redbridge . (He, became, A Player-Coach) ! (neville southall.Q436650, occupation.P106, A Player-Coach) - John Maynard Keynes In January 1889 at the age of five and a half, Keynes started at the kindergarten of the Perse School for Girls for five mornings a week. - (Keynes, at, The Perse School) ! (john maynard keynes.Q9317, at, the perse school.Q7756751) John Maynard Keynes In January 1892, at eight and a half, he started as a day pupil at St Faith’s preparatory school. - (He, at, St Faith’S Preparatory School) ! (john maynard keynes.Q9317, at, st faith’s school.Q7593056) John Maynard Keynes After the war , Winston Churchill attempted to check the rise of Keynesian policy - making in the United Kingdom and used rhetoric critical of the mixed economy in his 1945 election campaign . (Keynesian Policy-Making, in, The United Kingdom) ! (john maynard keynes.Q9317, in, united kingdom.P145) - John Maynard Keynes Keynes was a lifelong member of the Liberal Party, which until the 1920s had been one of the two main political parties in the United Kingdom, and as late as 1916 had often been the dominant power in government. - (Keynes, was a lifelong member of, The liberal party) ! (john maynard keynes.Q9317, was a lifelong member of, liberal party (uk).Q622441) Ernst Haeckel Ernst Haeckel Ernst Heinrich Philipp August Haeckel was a German zoologist , naturalist , philosopher , physician , professor , marine biologist , and artist who discovered , described and named thousands of new species , mapped a genealogical tree relating all life forms , and coined many terms in biology , including ” ecology ” , ” phylum ” , ” phylogeny ” , and ” Protista . ” (August Haeckel, was, A German Zoologist) ! (ernst haeckel.Q48246, was, germany.Q183) - Ernst Haeckel Ernst Haeckel was born on 16 February 1834, in Potsdam. - (Haeckel, was born on 17 February 1834, in, Potsdam) ! (ernst haeckel.Q48246, was born on 17 February 1834, in, potsdam.Q1711) Ernst Haeckel Haeckel was awarded the title of Excellency by Kaiser Wilhelm II in 1907 and the Linnean Society of London’s prestigious Darwin-Wallace Medal in 1908. (Haeckel, was awarded the, Excellency) ! (ernst haeckel.Q48246, award received.P166, Excellency) (Haeckel, was awarded the, Excellency) ! (ernst haeckel.Q48246, award received.P166, Excellency) Ernst Haeckel He was also a pacifist until the First World War, when he wrote propaganda in favor of the war. (He, was, A Pacifist) ! (ernst haeckel.Q48246, was, A Pacifist) (He, was, A Pacifist) ! (ernst haeckel.Q48246, was, A Pacifist)
Figure 15: Unmapped facts: MAMA-BERTLARGE vs. MAMA-GPT-2XL on Wikidata.
Figure 16: A snapshot subgraph of the open KG generated by MAMA using BERTLARGE from Wikipedia pages neighboring “Bob Dylan”. The blue node and arrow represent the mapped facts in the Wikidata schema,
while the yellow node and arrow denote the unmapped facts in the open schema. We also visualize the correct facts that are new in Wikidata in yellow.
25
Under review as a conference paper at ICLR 2021
26
Under review as a conference paper at ICLR 2021
27
Under review as a conference paper at ICLR 2021
28
Under review as a conference paper at ICLR 2021
29
Under review as a conference paper at ICLR 2021
30
Language M odels are O pen Know ledge G raphs Supplem entary M aterials
Problem •Know ledge graph construction requires hum an supervision •Language m odels store know ledge
2
Problem •How to use language m odels to construct know ledge graphs? ?
3
Know ledge graph
Challenges
Target corpus
Language m odel General know
ledge Dom
ain specific know
ledge Structured representation
4
Proposed Approach • M AM A constructs an open know ledge graph w ith a single forw ard pass of the language m odel (w ithout fine-tuning) over the corpus • M atch: generates a set of candidate facts from a textual corpus • M ap: produces an open know ledge graph from the m atched candidates Corpus Dylan is a songw riter. Dylan w as aw arded N obel Prize. Dylan signed Albert Grossm an. … Candidate Facts (Dylan, is, songw riter) (Dylan, aw arded, N obel Prize) (Dylan, signed, Albert Grossm an) … O pen Know ledge G raph Bob_Dylan.Q 392 occupation.P106 Songw riter.Q 753110 award_received.P166 N obel_Prize.Q 7191 … … M atch M ap Pre-trained language m odel BERT GPT-2 … sign Albert_Grossm an.Q 708584 5
Results on TAC KBP 23.16 27.07 28.79
29.05 28.07
28.69 29.12
29.72
F1 (%)
O penIE 5.1
Stan fo rd O penIE
M A M A -BER Tbase
M A M A -BER Tlarge
M A M A -G PT2
M A M A -G PT2m edium M A M A -G PT2large M A M A -G PT2xl
M AM A outperform s the previous best system by 2.6%6
Results on TAC KBP 23.16 27.07 28.79
29.05 28.07
28.69 29.12
29.72
F1 (%)
O penIE 5.1
Stanford O penIE
M AM A-BERTbase M AM A-BERTlarge
M AM A-G PT2
M AM A-G PT2m edium M AM A-G PT2large M AM A-G PT2xl
Larger/deeper LM s produce KGs of higher quality
M AM A outperform s the previous best system by 2.6%7
Results on TAC KBP 23.16 27.07 28.79
29.05 28.07
28.69 29.12
29.72
F1 (%)
O penIE 5.1
Stan fo rd O penIE
M A M A -BER Tbase
M A M A -BER Tlarge
M A M A -G PT2
M A M A -G PT2m edium M A M A -G PT2large M A M A -G PT2xl
M AM A outperform s the previous best system by 2.6% Larger/deeper LM s produce KG s of higher quality
BERT LM s outperform
G PT-2 LM s under sim ilar m odel sizes
8
Results on W ikidata 16.77
21.22 22.39
F1 (%)
Stanford O penIE
M AM A-BERTlarge M AM A-G PT2xl
M AM
A outperform s the previous
best system by over 5.6% M AM
A is scalable to larger corpora
Larger corpora em bed m ore com plete KGs
9
An O pen KG
Exam ple
Contains know ledge in existing KGs (in blue) Features new know
ledge not in existing KGs (in yellow ) 10
An O pen Q uestion •Assum ption: Language m
odels and know ledge graphs just encode the
sam e w orld know ledge in tw o different form ats, thus the tw o form ats should be equal. But, Is the assum ption TRUE? Language M odels Know ledge Graphs
11
Contributions •Problem : How
to construct know ledge graphs from pre-trained language m odels. •Approach: An unsupervised tw o-stage approach that constructs know ledge graphs w ith a single forw ard pass of the pre-trained language m odels w ithout fine-tuning over the textual corpora (outperform ing com pared m ethods by over 5.6%
in F1 on W ikidata).
•Result: O pen know ledge graphs not only cover the know ledge already in existing know ledge graphs (e.g., W ikidata), but also features open factual know ledge that is new .
12

Review summaries:
This paper presents an unsupervised approach for extracting OpenIE style triples from a corpus. The approach leverages the internal attention maps of pretrained transformers to identify paths which correspond to relations between a head entity and a tail entity. The extracted open triples are then mapped, wherever possible, to an existing KG to create what is referred to as an Open KG.

Paper summary: The paper introduces an unsupervised method that utilizes an off-the-shelf BERT (without any fine-tuning) to create an information extraction system without any training data. It first creates ungrounded triples (a.k.a. OpenIE) from raw text by looking into the attention weights between words and finding a sequence of words through a beam search that has high attention weights between every consecutive words. When such sequence is created, the first and the last word become the head and the tail entities and the words between them becomes the relation. The next step is grounding each triplet to a known Knowledge Graph by utilizing off-the-shelf entity and relation grounding mechanisms. The proposed method shows 1-2% F1 score advantage over Stanford OpenIE on TAC KBP and Wikidata.