Language Models are Open Knowledge Graphs

1 INTRODUCTION . Knowledge graphs ( KGs ) are an important resource for both humans and machines . Factual knowledge in KGs is injected into AI applications to imitate important skills possessed by humans , e.g. , reasoning and understanding . KG construction is mainly supervised , requiring humans to handwrite every fact , such as Freebase ( Bollacker et al. , 2008 ) and Wikidata . KGs can also be constructed in a semi-supervised way , in which a semi-automatic extractor is used to obtain the facts from web corpora ( e.g. , NELL ( Carlson et al. , 2010 ) and Knowledge Vault ( Dong et al. , 2014 ) ) . Humans however still need to interact with the extractor to improve the quality of the discovered facts . Therefore , human supervision , which is often expensive , is required in constructing KGs . Recent progress in language models ( LMs ) , such as BERT ( Devlin et al. , 2018 ) and GPT-2/3 ( Radford et al. , 2019 ; Brown et al. , 2020 ) , has led to superior results even outperforming humans in a wide range of tasks , e.g. , sentence classification ( Wang et al. , 2018 ) , question answering ( Brown et al. , 2020 ) . Pre-trained LMs are also capable to write poetry , music , and code , while such tasks often require we human to spend a significant amount of time in learning the relevant knowledge to work well . In fact , these pre-trained LMs automatically acquire factual knowledge from large-scale corpora ( e.g. , BookCorpus ( Zhu et al. , 2015 ) , Common Crawl ( Brown et al. , 2020 ) ) via pre-training . The learned knowledge in pre-trained LMs is the key to the current success . We therefore consider the following question : instead of using the manually created knowledge , can we use the knowledge stored in pre-trained LMs to construct KGs ? In this paper , we design an unsupervised approach called MAMA that successfully recovers the factual knowledge stored in LMs to build KGs from scratch . MAMA constructs a KG with a single forward pass of a pre-trained LM ( without fine-tuning ) over a textual corpus . As illustrated in Figure 1 , MAMA has two stages : Match and Map . Match stage generates a set of candidate facts by matching the facts in the textual corpus with the knowledge in the pre-trained LM . General or world knowledge from large-scale corpora is embedded in the LM , thus candidate facts in the target corpus are often covered by the knowledge in the LM . The candidate facts are matched through an efficient beam search in the attention weight matrices of the pre-trained LM without fine-tuning . Map stage produces an open KG via mapping the matched candidate facts from Match stage to both fixed KG schema and open schema . If the schema of candidate facts exists in the KG schema , we map the candidate facts directly to the fixed KG schema . Otherwise , we reserve the unmapped candidate 1 Under review as a conference paper at ICLR 2021 with a single forward pass of the pre-trained language model ( LM ) ( without fine-tuning ) over the corpus . Given the input : a textual corpus containing passages and sentences , e.g. , English Wikipedia , and a pre-trained LM , e.g. , BERT , GPT-2/3 , MAMA ( 1 ) generates a set of candidate facts via matching the knowledge in the pretrained LM with facts in the textual corpus , e.g. , a candidate fact ( Dylan , is , songwriter ) from the sentence “ Dylan is a songwriter. ” , and ( 2 ) produces an open KG by mapping the matched candidate facts to both an existing KG schema , e.g. , ( Bob Dylan.Q392 , occupation.P106 , Songwriter.Q753110 ) in Wikidata schema , and an open schema , e.g. , ( Bob Dylan.Q392 , sign , Albert Grossman.Q708584 ) . facts in the open schema . This results in a new type of KG , open KG , with a mixture of mapped facts in fixed KG schema and unmapped facts in the open schema . Our contributions are as follows : 1 . We show how to construct KGs from pre-trained LMs . The KGs are constructed with a single forward pass of the pre-trained LMs without fine-tuning over the textual corpora . This helps researchers explicitly understand what the language models learn , bridging the deep LM and KG communities through enhanced model transparency . 2 . We propose an unsupervised two-stage approach , MAMA , to first match the candidate facts in the corpora with the knowledge stored in LMs , then map the matched candidate facts to both fixed and open schema to produce a KG . 3 . We generate a new type of KG , namely open KG , consists of mapped facts in the fixed KG schema of existing KGs ( Wikidata and TAC KBP ) annotated by humans ; and unmapped facts in the open schema that are new in the reference KG schema . The reach of this result is broad and has downstream utility for knowledge graph construction , deep neural network interpretation , and information extraction . 2 MAMA . We introduce an unsupervised end-to-end approach Match and Map ( MAMA ) as illustrated in Figure 1 to construct open knowledge graphs ( KGs ) from language models ( LMs ) . MAMA constructs the KGs with a single forward pass of the pre-trained LMs ( without fine-tuning ) over the corpora . The two stages of MAMA are : Match generates a set of candidate facts from a textual corpus . LMs contain global or world knowledge learned from large-scale corpora , which often does not perfectly match the knowledge in the target corpus . The goal of this stage is to match the knowledge stored in pre-trained LMs with facts in the corpus . Each fact is represented as a triplet ( head , relation , tail ) 1 , in short , ( h , r , t ) , and passed to Map stage . Match procedure is detailed in Sec . 2.1 . Map produces an open KG using the matched candidate facts from Match stage . The constructed open KG has two portions : ( a ) mapped candidate facts that are in a fixed KG schema , e.g. , ( Dylan , is , songwriter ) is mapped to ( Bob Dylan.Q392 , occupation.P106 , Songwriter.Q753110 ) according to Wikidata schema ; and ( b ) unmapped candidate facts that are in an open schema , e.g. , a candidate fact ( Dylan , signed , Albert Grossman ) is partially mapped to ( Bob Dylan.Q392 , sign , Albert Grossman.Q708584 ) in the open schema . This stage is described in Sec . 2.2 . 2 Under review as a conference paper at ICLR 2021 the best matched candidate fact ( Dylan , is , songwriter ) from the sentence “ Dylan is a songwriter. ” The lower portion shows the corresponding step-by-step process . Given a head-tail pair ( Dylan , songwriter ) , at each step , the search chooses one of the actions , i.e. , START , YIELD , STOP to produce an intermediate candidate fact . The search starts by adding the head “ Dylan ” as an initial candidate ( step 0 ) . The matching degree of the candidate is initialized as 0 . Next , a new candidate is yielded if the candidate has not reached the tail “ songwriter ” ( step 1 and step 2 ) , by appending the next largest attended token ( with the largest score from the attention matrix ( b ) of the sentence ) to the end of the current candidate , and the corresponding matching degrees are increased by the associated attention scores ( 0.3 and 0.4 ) to 0.3 ( 0+0.3 ) and 0.7 ( 0.3+0.4 ) respectively . Otherwise , the search stops , and the candidate fact with the best matching degree is returned for the head-tail pair ( step 3 ) . The attention matrix ( b ) is from the forward pass of the LM without fine-tuning over the sentence . “ x ” marks the tokens to prevent searching backward . 2.1 MATCH . We frame the matching procedure as a search problem . To obtain the best matched candidate facts of an input sentence , the candidates with the top matching degrees are returned from a search process . The matching degree is derived from the search in the attention weight matrices of the pre-trained LM , since the attention weight matrices are one of the main containers of the knowledge in the pre-trained LM . The attention weight matrices are simply from the forward pass of the LM without fine-tuning over the sentence . 2.1.1 BEAM SEARCH . We design a simple-yet-effective beam search to find the best matched candidate facts . For every head-tail pair ( h , t ) in a sentence , the search maintains the k-best matched candidate facts of the pair . Let ’ s first consider the search from left to right with beam size equals to 1 . An example search process is shown in Figure 2 . Given a head-tail pair ( Dylan , songwriter ) , at each step , the search performs one of the following actions : START the search from the head . The head h is added as an initial candidate into the beam . For simplicity , we use START ( h ) to denote the action , which returns a candidate ( h , . In Figure 2 ( a ) , at step 0 , the head “ Dylan ” is added as ( Dylan , into the beam . The matching degree is initialized to 0 . YIELD a new intermediate candidate in the beam if the current candidate has not reached the tail . The next largest attended token ( with the largest score from the attention matrix ) is appended to the end of the current candidate to yield the new candidate . The corresponding matching degrees are increased by the associated attention scores . At step 1 ( orange arrow in Figure 2 ( a ) ) , “ is ” is appended to the current candidate to yield ( Dylan , is , , since “ is ” has the largest attention score with “ Dylan ” in the attention matrix . The attention score is 0.3 as highlighted in orange in Figure 2 ( b ) . The matching degree becomes 0.3 ( i.e . 0+0.3 ) . The multi-head attention is reduced to a single head so that every two tokens of the sentence are associated with one attention weight . We experiment with different reduction 1 We use the term “ head ” and “ tail ” to denote head and tail ’ s “ entities ” or “ entity mentions ” for simplicity . 3 Under review as a conference paper at ICLR 2021 Algorithm 1 Beam search for matching candidate facts . Input : Head-tail pair ( h , t ) , sentence s , attention matrix As , action manager O = { START , YIELD , STOP } , beam size k Output : Candidate facts T ( h , t ) 1 : T ( h , t ) { START ( h ) } . Start by adding the head as a candidate in the beam 2 : while 9c 2 T ( h , t ) [ O ( c ) = YIELD ] do 3 : eT ( h , t ) ; . Initialize a new beam 4 : for each c 2 T ( h , t ) do 5 : if O ( c ) = YIELD then 6 : eT ( h , t ) eT ( h , t ) [ { YIELD ( c , s , As ) } . Yield a new candidate if not reached the tail 7 : else 8 : eT ( h , t ) eT ( h , t ) [ { STOP ( c , t ) } . Stop then produce a valid fact if reached the tail 9 : end if 10 : end for 11 : T ( h , t ) TOP ( k , eT ( h , t ) ) . Maintain k-best candidates in the beam 12 : end while 13 : return T ( h , t ) setups in Sec . A.3 . “ x ” marks the tokens ( prior to the current token ) that are not considered in the search to prevent searching backward . Step 2 similarly takes YIELD action to produce ( Dylan , is songwriter , . The matching degree is now 0.7 ( i.e . 0.3+0.4 ) . We use YIELD ( c , s , As ) to denote the action , where c is a current candidate , s represents the sentence , and As is the attention matrix from the forward pass of the pre-trained LM over s , which yields a new candidate . STOP the search step if the candidate has reached the tail , then add the candidate as a valid candidate fact into the beam . As beam size equals to 1 , ( Dylan , is , songwriter ) is the only returned candidate fact for the given pair . The final matching degree of the candidate is 0.7 . We denote this step using STOP ( c , t ) , which returns a valid fact . The details of the proposed beam search are in Algorithm 1 . The inputs of the search algorithm are a head-tail pair ( h , t ) , a sentence s , an attention matrix As of s. Both h and t are identified by the noun chunk in s. As is the attention matrix associated with s from the forward pass of LM without fine-tuning . The search gets started by adding the head h as the initial candidate in the beam ( line 1 ) . While there are still new candidates waiting to be yielded ( line 2 ) , the search continues , and the top k candidates sorted by the matching degrees are maintained ( line 3-11 ) in the beam . In practice , we implement an action manager O to decide which action to take at each step . Given a candidate c in the beam , O ( c ) = START always happens at the beginning of the search . If c has not reached the tail t yet , O ( c ) = YIELD . Otherwise , O ( c ) = STOP . We convert the subwords to the corresponding full words . We also notice some facts are in reverse order in the sentence , e.g. , “ · · · said Jason Forcier , a vice president at battery maker A123 Systems Inc. ” for facts of relation “ org : top members employees ” , thus enable bidirectionality by running the algorithm in both directions ( left to right and right to left ) . The beam search is implemented by the breadth-first search , which is efficient as the time complexity is O ( k · d ) , where d is the maximum depth of the search tree .

Summary: This paper presents an unsupervised approach for extracting OpenIE style triples from a corpus. The approach leverages the internal attention maps of pretrained transformers to identify paths which correspond to relations between a head entity and a tail entity. The extracted open triples are then mapped, wherever possible, to an existing KG to create what is referred to as an Open KG.