Language Models are Open Knowledge Graphs

1 INTRODUCTION . Knowledge graphs ( KGs ) are an important resource for both humans and machines . Factual knowledge in KGs is injected into AI applications to imitate important skills possessed by humans , e.g. , reasoning and understanding . KG construction is mainly supervised , requiring humans to handwrite every fact , such as Freebase ( Bollacker et al. , 2008 ) and Wikidata . KGs can also be constructed in a semi-supervised way , in which a semi-automatic extractor is used to obtain the facts from web corpora ( e.g. , NELL ( Carlson et al. , 2010 ) and Knowledge Vault ( Dong et al. , 2014 ) ) . Humans however still need to interact with the extractor to improve the quality of the discovered facts . Therefore , human supervision , which is often expensive , is required in constructing KGs . Recent progress in language models ( LMs ) , such as BERT ( Devlin et al. , 2018 ) and GPT-2/3 ( Radford et al. , 2019 ; Brown et al. , 2020 ) , has led to superior results even outperforming humans in a wide range of tasks , e.g. , sentence classification ( Wang et al. , 2018 ) , question answering ( Brown et al. , 2020 ) . Pre-trained LMs are also capable to write poetry , music , and code , while such tasks often require we human to spend a significant amount of time in learning the relevant knowledge to work well . In fact , these pre-trained LMs automatically acquire factual knowledge from large-scale corpora ( e.g. , BookCorpus ( Zhu et al. , 2015 ) , Common Crawl ( Brown et al. , 2020 ) ) via pre-training . The learned knowledge in pre-trained LMs is the key to the current success . We therefore consider the following question : instead of using the manually created knowledge , can we use the knowledge stored in pre-trained LMs to construct KGs ? In this paper , we design an unsupervised approach called MAMA that successfully recovers the factual knowledge stored in LMs to build KGs from scratch . MAMA constructs a KG with a single forward pass of a pre-trained LM ( without fine-tuning ) over a textual corpus . As illustrated in Figure 1 , MAMA has two stages : Match and Map . Match stage generates a set of candidate facts by matching the facts in the textual corpus with the knowledge in the pre-trained LM . General or world knowledge from large-scale corpora is embedded in the LM , thus candidate facts in the target corpus are often covered by the knowledge in the LM . The candidate facts are matched through an efficient beam search in the attention weight matrices of the pre-trained LM without fine-tuning . Map stage produces an open KG via mapping the matched candidate facts from Match stage to both fixed KG schema and open schema . If the schema of candidate facts exists in the KG schema , we map the candidate facts directly to the fixed KG schema . Otherwise , we reserve the unmapped candidate 1 Under review as a conference paper at ICLR 2021 with a single forward pass of the pre-trained language model ( LM ) ( without fine-tuning ) over the corpus . Given the input : a textual corpus containing passages and sentences , e.g. , English Wikipedia , and a pre-trained LM , e.g. , BERT , GPT-2/3 , MAMA ( 1 ) generates a set of candidate facts via matching the knowledge in the pretrained LM with facts in the textual corpus , e.g. , a candidate fact ( Dylan , is , songwriter ) from the sentence “ Dylan is a songwriter. ” , and ( 2 ) produces an open KG by mapping the matched candidate facts to both an existing KG schema , e.g. , ( Bob Dylan.Q392 , occupation.P106 , Songwriter.Q753110 ) in Wikidata schema , and an open schema , e.g. , ( Bob Dylan.Q392 , sign , Albert Grossman.Q708584 ) . facts in the open schema . This results in a new type of KG , open KG , with a mixture of mapped facts in fixed KG schema and unmapped facts in the open schema . Our contributions are as follows : 1 . We show how to construct KGs from pre-trained LMs . The KGs are constructed with a single forward pass of the pre-trained LMs without fine-tuning over the textual corpora . This helps researchers explicitly understand what the language models learn , bridging the deep LM and KG communities through enhanced model transparency . 2 . We propose an unsupervised two-stage approach , MAMA , to first match the candidate facts in the corpora with the knowledge stored in LMs , then map the matched candidate facts to both fixed and open schema to produce a KG . 3 . We generate a new type of KG , namely open KG , consists of mapped facts in the fixed KG schema of existing KGs ( Wikidata and TAC KBP ) annotated by humans ; and unmapped facts in the open schema that are new in the reference KG schema . The reach of this result is broad and has downstream utility for knowledge graph construction , deep neural network interpretation , and information extraction . 2 MAMA . We introduce an unsupervised end-to-end approach Match and Map ( MAMA ) as illustrated in Figure 1 to construct open knowledge graphs ( KGs ) from language models ( LMs ) . MAMA constructs the KGs with a single forward pass of the pre-trained LMs ( without fine-tuning ) over the corpora . The two stages of MAMA are : Match generates a set of candidate facts from a textual corpus . LMs contain global or world knowledge learned from large-scale corpora , which often does not perfectly match the knowledge in the target corpus . The goal of this stage is to match the knowledge stored in pre-trained LMs with facts in the corpus . Each fact is represented as a triplet ( head , relation , tail ) 1 , in short , ( h , r , t ) , and passed to Map stage . Match procedure is detailed in Sec . 2.1 . Map produces an open KG using the matched candidate facts from Match stage . The constructed open KG has two portions : ( a ) mapped candidate facts that are in a fixed KG schema , e.g. , ( Dylan , is , songwriter ) is mapped to ( Bob Dylan.Q392 , occupation.P106 , Songwriter.Q753110 ) according to Wikidata schema ; and ( b ) unmapped candidate facts that are in an open schema , e.g. , a candidate fact ( Dylan , signed , Albert Grossman ) is partially mapped to ( Bob Dylan.Q392 , sign , Albert Grossman.Q708584 ) in the open schema . This stage is described in Sec . 2.2 . 2 Under review as a conference paper at ICLR 2021 the best matched candidate fact ( Dylan , is , songwriter ) from the sentence “ Dylan is a songwriter. ” The lower portion shows the corresponding step-by-step process . Given a head-tail pair ( Dylan , songwriter ) , at each step , the search chooses one of the actions , i.e. , START , YIELD , STOP to produce an intermediate candidate fact . The search starts by adding the head “ Dylan ” as an initial candidate ( step 0 ) . The matching degree of the candidate is initialized as 0 . Next , a new candidate is yielded if the candidate has not reached the tail “ songwriter ” ( step 1 and step 2 ) , by appending the next largest attended token ( with the largest score from the attention matrix ( b ) of the sentence ) to the end of the current candidate , and the corresponding matching degrees are increased by the associated attention scores ( 0.3 and 0.4 ) to 0.3 ( 0+0.3 ) and 0.7 ( 0.3+0.4 ) respectively . Otherwise , the search stops , and the candidate fact with the best matching degree is returned for the head-tail pair ( step 3 ) . The attention matrix ( b ) is from the forward pass of the LM without fine-tuning over the sentence . “ x ” marks the tokens to prevent searching backward . 2.1 MATCH . We frame the matching procedure as a search problem . To obtain the best matched candidate facts of an input sentence , the candidates with the top matching degrees are returned from a search process . The matching degree is derived from the search in the attention weight matrices of the pre-trained LM , since the attention weight matrices are one of the main containers of the knowledge in the pre-trained LM . The attention weight matrices are simply from the forward pass of the LM without fine-tuning over the sentence . 2.1.1 BEAM SEARCH . We design a simple-yet-effective beam search to find the best matched candidate facts . For every head-tail pair ( h , t ) in a sentence , the search maintains the k-best matched candidate facts of the pair . Let ’ s first consider the search from left to right with beam size equals to 1 . An example search process is shown in Figure 2 . Given a head-tail pair ( Dylan , songwriter ) , at each step , the search performs one of the following actions : START the search from the head . The head h is added as an initial candidate into the beam . For simplicity , we use START ( h ) to denote the action , which returns a candidate ( h , . In Figure 2 ( a ) , at step 0 , the head “ Dylan ” is added as ( Dylan , into the beam . The matching degree is initialized to 0 . YIELD a new intermediate candidate in the beam if the current candidate has not reached the tail . The next largest attended token ( with the largest score from the attention matrix ) is appended to the end of the current candidate to yield the new candidate . The corresponding matching degrees are increased by the associated attention scores . At step 1 ( orange arrow in Figure 2 ( a ) ) , “ is ” is appended to the current candidate to yield ( Dylan , is , , since “ is ” has the largest attention score with “ Dylan ” in the attention matrix . The attention score is 0.3 as highlighted in orange in Figure 2 ( b ) . The matching degree becomes 0.3 ( i.e . 0+0.3 ) . The multi-head attention is reduced to a single head so that every two tokens of the sentence are associated with one attention weight . We experiment with different reduction 1 We use the term “ head ” and “ tail ” to denote head and tail ’ s “ entities ” or “ entity mentions ” for simplicity . 3 Under review as a conference paper at ICLR 2021 Algorithm 1 Beam search for matching candidate facts . Input : Head-tail pair ( h , t ) , sentence s , attention matrix As , action manager O = { START , YIELD , STOP } , beam size k Output : Candidate facts T ( h , t ) 1 : T ( h , t ) { START ( h ) } . Start by adding the head as a candidate in the beam 2 : while 9c 2 T ( h , t ) [ O ( c ) = YIELD ] do 3 : eT ( h , t ) ; . Initialize a new beam 4 : for each c 2 T ( h , t ) do 5 : if O ( c ) = YIELD then 6 : eT ( h , t ) eT ( h , t ) [ { YIELD ( c , s , As ) } . Yield a new candidate if not reached the tail 7 : else 8 : eT ( h , t ) eT ( h , t ) [ { STOP ( c , t ) } . Stop then produce a valid fact if reached the tail 9 : end if 10 : end for 11 : T ( h , t ) TOP ( k , eT ( h , t ) ) . Maintain k-best candidates in the beam 12 : end while 13 : return T ( h , t ) setups in Sec . A.3 . “ x ” marks the tokens ( prior to the current token ) that are not considered in the search to prevent searching backward . Step 2 similarly takes YIELD action to produce ( Dylan , is songwriter , . The matching degree is now 0.7 ( i.e . 0.3+0.4 ) . We use YIELD ( c , s , As ) to denote the action , where c is a current candidate , s represents the sentence , and As is the attention matrix from the forward pass of the pre-trained LM over s , which yields a new candidate . STOP the search step if the candidate has reached the tail , then add the candidate as a valid candidate fact into the beam . As beam size equals to 1 , ( Dylan , is , songwriter ) is the only returned candidate fact for the given pair . The final matching degree of the candidate is 0.7 . We denote this step using STOP ( c , t ) , which returns a valid fact . The details of the proposed beam search are in Algorithm 1 . The inputs of the search algorithm are a head-tail pair ( h , t ) , a sentence s , an attention matrix As of s. Both h and t are identified by the noun chunk in s. As is the attention matrix associated with s from the forward pass of LM without fine-tuning . The search gets started by adding the head h as the initial candidate in the beam ( line 1 ) . While there are still new candidates waiting to be yielded ( line 2 ) , the search continues , and the top k candidates sorted by the matching degrees are maintained ( line 3-11 ) in the beam . In practice , we implement an action manager O to decide which action to take at each step . Given a candidate c in the beam , O ( c ) = START always happens at the beginning of the search . If c has not reached the tail t yet , O ( c ) = YIELD . Otherwise , O ( c ) = STOP . We convert the subwords to the corresponding full words . We also notice some facts are in reverse order in the sentence , e.g. , “ · · · said Jason Forcier , a vice president at battery maker A123 Systems Inc. ” for facts of relation “ org : top members employees ” , thus enable bidirectionality by running the algorithm in both directions ( left to right and right to left ) . The beam search is implemented by the breadth-first search , which is efficient as the time complexity is O ( k · d ) , where d is the maximum depth of the search tree .

Summary: Paper summary: The paper introduces an unsupervised method that utilizes an off-the-shelf BERT (without any fine-tuning) to create an information extraction system without any training data. It first creates ungrounded triples (a.k.a. OpenIE) from raw text by looking into the attention weights between words and finding a sequence of words through a beam search that has high attention weights between every consecutive words. When such sequence is created, the first and the last word become the head and the tail entities and the words between them becomes the relation. The next step is grounding each triplet to a known Knowledge Graph by utilizing off-the-shelf entity and relation grounding mechanisms. The proposed method shows 1-2% F1 score advantage over Stanford OpenIE on TAC KBP and Wikidata.