Combining Analogy with Language Models for Knowledge Extraction

1 . Introduction . Knowledge bases ( KBs ) have proven to be a very useful resource in many artificial intelligence applications including Dialogue , Question Answering , and Semantic Search [ Bradeško et al. , 2010 , Lukovnikov et al. , 2017 , Ribeiro et al. , 2019 ] . While KBs serve as a good source of explicit knowledge , they often suffer from missing relations and facts . For this reason , extracting knowledge from text with the intent of improving existing KBs has been a longstanding goal in the field of Natural Language Processing . This task is usually referred to as knowledge extraction and is considered challenging because of the inherent diversity and ambiguity of natural language text . In recent years , there has been substantial progress in automated knowledge extraction , but the resources built by these methods are typically limited by at least one of these four constraints : ( 1 ) They lack general world knowledge about common nouns , targeting named entities instead . Examples include NELL [ Carlson et al. , 2010 ] and FreeBase [ Bollacker et al. , 2008 ] ( 2 ) They work with a small set of semantic relations . For instance , WordNet [ Miller , 1998 ] has less than a dozen . ( 3 ) They are supervised in nature , requiring vast amounts of training data . Examples include KBP37 [ Zhang and Wang , 2015 ] and TACRED [ Zhang et al. , 2017 ] . ( 4 ) They represent entities and relations as text strings , which are often ambiguous and harder to manipulate . Examples include ConceptNet [ Speer et al. , 2017 ] , Aristo Tuple KB [ Mishra et al. , 2017 ] , ATOMIC [ Sap et al. , 2019 ] , Dice [ Chalier et al. , 2020 ] , and Ascent [ Nguyen et al. , 2021 ] . By contrast , our goal is to populate an existing ontology , seeded by the facts and schemas already defined in the KB . We use the Companion Cognitive Architecture [ Forbus et al. , 2017 ] which has a general-purpose semantic parser capable of generating semantic representations grounded by relations and entities present in the architecture ’ s KB . In general , KBs containing common-sense world knowledge are incomplete . Consequently , a system has to rely on a small amount of training samples to learn new facts ( in our KB around 94.8 % of relations are associated with less than 100 facts ) . To solve this problem , we propose a novel technique based on analogical Q/A training [ Crouse et al. , 2018 ] . Our approach efficiently creates query cases which are later analogically retrieved and combined to interpret new texts . The system uses distant supervision to automatically generate training data for analogical training which extracts general knowledge from Simple English Wikipedia1 articles . In this work we combine structural similarity with BERT-guided word sense disambiguation and fact classification for few-shot learning of common-sense facts . To the best of our knowledge , we are the first to combine these two techniques . 2 . Related Work . Our work relates to learning structured knowledge from natural language text with the goal of extending a KB without relying on labeled corpora . Such tasks are often classified as information extraction ( IE ) or relation extraction ( RE ) . Learning under distant supervision for RE without labeled data was initially introduced by Mintz et al . [ 2009 ] . They showed how text can be heuristically aligned with a knowledge graph ( KG ) to generate instances used to train a classifier . Subsequent systems incorporated not only syntactic textual features but also jointly learned KG information to perform extraction , either by syntactic-semantic rules from random walks [ Lao et al. , 2012 ] , scoring functions of low-dimensional embeddings [ Weston et al. , 2013 ] , mutual attention between KG and text [ Han et al. , 2018a ] , or neural encoder-decoder models [ Distiawan et al. , 2019 ] . Other recent methods have focused on using neural networks in the context on few-shot relation extraction [ Han et al. , 2018b , Gao et al. , 2019 , Qu et al. , 2020 , Wang et al. , 2021 ] . Such methods have achieved impressive results on RE but focused on resources that either require many hundreds of training examples or mostly extract facts about named entities . The work of Sharma et al . [ 2019 ] is more related to ours as they used distant supervision to learn Cyc style facts [ Lenat and Guha , 1993 ] and were able to leverage the structure of the KB ’ s ontology to improve accuracy . Their experiments however were limited to a small set of relations and they used more shallow textual patterns instead of semantic-parsed interpretation of sentences . To avoid dependency on pre-defined schemas , Open IE [ Yates et al. , 2007 , Fader et al. , 2011 , Angeli et al. , 2015 , Romero et al. , 2019 , Romero and Razniewski , 2020 ] extracts knowledge from text using an open set of relations and entities . Commonly , the relation names are represented as text linking two arguments . This representation leads systems to commonly generate multiple instances of the same relation . Riedel et al . [ 2013 ] tackled the problem of identifying equivalence between extracted relations by learning canonical schemas . Mishra et al . [ 2017 ] used this approach to create a large KB with subject-predicate- 1. http : //simple.wikipedia.org/ object triples of generic statements about the world . The advantage of our approach over Open IE is that we output symbolic , unambiguous representation of knowledge . Textual representations make reasoning harder , and even though some of the predicates are typed , they often suffer from word sense ambiguity . Analogy uses the relational structure from semantic representations of texts as a way of learning abstract concepts without vast amounts of labeled data . This approach has been previously applied to solve many natural language problems such as word sense disambiguation [ Barbella and Forbus , 2013 ] , question answering [ Crouse et al. , 2018 ] , multimodal dialog systems [ Wilson et al. , 2019 ] and process understanding [ Ribeiro et al. , 2019 ] . In our work , analogical similarity is used to retrieve rule-like constructs called query cases . Retrieval of stored cases has been extensively used by many case-based reasoning systems [ Daniels and Rissland , 1995 , Burke et al. , 1997 ] . However , these case-based reasoning approaches differ from analogical learning since they use shallow pattern matching or standard information retrieval methods instead of structural similarity applied to parsed semantics . 3 . Problem Definition . The task is to populate an existing KB with facts extracted from natural language text , constrained by the entities and relations already defined in such KB . The standard relation extraction task represents structured knowledge as graphs [ Weston et al. , 2013 , Zhang et al. , 2020 ] . Relations are edges in the graph and are stored as triples ( h , r , t ) , as in ( Spain , borders , France ) . Here , the knowledge is represented using the CycL higher-order logic and OpenCyc ontology [ Lenat and Guha , 1993 ] , which is more expressive . Further details on CycL can be found in Appendix A . Formally , we denote the knowledge base as KB = ( C , R , F ) , where C is the set of concepts ( in CycL they can be either collections , entities , or logical functions ) , R is the set of relations , and F is the set of facts . Each fact f ∈ F is a nested tuple , represented as a tree data structure , where the root node f1 ∈ R is the relation and each internal node f2 , · · · , fn are the nested arguments . A text corpus is used as input , where each sentence is a sequence of tokens s = t1 , · · · , tm . The final task is to find all arguments for relations r ∈ R ; that express common-sense beliefs extracted from sentences in the corpus . Note that our proposed task is harder than common RE tasks . They often only attempt to infer either r given ( h , ? , t ) or t given ( h , r , ? ) , instead of predicting all arguments given a relation . 4 . Background . We use the Companion Cognitive Architecture as a foundation for our system , as it contains key components that allow for efficient knowledge extraction . Namely , we use the NextKB knowledge base , the CNLU natural language processing system [ Tomai and Forbus , 2009 ] , and analogical capabilities provided by SME , the Structure Mapping Engine [ Forbus et al. , 2017 ] . We briefly explain each of these components below . Knowledge Base : The NextKB 2 knowledge base integrates material from OpenCyc and FrameNet [ Baker et al. , 1998 ] , as well as a large-scale lexicon [ McFate and Forbus , 2011 ] . 2. http : //www.qrg.northwestern.edu/nextkb/index.html All facts in NextKB are represented using the CycL ontology language and they are used as seeds for further inference of missing facts . Natural Language Understanding : The CNLU language system combines Allen ’ s ( 1995 ) bottom-up chart parser with ideas from Discourse Representation Theory [ Kamp and Reyle , 2013 ] to produce semantic representations from the input sentences . Specifically , CNLU produces predicate calculus interpretations represented in CycL language . CNLU is task agnostic and the syntactic parse and word sense ambiguities are explicitly represented as distinct and internally consistent choice sets . Choice sets are often derived from FrameNet semantic frames . Logical constraints among the statements are designed such that by choosing one semantic interpretation , other conflicting choices are ruled out . Analogy : We use the Structure-Mapping Engine ( SME ) by Forbus et al . [ 2017 ] to match stored query cases . This component compares two structured , relational representations and creates one or more mappings between them . These mappings consist of ( 1 ) a set of correspondences , specifying what entities and predicates in one description go with entities and predicates in the other , ( 2 ) a score indicating the similarity between the representations , and ( 3 ) candidate inferences that project information from one description to the other . For more computationally efficient retrieval , our approach uses MAC/FAC [ Forbus et al. , 1995 ] to apply query cases learned during training for extraction of structured knowledge . BERT Language Model : The Bidirectional Encoder Representations from Transformers ( BERT ) [ Devlin et al. , 2018 ] is a language model trained on a large unlabeled corpus ( around 3.3B words from English Wikipedia and BooksCorpus ) that jointly conditions on both left and right context for all layers . We use BERT for word sense disambiguation and classification of extracted facts , as described in the following sections . The first input token is always a special token [ CLS ] which is used for sentence classification while sentence boundaries are marked with the special token [ SEP ] . The final hidden vector of the special [ CLS ] token is defined as TCLS ∈ RH , and the final hidden vector for the ith input token ti as Ti ∈ RH , where H is the hidden state size .

Summary: This paper presents an approach for augmenting a knowledge base with general knowledge facts by extracting them from text. The system parses new sentences, converts them into semantic relations, ties them to the existing knowledge graph to compute a connectivity with the current knowledge base, and uses BERT to evaluate plausibility. An experiment shows that the precision of the proposed system in extracting facts is quite high, higher than the baselines.