Combining Analogy with Language Models for Knowledge Extraction

1 . Introduction . Knowledge bases ( KBs ) have proven to be a very useful resource in many artificial intelligence applications including Dialogue , Question Answering , and Semantic Search [ Bradeško et al. , 2010 , Lukovnikov et al. , 2017 , Ribeiro et al. , 2019 ] . While KBs serve as a good source of explicit knowledge , they often suffer from missing relations and facts . For this reason , extracting knowledge from text with the intent of improving existing KBs has been a longstanding goal in the field of Natural Language Processing . This task is usually referred to as knowledge extraction and is considered challenging because of the inherent diversity and ambiguity of natural language text . In recent years , there has been substantial progress in automated knowledge extraction , but the resources built by these methods are typically limited by at least one of these four constraints : ( 1 ) They lack general world knowledge about common nouns , targeting named entities instead . Examples include NELL [ Carlson et al. , 2010 ] and FreeBase [ Bollacker et al. , 2008 ] ( 2 ) They work with a small set of semantic relations . For instance , WordNet [ Miller , 1998 ] has less than a dozen . ( 3 ) They are supervised in nature , requiring vast amounts of training data . Examples include KBP37 [ Zhang and Wang , 2015 ] and TACRED [ Zhang et al. , 2017 ] . ( 4 ) They represent entities and relations as text strings , which are often ambiguous and harder to manipulate . Examples include ConceptNet [ Speer et al. , 2017 ] , Aristo Tuple KB [ Mishra et al. , 2017 ] , ATOMIC [ Sap et al. , 2019 ] , Dice [ Chalier et al. , 2020 ] , and Ascent [ Nguyen et al. , 2021 ] . By contrast , our goal is to populate an existing ontology , seeded by the facts and schemas already defined in the KB . We use the Companion Cognitive Architecture [ Forbus et al. , 2017 ] which has a general-purpose semantic parser capable of generating semantic representations grounded by relations and entities present in the architecture ’ s KB . In general , KBs containing common-sense world knowledge are incomplete . Consequently , a system has to rely on a small amount of training samples to learn new facts ( in our KB around 94.8 % of relations are associated with less than 100 facts ) . To solve this problem , we propose a novel technique based on analogical Q/A training [ Crouse et al. , 2018 ] . Our approach efficiently creates query cases which are later analogically retrieved and combined to interpret new texts . The system uses distant supervision to automatically generate training data for analogical training which extracts general knowledge from Simple English Wikipedia1 articles . In this work we combine structural similarity with BERT-guided word sense disambiguation and fact classification for few-shot learning of common-sense facts . To the best of our knowledge , we are the first to combine these two techniques . 2 . Related Work . Our work relates to learning structured knowledge from natural language text with the goal of extending a KB without relying on labeled corpora . Such tasks are often classified as information extraction ( IE ) or relation extraction ( RE ) . Learning under distant supervision for RE without labeled data was initially introduced by Mintz et al . [ 2009 ] . They showed how text can be heuristically aligned with a knowledge graph ( KG ) to generate instances used to train a classifier . Subsequent systems incorporated not only syntactic textual features but also jointly learned KG information to perform extraction , either by syntactic-semantic rules from random walks [ Lao et al. , 2012 ] , scoring functions of low-dimensional embeddings [ Weston et al. , 2013 ] , mutual attention between KG and text [ Han et al. , 2018a ] , or neural encoder-decoder models [ Distiawan et al. , 2019 ] . Other recent methods have focused on using neural networks in the context on few-shot relation extraction [ Han et al. , 2018b , Gao et al. , 2019 , Qu et al. , 2020 , Wang et al. , 2021 ] . Such methods have achieved impressive results on RE but focused on resources that either require many hundreds of training examples or mostly extract facts about named entities . The work of Sharma et al . [ 2019 ] is more related to ours as they used distant supervision to learn Cyc style facts [ Lenat and Guha , 1993 ] and were able to leverage the structure of the KB ’ s ontology to improve accuracy . Their experiments however were limited to a small set of relations and they used more shallow textual patterns instead of semantic-parsed interpretation of sentences . To avoid dependency on pre-defined schemas , Open IE [ Yates et al. , 2007 , Fader et al. , 2011 , Angeli et al. , 2015 , Romero et al. , 2019 , Romero and Razniewski , 2020 ] extracts knowledge from text using an open set of relations and entities . Commonly , the relation names are represented as text linking two arguments . This representation leads systems to commonly generate multiple instances of the same relation . Riedel et al . [ 2013 ] tackled the problem of identifying equivalence between extracted relations by learning canonical schemas . Mishra et al . [ 2017 ] used this approach to create a large KB with subject-predicate- 1. http : //simple.wikipedia.org/ object triples of generic statements about the world . The advantage of our approach over Open IE is that we output symbolic , unambiguous representation of knowledge . Textual representations make reasoning harder , and even though some of the predicates are typed , they often suffer from word sense ambiguity . Analogy uses the relational structure from semantic representations of texts as a way of learning abstract concepts without vast amounts of labeled data . This approach has been previously applied to solve many natural language problems such as word sense disambiguation [ Barbella and Forbus , 2013 ] , question answering [ Crouse et al. , 2018 ] , multimodal dialog systems [ Wilson et al. , 2019 ] and process understanding [ Ribeiro et al. , 2019 ] . In our work , analogical similarity is used to retrieve rule-like constructs called query cases . Retrieval of stored cases has been extensively used by many case-based reasoning systems [ Daniels and Rissland , 1995 , Burke et al. , 1997 ] . However , these case-based reasoning approaches differ from analogical learning since they use shallow pattern matching or standard information retrieval methods instead of structural similarity applied to parsed semantics . 3 . Problem Definition . The task is to populate an existing KB with facts extracted from natural language text , constrained by the entities and relations already defined in such KB . The standard relation extraction task represents structured knowledge as graphs [ Weston et al. , 2013 , Zhang et al. , 2020 ] . Relations are edges in the graph and are stored as triples ( h , r , t ) , as in ( Spain , borders , France ) . Here , the knowledge is represented using the CycL higher-order logic and OpenCyc ontology [ Lenat and Guha , 1993 ] , which is more expressive . Further details on CycL can be found in Appendix A . Formally , we denote the knowledge base as KB = ( C , R , F ) , where C is the set of concepts ( in CycL they can be either collections , entities , or logical functions ) , R is the set of relations , and F is the set of facts . Each fact f ∈ F is a nested tuple , represented as a tree data structure , where the root node f1 ∈ R is the relation and each internal node f2 , · · · , fn are the nested arguments . A text corpus is used as input , where each sentence is a sequence of tokens s = t1 , · · · , tm . The final task is to find all arguments for relations r ∈ R ; that express common-sense beliefs extracted from sentences in the corpus . Note that our proposed task is harder than common RE tasks . They often only attempt to infer either r given ( h , ? , t ) or t given ( h , r , ? ) , instead of predicting all arguments given a relation . 4 . Background . We use the Companion Cognitive Architecture as a foundation for our system , as it contains key components that allow for efficient knowledge extraction . Namely , we use the NextKB knowledge base , the CNLU natural language processing system [ Tomai and Forbus , 2009 ] , and analogical capabilities provided by SME , the Structure Mapping Engine [ Forbus et al. , 2017 ] . We briefly explain each of these components below . Knowledge Base : The NextKB 2 knowledge base integrates material from OpenCyc and FrameNet [ Baker et al. , 1998 ] , as well as a large-scale lexicon [ McFate and Forbus , 2011 ] . 2. http : //www.qrg.northwestern.edu/nextkb/index.html All facts in NextKB are represented using the CycL ontology language and they are used as seeds for further inference of missing facts . Natural Language Understanding : The CNLU language system combines Allen ’ s ( 1995 ) bottom-up chart parser with ideas from Discourse Representation Theory [ Kamp and Reyle , 2013 ] to produce semantic representations from the input sentences . Specifically , CNLU produces predicate calculus interpretations represented in CycL language . CNLU is task agnostic and the syntactic parse and word sense ambiguities are explicitly represented as distinct and internally consistent choice sets . Choice sets are often derived from FrameNet semantic frames . Logical constraints among the statements are designed such that by choosing one semantic interpretation , other conflicting choices are ruled out . Analogy : We use the Structure-Mapping Engine ( SME ) by Forbus et al . [ 2017 ] to match stored query cases . This component compares two structured , relational representations and creates one or more mappings between them . These mappings consist of ( 1 ) a set of correspondences , specifying what entities and predicates in one description go with entities and predicates in the other , ( 2 ) a score indicating the similarity between the representations , and ( 3 ) candidate inferences that project information from one description to the other . For more computationally efficient retrieval , our approach uses MAC/FAC [ Forbus et al. , 1995 ] to apply query cases learned during training for extraction of structured knowledge . BERT Language Model : The Bidirectional Encoder Representations from Transformers ( BERT ) [ Devlin et al. , 2018 ] is a language model trained on a large unlabeled corpus ( around 3.3B words from English Wikipedia and BooksCorpus ) that jointly conditions on both left and right context for all layers . We use BERT for word sense disambiguation and classification of extracted facts , as described in the following sections . The first input token is always a special token [ CLS ] which is used for sentence classification while sentence boundaries are marked with the special token [ SEP ] . The final hidden vector of the special [ CLS ] token is defined as TCLS ∈ RH , and the final hidden vector for the ith input token ti as Ti ∈ RH , where H is the hidden state size .

Summary: The paper presents a method for combining the Companion Cognitive Architecture with BERT for the purpose of discovering new Cyc-style common-sense facts. By using BERT at relevant points in the Companion pipeline, the paper achieves a significant performance improvement over an end-to-end T5 model. In particular the BERT network for filtering facts proposed by the cognitive systems results in a huge jump in precision from 46 to 71.