Fact-driven Logical Reasoning

1 INTRODUCTION . To understand human language , deep neural networks have been widely applied and achieved impressive benchmark results ( Chen et al. , 2016b ; Sachan & Xing , 2016 ; Seo et al. , 2017 ; Dhingra et al. , 2017 ; Cui et al. , 2017 ; Song et al. , 2018 ; Hu et al. , 2019 ; Zhang et al. , 2020a ; Back et al. , 2020 ; Zhang et al. , 2020b ; Hermann et al. , 2015 ) . However , the core requirement among the natural language understanding , logic reasoning , can not be simply solved by the current design philosophy of extracting statistical patterns from data ( Shi et al. , 2020 ) . In order to solve such a problem , there emerges an interest that accounts for human intuition about the entailment of sentences and reflects the semantic relations between sentential constituents ( Iwańska , 1993 ) . In this paper , we focus on logic reasoning in the form of natural language understanding ( NLU ) as logic reasoning may be naturally embodied in such a task and natural language offers sufficient enough clues for effective logic reasoning . In detail , we concern about a logic reasoning question-answering ( QA ) task , where given passage , question , and candidate answer options , the model has to make a proper decision with its logic reasoning ability . There are examples shown in Figure 1 from logic reasoning benchmark datasets , ReClor ( Yu et al. , 2020 ) and LogiQA ( Liu et al. , 2020 ) . Recent neural models usually exploit a pretrained language model ( PrLM ) as a key encoder for effective contextualized representation . According to diagnostic tests ( Ettinger , 2020 ; Rogers et al. , 2020 ) , though PrLMs like BERT ( Devlin et al. , 2019 ) have encoded syntactic and semantic information after large-scale pre-training , they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge , which are fundamental for reasoning whose major challenge is to uncover logical structures , and reason with the candidate options and questions to predict the correct answer . It is difficult for PrLMs to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training . Existing logic reasoning has shown serious dependence on knowledge-like clues . This is due to the lengthy , noisy text in human language which is though a natural carrier of knowledge but does not provide a clean , exact knowledge form . Thus , an increasing interest is in using graph networks to model the entity-aware relationships in the passages ( Yasunaga et al. , 2021 ; Ren & Leskovec , 2020 ; Huang et al. , 2021 ; Krishna et al. , 2020 ; Lv et al. , 2020 ) . However , all these methods may insufficiently capture indispensable logical units from two perspectives . First , they mostly focus on entity-aware commonsense knowledge , but pay little attention to those non-entity , non-commonsense clues ( Zhong et al. , 2021 ) . Second , when existing models extract predicate logic inside language into knowledge , they only exploit quite limited predicates like hasA and isA but ignore a broad range of predicates in real language . From either of the perspectives , the existing methods actually only concern about those `` global '' knowledge that keeps valid across the entire data , without sufficient `` local '' perception of complete facts or events in the given specific part of logic reasoning task . We argue such insufficient modeling on logic units roots from the ignorance of language itself being the complete knowledge/clue carrier . Thus , we propose extracting a kind of broad facts according to backbone constituents of a sentence to effectively cover such indispensable logic reasoning basis , filling the gap of local , non-commonsense , non-entity , or even non-knowledge clues in existing methods as shown in Figure 2 . For example , these units may reflect the facts of who did what to whom , or who is what in Figure 3 . Such groups can be defined as `` fact unit '' following Nakashole & Mitchell ( 2014 ) in Definition 1 . The fact units are further organized into a supergraph following Definition 2 . Definition 1 ( Fact Unit ) Given an triplet T = { E1 , P , E2 } , where E1 and E2 are arguments ( including entity and non-entity ) , P is the predicate , a fact unit F is the set of all entities in T and their corresponding relations . Definition 2 ( Supergraph ) A supergraph is a structure made of fact units ( regarded as subgraphs ) as the vertices , and the relations between fact units as undirected edges . As shown in Figure 2 , we regard the defined fact as the results of syntactic processing , rather than those from semantic role labeling ( SRL ) as in previous study , thus the proposed fact also extends the processing means in existing work . Correspondingly , in this work , we propose a fact-driven logical reasoning model , called FOCAL REASONER , which builds supergraphs on top of fact units as the basis for logic reasoning , to capture both global connections between facts and the local concepts or actions inside the fact . Our model FOCAL REASONER is evaluated on two challenging logic reasoning benchmarks including ReClor , LogiQA , one dialogue reasoning dataset Mutual , for verifying the effectiveness and the generalizability across different domains and question formats . 2 RELATED WORK . Machine Reading Comprehension Recent years have witnessed massive researches on Machine Reading Comprehension ( MRC ) whose goal is training machines to understand human languages , which has become one of the most important areas of NLP ( Chen et al. , 2016b ; Sachan & Xing , 2016 ; Seo et al. , 2017 ; Dhingra et al. , 2017 ; Cui et al. , 2017 ; Song et al. , 2018 ; Hu et al. , 2019 ; Zhang et al. , 2020a ; Back et al. , 2020 ; Zhang et al. , 2020b ) . Despite the success of MRC models on various datasets such as CNN/Daily Mail ( Hermann et al. , 2015 ) , SQuAD ( Rajpurkar et al. , 2016 ) , RACE ( Lai et al. , 2017 ) and so on , researchers began to rethink what extent does the problem been solved . Nowadays , there are massive researches into the reasoning ability of machines . According to ( Kaushik & Lipton , 2018 ; Zhou et al. , 2020 ; Chen et al. , 2016a ) , reasoning abilities can be broadly categorized into ( 1 ) commonsense reasoning ( Davis & Marcus , 2015 ; Bhagavatula et al. , 2019 ; Talmor et al. , 2019 ; Huang et al. , 2019 ) ; ( 2 ) numerical reasoning ( Dua et al. , 2019 ) ; ( 3 ) multi-hop reasoning ( Yang et al. , 2018 ) and ( 4 ) logic reasoning ( Yu et al. , 2020 ; Liu et al. , 2020 ) , among which logic reasoning is essential in human intelligence but has merely been delved into . Natural Language Inference ( NLI ) ( Bowman et al. , 2015 ; Williams et al. , 2018 ; Nie et al. , 2020 ) is a task closely related to logic reasoning . However , it has two obvious drawbacks in measuring logic reasoning abilities . One is that it only has three logical types which are entailment , contradiction and neutral . The other is its limitation on sentence-level reasoning . Hence , it is important to research more comprehensive and deeper logic reasoning abilities . logic reasoning in MRC There are two main kinds of features in language data that would be the necessary basis for logic reasoning : 1 ) knowledge : global facts that keep consistency regardless of the context , such as commonsense , mostly derived from named entities ; 2 ) non-knowledge : local facts or events that may be sensitive to the context , mostly derived from detailed language . Existing works have made progress in improving logic reasoning ability ( Yasunaga et al. , 2021 ; Ren & Leskovec , 2020 ; Huang et al. , 2021 ; Krishna et al. , 2020 ; Zhong et al. , 2021 ; Wang et al. , 2021 ) . However , these approaches are barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations , which are obviously not sufficient . In this work , we strengthen the basis for logic reasoning by unifying both types of the features as `` facts '' . Different from previous studies that focus on the knowledge components , we propose a fact-driven logic reasoning framework that builds supergraphs on top of fact units to capture both global connections between entityaware facts and the local concepts or events inside the fact . 3 METHODOLOGY . In this section , we present a fact-driven approach for logic reasoning and the overall architecture of the model is shown in Figure 4 . The framework can be divided into three steps as following . We first extract fact units from raw texts via syntactic processing to construct a supergraph . Then it performs reasoning over the supergraph along with a logical fact regularization . Finally , it aggregates the learned representation to decode for the right answer . 3.1 FACT UNIT EXTRACTION AND SUPERGRAPH CONSTRUCTION . Fact Unit Extraction . Figure 5 illustrates our method for constructing a supergraph from raw text inputs . The first step is to obtain triplets that constitute a fact unit . To keep the framework generic , we use a fairly simple fact unit extractor based on the syntactic relations . Given a context consisting of multiple sentences , we first conduct dependency parsing of each sentence . After that , we extract the subject , the predicate , and the object tokens to get the `` Argument-Predicate-Argument '' triplets corresponding to each sentence in the context . Supergraph Construction . With the obtained triplets , the fact units are organized in the form of Levi graph ( Levi , 1942 ) , which turns arguments and predicates all into nodes . An original fact unit is in the form of F = ( V , E , R ) , where V is the set of the arguments , E is the set of edges connected between arguments , and R is the relations of each edge which are predicates here . The corresponding Levi graph is denoted as Fl = ( VL , EL , RL ) where VL = V ∪ R , which makes the originally directly connected arguments be intermediately connected via relations . As for RL , previous works such as ( Marcheggiani & Titov , 2017 ; Beck et al. , 2018a ) designed three types of edges RL = { default , reverse , self } to enhance information flow . Here in our settings , we extend it into five types : default-in , default-out , reverse-in , reverse-out , self , corresponding to the directions of edges towards the predicates . Detailed description for edge types can be found in Appendix A . We construct the supergraph by making connections between fact units Fl . In particular , we take three strategies according to global information , identical concept and co-reference information . ( 1 ) We add a node Vg initialized with the question-option representation and connect it to all the fact unit nodes . The edge type is set as aggregate for better information interaction . ( 2 ) There can be identical mentions in different sentences , resulting in repeated nodes in fact units . We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same . ( 3 ) We conduct co-reference resolution on context using an off-to-shelf model1 in order to identify arguments in fact units that refer to the same one . We add edges with type coref between them . The final supergraph is denoted as S = ( Fl ∪ Vg , E ) where E is the set of edges added with the previous three strategies .

Summary: The authors propose a framework, called Focal Reasoner, to perform logical reasoning to answer questions. The proposed approach first extracts the facts from raw text. The authors describe a fact unit as a triple of (argument, predicate, argument). These collections of facts can be viewed as a graph with predicates as undirected edges between arguments. After forming a subgraph, reasoning is performed by a graph convolution module to predict the correct answer.  The authors show that their approach outperforms existing approaches on two benchmarked datasets. The authors also perform a detailed analysis that sheds more light on the internal workings of the model.