Fact-driven Logical Reasoning

1 INTRODUCTION . To understand human language , deep neural networks have been widely applied and achieved impressive benchmark results ( Chen et al. , 2016b ; Sachan & Xing , 2016 ; Seo et al. , 2017 ; Dhingra et al. , 2017 ; Cui et al. , 2017 ; Song et al. , 2018 ; Hu et al. , 2019 ; Zhang et al. , 2020a ; Back et al. , 2020 ; Zhang et al. , 2020b ; Hermann et al. , 2015 ) . However , the core requirement among the natural language understanding , logic reasoning , can not be simply solved by the current design philosophy of extracting statistical patterns from data ( Shi et al. , 2020 ) . In order to solve such a problem , there emerges an interest that accounts for human intuition about the entailment of sentences and reflects the semantic relations between sentential constituents ( Iwańska , 1993 ) . In this paper , we focus on logic reasoning in the form of natural language understanding ( NLU ) as logic reasoning may be naturally embodied in such a task and natural language offers sufficient enough clues for effective logic reasoning . In detail , we concern about a logic reasoning question-answering ( QA ) task , where given passage , question , and candidate answer options , the model has to make a proper decision with its logic reasoning ability . There are examples shown in Figure 1 from logic reasoning benchmark datasets , ReClor ( Yu et al. , 2020 ) and LogiQA ( Liu et al. , 2020 ) . Recent neural models usually exploit a pretrained language model ( PrLM ) as a key encoder for effective contextualized representation . According to diagnostic tests ( Ettinger , 2020 ; Rogers et al. , 2020 ) , though PrLMs like BERT ( Devlin et al. , 2019 ) have encoded syntactic and semantic information after large-scale pre-training , they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge , which are fundamental for reasoning whose major challenge is to uncover logical structures , and reason with the candidate options and questions to predict the correct answer . It is difficult for PrLMs to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training . Existing logic reasoning has shown serious dependence on knowledge-like clues . This is due to the lengthy , noisy text in human language which is though a natural carrier of knowledge but does not provide a clean , exact knowledge form . Thus , an increasing interest is in using graph networks to model the entity-aware relationships in the passages ( Yasunaga et al. , 2021 ; Ren & Leskovec , 2020 ; Huang et al. , 2021 ; Krishna et al. , 2020 ; Lv et al. , 2020 ) . However , all these methods may insufficiently capture indispensable logical units from two perspectives . First , they mostly focus on entity-aware commonsense knowledge , but pay little attention to those non-entity , non-commonsense clues ( Zhong et al. , 2021 ) . Second , when existing models extract predicate logic inside language into knowledge , they only exploit quite limited predicates like hasA and isA but ignore a broad range of predicates in real language . From either of the perspectives , the existing methods actually only concern about those `` global '' knowledge that keeps valid across the entire data , without sufficient `` local '' perception of complete facts or events in the given specific part of logic reasoning task . We argue such insufficient modeling on logic units roots from the ignorance of language itself being the complete knowledge/clue carrier . Thus , we propose extracting a kind of broad facts according to backbone constituents of a sentence to effectively cover such indispensable logic reasoning basis , filling the gap of local , non-commonsense , non-entity , or even non-knowledge clues in existing methods as shown in Figure 2 . For example , these units may reflect the facts of who did what to whom , or who is what in Figure 3 . Such groups can be defined as `` fact unit '' following Nakashole & Mitchell ( 2014 ) in Definition 1 . The fact units are further organized into a supergraph following Definition 2 . Definition 1 ( Fact Unit ) Given an triplet T = { E1 , P , E2 } , where E1 and E2 are arguments ( including entity and non-entity ) , P is the predicate , a fact unit F is the set of all entities in T and their corresponding relations . Definition 2 ( Supergraph ) A supergraph is a structure made of fact units ( regarded as subgraphs ) as the vertices , and the relations between fact units as undirected edges . As shown in Figure 2 , we regard the defined fact as the results of syntactic processing , rather than those from semantic role labeling ( SRL ) as in previous study , thus the proposed fact also extends the processing means in existing work . Correspondingly , in this work , we propose a fact-driven logical reasoning model , called FOCAL REASONER , which builds supergraphs on top of fact units as the basis for logic reasoning , to capture both global connections between facts and the local concepts or actions inside the fact . Our model FOCAL REASONER is evaluated on two challenging logic reasoning benchmarks including ReClor , LogiQA , one dialogue reasoning dataset Mutual , for verifying the effectiveness and the generalizability across different domains and question formats . 2 RELATED WORK . Machine Reading Comprehension Recent years have witnessed massive researches on Machine Reading Comprehension ( MRC ) whose goal is training machines to understand human languages , which has become one of the most important areas of NLP ( Chen et al. , 2016b ; Sachan & Xing , 2016 ; Seo et al. , 2017 ; Dhingra et al. , 2017 ; Cui et al. , 2017 ; Song et al. , 2018 ; Hu et al. , 2019 ; Zhang et al. , 2020a ; Back et al. , 2020 ; Zhang et al. , 2020b ) . Despite the success of MRC models on various datasets such as CNN/Daily Mail ( Hermann et al. , 2015 ) , SQuAD ( Rajpurkar et al. , 2016 ) , RACE ( Lai et al. , 2017 ) and so on , researchers began to rethink what extent does the problem been solved . Nowadays , there are massive researches into the reasoning ability of machines . According to ( Kaushik & Lipton , 2018 ; Zhou et al. , 2020 ; Chen et al. , 2016a ) , reasoning abilities can be broadly categorized into ( 1 ) commonsense reasoning ( Davis & Marcus , 2015 ; Bhagavatula et al. , 2019 ; Talmor et al. , 2019 ; Huang et al. , 2019 ) ; ( 2 ) numerical reasoning ( Dua et al. , 2019 ) ; ( 3 ) multi-hop reasoning ( Yang et al. , 2018 ) and ( 4 ) logic reasoning ( Yu et al. , 2020 ; Liu et al. , 2020 ) , among which logic reasoning is essential in human intelligence but has merely been delved into . Natural Language Inference ( NLI ) ( Bowman et al. , 2015 ; Williams et al. , 2018 ; Nie et al. , 2020 ) is a task closely related to logic reasoning . However , it has two obvious drawbacks in measuring logic reasoning abilities . One is that it only has three logical types which are entailment , contradiction and neutral . The other is its limitation on sentence-level reasoning . Hence , it is important to research more comprehensive and deeper logic reasoning abilities . logic reasoning in MRC There are two main kinds of features in language data that would be the necessary basis for logic reasoning : 1 ) knowledge : global facts that keep consistency regardless of the context , such as commonsense , mostly derived from named entities ; 2 ) non-knowledge : local facts or events that may be sensitive to the context , mostly derived from detailed language . Existing works have made progress in improving logic reasoning ability ( Yasunaga et al. , 2021 ; Ren & Leskovec , 2020 ; Huang et al. , 2021 ; Krishna et al. , 2020 ; Zhong et al. , 2021 ; Wang et al. , 2021 ) . However , these approaches are barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations , which are obviously not sufficient . In this work , we strengthen the basis for logic reasoning by unifying both types of the features as `` facts '' . Different from previous studies that focus on the knowledge components , we propose a fact-driven logic reasoning framework that builds supergraphs on top of fact units to capture both global connections between entityaware facts and the local concepts or events inside the fact . 3 METHODOLOGY . In this section , we present a fact-driven approach for logic reasoning and the overall architecture of the model is shown in Figure 4 . The framework can be divided into three steps as following . We first extract fact units from raw texts via syntactic processing to construct a supergraph . Then it performs reasoning over the supergraph along with a logical fact regularization . Finally , it aggregates the learned representation to decode for the right answer . 3.1 FACT UNIT EXTRACTION AND SUPERGRAPH CONSTRUCTION . Fact Unit Extraction . Figure 5 illustrates our method for constructing a supergraph from raw text inputs . The first step is to obtain triplets that constitute a fact unit . To keep the framework generic , we use a fairly simple fact unit extractor based on the syntactic relations . Given a context consisting of multiple sentences , we first conduct dependency parsing of each sentence . After that , we extract the subject , the predicate , and the object tokens to get the `` Argument-Predicate-Argument '' triplets corresponding to each sentence in the context . Supergraph Construction . With the obtained triplets , the fact units are organized in the form of Levi graph ( Levi , 1942 ) , which turns arguments and predicates all into nodes . An original fact unit is in the form of F = ( V , E , R ) , where V is the set of the arguments , E is the set of edges connected between arguments , and R is the relations of each edge which are predicates here . The corresponding Levi graph is denoted as Fl = ( VL , EL , RL ) where VL = V ∪ R , which makes the originally directly connected arguments be intermediately connected via relations . As for RL , previous works such as ( Marcheggiani & Titov , 2017 ; Beck et al. , 2018a ) designed three types of edges RL = { default , reverse , self } to enhance information flow . Here in our settings , we extend it into five types : default-in , default-out , reverse-in , reverse-out , self , corresponding to the directions of edges towards the predicates . Detailed description for edge types can be found in Appendix A . We construct the supergraph by making connections between fact units Fl . In particular , we take three strategies according to global information , identical concept and co-reference information . ( 1 ) We add a node Vg initialized with the question-option representation and connect it to all the fact unit nodes . The edge type is set as aggregate for better information interaction . ( 2 ) There can be identical mentions in different sentences , resulting in repeated nodes in fact units . We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same . ( 3 ) We conduct co-reference resolution on context using an off-to-shelf model1 in order to identify arguments in fact units that refer to the same one . We add edges with type coref between them . The final supergraph is denoted as S = ( Fl ∪ Vg , E ) where E is the set of edges added with the previous three strategies .

Summary: The paper claims that a lot of previous work in QA-based MRC focused only on entity-aware common sense knowledge.  To overcome this, the paper proposes to build a finer-grained local fact, (entity, predicate, entity) triplet, information based on dependency parsing, and then connect the triplet nodes with global information such as coreference, entity information.  Architecture-wise, a passage, question, and options are passed to RoBERTa (or DeBERTa) and then this information goes through the graph-attention network (GAN), based on the graph structure described above. In the experiment section, the authors show that the proposed FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 5,6).  The main contribution of this paper, as I understand, is bringing more fine-grained local facts and connecting them with global information such as coreference and entity linking.