Year: 2021

Abstract:
Recent years have witnessed an increasing interest in training machines with reasoning ability, which deeply relies on accurate, clearly presented clue forms that are usually modeled as entity-like knowledge in existing studies. However, in real hierarchical reasoning motivated machine reading comprehension, such one-sided modeling is insufficient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to. Thus, in view of language being a complete knowledge/clue carrier, we propose a general formalism to support representing logic units by extracting backbone constituents of the sentence such as the subject-verb-object formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logic reasoning. Beyond building the ad-hoc graphs, we propose a more general and convenient fact-driven approach to construct a supergraph on top of our newly defined fact units, benefiting from both sides of the connections between facts and internal knowledge such as concepts or actions inside a fact. Experiments on two challenging logic reasoning benchmarks show that our proposed model, FOCAL REASONER, outperforms the baseline models dramatically and achieves state-of-the-art results.

heading: 1 INTRODUCTION
text: To understand human language, deep neural networks have been widely applied and achieved impressive benchmark results (Chen et al., 2016b; Sachan & Xing, 2016; Seo et al., 2017; Dhingra et al., 2017; Cui et al., 2017; Song et al., 2018; Hu et al., 2019; Zhang et al., 2020a; Back et al., 2020; Zhang et al., 2020b; Hermann et al., 2015). However, the core requirement among the natural language understanding, logic reasoning, cannot be simply solved by the current design philosophy of extracting statistical patterns from data (Shi et al., 2020). In order to solve such a problem, there emerges an interest that accounts for human intuition about the entailment of sentences and reflects the semantic relations between sentential constituents (Iwańska, 1993). In this paper, we focus on logic reasoning in the form of natural language understanding (NLU) as logic reasoning may be naturally embodied in such a task and natural language offers sufficient enough clues for effective logic reasoning. In detail, we concern about a logic reasoning question-answering (QA) task, where given passage, question, and candidate answer options, the model has to make a proper decision with its logic reasoning ability. There are examples shown in Figure 1 from logic reasoning benchmark datasets, ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020).
Recent neural models usually exploit a pretrained language model (PrLM) as a key encoder for effective contextualized representation. According to diagnostic tests (Ettinger, 2020; Rogers et al., 2020), though PrLMs like BERT (Devlin et al., 2019) have encoded syntactic and semantic information after large-scale pre-training, they perform sensitivity to role reversal and struggles with pragmatic inference and role-based event knowledge, which are fundamental for reasoning whose major challenge is to uncover logical structures, and reason with the candidate options and questions to predict the correct answer. It is difficult for PrLMs to
capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training. Existing logic reasoning has shown serious dependence on knowledge-like clues. This is due to the lengthy, noisy text in human language which is though a natural carrier of knowledge but does not provide a clean, exact knowledge form. Thus, an increasing interest is in using graph networks to model the entity-aware relationships in the passages (Yasunaga et al., 2021; Ren & Leskovec, 2020; Huang et al., 2021; Krishna et al., 2020; Lv et al., 2020). However, all these methods may insufficiently capture indispensable logical units from two perspectives. First, they mostly focus on entity-aware commonsense knowledge, but pay little attention to those non-entity, non-commonsense clues (Zhong et al., 2021). Second, when existing models extract predicate logic inside language into knowledge, they only exploit quite limited predicates like hasA and isA but ignore a broad range of predicates in real language. From either of the perspectives, the existing methods actually only concern about those "global" knowledge that keeps valid across the entire data, without sufficient "local" perception of complete facts or events in the given specific part of logic reasoning task. We argue such insufficient modeling on logic units roots from the ignorance of language itself being the complete knowledge/clue carrier. Thus, we propose extracting a kind of broad facts according to backbone constituents of a sentence to effectively cover such indispensable logic reasoning basis, filling the gap of local, non-commonsense, non-entity, or even non-knowledge clues in existing methods as shown in Figure 2. For example, these units may reflect the facts of who did what to whom, or who is what in Figure 3. Such groups can be defined as "fact unit" following Nakashole & Mitchell (2014) in Definition 1. The fact units are further organized into a supergraph following Definition 2.
Definition 1 (Fact Unit) Given an triplet T = {E1, P, E2}, where E1 and E2 are arguments (including entity and non-entity), P is the predicate, a fact unit F is the set of all entities in T and their corresponding relations.
Definition 2 (Supergraph) A supergraph is a structure made of fact units (regarded as subgraphs) as the vertices, and the relations between fact units as undirected edges.
As shown in Figure 2, we regard the defined fact as the results of syntactic processing, rather than those from semantic role labeling (SRL) as in previous study, thus the proposed fact also extends the processing means in existing work. Correspondingly, in this work, we propose a fact-driven logical reasoning model, called FOCAL REASONER, which builds supergraphs on top of fact units as the basis for logic reasoning, to capture both global connections between facts and the local concepts or actions inside the fact. Our model FOCAL REASONER is evaluated on two challenging logic reasoning benchmarks including ReClor, LogiQA, one dialogue reasoning dataset Mutual, for verifying the effectiveness and the generalizability across different domains and question formats.

heading: 2 RELATED WORK
text: Machine Reading Comprehension Recent years have witnessed massive researches on Machine Reading Comprehension (MRC) whose goal is training machines to understand human languages,
which has become one of the most important areas of NLP (Chen et al., 2016b; Sachan & Xing, 2016; Seo et al., 2017; Dhingra et al., 2017; Cui et al., 2017; Song et al., 2018; Hu et al., 2019; Zhang et al., 2020a; Back et al., 2020; Zhang et al., 2020b). Despite the success of MRC models on various datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017) and so on, researchers began to rethink what extent does the problem been solved. Nowadays, there are massive researches into the reasoning ability of machines. According to (Kaushik & Lipton, 2018; Zhou et al., 2020; Chen et al., 2016a), reasoning abilities can be broadly categorized into (1) commonsense reasoning (Davis & Marcus, 2015; Bhagavatula et al., 2019; Talmor et al., 2019; Huang et al., 2019); (2) numerical reasoning (Dua et al., 2019); (3) multi-hop reasoning (Yang et al., 2018) and (4) logic reasoning (Yu et al., 2020; Liu et al., 2020), among which logic reasoning is essential in human intelligence but has merely been delved into. Natural Language Inference (NLI) (Bowman et al., 2015; Williams et al., 2018; Nie et al., 2020) is a task closely related to logic reasoning. However, it has two obvious drawbacks in measuring logic reasoning abilities. One is that it only has three logical types which are entailment, contradiction and neutral. The other is its limitation on sentence-level reasoning. Hence, it is important to research more comprehensive and deeper logic reasoning abilities.
logic reasoning in MRC There are two main kinds of features in language data that would be the necessary basis for logic reasoning: 1) knowledge: global facts that keep consistency regardless of the context, such as commonsense, mostly derived from named entities; 2) non-knowledge: local facts or events that may be sensitive to the context, mostly derived from detailed language. Existing works have made progress in improving logic reasoning ability (Yasunaga et al., 2021; Ren & Leskovec, 2020; Huang et al., 2021; Krishna et al., 2020; Zhong et al., 2021; Wang et al., 2021). However, these approaches are barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations, which are obviously not sufficient. In this work, we strengthen the basis for logic reasoning by unifying both types of the features as "facts". Different from previous studies that focus on the knowledge components, we propose a fact-driven logic reasoning framework that builds supergraphs on top of fact units to capture both global connections between entityaware facts and the local concepts or events inside the fact.

heading: 3 METHODOLOGY
text: In this section, we present a fact-driven approach for logic reasoning and the overall architecture of the model is shown in Figure 4. The framework can be divided into three steps as following. We first extract fact units from raw texts via syntactic processing to construct a supergraph. Then it performs reasoning over the supergraph along with a logical fact regularization. Finally, it aggregates the learned representation to decode for the right answer.

heading: 3.1 FACT UNIT EXTRACTION AND SUPERGRAPH CONSTRUCTION
text: Fact Unit Extraction. Figure 5 illustrates our method for constructing a supergraph from raw text inputs. The first step is to obtain triplets that constitute a fact unit. To keep the framework generic, we use a fairly simple fact unit extractor based on the syntactic relations. Given a context consisting of multiple sentences, we first conduct dependency parsing of each sentence. After that, we extract the subject, the predicate, and the object tokens to get the "Argument-Predicate-Argument" triplets corresponding to each sentence in the context.
Supergraph Construction. With the obtained triplets, the fact units are organized in the form of Levi graph (Levi, 1942), which turns arguments and predicates all into nodes. An original fact unit is in the form of F = (V,E,R), where V is the set of the arguments, E is the set of edges connected between arguments, and R is the relations of each edge which are predicates here. The corresponding Levi graph is denoted as Fl = (VL, EL, RL) where VL = V ∪ R, which makes the originally directly connected arguments be intermediately connected via relations. As for RL, previous works such as (Marcheggiani & Titov, 2017; Beck et al., 2018a) designed three types of edges RL = {default, reverse, self} to enhance information flow. Here in our settings, we extend it into five types: default-in, default-out, reverse-in, reverse-out, self, corresponding to the directions of edges towards the predicates. Detailed description for edge types can be found in Appendix A.
We construct the supergraph by making connections between fact units Fl. In particular, we take three strategies according to global information, identical concept and co-reference information. (1) We add a node Vg initialized with the question-option representation and connect it to all the fact unit nodes. The edge type is set as aggregate for better information interaction. (2) There can be identical mentions in different sentences, resulting in repeated nodes in fact units. We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same. (3) We conduct
co-reference resolution on context using an off-to-shelf model1 in order to identify arguments in fact units that refer to the same one. We add edges with type coref between them. The final supergraph is denoted as S = (Fl ∪ Vg, E) where E is the set of edges added with the previous three strategies.

heading: 3.2 REASONING PROCESS
text: Graph Reasoning. A natural way to model the supergraph is via Relational Graph Convolution Networks (Schlichtkrull et al., 2018). We first feed the contexts, stated as [CLS]C[SEP ]q||o[SEP ], into a pre-trained model to get the encoded representation. We initialize the nodes with averaged hidden states of its tokens because our triplets extraction performs in word-level. For edges, we use a one-hot embedding layer to encode the relations.
Based on the relational graph convolutional network and given the initial representation h0i for every node vi, the feed-forward or the message-passing process with information control can be written as h (l+1) i = ReLU( ∑ r∈RL ∑ vj∈Nr(vi) g (l) q 1 ci,r w (l) r h (l) j ), where Nr(vi) denotes the neighbors of node vi under relation r and ci,r is the number of those nodes. w (l) r is the learnable parameters of layer l. g (l) q is a gated value between 0 and 1.
Through the graph encoder FG(.), we then obtain the hidden representations of nodes in fact units as {hF0 , ...hFm} = FG({vL,0, ...vL,m}, EL). hFi is the node representation inside fact unit. They are then concatenated as the representation for supernode as hS0 . {h0, ...hm} = FG({hS0 , ...hSm}, EC). For node features on the supergraph, it is fused via the attention and gating mechanisms with the original representations of the context encoder HC . We apply attention mechanism to append the supergraph representation to the original one H̃ = Attn(Hc,Kf , Vf ), where {Kf , Vf} are packed from the learned representations of the supergraph. We compute λ ∈ [0, 1] to weigh the expected importance of supergraph representation of each source word λ1 = σ(WλH̃ + UλHC), where Wλ and Uλ are learnable parameters. HC and H̃ are then fused for an effective representation H = HC + λH̃ ∈ R4×d.
Interaction. For the application to the concerned QA tasks that require reasoning, options have their inherent logical relations, which can be leveraged to aid answer prediction. Inspired by Ran et al. (2019), we use an attention-based mechanism to gather option correlation information.
Specifically for an option Oi, the information it get by interaction with option Oj is calculated as O(j)i = [O q i − O q iAttn(O q i , O q j ; v);O q i ◦ O q iAttn(O q i , O q j ; v)], where O q i is the representation of the concatenation for the i-th option and question after the context encoder. Then the option-wise information are gathered to fuse the option correlation information Ôi = tanh(Wc[O q i ; {O (j) i }i 6=j ] + bc), where Wc ∈ Rd×7d and bc ∈ Rd. For answer prediction, We seek to minimize the cross entropy loss byLans = − log softmax(WzC+ bz)l ∈ R4, where C is the combined representations of Ô and H .
Logical Fact Regularization. Since the subject, verb, and object in a fact should be closely related with some explicit relationships, we design logical fact regularization technique to make the logical facts more of factual correctness. Without loss of generality, we assume that in our settings, the summation of the subject vector and the relation vector should be close to the object vector as much as possible, i.e., vsubject + vpredicate → vobject. Specifically, given the hidden states of the sequence hi where i = 1, . . . , L and L is the total length of the sequence, The regularization is defined as Llfr = ∑m k=1(1− cos(hsubk + hpredk , hobjk)), where m is the total number of logical fact triplets extracted from the context as well as the option and k indicates the k-th fact triplet.

heading: 3.3 TRAINING OBJECTIVE
text: During training, the overall loss for answer prediction is L = αLans + βLlfr, where α and β are two parameters. In our implementation, we set α = 1.0 and β = 0.5.
1https://github.com/huggingface/neuralcoref.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: We conducted the experiments on three datasets. Two for specialized logic reasoning ability testing: ReClor (Yu et al., 2020) and LogiQA (Liu et al., 2020) and one for logic reasoning in dialogues: MuTual (Cui et al., 2020).
We take RoBERTa-large (Liu et al., 2019) and DeBERTa-xlarge (He et al., 2020) as our backbone models for convenient comparison with previous works. We also compare our model with DAGN (Huang et al., 2021), a framework leveraging RoBERTa-large as the backbone and LReasoner (Wang et al., 2021), the previous state-of-the-art model on the leaderboard using DeBERTa-xlarge. For more details on datasets and baseline models, one can refer to Appendix A.

heading: 4.2 IMPLEMENTATION DETAILS
text: The overall model is end-to-end trained and updated by Adam (Kingma & Ba, 2015) optimizer with an overall learning rate 8e-6 for ReClor and LogiQA, and 4e-6 for MuTual. The weight decay is 0.01. We set the warm-up proportion during training to 0.1. Graph encoders are implemented using DGL, an open-source lib of python. The layer number of the graph encoder is 2 for ReClor and 3 for LogiQA. The maximum sequence length is 256 for LogiQA and MuTual, and 384 for ReClor. The model is trained for 10 epochs with a total batch size 16 and an overall dropout rate 0.1 on 4 NVIDIA Tesla V100 GPUs, which takes around 2 hours for ReClor and 4 hours for LogiQA2.

heading: 4.3 RESULTS
text: Tables 1 and 2 show the results on ReClor, LogiQA, and MuTual, respectively. All the best results are shown in bold. From the results, we have the following observations:
1) Based on our implemented baseline models (basically consistent with public results), we observe dramatic improvements on both of the logic reasoning benchmarks, e.g., on ReClor test set, FOCAL REASONER achieves +4.2% on dev set and +3.3% on the test set. FOCAL REASONER also outperforms the prior best system LReasoner3, reaching 77.05% on the EASY subset, and 44.64% on the HARD subset. The performance suggests that FOCAL REASONER makes better use of logical
2Our code has been submitted along with this submission, which will be open after the blind review period. 3The test results are from the official leaderboard https://eval.ai/web/challenges/
challenge-page/503/leaderboard/1347.
structure inherent in the given context to perform reasoning than existing methods. More detailed ablations will be shown in Section 5.
2) Table 3 lists the accuracy of our model on the dev set of ReClor of different question types. Results show that our model can perform well on most of the question types, especially "Strengthen" and "Weaken". This means that our model can well interpret the question type from the question statement and make the correct choice corresponding to the question.
3) Our model also achieves comparable performance with the unpublished LReasoner. It employs symbolic reasoning and data augmentation techniques, which is in a different research line from ours. Without data argumentation, LReasoner shows relatively poorer results, showing that our fact-driven approach would be beneficial compared with the symbolic-driven technique. Compared with the neural methods for logic reasoning, symbolic approaches would rely heavily on dataset-related predefined patterns which entail massive manual labor, potentially restricting the generalizability of models.
4) On the dialogue reasoning dataset MuTual, our model achieves quite a jump compared with the RoBERTa-base LM.4 This verifies our model’s generalizability on other downstream reasoning task settings.
5) For the model complexity, our method basically keeps as simple as previous models like DAGN. Our model only has 414M parameters compared with 355M in the baseline RoBERTa, and 400M in DAGN which also employs GNN.

heading: 5 ANALYSIS

heading: 5.1 ABLATION STUDY
text: To dive into the effectiveness of different components in FOCAL REASONER, we conduct analysis by taking RoBERTa as the backbone on the ReClor dev set. Tables 5-6 summarize the results.
4Since there are no official results on RoBERTa-large LM, we use RoBERTa-base LM instead for consistency.
Table 4: Statistics for fact unit entities and traditional named entities in datasets.
Number ReClor LogiQA
Train Dev Train Dev
Fact Unit Argument 14,895 1,665 20,676 1,981 Named Entity 9,495 984 12,439 1,515
Table 5: Replacing fact units with named entities or semantic roles on the ReClor dev set.
Model Accuracy
FOCAL REASONER 66.8±0.13 w/ named entity only 62.8±0.26 w/ semantic role only 62.2±0.32
Fact Units Variants Apart from our syntactically constructed fact units, there are two other ways in different granularities for construction. We replace the fact units with named entities that are used in previous works like Chen et al. (2019a). The statistics of fact units and named entities of ReClor and LogiQA are stated in Table 4, from which we can infer that there are indeed more fact units than named entities. Thus using fact units can better incorporate the logical information within the context. When replacing all the fact units with named entities, we can see from Table 5 that it significantly decreases the performance. We also explore the performance using semantic role labeling a similar way as in Zhong et al. (2020). We can see that SRL, leveraging much more complex information as well as computation complexity, fails to achieve performance as good as our original fact unit.
Supergraph Reasoning: The first key component is supergraph reasoning. We ablate the global atom and erase all the edges connected with it. The results suggest that the global atom indeed betters message propagation, leveraging performance from 64.6% to 66.8%. We also find that replacing the initial QA pair representation of the global atom with only question representation hurts the performance. In addition, without the logical fact regularization, the performance drops from 66.8% to 64.2%, indicating its usefulness. For edge analysis, when (1) all edges are regarded as a single type rather than the original designed 8 types in total and (2) co-reference edges are removed, the accuracy
drops to 63.7% and 64.8%, respectively. It is proved that in our supergraph, edges link the fact units in reasonable manners, which properly uncovers the logical structures.
Interactions: We further experimented with the query-option-interactions setting to see how it affects the performance. The results suggest that the features learned from the interaction process enhance the model. Considering that the logical relations between different options are a strong indicator of the right answer, this means that the model learns from a comparative reasoning strategy.

heading: 5.2 EFFECTS OF FACT UNITS NUMBERS
text: To inspect the effects of the number of fact units, we split the original dev set of ReClor and LogiQA into 5 subsets. The statistics of the fact unit distribution on the datasets are shown in Table 7. The numbers of fact units for most contexts in ReClor and LogiQA are in [3, 6) and [0, 3), respectively.
Comparing the accuracies of RoBERTa-large baseline, prior SOTA LReasoner and our proposed FOCAL REASONER in Figure 6, our model outperforms baseline models on all the divided subsets, which demonstrates the effectiveness and robustness of our proposed method. Specifically, for ReClor, FOCAL REASONER performers better when there are more fact units in the context, while for LogiQA, FOCAL REASONER works better when the number of fact units locates in [0, 3) and [9, 12). The reason may lie in the difference in style of the two datasets. However, all the models include ours struggle when the number of fact units is above certain thresholds, i.e., the logical structure is more complicated, calling for better mechanisms to cope with.

heading: 5.3 INTERPRETABILITY: A CASE STUDY
text: We aim to interpret FOCAL REASONER’s reasoning process by analyzing the node-to-node attention weights induced in the supergraph in Figure 7. We can see that our FOCAL REASONER can well bridge the reasoning process between context, question and option. Specifically, in the graph, "students rank 30%" attends strongly to "playing improve performance". Under the guidance of question to select the option that weakens the statement and option interac-
tion, our model is able to tell that "students rank 30% can play" mostly undermines the conclusion that "playing improves performance".

heading: A recent survey in a key middle school showed that high school students in this school have a special preference for playing football, and it far surpasses other balls.The survey also found that students who regularly play football are better at academic performance than students who do not often play football.This shows that often playing football can improve students' academic performance.

heading: 6 CONCLUSION
text: For logic reasoning arising from machine reading comprehension, it is well known that clear and accurate forms like global knowledge are crucial. In this work, we make a finding that existing studies miss focusing on quite a lot of non-knowledge parts which is also indispensable for better reasoning. Thus we propose extracting a general form called "fact unit" to cover both global and local logical units, hoping to shed light on the basis of structural modeling for logic reasoning. Our proposed FOCAL REASONER not only better uncovers the logical structures within the context, which can be a general method for other sophisticated reasoning tasks, but also better captures the logical interactions between context and options. The experimental results verify the effectiveness of our method.

heading: A LEVI GRAPH CONSTRUCTION
text: Levi graph transformation turns labeled edges into additional vertices. There are two types of edges in a traditional Levi graph: default and reverse. For example, an edge (E1, R1, E2) in the original graph becomes (E1, default, R1), (R1, default, E2), (R1, reverse, E1) and (E2, reverse,R1).
However, the type of source and target vertices in the graph also matters (Beck et al., 2018b). Specifically, previous works use the same type of edge to pass information, which may reduce the effectiveness. Thus we propose to transform the default edges into default-in and default-out edges, and the reverse edges into reverse-in and reverse-out edges.

heading: B DETAILS FOR DATASETS AND BASELINE MODELS
text: In this section, we describe the datasets and baseline models used in the experiments.
B.1 DATASETS
ReClor ReClor contains 6,138 multiple-choice questions modified from standardized tests such as GMAT and LSAT, which are randomly split into train/dev/test sets with 4,638/500/1,000 samples respectively. It contains multiple logic reasoning types. The held-out test set is further divided into EASY and HARD subsets based on the performance of BERT-based model Devlin et al. (2019).
LogiQA LogiQA consists of 8,678 multiple-choice questions collected from National Civil Servants Examinations of China and are manually translated into English by experts. The dataset is randomly split into train/dev/test sets with 7,376/651/651 samples correspondingly. LogiQA also contains various logic reasoning types.
MuTual MuTual has 8,860 dialogues annotated by linguist experts and high-quality annotators from Chinese high school English listening comprehension test data. It is randomly split into train/dev/test sets with 7,088/886/886 samples respectively. There more than 6 types of reasoning abilities reflected in MuTual. MuTualplus is an advanced version, where one of the candidate responses is replaced by a safe response (e.g., "could you repeat that?") for each example.
B.2 BASELINE MODELS
DAGN explores passage-level discourse-aware clues used for solving logical reasoning QA. Specifically, they leverage discourse relations annotated in Penn Discourse TreeBank 2.0 (PDTB 2.0) (Prasad et al., 2008) and punctuation as the delimiters to split the context into elementary discourse units (EDUs). They are further organized into a logical graph and feed into a GNN to get the representation.
LReasoner is a symbolic-driven framework for logical reasoning of text. It firstly identifies the logical symbols and expressions explicitly for the context and options based on manually designed rules. Then it performs logical inference over the expressions according to logical equivalence laws such as contraposition (Russell & Norvig, 2002). Finally, it verbalizes the expressions to match the answer.
FOCAL REASONER enjoys two major merits. (1) Broader knowledge: Compared with DAGN which uses sententious knowledge such as logical connectives (e.g., becaus, however), FOCAL REASONER leverages a broader type of knowledge characterized by "fact unit", including global knowledge and local knowledge. (2) More transferable: Compared with LReasoner which manually design rules to extract logical patterns and perform logical reasoning in a symbolic way, Focal Reasoner is neural-based and manual-free, which is more generalizable to other datasets.

heading: C MODEL COMPLEXITY
text: In this section, we display the statistics for the parameters of FOCAL REASONER and baseline models to demonstrate the model complexity and the effectiveness.
From Table 8 we can see that the increase of parameters is no more than 15% compared with the baseline models, and is comparable with strong baseline model DAGN. This indicates that, our model, being quite effective, is not a result of stacking complicated modules.
D VARIANCES FOR FOCAL REASONER W.R.T EXPERIMENT RESULTS
In this section, we report the average and variances run on 5 random seeds for FOCAL REASONER with different pre-trained language models.

heading: E FURTHER INTERPRETATION
text: We change the example in Figure 7 a bit. Specifically, we change the conclusion from "playing football can improve students’ academic performance" to "football players are held to higher academic standards than non-athletes". We can observe that our model can select the relatively correct answer, which indicates that FOCAL REASONER has some logical reasoning ability, instead of simple text matching and mining ability.

Review summaries:
The paper claims that a lot of previous work in QA-based MRC focused only on entity-aware common sense knowledge.  To overcome this, the paper proposes to build a finer-grained local fact, (entity, predicate, entity) triplet, information based on dependency parsing, and then connect the triplet nodes with global information such as coreference, entity information.  Architecture-wise, a passage, question, and options are passed to RoBERTa (or DeBERTa) and then this information goes through the graph-attention network (GAN), based on the graph structure described above. In the experiment section, the authors show that the proposed FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 5,6).  The main contribution of this paper, as I understand, is bringing more fine-grained local facts and connecting them with global information such as coreference and entity linking.

The authors propose a framework, called Focal Reasoner, to perform logical reasoning to answer questions. The proposed approach first extracts the facts from raw text. The authors describe a fact unit as a triple of (argument, predicate, argument). These collections of facts can be viewed as a graph with predicates as undirected edges between arguments. After forming a subgraph, reasoning is performed by a graph convolution module to predict the correct answer.  The authors show that their approach outperforms existing approaches on two benchmarked datasets. The authors also perform a detailed analysis that sheds more light on the internal workings of the model.

This paper proposes a model architecture for question answering which takes advantage of shallow proposition structure and coreference links using (relational) graph convolutional networks. Subject-predicate-object triples (dubbed "fact units") are extracted from the text using dependency parsing, extra coreference links are added using a coreference system, and the resulting graph is initialized with representations from a contextualizing encoder, passed through a GCN, and recombined with the contextual representations via multi-headed attention. The resulting representations serve as input to the final classification layer for answer prediction, as well as a "logical facts regularization" step which encourages the final representation of the object in each fact unit to align (via cosine) with the sum of the subject and predicate representations.  In experiments on several datasets designed to test logical reasoning, the proposed architecture scores higher than several baselines.

The paper claims that a lot of previous work in QA-based MRC focused too much on entity-aware common sense knowledge and other clues in the natural language (in passage, questions & candidate options). To overcome this, the authors extract "fact units" in the form of "subject-verb-object" from syntactic processing (dependency parsing).  With these fact units as a node, they form a supergraph and uses GNNs to process this. There are also many more components: global node, logical fact regularization, enlarged edge types compared to previous work and interactions. In the experiment section, they show that FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 3). 

This paper presents a system for graph-based reasoning about scenarios. Facts are represented as subgraphs which are connected into a supergraph via a single "global" node and coreference nodes. A passage, question, and options are first passed through RoBERTa, then a neural network (graph attention network) passes information through the graph representation before an answer is predicted. Results show that this model works better than a related graph-based technique (the DAGN model), with particularly strong gains on the LogiQA test set.

This paper presents a graph-based approach to solve logical reasoning questions. This method first extracts fact units from given questions, passages and answer options and builds a graph based on them, where a dependency parser is used to extract <subject, verb, object> triples. Then, the pre-trained RoBERTa large is used to generate representations of the concatenated question-passage-option sequences. Next, a graph is built based on the dependency parsing results of the sequence, and a graph attention network is used to perform neural message passing to exchange information between graph nodes. Last, all information are aggregated and used to do the prediction. Evaluations are conducted on 3 reasoning tasks, ReClor, LogiQA and MuTual, with ablation study performed for the effects of global atoms, fact unit variants/numbers and interaction layers.