Fact-driven Logical Reasoning
Authors: Siru Ouyang
Zhuosheng Zhang
hai zhao
Venue: ICLR 2022 Submitted (2022)
Keywords: logical reasoning
machine reading comprehension
language understanding

Abstract: Recent years have witnessed an increasing interest in training machines with reasoning ability, which deeply relies on accurate, clearly presented clue forms that are usually modeled as entity-like knowledge in existing studies. However, in real hierarchical reasoning motivated machine reading comprehension, such one-sided modeling is insufficient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to. Thus, in view of language being a complete knowledge/clue carrier, we propose a general formalism to support representing logic units by extracting backbone constituents of the sentence such as the subject-verb-object formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logical reasoning. Beyond building the ad-hoc graphs, we propose a more general and convenient fact-driven approach to construct a supergraph on top of our newly defined fact units, benefiting from both sides of the connections between facts and internal knowledge such as concepts or actions inside a fact. Experiments on two challenging logical reasoning benchmarks show that our proposed model, \textsc{Focal Reasoner}, outperforms the baseline models dramatically and achieves state-of-the-art results.

Under review as a conference paper at ICLR 2022
FACT-DRIVEN LOGIC REASONING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent years have witnessed an increasing interest in training machines with reasoning ability, which deeply relies accurate, clearly presented clue forms that are usually modeled as entity-like knowledge in existing studies. However, in real hierarchical reasoning motivated machine reading comprehension, such one-sided modeling is insufficient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to. Thus, view of language being a complete knowledge/clue carrier, we propose general formalism to support representing logic units by extracting backbone constituents the sentence such as the subject-verb-objeet formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logic reasoning. Beyond building the ad-hoc graphs, propose more general and convenient fact-driven approach construct a supergraph on top of our newly defined fact units, benefiting from both sides of the connections between facts and internal knowledge such as concepts actions inside a fact. Experiments on two challenging logic reasoning benchmarks show that our proposed model, FOCAL REASONER, outperforms the baseline models dramatically and achieves state-of-the-ar results.
INTRODUCTION
To understand human language, deep neural networks have been widely applied and achieved impressive benchmark results (Chen al.
2016bb /aahan & Xing 2016 Seo et al. 2017 Dhingra
et al. 2017: Cui et al.] 2017|SSon et al.,
2018 Hu et al.] [019 Zhang et al.) [020a]] Back et al.
2020; Zhang et al.) [2020b; Hermann et al., 2015). However, the core requirement among the natural
language understanding, logic reasoning, cannot be simply solved by the current design philosophy of extracting statistical patterns from data (Shi et al. 2020). In order to solve such a problem, there emerges an interest that accounts for human intuition about the entailment of sentences and reflects the semantic relations between sentential constituents (Iwańska (993). In this paper, we focus on logic reasoning in the form of natural language understanding (NLU) as logic reasoning may be naturally embodied in such task and natural language offers sufficient enough clues for effective logic reasoning. In detail, we concern about logic reasoning question-answee answering (QA) task. where given passage, question, and candidate answer options, the model has to make a proper decision with its logic reasoning ability. There examples shown in Figurel T from logic reasoning benchmark datasets, ReClor Yuu et al., (020)) and LogiQA (Liu et al. 2020).
Recent neural models usually exploit pre- trained language model (PrLM) as key en- coder for effective contextualized representa- tion. According to diagnostic tests (Ettinger) Rogers et al., (020), though PrLMs like BERT (Devlin et al. 20199) have encoded syn- tactic and semantic information after large-scale pre-training, they perform sensitivity to role re- versal and struggles with pragmatic inference and role-based event knowledge, which are fun- damental for reasoning whose major challenge is uncover logical structures, and reason with the candidate options and questions to predict the correct answer. It is difficult for PrL.Ms to
approaches
(b) obtained knowledge
Existing Works
FACT (ogical Structure)
Non-
Common- sense
Common-
SRL
sense
(semantic)
Named Entity
Non-
Named
Ours syytacttd
global
Entity
Figure 2: Our "fact" V.S. existing approaches.

---

Under review as a conference paper at ICLR 2022
Question
Passage
Answer
Exammpe
Xiso Wang is taller than Xiso Li, Xiso Zhao taller than Xiao Qlan, Xiso shorter than Xiao Sun, and Xiso Sun is shorter than Xieo Qien.
A, Xiso Li is shorter then Xiso Zhao.
B. Xiso Wang taller than Xiao Zhao, Sun is shorter than D. Xiso Sun is taller then Xiso Zheo.
From this wa know
Wang.
1Example? 2]
A. Many other animal species from same era did not
A large enough comet colliding with Earth could have caused a cloud dust that enshrouded the planet and cooled the climate long enough to result in the dinosaurs' demise,
become extinct at the same time the dinosaurs did.
B. cannot determined from dinossur skeletons whether
Which one of the fallow- ing statements, most sariouzly weakens the argument?
animals died
effects dust cloud.
C. consequancees for vegetation
animals comet colliding with Earth are not fully understoed.
D. Verious species of animals from the same ara and similar habitat physiology
become extinct.
Two examples from LogiQA and ReClor respectively are illustrated. There are arguments and relations between arguments. Both are emphasized by different colors: arguments, relations. Key words in questions are highlighted in Purple. Key options are highlighted in gray.
Figure 1:
capture the logical structure inherent in the texts
since logical supervision is rarely available during pre-training. Existing logic reasoning has shown serious dependence on knowledge-like clues. This is due to the lengthy, noisy text in human language which is though natural carrier of knowledge but does not provide clean, exact knowledge form. Thus, an increasing interest is in using graph networks to model the entity-aware relationships in the passages ( (Yasunaga et al., 202T Ren & Leskovec 2020 Huang et al.] 2021 Krishna al., 2020 [Lv et al., 2020). However, all these methods may insufficienlly capture indispensable logical units from two perspectives. First, they mostly focus on entity- commonsense knowledge, but pay little attention to those non-entity, non-commonsense clues (Zhong et al. (021). Second, when existing models extract predicate logic inside language into knowledge, they only exploit quite limited predicates like hasA and isA but ignore broad range of predicates in real language. From either of the perspectives, the existing methods actually only concern about those "global" " knowledge that keeps valid across the entire data, without sufficient "local" perception of complete facts or events in the given specific part of logic reasoning task. We argue such insufficient modeling on logic units roots from the ignorance of language itself being the complete knowledge/clue carrier. Thus, we propose extracting kind of broad facts according to backbone constituents of a sentence to effectively cover such indispensable logic reasoning basis, filling the gap of local, non-commonsense, non-entity, or even non- noo-lneee clues in existing methods as shown in Figure 2 For example, these units may reflect the facts of who did what to whom, or who is what Figure
Such groups
can be defined as "fact unit" following Nakashole & Mitchell (2014) in Definition are further organized into a supergraph following Definition
The fact units
Definition 1 (Fact Unit) Given an triplet T (E,, E2). where E① and Eg are arguments
(including entity and non-entity), P is the predicate, a fact unit F is the set of all entities in T and their corresponding relations.
Definition 2 (Supergraph) A supergraph is a structure made of fact units (regarded as subgraphs) as the vertices, and the relations between fact units as undirected edges.
As shown in Figure 2 we regard the defined fact the results of syntactic processing, rather than those from semantic role labeling (SRL) as in previous study, thus the proposed fact also extends the processing means in existing work. Correspondingly, in this work, we propose fact-driven logical reasoning model, called FOCAL REASONER, which builds supergraphs on top of fact units as the basis for logic reasoning, to capture both global connections between facts and the local concepts or actions inside the fact. Our model FOCAL REASONER is evaluated on two challenging logic reasoning benchmarks including ReClor, LogiQA, one dialogue reasoning dataset Mutual, for verifying the effectiveness and the generalizability across different domains and question formats.
2 RELATED WORK
Machine Reading Comprehension Recent years have witnessed massive researches on Machine Reading Comprehension (MRC) whose goal is training machines to understand human languages,
2

---

Under review as a conference paper at ICLR 2022
which has become one of the most important areas of NLP
CChnn et al.
2016b; Sachan & Xing 2018:
2016; Seo et al. 2017: Dhingra et al., 2017| Cui et al. 2017 Song et al.
Hu et al. 2019
Zhang et al. 2020a; Back et al. 22200] Zhang al. (222bb) Despite the success of MRC models on
various datasets such as CNN/Daill Mail (Hermann et al. 2015 SQuAD (Rajpurkar et al. 2016). RACE (Lai et al.
2017) and so on, researchers began rethink what extent does the problem been solved. Nowadays, there are massive researches into the reasoning ability of machines. According to 2018: Zhou et al., 2020 Chen al. [2]66a), reasoning abilities can be broadly
(Kaushik & Lipton
categorized into (1) commonsense reasoning (Davis & Marcus 2015 Bhagavatula et al. 2019]
Talmor et al. 2019 Huang et al., ) (2) numerical reasoning (Dua al. 2019); (3) multi hop
reasoning (Yang et 2018) and (4) logic reasoning (Yu et al. 2020; Liu et al. (020), among which
logic reasoning is essential human intelligence but has merely been delved into. Natural Language
Inference (NI.)) (Bowman et al.) 2015 Williams et al. 2018 Nie et al. 2020) is task closely related to logic reasoning. However, it has two obvious drawbacks in measuring logic reasoning abilitics. One is that it only has three logical types which are entailment, contradiction and neutral. The other its limitation on sentence-level reasoning. Hence, it is important to research more comprehensive and deeper logic reasoning abilities.
logic reasoning in MRC There are two main kinds of features in language data that would be the neces- sary basis for logic reasoning: 1) knowledge: global facts that keep consistency regardless of the context, such as commonsense, mostly derived from named entities; 2) non- non-llewwedge: local facts or events that may be sensitive to the context, mostly derived from detailed language. Existing works have made progress in improving logic reasoning ability Ya- sunaga et al. [202T] Ren & Leskovec 2220;; Huang
colliding
coreference same entity
comet
Earth:
result
caused
comet
demise
comet
dust
al. 2021
Krishna al. [2200] Zhong et al. [202] 2021). However, these approaches are
Wang et al.
enshrouded
cooled
barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations, which are obviously not sufficient. In this work, we strengthen the basis for logic reasoning by unifying both types of the features as "facts". Different from previous studies that focus the knowledge com- ponents, we propose a fact-driven logic reasoning framework that builds supergraphs on top of fact units to capture both global connections between entity- aware facts and the local concepts or events inside
dust
Planet,
dust
climate
Figure 3: An example of constructed super- graph. In contrast, the dotted vertices and edges are focused in most existing studies (Qiu et al. 2019 Ding et al. 2019) Chen et al. 2019b).
the fact.
3 METHODOLOGY
In this section, we present fact-driven approach for logic reasoning and the overall architecture of the model is shown in Figure The framework can be divided into three steps as following. We first extract fact units from raw texts via syntactic processing to construct a supergraph. Then performs reasoning over the supergraph along with logical fact regularization. Finally, it aggregates the learned representation to decode for the right answer.
3.1 FACT UNIT EXTRACTION AND SUPERGRAPH CONSTRUCTION
Fact Unit Extraction. Figure 5 illustrates our method for constructing a supergraph from raw text inputs. The first step is to obtain triplets that constitute fact unit. To keep the framework generic, we use a fairly simple fact unit extractor based on the syntactic relations. Given a context consisting of multiple sentences, we first conduct dependency parsing of each sentence. After that, extract the subject, the predicate, and the object tokens to get the "Argument -P -Predi maticcate-i www.ecttttttuauaaaee Argument triplets corresponding each sentence in the context.
3

---

Under review as a conference paper at ICLR 2022
Logical Facts Regularization
passage
Answer Prediction
Cross Attention
RoBERTa
Fact-driven Reasoning Process
question
OQ-Pair
Co -o-attention
Interaction
MaxPooling
OQ-Pair
option
OQ-Pair
00
OQ-Pair 4
00
Figure 4: The framework or our model. For supergraph reasoning, in each iteration, each node selectively receives the message from the neighboring nodes to update its representation. The dashed circle means zero vector.
A large enough comet colliding with Earth
colliding
coreference same entity
aggregate
coreference
could have caused a cloud shrouded the planet cooled the climate long enough to result in dinosaurs' demise.
default-in/out
comet
Earth
reverse-in/out
result
caused
colliding Earth coreet caused dust dust enshrouded - planet dust cooled - climate
comet demise
comet
dust
enshrouded
cooled
self same
comet result - demise
dust
planet
dust
climate
Which one of the following most seriously weakens the argument?
Various species of animals from the same era as dinosaurs and similar them did not become extinct when..
The process of constructing the fact chain and its corresponding Levi graph form of an example in Figure Entities and relations are illustrated in their corresponding color.
Figure 5:
Supergraph Construction. With the obtained triplets, the fact units organized in the form of Levi graph (Levi) 1942), which turns arguments and predicates all into nodes. An original fact unit in the form of (V, E, R), where is the set of the arguments, É the of edges connected between arguments, and R is the relations of each edge which are predicates here. The corresponding Levi graph is denoted as Fi (Vz, EL, R2) where VL V R. which makes the originally directly connected arguments be intermediately connected via relations. As for RL. previous works such (Marcheggiani & Titov| 2017;; Beck et al., 2018a) designed three types of edges R2, = (delauul, reverse, sel to enhance information flow. Here in our settings, we extend into five types: default-in, default-out, reverse-in, reverse-out, self, corresponding to the directions of edges towards the predicates. Detailed description for edge types be found in Appendix A
We construct the supergraph by making connections between fact units Fi. In particular, we take three strategies according global information, identical concept and co-reference information. (1) We add a node Vy initialized with the question-option representation and connect it to all the fact unit nodes. The edge type is set aggregate for better information interaction. (2) There can be identical mentions in different sentences, resulting in repeated nodes in fact units. We connect nodes corresponding to the non-pronour arguments by edges with edge type same. (3) We conduct
4

---

Under review as a conference paper at ICLR 2022
co-reference resolution context using an off-to-shelf mode in order identify arguments in fact units that refer to the same one. We add edges with type coref between them. The final supergraph is denoted as S (Fi U V. E) where E is the set of edges added with the previous three strategies.
3.2 REASONING PROCESS
Graph Reasoning. A natural way model the supergraph is via Relational Graph Convolution Networks (Schlichtkrull et al. 2018). We first feed the contexts, stated as [CLS][[SEP|gglo(EER into a pre-trained model get the encoded representation. We initialize the nodes with averaged hidden states of its tokens because our triplets extraction performs in word-level. For edges, we use one-hot embedding layer to encode the relations.
Based on the relational graph convolutional network and given the initial representation h& for every node vys the feed-forward or the message-passingg process with information control can written as G,r We hi h where (V;(ss denotes the neighbors of node (1)
h ((+1) ReLU EejEAs(ex) 94 L
o under relation and Ci,r is the number of those nodes. w/ (!) is the learnable parameters of layer I. Po is gated value between 0 and 1.
Through the graph encoder F.).), we then obtain the hidden representations nodes in fact units as (ho....h } = F&
EL). h." is the node representation inside fact unit. They are then concatenated as the representation for supernode as h. !h.m = F A. Ec).
For node features on the supergraph, it fused via the attention and gating mechanisms with the original representations of the context encoder H We apply attention mechanism to append the supergraph representation to the original one H = Attn H" K,, Vy). where {Ky Vý} are packed from the learned representations of the supergraph. We compute À E [o, to weigh the expected importance of supergraph representation of each source word A oWW + UCH where WA and UA are learnable parameters. HC and H are then fused for an effective representation H. = HC + AH 124xd
Interaction. For the application to the concerned QA tasks that require reasoning, options have their inherent logical relations, which can be leveraged to aid answer prediction. Inspired by |Ran ct al. (019), we use an attention-based mechanism to gather option correlation information.
Specifically for option O,, the information it get by interaction with option O, is calculated as 0 = 10 - O AOt, 0%: v); O² O AOt, 0y: v)]. where Of is the representation of the concatenation for the i-th option and question after the context encoder. Then the option-wise information are gathered to fuse the option correlation information Ô, = tanh(W.[O,; (0, Jess be), where W. E Rdx7d and be RRd.
For answer prediction, We seek to minimize the cross entropy loss by Cans = log wwof///aaWWWCCC ba)x Rd, where C' is the combined representations of Ô and H.
Logical Fact Regularization. Since the subject, verb, and object in fact should be closely related with some explicit relationships, we design logical fact regularization technique to make the logical facts more of factual correctness. Without loss of generality, summation of the subject vector and the relation vector should be close to the object vector as much as possible, i.e., Usubject + Upredicate - Uobject- Specifically, given the hidden states of the sequence h4 where i = 1,..., L and L is the total length of the sequence, The regularization is defined as Lift Σx= k=1 cos(hsude hprede- hobjs) )), where is the total number of logical fact triplets extracted from the context as well as the option and k indicates the k-th fact triplet.
assume that in settings, the
3.3 TRAINING OBJECTIVE
During training, the overall loss for answer prediction is L = aLans + BLifr, where a and /3 are two parameters. In our implementation, we set a 1.0 and B = 0.5.
coref
5

---

Under review as a conference paper at ICLR 2022
Table 1: Experimental results of model compared with baseline models ReClor and LogiQA dataset. Test-E and Test-H denote Test-Easy and Test-Hard respectively. The results in bold are the best performance except for the human performance. indicates that the results are taken from Yu et al. 2020) and Liu et al. (2020). Results with is taken from Wang et al. (02TT).
ReClor
LogiQA
Model
Dev
Test
Test-E
Test-H
Dev
Test
Human Performance*
63.0
57.1
67.2
86.0
BERT*
53.8
49.8
72.0
32.3
34.1
31.0
XLNet*
62.0
56.0
75.7
40.5
RoBERTa
62.6
55.6
75.5
40.0
35.0
35.3
DeBERTaf
74.4
68.9
83.4
57.5
44.4
41.5
DAGNR-BERT. &
65.8
75.9
44.5
39.3
data augmentation
65.2
58.2
76.1
44.1
35.5
38.7
66.2
62.4
81,4
47.5
38.1
40.6
data augmentation
65.2
58.3
78.6
42.3
LReasonerDassen.
74.6
83.4
62.7
43.3
FOCAL REASONER
66.8(14.2)
58.9(13.3)
77.1(1166)
44.6(14.6)
41.0(16.0) 47.3(12.9)
40.3(15.0)
FOCAL REASONERDEBERT
78.6(14.2)
73.3(14.4)
86.4(13.0)
63.0(15.5)
45.8(14.3)
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We conducted the experiments on three datasets. Two for specialized logic reasoning ability testing:
ReClor Yu et al. 2020) and LogiQA (Liu et al. 2020 and one for logic reasoning in dialogues:
MuTual (Cui et al. 2020).
take RoBERTa- large (Liu al. 2019) and DeBeRTa-xlarge (He et al. 2020) as our backbone models for convenient comparison with previous works. We also compare model with DAGN (Huang et al. 2021), a framework leveraging RoBERTa-Larg as the backbone and LReasoner (Wang al. 2021), 1), the previous state-of-the-ard model on the leaderboard using DeBERRaa-xlage.. For more details on datasets and baseline models, one can refer to Appendix A.
4.2
IMPLEMENTATION DETAILS
The overall model end-to-end trained and updated by Adam (Kingma & Ba 2015) optimizer with an overall learning rate 8e-6 for ReClor and LogiQA, and 4e-6 for Mu'Tual. The weight decay is 0.01. We set the warm-up proportion during training to 0.1. Graph encoders are implemented using DGL, an open-source lib of python. The layer number of the graph encoder is 2 for ReClor and 3 for LogiQA. The maximum sequence length is 256 for LogiQA and MuTual, and 384 for ReClor. The model is trained for 10 epochs with total batch size 16 and overall dropout rate 0.1 on NVIDIA Tesla V100 GPUs, which takes around 2 hours for ReClor and hours for LogiQA
4.3 RESULTS
Tables and show the results ReClor, LogiQA, and MuTual, respectively, All the best results are shown in bold. From the results, we have the following observations:
Based implemented baseline models (basically consistent with public results), observe dramatic improvements on both of the logic reasoning benchmarks, c.g., on ReClor test set, FOCAL REASONER achieves +4.2% on dev set and +3.3% on the test set. FOCAL REASONER also out- performs the prior best system I.eeasone reaching 77.05% the EASY subset, and 44.64% on the HARD subset. The performance suggests that FOCAL REASONER makes better use of logical
2 Our code has been submitted along with this submission, which will open after blind review period. lenges/
Thhe test results from the official leaderboard https: / a /ebb/ web/chal chal
cha Shallenee eeeenge-page555:5 http://////////aaeaaarrr////////

---

Under review as a conference paper at ICLR 2022
Table 2: Experimental results of our model compared with baseline on MuTual dataset. indicates that the results are taken from Cui et al. (2020). For fair comparison with our method, we also report the multi-choice method (RoBERTa-MC) in addition the default Individual scoring method (RoBERTa).
MuTual
Model
Set
Test Set
Rg
R,@2
MRR
Rg01
R4 R462
MRR
RoBERTa* 69.5
87.8
82.4
71.3
89.2
83.6
RoBERTaMMC
88.7
82,5
68.6
88.7
82.2
FOCAL REASONER 73.4(14.1)
90.3(11.6)
84.9(12.4)
72.7(14.1)
91.0(12.3)
84.6(12.4)
Table 3: Accuracy on the dev set of ReClor corresponding to several representative question types. S:
Strengthen, W: Weaken, I: Implication, CMP: Conclusion/Main Point, ER: Explain Resolve, D:
Dispute, R: Role, IF: Identify Flaw, MS: Match Structures. All results are reported on the same PrLM RoBERTa.
Model
W
CMP
ER
P
D
R
IF
MS
RoBERTa
61.7
47.8
39.1
63.9
58.3
50.8
50.0
56.3
61.5
56.7
DAGN
63.8
46.0
39.1
69.4
57.1
53.9
46.7
62.5
62.4
56.7
FOCAL REASONER
72.3
66.4
47.8
91.7
76.2
76.9
66.7
68.8
73.5
86.7
structure inherent in the given context to perform reasoning than existing methods. More detailed ablations will be shown in Section 5
2) Table 3 lists the accuracy of our model the dev set of ReClor of different question types. Results show that our model can perform well on most of the question types, especially "Strengthen" " and "Weaken". This means that model can well interpret the question type from the question statement and make the correct choice corresponding to the question.
3) Our model also achieves comparable performance with the unpublished LReasoner. employs symbolic reasoning and data augmentation techniques, which is in a different research line from ours. Without data argumentation, LReasoner shows relatively poorer results, showing that our fact-driven approach would be beneficial compared with the symbolic-driven technique. Compared with the neural methods for logic reasoning, symbolic approaches would rely heavily on dataset-related predefined patterns which entail massive manual labor, potentially restricting the generalizability of models.
4) On the dialogue reasoning dataset MuTual, our model achieves quite a jump compared with the RoBERTa-base LM This verifies our model's generalizability on other downstream reasoning task settings.
5) For the model complexity, our method basically keeps as simple as previous models like DAGN. Our model only has 414M parameters compared with 355M in the baseline RoBERTa, and 400M in DAGN which also employs GNN.
ANALYSIS
5.1
ABLATION STUDY
To dive into the effectiveness different components in FOCAL REASONER, we conduct analysis by taking RoBERTa the backbone on the ReClor dev set. Tables 5 summarize the results.
SSine there official results RoBERTa-large L.M, RoBERTa-hase I. instead for consistency.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Statistics for fact unit entities and tradi-
Table 5: Replacing fact units with named entities or semantic roles on the ReClor dev set.
tional named entities in datasets.
ReClor
LogiQA
Model
Accuracy
Number
Train
Dev
Train
Dev
FOCAL REASONER
66,8=0.13
Fact Unit Argument
14,895
1,665
20,676
1,981
named entity only
62.8
Named Entity
9,495
984
12,439
1,515
w/ semantic role only
62.2=0.32
Fact Units Variants Apart from our syntactically constructed fact units, there are two other ways in different granularities for construction. We replace the fact units with named entities that are used in previous works like Chen et al. (2019)). The statisties of fact units and named entities of ReClor and LogiQA are stated in Table 4 from which we can infer that there are indeed more fact units than named entities. Thus using fact units can better incorporate the logical information within the context. When replacing all the fact units with named entities, we can see from Table 5 that it significantly decreases the performance. We also explore the performance using semantic role labeling a similar way as in Zhong et al. (2020). We can see that SRL, leveraging much more complex information as well as computation complexity, fails to achieve performance as good as our original fact unit.
The first key com- ponent is supergraph reasoning. We ablate the global atom and erase all the edges connected with it. The results suggest that the global atom indeed betters message propagation, leveraging performance from 64.6% to 66.8%. We also find that replacing the initial QA pair represen- tation of the global atom with only question representation hurts the performance. In addi- tion, without the logical fact regularization, the performance drops from 66.8% 64. 64.2%. indi- cating its usefulness. For edge analysis, when (1) all edges are regarded as single type rather than the original designed 8 types in total and (2) co-reference edges are removed, the accuracy drops to 63.7% and 64.8%, respectively. It is proved that in supergraph, edges link the fact units in reasonable manners, which properly uncovers the logical structures.
Supergraph Reasoning:
Table 6: Ablation results on the ReClor dev set.
Model
Accuracy
FOCAL REASONER
66.8+0.1v 13
Supergraph Reasoning global edge
64.6:t0.33
co-reference edges
64.8:t0.24
logical fact regularization edge type
64.2:10.12
63.7=00.19
Interactions
interactions
65.5.10.52
Interactions: We further experimented with the query-option suery-oppioon-nteraaciiosss setting see how it affects the performance. The results suggest that the features learned from the interaction process enhance the model. Considering that the logical relations between different options are strong indicator of the right answer, this means that the model learns from a comparative reasoning strategy.
5.2 EFFECTS OF FACT UNITS NUMBERS
To inspect the effects of the number of fact units, we split the original dev set of ReClor and LogiQA into 5 subsets. The statisties of the fact unit distribution on the datasets are shown Table The numbers of fact units for most contexts in ReClor and LogiQA are in [3, 6) and (0,3), respectively.
Comparing the accuracies of RoBERTa-large baseline, prior SOTA LReasoner and our proposed FOCAL REASONER in Figure 6 our model outperforms baseline models on all the divided subsets, which demonstrates the effectiveness and robustness of our proposed method. Specifically, for ReClor, FOCAL REASONER performers better when there are more fact units in the context, while for LogiQA, FOCAL REASONER works better when the number of fact units locates in [0. 3) and [9, 12). The reason may lie in the difference in style of the two datasets. However, all the models include struggle when the number of fact units is above certain thresholds, i.c., the logical structure is more complicated, calling for better mechanisms to cope with.
8

---

Under review as a conference paper at ICLR 2022
RoBERTa
DAON Reasoner
RoBERTa DAAN
Focal Reaxoner
060
580
100
60
372
408
35.2
386
365
ACCURACY
ACCURACY
9
18 A 18
(0.3]
(3.6]
16,9]
12,
[0,30
[3,6]
[6
[9.22]
112. R)
NUMBER OF FACT UNITS
NUMBER OF FACT UNITS
Figure 6: Accuracy of models on number of fact units on dev set of ReClor (left) and LogiQA (right).
5.3 INTERPRETABILITY: A CASE STUDY
We aim to interpret FOCAL REASONER's 's rea- soning process by analyzing the node-to-node attention weights induced in the supergraph in Figure 7 We can see that our FOCAL REA- SONER can well bridge the reasoning process between context, question and option. Specifi- cally, in the graph, "students rank 30%" attends strongly to "playing improve performance". Un- der the guidance of question to select the option that weakens the statement and option interac-
Table 7: Distribution of fact unit number on dev set of the training datasets.
Dataset
[0,3)
[3, 6)
(6,9) 9)
[9, 12)
[12, no)
ReClor
37.2%
48.6%
12.6%
1.2% 0.6%
LogiQA 47.5%
37.5%
10.9%
3.5%
0.6%
tion, our model is able to tell that "students rank 30% can play" mostly undermines the conclusion that "playing improves performance".
A recent survey in key middle school showed that high school stucients in this school have special preference for playing football, and surpasses other balls.The survey also found that studentss who regularly play football better at academic performance than students who do not often play footbell.This shows thet often playing footbell can improve students' academic performance.
Only high school students who are ranked the top 30% of grades can often play football. Which of the following can weaken the above conclusion most?
B. Regular football can exercise and maintain a strong learning energy.
Often playing football delays study time.
D. Research has not proved that playing football can contribute to intellectual development.
1. students have preferences
2. preference playing football
3. it surpasses balls
4. who play football
QA
5. students better performance
6. who lplay football
7. playing improve performance students rank 30%
Option Similarity Matrix after Interaction
9. students play football
A: 4. 1918
C: -12.37188
Fact Units
buge Wergf
-5.3050
D: -6.922
Figure 7: An example of how our model reasons to get the final answer.
CONCLUSION
For logic reasoning arising from machine reading comprehension, is well known that clear and accurate forms like global knowledge are crucial. In this work, we make finding that existing studies miss focusing on quite a lot of non-knowledge parts which is also indispensable for better reasoning. Thus propose extracting a general form called "fact unit" to cover both global and local logical units, hoping to shed light on the basis of structural modeling for logic reasoning. Our proposed FOCAL REASONER not only better uncovers the logical structures within the context, which can be general method for other sophisticated reasoning tasks, but also better captures the logical interactions between context and options. The experimental results verify the effectiveness of our method.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Seohyun Back, Sai Chetan Chinthakindi, Akhil Kedia, Haejun Lee, and Jaegul Choo. NeurQuRI: Neural question requirement inspector for answerability prediction in machine reading com- prehension. In International Conference on Learning Representations, 2020. URL https
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 273-283, Melbourne, Australia, July 201 8a. Association for Computational Linguistics. doi: 111118665/1/11... 1026.
URLptttss: /tt/://wwwww /www acwweb. org/
antho anthellogyPI331128
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequencc learning using gated graph neural networks. arXiv preprint h00v:/806..333 2018b.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2019.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632 642, 2015.
Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Berlin, Germany, August 201 2a. Association for Computational Linguistics. doi: 10. 10.18633/11/P11-123 12233. URL. hntppa
/ //ww, wwwwwb.wwwwbco.... crg/ http:///rrr.....ooooooo//-------
Danqi Chen, Jason Bolton, and Christopher Manning. thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2358-2367, 2016b.
Jifan Chen, Shih-Ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. ArXiv, abs/1910.022101 2019a.
Jifan Chen, Shih-tin ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. arXiv preprint arXiv:1910.02610, 2019b.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang. Ting Liu, and Guoping Hu. Attention-ovee-attentiin neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 593-602, 2017. URL hstps://wwwk ac Lweb. org/ http://aanttooooooo1-11555 .pÉ
Eresst Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 9009):92-103, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp.
Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//19- 1423. URL attps
wwww.acc. web. wwwp.e.....tahaaa//eeeaa http://aannallloyllllllga 1423
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang. William Cohen, and Ruslan Salakhutdinov. Gated- attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1832-1846, 2017. URL //1.//11171118..11111 16..pdf
https: https///www... ac wew. wwwpw//...//.ooo.oooogoooo/////e
10

---

Under review as a conference paper at ICLR 2022
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computatiinnal Linguistics, pp. 2694-2703 2019.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 2378-2378, 2019.
Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguisties, 8:34-48, 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanee bert with disentangled attention. arXiv preprint arXiv:2006. 03654, 2020.
Karl Moritz Hermann, Tomá Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Proceedings of the 28th International Conference Neural Information Processing Systems- Volume 1, pp. 1993-1701, 2015.
Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. Read+ verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6529-6537, 2019. URL https:://wvv
aaai.org/o index.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2391-2401, 2019.
Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. DAGN: Discourse-aware graph network for logical reasoning. In NAACL, 2021.
Lucja Iwarska. Logical reasoning in natural language: is all about knowledge. Minds and Machines, 3(4):475-510, 1993.
Divyansh Kaushik and Zachary C. Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference Empirical Methods Natural Language Processing, pp. 5010-5015, Brussels, Belgium, October-November
2018. Association for Computational Linguisties. doi: 10.18653/ URL https
7/www. acl
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL https //axxiv abs////////////
abs/ 1412. 6980
Siddharth Krishna, Alexander Summers, and Thomas Wies. Local reasoning for global graph properties. In European Symposium Programming, pp. 308-335 Springer, Cham, 2020.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 20/7 Conference on Empirical Methods Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017, Association for Computational Linguistics. doi: 10. 10.18653/11/D17-11000 URL
https
Friedrich Wilhelm Levi. Finite geometrical systems: six public lectues delivered in February, 1940,
at the University of Calcutta. University of Calcutta, 1942.
11

---

Under review as a conference paper at ICLR 2022
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI-20, pp- 3622-3628. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 0..24963/jccai.222002
URL httpttss.///....r///....
Main track.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints, art. MIXiv:190..11992 July 2019.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8449-8456, 2020.
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1506-1515, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10.18653w1/D17-11559 URL
https: /tps::/wwwes /www ac I web. org htt.aaooooo//bl-111/5555
Ndapandula Nakashole and Tom Mitchell. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1009-1019, 2014.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, 2020.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. The penn discourse treebank 2.0. In LREC. Citeseer, 2008
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6140-6150, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 10111665/11/11-11266 URL https: wttp://wwwwwc/.o hwtp::wwwwwww........... acwweb. org/
Sethollogy/D16-1255
Qiu Ran, Peng Li, Weiwei Hu, and Jie Zhou. Option comparison network for multiple-choice reading comprehension. arXiv preprint arXiv:1903.03033, 2019.
Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances Neural Information Processing Systems, 33, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguisties, 8:642-866, 2020.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. 2002.
Mrinmaya Sachan and Eric Xing. Machine comprehension using rich semantic representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 486-492. 2016.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne vanden Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (eds.), The Semantic Web, pp. 593-607. Springer International Publishing, 2018.
12

---

Under review as a conference paper at ICLR 2022
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR 2017, 2017. URL https::/openreviev.. net/ feterences//od22.d4BB..1111
Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. Neural logic reasoning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1365-1374, 2020.
Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks. arXiv preprini arXiv: 1809002000 2018.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegar question answering challenge targeting commonsense knowledge. In NAACL-HLT (I), 2019.
Siyuan Wang. Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. Logic-driven context extension and data augmentation for logical reasoning of text. arXiv preprint arXiv:2105.03655 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers), pp. 11 1112-1122, 2018.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2380, 2018
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. SG-Net: Syntax- guided machine reading comprehension. In Proceedings of the Thirty- Fourthh AAAI Conference on Artificial Intelligence (AAAI), 2020a.
9988 05w.05114477 .p.
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. Retrospective reader reader for machine reading comprehen- sion. arXiv preprint wwXiv22001.09694, 2020b.
09694
pdf
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, M. Zhou, Jiahai Wang, and Jian Yin. Reasoning over semantic-level graph for fact checking. In ACL, 2020.
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. AR-LSAT: Investigating Analytical Reasoning of Text. arXiv e-prints, art. arXiv:2104.06598, April 2021.
Ming Zhou, Nan Duan, Shujie Liu, and Heung-Yeun Shum. Progress in neural nlp: Modeling, learning, and reasoning. Engineering, 6(3):275-290 2020. ISSN 2095-8099. doi: https://doi.org/ 10. 10.1016/jengg2201112..444 URL https: http://wwwweeeeeeeeeeee/e/////ee com/ wwtps//iee///////ccccl.lo. pi b11/2295800919304922
13

---

Under review as a conference paper at ICLR 2022
A
LEVI GRAPH CONSTRUCTION
Levi graph transformation turns labeled edges into additional vertices. There are two types of edges in traditional Levi graph: default and reverse. For example, an edge (E1, RI, E2) in the original graph becomes (E1, de fault, R1), (RI, fault,, E2), (RI, reverse, E1) and (E2, reverse, RI)
However, the type of source and target vertices in the graph also matters (Beck al.) 2018b). Specifically, previous works use the same type of edge to pass information, which may reduce the effectiveness. Thus we propose to transform the default edges into default-in and default-out edges, and the reverse edges into reverse-in and reverse-out edges.
B DETAILS FOR DATASETS AND BASELINE MODELS
In this section, we describe the datasets and baseline models used in the experiments.
B.1 DATASETS
ReClor contains 6,138 multiple-choice questions modified from standardized tests such as GMAT and LSAT, which are randomly split into train/dev/test sets with 4,638/500/ 1,000 samples respectively. It contains multiple logic reasoning types. The held-out test set is further divided into EASY and HARD subsets based on the performance of BERT-based model Devlin et al. (209)).
ReClor
LogiQA LogiQA consists of 8,678 multiple-choie questions collected from National Civil Servants Examinations of China and are manually translated into English by experts. The dataset randomly split into train/dev/test sets with 7,376/651/65) samples correspondingly. LogiQA also contains various logic reasoning types.
MuTual MuTual has 8,860 dialogues annotated by linguist experts and high-quality annotators from Chinese high school English listening comprehension test data. is randomly split into train/dev/test sets with 7,088/886/886 samples respectively. There more than 6 types of reasoning abilities reflected in MuTual. MuTual is an advanced version, where one of the candidate responses is replaced by a safe response (e.g., "could you repeat that?") for each example.
B.2 BASELINE MODELS
DAGN explores passage-level discourse-aware clues used for solving logical reasoning QA. Speciiii cally, they leverage discourse relations annotated Penn Discourse TreeBank 2.0 (PDTB 2.0) (Prasad ct al. 2008) and punctuation the delimiters to split the context into elementary discourse units (EDUs). They are further organized into logical graph and feed into a GNN to get the representation.
LReasoner is symbolic-driven framework for logical reasoning of text. It firstly identifies the logical symbols and expressions explicitly for the context and options based on manually designed rules. Then it performs logical inference over the expressions according to logical equivalence laws such as contraposition (Russell & Norvig 2002). Finally, it verbalizes the expressions to match the answer.
FOCAL REASONER enjoys two major merits. (1) Broader knowledge: Compared with DAGN which uses sententious knowledge such logical connectives (e.g., becaus, however), FOCAL REASONER leverages broader type of knowledge characterized by "fact unit", including global knowledge and local knowledge. (2) More transferable: Compared with LReasoner which manually design rules to extract logical patterns and perform logical reasoning in symbolic way, Focal Reasoner is neural-based and manual-free, which is more generalizable to other datasets.
MODEL COMPLEXITY
this section, we display the statistics for the parameters of FOCAL REASONER and baseline models to demonstrate the model complexity and the effectiveness.
14

---

Under review as a conference paper at ICLR 2022
that the increase of parameters no more than 15% compared with the bascline models, and is comparable with strong baseline model DAGN. This indicates that, model, being quite effective, is not a result of stacking complicated mod-
From Table g
Table 8: Distribution of fact unit number on dev set of the training datasets.
RoBERTa DAGN
FOCAL REASONER
355M
395M
409M
ules.
D VARIANCES FOR FOCAL REASONER W.R.T EXPERIMENT RESULTS
In this section, we report the average and variances run on random seeds for FOCAL REASONER with different pre-trained language models.
Table 9: Experimental results for FOCAL REASONER with average results and variances run no random seeds.
ReClor
LogiQA
Model
Dev
Test
Test-F
Test-H
Dev
Test
FOCAL REASONERROEERT
66.8110.13
58.810.14 14
76.910.16
44.5:1.112 62.910.13
41.010.11 47.310.16
40.3+0.155 45.810.17
FOCAL REASONERDaBERI.
78.6:10.18
73.2:10.17
86.2110.1
E
FURTHER INTERPRETATION
We change the example in Figure 7 bit. Specifically, we change the conclusion from "playing football can improve students' academic performance" "football players are held to higher academic standards than non-athletes". We can observe that our model can select the relatively correct answer, which indicates that FOCAL REASONER has some logical reasoning ability, instead of simple text matching and mining ability.
A recent survey in key middle school showed that high school students in this school have special preference for playing foctball, surpasses other balls. survey also than students who do not often play football, Therefore, football plavers are held to higher academic standards than non-athletes.
students regularly play football better academic performance
A, Only high school students who ranked in the 30% grades often play football.
Which of the following
B. Regular football can exercise and maintain strong learning energy.
C. Often playing football delays the study
weaken the above conclus cosncusion
D. Research has not proved that playing football can contribute to intellectual development.
1
1. students have preferences
2. preference playing football
it surpasses balls
4. who play football
QA
students better porformance
who lplay football
7. players higher standards play Icontribute davelopment
Fact Units
Anigi
Figure 8: An example of how our model reasons get the final answer the modified example.
15


Reviews:
[{"review_id": "swfjZpjnLnH", "reviewer": "Reviewer_vNnA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The approach is interesting boasts great results on two benchmarks compared to previous approaches. \n2. The authors also perform useful analysis on the model which is interesting and would be useful to the community. \n3. The approach is, to some extent, described well. \n4. The authors also plan to open source their code which would be beneficial to the community. \n\nConcerns, questions, and suggestions:\n1. In the related works section, the authors fail to describe and differentiate themselves from the two baselines that they compare against (DAGN and LReasoner).  \n2. In the related work section, the authors describe a slew of other methods that the model is not compared against. This makes it hard to place their approach in comparison with prior work thus making it hard to understand the novelty of the approach. \n3. Perhaps, the authors could add a section describing related works that are used for the tasks described. (The authors claim that the baselines are described in the appendix but that is not the case)\n", "summary_of_the_paper": "The authors propose a framework, called Focal Reasoner, to perform logical reasoning to answer questions. The proposed approach first extracts the facts from raw text. The authors describe a fact unit as a triple of (argument, predicate, argument). These collections of facts can be viewed as a graph with predicates as undirected edges between arguments. After forming a subgraph, reasoning is performed by a graph convolution module to predict the correct answer. \nThe authors show that their approach outperforms existing approaches on two benchmarked datasets.\nThe authors also perform a detailed analysis that sheds more light on the internal workings of the model.", "summary_of_the_review": "The approach is interesting but it is hard to understand what aspect is novel compared to other competitive approaches. Perhaps the authors should describe other methods that have been used to solve this task. This would help make it easier for the reader to understand the novel contribution of the paper and also makes the paper more complete by itself. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kkXt0oJFGW1", "reviewer": "Reviewer_UKpM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As I served as one of the reviewers for this paper before, I am also sharing a summary or quote of reviews from the last round here as the paper did not go through major changes since the last submission. Content-wise, $LReasoner_{RoBERTa}$ , $LReasoner_{DeBERTa}$ , as well as $FocalReasoner_{DeBERTa}$ were added as new result in Table 1. \n\n### Strength\n\n- (From the previous review) All reviewers noted that the main strength of the work to be its good empirical results on the 3 benchmarks considered.\n- This strength is corroborated with the additional results of competitive baselines and  $FocalReasoner_{DeBERTa}$.\n- The paper is well-written overall. I appreciate the authors making an effort to organize and clarify the complex steps that this paper takes. \n\n### Weakness\n-  (From the previous review) The unanimous criticism was that while the results are impressive, given the complex nature of the proposed system, it was difficult to understand which aspects of the proposed techniques are truly effective.\n- While the coreference, and entity linking are the major resource in creating the supergraph, I don't think the challenges nor machinery in getting this information is well described.  (It might be in the text, but I am not sure how the authors get dependency parse trees as well.) It would be nice to have  more description on the challenges of identifying the global relations (same entity, coreference)\n- As described in the *summary* section, I think the main contribution of this paper is in getting fine-grained facts and connecting them with global information. I am not entirely convinced whether this is a very novel approach. What is the major scientific contribution here? \n- The authors attempt to decompose some parts of the method in Tables 5 and 6. Related to the earlier weakness on the complex nature of this work, I am not sure what role the *logical fact regularization loss* serves in this paper. Is this one of the contributions of this paper? Or is it simply a tool to get a better results? Without it, the performance 66.8 --> 64.2 which is lower than competing models.\n \nSome of the comments that other reviewers made from the last round of review (which resonated with me):\n- \" **Significance**: This is where I am most conflicted about the paper. The results are difficult to interpret. Certainly, there are gains from the paper's technique and the ablations show that the different components of the model all contribute to the performance. But when fairly important elements of the model like coreference edges contribute only around 2% accuracy, it's hard to know how much to trust the narrative here about reasoning.\"\n- \"More generally, in a multiple-choice task, it's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines. It's not easy to form an apples-to-apples comparison in terms of number of parameters. The case study in Section 5.3 shows that something is happening in the graph component, but whether this is *causally associated* with the ability to get the right answer as opposed to a byproduct is hard to determine.\"\n\n### Questions/Suggestions\n\n- I am not sure what each edges really do: (*default-in, default-out, reverse-in, reverse-out, self*). It would be nice if authors can describe the roles of these edges.\n- Figure 5's *global edge* and what this paper claims as *global (typical entity or sentence relation)* is confusing. Maybe try to use distinguishable terms?\n- Why don't you update 4.3 (1) sentence to match newly updated best result table 1?\n    - \"FOCAL REASONER also outperforms the prior best system LReasoner, reaching 77:05% on the EASY subset, and 44:64% on the HARD subset.\"", "summary_of_the_paper": "The paper claims that a lot of previous work in QA-based MRC focused only on entity-aware common sense knowledge.  To overcome this, the paper proposes to build a finer-grained local fact, (entity, predicate, entity) triplet, information based on dependency parsing, and then connect the triplet nodes with global information such as coreference, entity information. \nArchitecture-wise, a passage, question, and options are passed to RoBERTa (or DeBERTa) and then this information goes through the graph-attention network (GAN), based on the graph structure described above. In the experiment section, the authors show that the proposed FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 5,6).\n\nThe main contribution of this paper, as I understand, is bringing more fine-grained local facts and connecting them with global information such as coreference and entity linking.", "summary_of_the_review": "FocalReasoner exhibits a strong experimental result and it is corroborated in the new submission with added experiment results in $LReasoner_{RoBERTa}$ , $LReasoner_{DeBERTa}$ , as well as $FocalReasoner_{DeBERTa}$ in Table 1. \n\nAlbeit its high-performance, as the previous review pointed out, I am not sure what is the major factor in the improvement when many parts of the neural machinery are used together. I view the main contribution as using the Levi graph from dependency trees as  Beck et al. 2018 and connecting them with coreference and entity linking.\n\nOverall, I'm borderline about this paper. I think this paper tests whether fine-grained pieces of information in the paragraph are useful or not (intuitively they should, and this paper shows that it does). However, I am not sure whether the usage graphs are novel or significantly better than previous work. \n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZwfEsJzN7re", "reviewer": "Reviewer_MQGH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well-written with clear motivations and structure. The concept of fact units is interesting and novel which are easily constructed via dependency trees. Compared with commonsense and entity-relation knowledge, the fact units are more informative to each specific question. By associating nodes across different fact units based on coreferences and mentions, a supergraph is built that connects all related information and conducts graph reasoning for answer predictions. The proposed model is then experimentally verified on three logic-driven datasets which demonstrates some performance gain.\n\nWeaknesses:\n1. Some of the details for model description are missing and confusing, e.g., (1) How the representation is initialized for a supernode corresponding to a fact triplet and how does the supernode propagate information to the sub-nodes? (2) Equation 2 and 3 use the same graph encoder $F_{G}$. Are they the same? Since they both use $m$ nodes, does it mean the supergraph also operates on each sub fact node? If so, does the global node $V_g$ connect to all the sub fact nodes? It is then not clear how each fact block (grey squares in Figure 4) functions as a whole. The notations in equation 2 and 3 are also inconsistent with equation 5. What does L, S, F and m represent?  (3) The loss for answer prediction in Eq 6 is not clearly described: do you use an aggregated embedding over all nodes for prediction? (4) How exactly is the interaction module processed?\n2. According to the description, the fact units are constructed using the dependency parser. Hence, the model relies on whether the parser accurately discovers the crucial information. I am also wondering if the extracted facts could bring too much noise to the question. As shown in Figure 6, when more than 12 facts are constructed, the performance becomes worse. \n3. More discussions on comparing with symbolic logic reasoner model LReasoner are needed. Indeed, the proposed graph model seems to only implicitly convey knowledge across facts in terms of local reasoning. On the other hand, the symbolic logic rules are able to express global patterns. Hence, to my understanding, it may not be suitable to use the term \"global reasoning\" in this work.\n4. For experiments, Table 1 shows the FOCAL Reasoner (DeBERTa) brings much performance gain, compared to FOCAL Reasoner (RoBERTa) (the performance seems comparable with DAGN and LReasoner using RoBERTa). Can you explain why? Is it possible to run DAGN (DeBERTa)?", "summary_of_the_paper": "This paper presents a novel graph reasoning model based on supergraphs constructed via fact triplets. The authors define the fact triplets as subject-predicate-object relational path extracted from the dependency parser and use them as local knowledge, compared with the commonly used entity-relation knowledge to enhance the task of logic-driven question answering.  ", "summary_of_the_review": "Novel concept of fact units to answer logic-driven questions. More discussions are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JTAlexPz0XG", "reviewer": "Reviewer_fjem", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* The model seems fairly intuitive. The idea of representing the (entity + relational) structure of a text directly and propagating information through that structure has been prevalent in the IE / RC space for a while but it seems that many have had difficulty getting it to work well. This paper shows improvements which are big enough to indicate that there's probably something to it (though I have a few concerns about this, described in the weaknesses). It seems to me that the key to the success of the model is probably just in the distribution of data that it's applied on. Such approaches may not have worked for earlier datasets (such as SQuAD) simply because they were not suited to teach or test the kind of information aggregating abilities that this model is designed to do.\n\n* The ablations are nice.\n\n## Weaknesses\n\n* I take issue with some of the terminology — use of the terms \"fact\" and \"logical reasoning\" are, in my view, a bit misleading as descriptions of what a model like this one is doing. The tuples being extracted from the text are more like \"propositions\" than \"facts\" and the operations in the GCN don't seem to have much in common with \"logical reasoning\" (which normally involves the application of rules from a formal system). I understand there seems to be a bit of precedent for the terms here, as the datasets being used were designed to capture aspects of logical reasoning, but I think it's less correct to reuse them when describing model components such as \"Logical Fact Regularization.\" Something like \"propositional structure regularization\" might be more appropriate.\n\n* I'm worried that this model's improvements will end up being specific to the distribution in \"logic\"-focused datasets like ReClor and LogiQA, because of these datasets' focus on aggregating and combining information about a small set of entities in a paragraph. While I understand that this capability is the focus of this work, I think it's important to know if the proposed architecture is too specialized: does it maintain performance over its RoBERTa or DeBERTa baseline if applied to more naturalistic, extractive reading comprehension datasets like Natural Questions, QuoRef, or HotpotQA? If not, is it because of the distribution of text, extractive format, or other factors? It's fine if the the model doesn't end up improving these cases — I think investigating these issues would greatly improve the paper either way.\n\n* I would suggest more caution when it comes to interpretability. Attention weights cannot be relied on to provide explanations of model behavior. To make the interpretability case study in Section 5.3 more believable, I would suggest modifying the attended element and seeing if the answer changes: for example, change \"playing football can improve students' academic performance\" to \"football players are held to higher academic standards than non-athletes\" and see if the attention pattern or the answer changes. If it doesn't, then the attention pattern — while still indicative of the information relevant to answer the question - might not be indicative of the kind of logical reasoning we suspect.\n\n* I'm not sure if the findings are statistically robust. All of the test sets here are very small — less than 1000 items. The numbers seem good but I'm not sure what the uncertainty looks like. Please provide confidence intervals for the Focal Reasoner and at least some of the stronger baselines.\n\n* Are the baselines well-tuned? Time and time again, modeling innovations are proposed which ultimately get beaten by a better-tuned baseline down the line. How many experiments did you run for each model variant? Are you reporting the best of several runs? How did the performance of each model vary over runs? Can you say that the distribution of the new model's performance is appreciably better than the baseline models? By how much, and with what confidence?\n\n## Minor comments\n\n* In equations 4 and 5, wouldn't it be better to use $v_\\text{predicate}$ instead of $v_\\text{relation}$? It's a bit confusing to me to use the word \"relation\" here since earlier the word is used to refer to the types of edges between vertices in the graph.", "summary_of_the_paper": "This paper proposes a model architecture for question answering which takes advantage of shallow proposition structure and coreference links using (relational) graph convolutional networks. Subject-predicate-object triples (dubbed \"fact units\") are extracted from the text using dependency parsing, extra coreference links are added using a coreference system, and the resulting graph is initialized with representations from a contextualizing encoder, passed through a GCN, and recombined with the contextual representations via multi-headed attention. The resulting representations serve as input to the final classification layer for answer prediction, as well as a \"logical facts regularization\" step which encourages the final representation of the object in each fact unit to align (via cosine) with the sum of the subject and predicate representations.\n\nIn experiments on several datasets designed to test logical reasoning, the proposed architecture scores higher than several baselines.", "summary_of_the_review": "The model seems nice and the results seem positive, but I think the experiments are weak. We don't really learn much from this paper about the strengths and limitations of the proposed model, and it seems plausible that the experimental results are not statistically robust, due to a combination of the small test set size and the limited experimental reporting that is done in the paper.\n\n---\n\nEDIT: Some of my concerns were addressed, particularly the issues regarding reporting more details on the model's performance (number of runs, variance, etc.) which I think were most critical. I am still not satisfied with the discussion of interpretability (which I think should be removed; see discussion below) and I think the paper could have done a better job with experiments demonstrating the relative strengths and weaknesses of the model, but the results that are present seem strong. So I wouldn't say I'm particularly eager to accept the paper but I won't gatekeep it either. Raising my score from a 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
* Well-written paper
* Strong empirical results on three benchmarks
* Interesting approach of producing semantically augmented LMs using dependency parses to extract svo triples, and finding coreferences between them across multiple sentences

Weaknesses:
* None of the reviewers seem particularly excited about the paper
* Stronger baseline comparisons would have improved the paper
* Authors re-define a lot of terminology, but the novelty of the method is more from the type of graph used to initialize their method, which seems to be a function of OpenIE triplets
Decision: Reject