Fact-driven Logical Reasoning
Authors: Siru Ouyang
Zhuosheng Zhang
hai zhao
Venue: NeurIPS 2021 Submitted (2021)
Keywords: logical reasoning
machine reading comprehension
language understanding

Abstract: Logical reasoning deeply relies on accurate, clearly presented clue forms which are usually modeled as entity-like knowledge in existing studies. However, in real hierarchical reasoning motivated machine reading comprehension (MRC), such one-side modeling are insufficient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to. Thus, in view of language being a complete knowledge/clue carrier, we propose a general formalism to support representing logic units by extracting backbone constituents of the sentence such as the subject-verb-object formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logical reasoning. Beyond building the ad-hoc graphs, we propose a more general and convenient fact-driven approach to construct a supergraph on top of our newly defined fact units, and enhance the supergraph with further explicit guidance of local question and option interactions. Experiments on two challenging logical reasoning MRC benchmarks show that our proposed model, \textsc{Focal Reasoner}, outperforms the baseline models dramatically.

Fact-driven Logical Reasoning
Anonymous Author(s)
Affiliation Address
email
Abstract
Logical reasoning deeply relies on accurate, clearly presented clue forms which are usually modeled entity-like knowledge in existing studies. However, in real hierarchical reasoning motivated machine reading comprehension (MRC), such one-side modeling insufficient for those indispensable local complete facts or events when only "global" knowledge is really paid attention to. Thus, view of language being complete knowledge/clue carrier, we propose general formalism to support representing logic units by extracting backhone constituents of the sentence such as the subject-verb-objeet formed "facts", covering both global and local knowledge pieces that are necessary as the basis for logical reasoning. Beyond building the ad-hoc graphs, we propose a more general and convenient fact-driven approach to construct supergraph on top of our newly defined fact units, and enhance the supergraph with further explicit guidance of local question and option interactions. Experiments on two challenging logical reasoning MRC benchmarks show that our proposed model, FOCAL REASONER, outperforms the baseline models dramatically.
3
5
10
11
12
13
14
Introduction
17
Machine reading comprehension (MRC) requires machine to answer question according to given Logical reasoning S from MRC accounts for human intuition about entailment
passage [
19
of sentences and reflects the semantic relations between sentential constituents 66] Recently, there is surging trend of research into logical reasoning ability, among which ReClor and LogiQA (5) two representative datasets introduced to promote the development of logical reasoning, where logical reasoning questions selected from standardized exams such as GMAT requiring models to read and comprehend the complicated logical relationships. Similar to the standard question-answeriig (QA)-based MRC tasks in form, our concerned logical reasoning QA tasks contain three elements: passage, question and the candidate options as examples shown in Figure [
20
21
22
23
24
25
MRC models usually exploit a pre-trained language model (PrLM) as key encoder for effective contextualized representation. Meanwhile, the major challenge of logical reasoning to uncover logical structures, and reasoning with the candidate options and questions predict the correct answer. However, it is difficult for PrLMs to capture the logical structure inherent in the texts since logical supervision is rarely available during pre-training. Existing logical reasoning has shown serious dependence on knowledge-like clues. This is due to the lengthy, noisy text in human language
27
29
30
31
32 which is though natural carrier of knowledge but does provide clean, exact knowledge form.
33 Thus, an increasing interest is using graph networks to model the entity-aware relationships in the
34 passages [8] 10 II]. However, all these methods may insufficiently capture indispensable logical
35 units from two perspectives. First, they mostly focus on entity-aware commonsense knowledge, but
36 pay little attention to those non-entity, non-commonsense clues [2] Second, when existing models
_est
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Question
Passage
Answer
|Example
Xiso Xiso Zhao is taller than Xiao Qian, Xino shorter than Xiao Sun, and Xiso Sun shorter than Xiso Qian.
Xiso Wang is
Xiso is shortor Xiso Zhao.
B. Xiso Wang is taller than Xiso Zhao,
From this we know
C. Xiso Sun is shorter than Xiao Wang.
D. Xiao Sun is taller than Xiao Zhao.
(Example 2
Many other animal species from same era
A large enough comet colliding Earth could have caused cloud
become extinct at the same time the dinosaurs did.
B. It cannot be determined from dinosaur skalotons whether
Which one the follow- ing statements, most seriously weakens the argument?
of dust that enshrouded the planet and cooled the climate long enough
the animals died from the effects of dust cloud.
consequences for vegetation collding with Earth are not fully understood.
animals comet
result in the dinosaurs' demise.
D. Verious species of animals from the same are and similar
to them habitat and physiology did not become extinct.
Two examples from LogiQA and ReClor respectively are illustrated. There are arguments and relations between arguments. Both emphasized by different colors: arguments, relations. Key words in questions are highlighted in Purple. Key options are highlighted in gray.
Figure I:
extract predicate logic inside language into knowledge, they only exploit quite limited predicates like hasA and isA but ignore a broad range of predicates in real language. From either of the perspectives, the existing methods actually only concern about those "global" knowledge that keeps valid across the entire data, without sufficient "local" perception of complete facts or events in the given specific part of MRC task. We argue such insufficient modeling logic units roots from the ignorance of language itself being the complete knowledge/clue carrier. Thus, we propose extracting a kind of broad facts according to backbone constituents of sentence to effectively cover such indispensable logic reasoning basis, filling the gap of local, non-commonsensee non-entity, or even non- knowleelwee For example, these units may reflect the facts of who did what to whom, or who is what in Figure 3 Such groups can be defined as "fact unit" following in Definition [ The fact units are further organized into a supergraph following Definition ?
38
39
40
41
42
43
44
45
clues in existing methods as shown in Figure 2
46
47
48
{E,, E2), where Ey and Eg are arguments (including entity and non-entity), P is the predicate between them, a fact unit F is the set of all entities in T and their corresponding relations.
Definition (Fact Unit) Given an triplet
49
50
Definition (Supergraph) supergraph is a structure made of fact units (regarded as subgraphs) 52 as the vertices, and the relations between fact units as undirected edges.
51
53 As shown in Figure 2 we regard the defined
54 fact as the results of syntactic processing, rather
(s) sporoaches
(6) obtained knowledge
55 than those from semantic role labeling (SRL) as
Existing Works
FACT (Logical Structure)
56 in previous study, thus the proposed fact also extends the processing means in existing work.
Non- Common-
57
Common- sense
58 Correspondingly. in this work, we propose fact-driven logical reasoning model, called FOCAL REASONER, which builds supergraphs 60
SRL
(semantic)
sense
61 on top of fact units as the basis for logical reasoning, to capture both global connections 82
Named Entity
Non-
Named
between facts and the local concepts or actions inside the fact. In addition, we strengthen our interaction.
Ours (syntactid
globa'
Entity
64
model by the question-option-nnn Specifically, we explicitly reformulate questions with negation expressions to compensate for the insensitiveness of PrLMs, all of which are interacted in our supergraph. Such resulted FOCAL REASONER is evaluated on two challenging logical reasoning benchmarks including ReClor, LogiQA, and one dialogue reasoning dataset Mutual for generalizability, achieving new state-of-the-art results.
Figure 2: Our "fact" V.S. existing approaches.
66
88
89
70
2 Related Work
71
72
Machine Reading Comprehension Recent years have witnessed massive researches on Machine Reading Comprchension, which has become of the most important areas of NLP [ 15 16

---

74
17
19
20
21 22
Despite the success of MRC models on various datasets such as CNN/Daily
75
Mail M. SQuAD 2
RACE B and so on, researchers began to rethink to what extent does the
76
problem been solved. Nowadays, there are massive researches into the reasoning ability of machines.
77
According to 23 24 25], reasoning abilities can be broadly categorized into (1) commonsense
78
reasoning 26 27
28
29]; (2) numerical reasoning [30; (3) multi-hop reasoning [31] and (4) logical
79
reasoning [5 7 among which logical reasoning is essential in human intelligence but has merely
been delved into. Natural Language Inference (NLI) [32 33 [4] is a task closely related to logical reasoning. However, it has two obvious backs measuring logical reasoning abilities. One is that it only has three logical types which are entailment, contradiction and neutral. The other is its limitation on sentence- levell reasoning. Hence, it is important to research more comprehensive and deeper logical reasoning abilities.
81
83
85 Logical Reasoning in MRC There are two main kinds of features in language data that 87 would be the necessary basis for logical reasoning: 1) knowledge: global facts that keep consistency regardless of the context, such commonsense, mostly derived from named entities; 2) non-knowledge: local facts
colliding
coreference same entity
89
comet
Earth
91
result
92 or events that may be sensitive to the context, mostly derived from detailed language. Existing works have made progress in improving logical reasoning ability (8 10 12, 38). However,
caused
93
comet
demise
comet
94
95
96
these approaches are barely satisfactory as they mostly focus on the global facts such as typical entity or sentence-level relations, which are obviously not sufficient. In this work, we strengthen the basis for logical reasoning by "facts".
enshrouded
cooled
97
dust
98
planot
dust
climate
99
100
unifying both types of the features
Figure 3: An example of constructed supergraph. In contrast, the dotted vertices and edges focused in most existing studies [35 36 37
102
Different from previous studies that focus on the knowledge components, we propose fact- driven logical reasoning framework that builds supergraphs top of fact units to capture both global connections between entity-aware facts and the local concepts or events inside the fact.
104
106
Approaches
107
108 In this section, we will describe our method in detail. The overall architecture of the model is shown in Figure 4. We first construct supergraph from the raw text based the fact units extracted.
110
Then we conduct reasoning over the supergraph with question-option guided approaches to learn and update the features, which are further incorporated answer prediction.
111
3.1 Supergraph Construction
Figure iillustrates our method for constructing supergraph from raw text inputs. The first step is to obtain triplets that constitute a fact unit. To keep the framework generic, we use a fairly simple fact unit extractor based on the syntactic relations. Given context consisting multiple sentences, we first conduct dependency parsing of each sentence. After that, we extract the subject, the predicate, triplets corresponding to
114
115
116
and the object tokens to get the "Argument each sentence in the context.
118
119
With the obtained triplets, the fact units are organized in the form of Levi graph 39 which turns arguments and predicates all into nodes. An original fact unit is in the form of F (V, E, R). where V is the set of the arguments, E is the set of edges connected between arguments, and R is relations of each edge which are predicates here. The corresponding Levi graph denoted as Fi (Vr, Fr,, R2.) where Vz = V R, which makes the originally directly connected arguments be intermediately connected via relations. As for R,, previous works such as 44) 41 designed three
120
121
122
124
types of edges Rt= fault, reverse, self) to enhance information flow. Here in our settings,
3

---

Logical Facts Regularization
coreference
passage
Answer Prediction
Cross Attention
RoBERTa
Fact-driven Ressoning
question option
OQ-Pair
Interaction
Co-attention
Maxlinging
OQ-Pair
OQ-Pair
OQ-Pair
Figure 4: The framework model. For supergraph reasoning, in each iteration, each node selectively receives the message from the neighboring nodes to update its representation. The dashed circle means zero vector.
128
we extend it into five types: default-in, default-out, reverse-in, reverse-out, self, corresponding to the directions of edges towards the predicates.
128
We construct the supergraph by making connections between fact units Fi. In particular, we take three strategies according to question-option, identical concept and co-reference information. (1) For question-option pair, We initialize global node VG with its representation and connect all the fact unit nodes. The edge type are set as global. The global node ensures that all fact units are connected (2) There can be identical mentions in different sentences, resulting in repeated nodes in fact units. We connect nodes corresponding to the same non-pronoun arguments by edges with edge type same. (3) We conduct co-reference context using off-to-shelf mode order identify arguments in fact units that refer to the same one. We add edges with type coref between them. The final supergraph is denoted S (F) U Vg.. E) where E is the set of edges added with the previous three strategies.
130
132
so that information can be exchanged during graph encoding.
134
resolution
136
138
3.2
Encoder
139
3.2.1
Context Encoder
140
Our context encoder Fc(.) is initialized with a pre-trained language model, i.c.. RoBERTa-large Question, context and option are concatenated and then fed into the encoder. If the question detected to contain negative meanings, we add special token <pos> before the question, else we add <neg>. whole, we get the hidden representation as following:
142
143
(h Medet11 hgg1,... ... hooto FoltFe.n..., Telle Ig I.0. To,1: a. Iolo+1t),
(1)
where Fe,0 =<S> Bel.t1 Lod.+1 = =
, Fq.00 =<pos><<neg> and h2 E Idd, d is the hidden size.
144
3.2.2
Supegraph Encoder
146
Graph Initialization Fe(-) encodes each token in nodes Vz, and then the averaged hidden state used as the initial representation of the original word of each node, because PrLMS like RoBERTa take subwords as input while our triplets extraction performs in word-level. For the global QA-context node, we averaged the embeddings of tokens in question and option for initialization. We also use one-hot embedding layer to encode the relations between two nodes.
148
150

---

A large enough comet colliding with Earth could have caused cloud of dust that en- shrouded the planet and cooled the olimate long enough to result in the dinosaurs' demiss..
coreference ently
Exrt
cornet colliding - Earth caused dust dust enshrouded planet dust cooled comett result demise
demile
coole))
Setgrapt coneecis
corf cOnnaCto1
[Mid
Which the following .... most sertously weakenss the argument?
Various species of animals from the same era as dinosaurs and simiarr them did not become extinct when the dinosaurs did.
The process of constructing the fact chain and its corresponding Levi graph form of an example in Figure Entities and relations are illustrated in its corresponding color.
Figure 5:
Graph Attention Network Based on the relational graph convolutional network 43 and given the initial representation hy for every node vi, the feed-forward or message-passing process with information control can be written as:
152
(1+1) ReLU Σ Σ h:
1 Crr
9
VERI. ByEN-(e))
154
where Nf (v:) denotes the neighbors of node v; under relation r and G.r is the number of those nodes.
WP (1)
(1)
155
the learnable parameters of layer 9q is gated value between 0 and 1.
156
Through the graph encoder F.(.), we then obtain the hidden representations nodes in fact units as: (he h = Fa(fvL.:.....L... } EL).
(3)
158
These features are further concatenated to get the final node representation of the supergraph:
(.. Pa(th Ec).
159
For node features on the supergraph, fused via the attention and gating mechanisms with the original representations of the context encoder. Specifically, denoting the original whole sequence representation after context encoder as HC, we apply attention mechanism to append the supergraph representation to the original one:
161
Attn Ky, Vy),
(5)
163
where {Ky, Vi} are packed from the learned representations of the supergraph. We compute A [0, to weigh the expected importance of supergraph representation of each source word:
184
A1 < + U2H*),
where WA and UΛ are learnable parameters. Hº and H are then fused for an effective representation: 186
H + AA RAx
187
3.2.3
-ptso-an-O Interaction
188
Options have their inherent logical relations, which can leveraged to aid answer prediction. Inspired
by
44
we use attention-based mechanism to gather option correlation information.
170
Specifically for option O,, the information it get by interaction with option O, is calculated
o. (1) = i 0² Att 03: v): 0f o Att,
where 0 is the representation of the concatenation for the i-th option and question after the context encoder. Then the option-wist information are gathered to fuse the option correlation information:
172
Ó tanh (0 Jesss + bc),
(9)
5

---

where We Idx and be E R Finally, gating mechanism is used to fuse the option features:
173
0 Li Di.sk O 1isk + (1 gi.ck) Ōi,kk
(10)
where the Ii.kk = ( k 0 de Q] + by) E R is the i-th column of gate g.
174
175
3.3
Hierarchical Decoder
To better incorporate the information obtained above, apart from getting the original pooled context- attended representation he E R we combine the attended vectors Of and H from the previous encoder through a fusing layer.
177
E, ReLU(FC(()) HH H2h o H))) F2 ReI.U(FC((C) H, - 0 a of))),
(11)
= (FC((E),, E2l)):
C = Po H + (1 P) of E Baddd
Then another linear layer is applied for final prediction as 2 = W&C +b R,, We seek to minimize the cross entropy loss over the correct decision / by
179
Lans log softmax (z),
(12)
181
Logical Fact Regularization Inspired by [5).. the embedding of the tail argument should be close to the embedding of the head argument plus a space. Without loss generality, we assume that in our settings, the summation of the subject vector and the relation vector should be close to the object vector much as possible, i.e.,
182
vector in the hidden representation
183
184
Usubject
Uralation
Vobject-
(13)
185
In order to make the logical facts more of factual correctness, we introduce regularization for the extracted logical facts based on the hidden states of the sequence h; where = 1,..., L and L is the total length of the sequence. The regularization is defined as:
186
187
172
Luf- = Σ (1 cos(haub. hrrei hobjs)
(14)
where m is the total number of logical fact triplets extracted from the context as well as the option 189 and & indicates the k-th fact triplet.
188
190
Training Objective. During training, the overall loss for answer prediction is:
L= aLans + BLifrs
(15)
where and & are two parameters. In our implementation, set 1.0 and 0.5.
4
Experiments
193
4.1
Datasets
194
We conducted the experiments on three datasets. Two for specialized logical reasoning ability testing: ReClor 7 and LogiQA [5] and one for logical reasoning in dialogues: MuTual [46). For more details, one can refer to Appendix A.
196
197
4.2
Implementation Details
198
We fine-tune RoBERTa as the backbone PrLM for FOCAL REASONER. The overall model is end-to- end trained and updated by Adam [47] optimizer with an overall learning rate 8e-6 for ReClor and LogiQA, and 4e-6 for MuTual. The weight decay is 0.01. We set the warm-up proportion during training to 0.1. Graph encoders are implemented using DGL, an open-source lib of python. The layer
199
200
201
202
number of the graph encoder is 2 for ReClor and 3 for LogiQA. The maximum sequence length is
203
256 for LogiQA and MuTual, and 384 for ReClor. The model is trained for epochs with a total
batch size 16 and an overall dropout rate 0.1 on 4 NVIDIA Tesla V100 GPUs, which takes around 2
205
hours for ReClor and hours for LogiQA
Our code has been submitted along with this submission, which will be open after the blind review period.

---

ReClor
LogiQA
Model
Dev
Test
Test-E
Test-H
Dev
Test
Human
63.00
57.10
67.20
86.00
BERT-Large
53.80
49.80
72.00
32.30
34.10
31.03
XLNet-Large
62.00
56.00
75.70
40.50
RoBERTa-Large
62.60
55.60
75.50
40.00
35.02
35.33
DAGN 100
65,20
58.20
76.14
44.11
35,48
38.71
DAGN (Aug)
65.80
58.30
75.91
44.46
36.87
39.32
FOCAL REASONER 66.80
58.90
77.05
44.64
41.01
40.25
Table Experimental results of our model compared with baseline models on ReClor and LogiQA dataset. Test-E and Test-H denote Test-Easy and Test-Hard respectively. We performed Pitman's s permutation test [48] and found that our model significantly outperformed the baseline (p<0.05).
MuTual
MuTuaIPius
Model
Dev Set
Test Set
Dev Set
Test Set
R4@1
R4 @2
MRR
Rg01
R4 #2
MRR R4 @1
R,@2
MRR R4 @1
02 MRR
RoBERTa
46
69.5
87.8
82.4
71.3
89.2
83.6
62.2
85.3
78.2
62.6
86.6
78.7
-MC 46
69.3
88.7
82.5
68.6
88.7
82.2
62.1
83.0
77.8
64.3
84.5
79.2
FOCAL REASONER
73.4
90.3
84.9
72.7
91.0
84.6
63.7
86.1
79.1
65.5
84.3
79.7
Table 2: Experimental results of our model compared with baseline PrLM on MuTual dataset.
206
4.3 Results
Tables and 2 show the results on ReClor, LogiQA, and MuTuul, respectively. All the best results are shown bold. Based on our implemented baseline models (basically consistent with public results),
208
we observe dramatic improvements on both of the logical reasoning benchmarks, e.g., on ReClor test set, FOCAL REASONER achieves +4.2% on dev set and +3.3. on the test set. FOCAL REASONER also outperforms the prior best system DAGN reaching 77. 77.05% on the EASY subset, and 44.64% on the HARD subset. The performance suggests that FOCAL REASONER makes better use of logical structure inherent in the given context to perform reasoning than existing methods. On the dialogue reasoning dataset MuTual, our model achieves quite a jump compared with the RoBERTa base LMf) other downstream reasoning task settings.
210
212
214
This verifies our model's generalizability
216 In addition, Table lists the accuracy of our model on the dev set of ReClor of different question types. Results show that our model can perform well on most of the question types, especially "Strengthen" " and "Weaken". This means that our model can well interpret the question type from the question statement and make the correct choice corresponding to the question.
218
Analysis
220
5.1
Ablation Study
222
To dive into the effectiveness different components in FOCAL REASONER, we conduct an ablation study which takes RoBERTa as the backbone on the ReClor dev set. Table summarizes the results.
223
Supergraph reasoning: The first key component is the supergraph reasoning. We ablate the global atom and erase all the edges connected with it. The results suggest that the global atom indeed betters message propagation, leveraging performance from 64.6% to 66.8%. We also find that replacing the initial QA pair representation of the global atom with only question representation hurts the performance. In addition, without the logical fact regularization, the performance drops from 66.8% 228
224
226
For a fair comparison, only compare public literatures with the same PrLM RoBERTa-lagee. The test results are from the official leaderboard aattps://eva.l. htg.:////g//aaaee/ccca.../////// 5003/eeach 1133/eacerbca 1347
Since there official results RoBERTa-lage LM, RoBERTa- base LM instead for consistency.

---

RoBERTe-lorg #DMON
50.1
00:1
07
36.6
50
35
00.9
90
25
25
25
35
35.3
90
20
(03]
(3,00
(8.9]
[9:22]
[12,a]
[0.3] [36 (6.90
[6.22]
[:200
Figure 6: Accuracy of models on number of fact units on dev set of ReClor (left) and LogiQA (right).
to 64.2%, indicating its usefulness. For edge analysis, when (1) all edges regarded as single type rather than the original designed 8 types in total and (2) co-reference edges are removed, the accuracy drops 63.7% 7% and 64.8%, respectively. is proved that in our supergraph, edges link the fact units in reasonable manners, which properly uncovers the logical structures.
230
232
233
Fact Units Variants Apart from our syn- lactically constructed fact units, there are another two ways in different granularities for construction. We replace the fact units with
Model
Accuracy
235
236
FOCAL REASONER
66.8+0.13
237
named entities which are used in previous works
Supergraph Reasoning
like 49). The statistics of fact units and named entities of ReClor and LogiQA are stated in
global node
64.6#00.2
239
co-reference
64.8:10.24
Table from which we can infer that there
logical fact regularization
64.2#1.12
241
are indeed more fact units than named entities. Thus using fact units better incorporate the logical information within the context. When replacing all the fact units with named entities,
QA context node - Q node
66.410.16
-uesston reformulation
65.210.16
243
edge type
63.71
Fact Unit Variants
245
we can see from Table 3 that it significantly decreases the performance. We also explore the
-named entity
62.8+0.20
SRRL Interactions
62.240.32
247
performance using semantic role labeling the similar way in [50]. We can that SRL, leveraging a much more complex information as
interactions
65.5
240
well as computation complexity, fails achieve a performance as good as our original fact unit.
Table Ablation results on the dev set of ReClor.
251
252 Interactions:
We further experimented with setting to see how it affects the performance. The results suggest that the features learned from the interaction process enhance the model. Considering that the logical relations between different options are a strong indicator of the right answer, this means that the model learns from comparative reasoning strategy.
253
the intery-opttoonineeracee
254
256
257
5.2
Effects Fact Units Numbers
To inspect the effects of the number of fact units, we split the original dev set of ReClor and LogiQA into 5 subsets. The statistics of the fact unit distribution on the datasets are shown in 6 Numbers of fact units for most contexts in ReClor and LogiQA are in [3, 6) and [0, 3). respectively.
259
ReClor
LogiQA
Number
281
Train
Dev
Train
Dev
Fact Unit Argument
14,895
1,665
20,676
1,981
263
Named Entity
9,495
984
12,439
1,515
265
Comparing the accuracies of RoBERTa-large Table 4:
Statistics for fact unit entities and baseline, prior SOTA DAGN and our proposed traditional named entities in datasets.
286
257
FOCAL REASONER Figure 6 our model outperforms baseline models on all the divided subsets,
which demonstrates the effectiveness and robustness of our proposed method. Specifically, for ReClor,
269
FOCAL REASONER performers better when there are more fact units in the context, while for LogiQA,
FOCAL REASONER works better when the number of fact units locates in [0, 3) and [9, 12). The

---

Model
S
W
I
CMP
ER
P
D
IF
MS
RoBERTa Tagge
61.70
47.79
39.13
63.89
58.33
50.77
50.00
56.25
61.54
56.67
DAGN 10]
63.83
46.02
39.13
69.44
57.14
53 85
46.67
62.50
62.39
56.67
FOCAL REASONER
65.96
51.33
43.48
72.22
67.86
53.85
50.00
62.50
62.39
60.0
Table 5: Accuracy on the dev set of ReClor corresponding to several representative question types. S:
Strengihen, W: Weaken, I: Implication, CMP: Conclusion/Main Point, ER: Explain or Resolve, D: Dispute, R: Role, IF: Identify Flaw, MS: Match Structures.
271
reason may lie in the difference in style of the two datasets. However, all the models include ours struggle when the number of fact units is above certain thresholds, i.e., the logical structure is more complicated, calling for better mechanisms to cope with.
272
273
274
5.3
Interpretability: a Case Study
275
We aim to interpret FOCAL REASONER's reasoning process by analyzing the node-
278
Dataset
[0,3)
[3.
[6, 9)
[9, 12)
[12, oo)
277
to-node attention weights induced in the supergraph in Figure 7 We can see that our FOCAL REASONER can well bridge the reasoning process between context, question and option. Specifically, in the graph, "students rank 30%" attends strongly to "playing improve performance". Under the guidance of question
278
ReClor
37.2%
48.6%
12.6%
0.6%
1.2%
LogiQA 47.5%
37.5%
10.9%
3.5%
0.6%
280
Distribution of fact unit number dev Table 6:
282
set of the training datasets.
284
to select the option that weakens the statement and option interaction, our model is able to tell that "students rank 30% can play" mostly undermines the conclusion that "playing improves performance"
A recent survey key middle school showed that high school stucients in this school have special preference for playing football, and surpasses other balls.The survey also found that students who regularly play football better at academic performance than students who do not often play footbell. This shows that often playing football can improve students' academic performance.
Only high school students who are ranked
top of grades can often play football.
of the following can weaken the above conclusion most?
B. Regular football can exercise and maintain strong leamning energy.
playing football delays study time.
D. Research has not proved that playing football can contribute to intellectual development.
1. students have preferences
2. preforence playing football
3. it surpasses balls
who play football
QA
5. students better performance
who Iplay football
7. playing improve performance students
Option Similarity Matrix after Interaction
9. students play football
A: 4. 1918
C: -12.37718
Fact Units
bog Werght
-5.3050
D: -6.9722
Figure 7: An example of how model reasons to get the final answer.
Conclusion
286
287
For logical reasoning arising from machine reading comprehension, it is well known that clear and accurate forms like global knowledge are crucial. In this work, we make a finding that existing studies miss focusing on quite lot of non onowledweedee parts which is also indispensable for better reasoning. Thus we propose extracting general form called "fact unit" to cover both global and local logical units, hoping to shed light on the basis of structural modeling for logical reasoning. Our proposed FOCAL. REASONER not only better uncovers the logical structures within the context, which can be a general method for other sophisticated reasoning tasks, but also better captures the logical interactions between context and options. The experimental results verify the effectiveness of
288
290
282
294
our method.

---

References
296
297
[1] Karl Moritz Hermann, Tomá Kočiský, Edward Grefenstette, Lasse Espcholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume Volume 1, pages 1693 1701, 2015.
299
[2]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics.
302
303
304
305
[3] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguisties.
306
307
308
309
[4] Zhuosheng Zhang, Hai Zhao, and Rui Wang. Machine reading comprehension: The role of contextualized language models and beyond. arXiv preprint arXiv:2005.06249 2020.
[5] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCA/-20, pages 3622-3628. International Joint Conferences on Artificial Intelligence Organization, 2020. Main track.
312
313
314
315
316
[6] Lucja Iwanska. Logical reasoning in natural language: It is all about knowledge. Minds and Machines, 344):475-500 1993.
317
318
[7] Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020.
319
[8] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
322
99] Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, 33, 2020.
325
328
[10] Yinya Huang, Meng Fang, Yu Cao, Liwei Wang, and Xiaodan Liang. DAGN: Discourse-aware graph network for logical reasoning. In NAACL, 2021.
327
328
[1]] Siddharth Krishna, Alexander J Summers, and Thomas Wies. Local reasoning for global graph properties. In European Symposium on Programming, pages 308-335. Springer, Cham, 2020.
329
[12] Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. AR-LSAT: Investigating Analytical Reasoning of Text. arXiv e-prints, page arXiv:2104.06599 April 2021.
331
[13] Ndapandula Nakashole and Tom Mitchell. Language- truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1019, 2014.
334
336
[14] Danqi Chen, Jason Bolton, and Christopher D Manning. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, 2016.
337
338
339
[15] Minjoon Sco, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR 2017, 2017.
340
[16] Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Gated- attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1832-1846, 2017.
342
10

---

344
[17] Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang. Ting Liu, and Guoping Hu. Attention- wver-atentioon neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 693-602, 2017.
345
346
347
348
[18] Linfeng Song. Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks. arXiv preprint arXiv: / 1809.02040 2018.
349
350
351
[19] Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang. Nan Yang, and Dongsheng Li. Read+ verify: Machine reading comprehension with unanswerable questions. In Proceedings of the 2019.
352
353
AAAI Conference on Artificial Intelligence, volume 33, pages
354
[20] Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. SG-Net: Syntax-guided machine reading comprehension. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAT), 2020.
355
356
357
[21] Seohyun Back, Sai Chetan Chinthakindi, Akhil Kedia, Haejun Lee, and Jaegul Choo. NeurQuRI: Neural question requirement inspector for answerability prediction in machine reading comprehension. In International Conference on Learning Representations, 2020.
358
359
360
[22] Zhuosheng Zhang, Junjie Yang, and Hai Zhao. Retrospective reader for machine reading comprehension. arXiv preprint arXiv:2001.0966 2020.
381
362
[23] Divyansh Kaushik and Zachary C. Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015 Brussels, Belgium, October-November 2018. Association for Computational Linguisties.
363
364
385
366
[24] Ming Zhou, Nan Duan, Shujie Liu, and Heung- Yeung Shum. Progress in neural nlp: Modeling, learning, and reasoning. Engineering, 6(3):275-25) 2902,, 2020.
367
368
[25] Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 2358-2367, Berlin, Germany, August 2016. Association for Computational Linguistics.
370
372
[26] Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92 103, 2015.
374
[27] Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2019.
376
377
[28] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL-HLT (1), 2019.
378
379
[29] Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 2391 1400,, 2019.
380
382
[30] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt 383
384
Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages
386
2019.
[31] Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, 388
and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380 2018
380
11

---

382
[32] Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Longuage Processing, pages 632-662, 2015.
393
394
395
[33] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, 2018.
396
397
398
399
[34] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kicla. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-490) 4901, 2020.
400
401
402
[35] Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1150-6150, 2019.
403
404
405
[36] Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2694-2703, 2019.
408
407
408
[37] Jifan Chen, Shih-ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. arXiv preprint arXiv:1900.022 02,100, 2019.
409
410
(38] Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. Logic-driven context extension and data augmentation for logical reasoning of text. arXiv preprint arXiv:2 100.06399,,, 2021.
412
[39] Friedrich Wilhelm Levi. Finite geometrical systems: six public lectues delivered in February. 1940, at the University of Calcutta. University of Calcutta, 1942.
414
[40] Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1515-1515, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
416
418
[41] Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequenc learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273-283, Melbourne, Australia, July 2018. Association for Computational Linguistics.
420
422
[42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints, page wwt.:1997..1199... July 2019.
424
426
[43] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne vanden Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam, editors, The Semantic Web, pages 593-607. Springer International Publishing, 2018
428
430
[44] Qiu Ran, Peng Li, Weiwei Hu, and Jie Zhou. Option comparison network for multiple-choice reading comprehension. arXiv preprint arXiv:1903.03033, 2019.
432
[45] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In C. C. Burges, L. Botton, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
434
436
[46] Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
438
12

---

440
[47] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
441
[48] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392. Association for Computational Linguistics, 2018
444
446
447
[49] Jifan Chen, Shih-Ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. ArXiv, abs/1910.02610, 2019.
448
449
[50] Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, M. Zhou, Jiahai Wang, and Jian Yin. Reasoning over semantic-level graph for fact checking. In ACL, 2020.
450
Checklist
451
1. For all authors...
453
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
454
contributions and scope? [Yes]
455
(b) Did you describe the limitations of your work? [Yes] See Section 5.2
456
(c) Did you discuss any potential negative societal impacts of your work? [No]
457
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
458
459
2. If you are including theoretical results...
460
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
462
3. If you ran experiments..
463
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
464
465
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
466
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
468
489
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
470
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
474
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
475
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
477
478
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
479
5. If you used crowdsourcing conducted research with human subjects...
481
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
482
485
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
486
13


Reviews:
[{"review_id": "Zbxn7sOMzbo", "reviewer": "Reviewer_Bigy", "summary": "This paper presents a graph-based approach to solve logical reasoning questions. This method first extracts fact units from given questions, passages and answer options and builds a graph based on them, where a dependency parser is used to extract <subject, verb, object> triples. Then, the pre-trained RoBERTa large is used to generate representations of the concatenated question-passage-option sequences. Next, a graph is built based on the dependency parsing results of the sequence, and a graph attention network is used to perform neural message passing to exchange information between graph nodes. Last, all information are aggregated and used to do the prediction. Evaluations are conducted on 3 reasoning tasks, ReClor, LogiQA and MuTual, with ablation study performed for the effects of global atoms, fact unit variants/numbers and interaction layers.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key contribution is to leverage the global and local knowledge to solve logical reasoning tasks. But based on my understanding, this approach is not new. For example, in the commonsense reasoning work https://arxiv.org/abs/1909.05311, a method was proposed to use graphs and graph neural networks for the commonsense qa task. Although there are some details different, such as the proposed work uses dependency parsing instead of semantic role labeling, uses given passage so it doesn’t need to retrieval external knowledge as evidences, it still shares a similar architecture with the previous work. Besides, for the ReClor task, the latest benchmark is https://arxiv.org/abs/2105.03659, I suggest the authors to include this paper as a reference and compare with it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don’t think this work has any negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "no concern", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hour"}, "extra_text": {}}, {"review_id": "YX4zSu_7Q9Z", "reviewer": "Reviewer_Tmgr", "summary": "The paper claims that a lot of previous work in QA-based MRC focused too much on entity-aware common sense knowledge and other clues in the natural language (in passage, questions & candidate options). To overcome this, the authors extract \"fact units\" in the form of \"subject-verb-object\" from syntactic processing (dependency parsing).  With these fact units as a node, they form a supergraph and uses GNNs to process this. There are also many more components: global node, logical fact regularization, enlarged edge types compared to previous work and interactions. In the experiment section, they show that FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 3). ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality/Quality]\n\nThe strength of this paper is their extensive experiments and good performance which beats the previous SOTA performance.\n\nHowever, given the complex system that the paper works on, I wasn't sure what this paper's main focus is nor what their scientific contribution/novelty is. \n\nAfter reading this paper back-and-forth, the claimed main contribution seems to be bringing 'fact\" from \"syntactic processing\" to bring \"local complete facts\". But I do not feel like the main story of the paper is revolving around this contribution of \"local fact extraction\" and most of the paper discusses the complex model structure. \nThe part that really discusses how different fact units impact the system is only discussed in section 5.1 \"Fact unit variants\" (233-251) and how they syntactically extract facts are in 115-118. And even if this is the main contribution of this paper, I am not sure if this contribution is novel enough to be accepted as a full paper in NeurIPS. \n\nGiven many of the complex model structures this paper has, and as many of them are borrowed from previous work (using GNN and  logical fact regularization), I think authors should set their stance on what their contribution exactly is.  For example, if the paper don't use logical fact regularization loss, the performance drops from 66.8 --> 64.2 which is lower than competing models. \nWhy all these components are required to claim that local syntactic facts are really important was not clear to me. Also, I wasn't sure what information to gain from the comparisons with previous models in Table 1, 2. What was lacking in Table 1,2 for me was answers to the following quesitons:\n* Are the baseline models only \"global\" information? \n* Is FOCAL REASONER just having better results because it has different model structure?\n* Ultimately, what information are the baseline models providing in order to satisfy the claims this paper makes?\n\nOne recommendation to authors is, rather than bringing best of everything, trying to create more comparable baselines that can justify the claim (such as analysis on \"Fact unit variants\")\nPerhaps, if this paper's real goal was improving SOTA by novel model structures, I think it should focus more on what is the new wheel they are bringing into this framework. \n\n[Clarity]\nThe paper needs to be written more clearly. I found many parts of the definitions to be ambiguous as can be seen in “question” section of the review.\n\n[Corrections]\n* Figure 1. Don’t see the “relations” in green, maybe you meant relations in orange?\n* 234-235:  there are another two ways  in different granularities for construction—> there are two other ways to construct ...\n\n[Questions]\n* I am not sure what definition 1 brings (From triplet to fact unit, it seems the same to me? Also, the fact unit is defined as the triplet on the cited paper [13] (as fact candidate in [13]).\n* Definitions are confusing as there are overloaded concepts here and there. It may be useful to have examples for each of them (although authors try to depict examples in the figure).  \ne.g. 120-123 Since V,E,R are overloaded between fact units and Levi graph, it might be nice to have examples to distinguish, V,R of F and V_L of F_L. \n* Figure 4 edge for “fact-driven reasoning process” box only relies on question & option vectors, is this correct? (Although they are contextualized by the passage, I thought the passage is the main source of the super graph as shown in figure 5?)\n* How can fact units be replaced with named entities when they don’t have any relations?\n* In Table 4, did you replace fact units to be SRL output? And used it in your current Focal Reasoner with all the E ={same, cored, global} and edges R_L={ default-in, default-out, reverse-in, reverse-out, self} ?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author did not address \"limitations and potential negative societal impact of their work\". \nAlthough this paper's main focus is in the reasoning from MRC, as it is merely learning from a supervised model, would this model be robust to social biases in the training texts? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "16l1gUv3nB", "reviewer": "Reviewer_hoPP", "summary": "This paper presents a system for graph-based reasoning about scenarios. Facts are represented as subgraphs which are connected into a supergraph via a single \"global\" node and coreference nodes. A passage, question, and options are first passed through RoBERTa, then a neural network (graph attention network) passes information through the graph representation before an answer is predicted. Results show that this model works better than a related graph-based technique (the DAGN model), with particularly strong gains on the LogiQA test set.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core idea of this paper is a pretty nice one. Merging global and local (instance-specific) information is a great one and is very well-motivated for improving machines' ability to reason. However,\nI don't actually see where the notion of global (cross-instance facts) is integrated into the paper? What I expected to see was local information from, say, a ReCLOR example combined with\nsome retrieved commonsense. The global node in the graph just seems to unify all the information in a single example.\n\nTherefore, I will judge the paper mostly in terms of its architecture for doing local reasoning over a single example.\n\nOriginality: The ideas here are original, although much of the neural machinery is known. In my view, a weakness of this paper is actually the sophistication of the neural model: many small changes to the neural architecture (compared to pre-trained models) make it a bit harder to attribute the gains to specific aspects of the approach, although the paper's ablations are well done.\n\nQuality: The work is technically sound as far as I can tell. The methods used are appropriate and the results are reasonably in-depth, including ablations of the various parts of the model.\n\nClarity: The paper is well-written overall.\n\nSignificance: This is where I am most conflicted about the paper.  The results are difficult to interpret. Certainly there are gains from the paper's technique and the ablations show that the different components of the model all contribute to the performance. But when fairly important elements of the model like coreference edges contribute only around 2% accuracy, it's hard to know how much to trust the narrative here about reasoning.\n\nMore generally, in a multiple-choice task, it's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines. It's not easy to form an apples-to-apples comparison in terms of number of parameters. The case study in Section 5.3 shows that something is happening in the graph component, but whether this is causally associated with the ability to get the right answer as opposed to a byproduct is hard to determine.\n\nThe authors may wish to cite Mrinmaya Sachan's work on using representations from AMR: https://www.aclweb.org/anthology/P16-2079.pdf\n\nThe LogiQA results show some nice improvements, but I'm curious about the statistical significance statement in Table 1: are all of these results (including ReCLOR) truly representing significant gains? Compared to which baseline(s)?\n\nOverall, I'm borderline about this paper. There are some nice core ideas but I don't know whether these ideas of how to use graphs are proven to be significantly better than prior work. The complexity of the work isn't necessarily justified by strong results on ReCLOR.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This has been addressed adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to extract subgraphs containing local factual information from the given QA context (in the form of subject-predicate-object triples), connect them via a single global node and coreference, and perform graph-based reasoning over the resulting structure. The reviewers note the main strength of the work to be its good empirical results on the 3 benchmarks considered. However, given the complex nature of the proposed system, it is difficult to understand which aspects of the proposed technique are truly effective.

For instance, I see from Table 1 that the gain on ReClor and LogiQA "test" sets is only about 1 percentage point, compared to the best existing baseline (DGAN). While this gain may be statistically significant (i.e., not a result of random chance) because the test set is large enough, it is difficult to rely mainly on a 1 point gain to justify a new approach.

The active response of the authors is much appreciated. It would certainly make the paper much stronger once the explanations and new results from the response are included in the next draft of the paper. I'm sorry the NeurIPS reviewing model does not allow uploading a revised version of the paper (similar to many conferences). The changes needed before publication, however, are substantial and would generally merit a fresh round of review.
Decision: Reject