Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper claims that a lot of previous work in QA-based MRC focused too much on entity-aware common sense knowledge and other clues in the natural language (in passage, questions & candidate options). To overcome this, the authors extract "fact units" in the form of "subject-verb-object" from syntactic processing (dependency parsing).  With these fact units as a node, they form a supergraph and uses GNNs to process this. There are also many more components: global node, logical fact regularization, enlarged edge types compared to previous work and interactions. In the experiment section, they show that FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 3). 
main_review: [Originality/Quality]

The strength of this paper is their extensive experiments and good performance which beats the previous SOTA performance.

However, given the complex system that the paper works on, I wasn't sure what this paper's main focus is nor what their scientific contribution/novelty is. 

After reading this paper back-and-forth, the claimed main contribution seems to be bringing 'fact" from "syntactic processing" to bring "local complete facts". But I do not feel like the main story of the paper is revolving around this contribution of "local fact extraction" and most of the paper discusses the complex model structure. 
The part that really discusses how different fact units impact the system is only discussed in section 5.1 "Fact unit variants" (233-251) and how they syntactically extract facts are in 115-118. And even if this is the main contribution of this paper, I am not sure if this contribution is novel enough to be accepted as a full paper in NeurIPS. 

Given many of the complex model structures this paper has, and as many of them are borrowed from previous work (using GNN and  logical fact regularization), I think authors should set their stance on what their contribution exactly is.  For example, if the paper don't use logical fact regularization loss, the performance drops from 66.8 --> 64.2 which is lower than competing models. 
Why all these components are required to claim that local syntactic facts are really important was not clear to me. Also, I wasn't sure what information to gain from the comparisons with previous models in Table 1, 2. What was lacking in Table 1,2 for me was answers to the following quesitons:
* Are the baseline models only "global" information? 
* Is FOCAL REASONER just having better results because it has different model structure?
* Ultimately, what information are the baseline models providing in order to satisfy the claims this paper makes?

One recommendation to authors is, rather than bringing best of everything, trying to create more comparable baselines that can justify the claim (such as analysis on "Fact unit variants")
Perhaps, if this paper's real goal was improving SOTA by novel model structures, I think it should focus more on what is the new wheel they are bringing into this framework. 

[Clarity]
The paper needs to be written more clearly. I found many parts of the definitions to be ambiguous as can be seen in “question” section of the review.

[Corrections]
* Figure 1. Don’t see the “relations” in green, maybe you meant relations in orange?
* 234-235:  there are another two ways  in different granularities for construction—> there are two other ways to construct ...

[Questions]
* I am not sure what definition 1 brings (From triplet to fact unit, it seems the same to me? Also, the fact unit is defined as the triplet on the cited paper [13] (as fact candidate in [13]).
* Definitions are confusing as there are overloaded concepts here and there. It may be useful to have examples for each of them (although authors try to depict examples in the figure).  
e.g. 120-123 Since V,E,R are overloaded between fact units and Levi graph, it might be nice to have examples to distinguish, V,R of F and V_L of F_L. 
* Figure 4 edge for “fact-driven reasoning process” box only relies on question & option vectors, is this correct? (Although they are contextualized by the passage, I thought the passage is the main source of the super graph as shown in figure 5?)
* How can fact units be replaced with named entities when they don’t have any relations?
* In Table 4, did you replace fact units to be SRL output? And used it in your current Focal Reasoner with all the E ={same, cored, global} and edges R_L={ default-in, default-out, reverse-in, reverse-out, self} ?


limitations_and_societal_impact: The author did not address "limitations and potential negative societal impact of their work". 
Although this paper's main focus is in the reasoning from MRC, as it is merely learning from a supervised model, would this model be robust to social biases in the training texts?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a system for graph-based reasoning about scenarios. Facts are represented as subgraphs which are connected into a supergraph via a single "global" node and coreference nodes. A passage, question, and options are first passed through RoBERTa, then a neural network (graph attention network) passes information through the graph representation before an answer is predicted. Results show that this model works better than a related graph-based technique (the DAGN model), with particularly strong gains on the LogiQA test set.
main_review: The core idea of this paper is a pretty nice one. Merging global and local (instance-specific) information is a great one and is very well-motivated for improving machines' ability to reason. However,
I don't actually see where the notion of global (cross-instance facts) is integrated into the paper? What I expected to see was local information from, say, a ReCLOR example combined with
some retrieved commonsense. The global node in the graph just seems to unify all the information in a single example.

Therefore, I will judge the paper mostly in terms of its architecture for doing local reasoning over a single example.

Originality: The ideas here are original, although much of the neural machinery is known. In my view, a weakness of this paper is actually the sophistication of the neural model: many small changes to the neural architecture (compared to pre-trained models) make it a bit harder to attribute the gains to specific aspects of the approach, although the paper's ablations are well done.

Quality: The work is technically sound as far as I can tell. The methods used are appropriate and the results are reasonably in-depth, including ablations of the various parts of the model.

Clarity: The paper is well-written overall.

Significance: This is where I am most conflicted about the paper.  The results are difficult to interpret. Certainly there are gains from the paper's technique and the ablations show that the different components of the model all contribute to the performance. But when fairly important elements of the model like coreference edges contribute only around 2% accuracy, it's hard to know how much to trust the narrative here about reasoning.

More generally, in a multiple-choice task, it's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines. It's not easy to form an apples-to-apples comparison in terms of number of parameters. The case study in Section 5.3 shows that something is happening in the graph component, but whether this is causally associated with the ability to get the right answer as opposed to a byproduct is hard to determine.

The authors may wish to cite Mrinmaya Sachan's work on using representations from AMR: https://www.aclweb.org/anthology/P16-2079.pdf

The LogiQA results show some nice improvements, but I'm curious about the statistical significance statement in Table 1: are all of these results (including ReCLOR) truly representing significant gains? Compared to which baseline(s)?

Overall, I'm borderline about this paper. There are some nice core ideas but I don't know whether these ideas of how to use graphs are proven to be significantly better than prior work. The complexity of the work isn't necessarily justified by strong results on ReCLOR.
limitations_and_societal_impact: This has been addressed adequately.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a graph-based approach to solve logical reasoning questions. This method first extracts fact units from given questions, passages and answer options and builds a graph based on them, where a dependency parser is used to extract <subject, verb, object> triples. Then, the pre-trained RoBERTa large is used to generate representations of the concatenated question-passage-option sequences. Next, a graph is built based on the dependency parsing results of the sequence, and a graph attention network is used to perform neural message passing to exchange information between graph nodes. Last, all information are aggregated and used to do the prediction. Evaluations are conducted on 3 reasoning tasks, ReClor, LogiQA and MuTual, with ablation study performed for the effects of global atoms, fact unit variants/numbers and interaction layers.
main_review: The key contribution is to leverage the global and local knowledge to solve logical reasoning tasks. But based on my understanding, this approach is not new. For example, in the commonsense reasoning work https://arxiv.org/abs/1909.05311, a method was proposed to use graphs and graph neural networks for the commonsense qa task. Although there are some details different, such as the proposed work uses dependency parsing instead of semantic role labeling, uses given passage so it doesn’t need to retrieval external knowledge as evidences, it still shares a similar architecture with the previous work. Besides, for the ReClor task, the latest benchmark is https://arxiv.org/abs/2105.03659, I suggest the authors to include this paper as a reference and compare with it.
limitations_and_societal_impact: I don’t think this work has any negative societal impact.
review_title: null

Meta-review: This paper proposes to extract subgraphs containing local factual information from the given QA context (in the form of subject-predicate-object triples), connect them via a single global node and coreference, and perform graph-based reasoning over the resulting structure. The reviewers note the main strength of the work to be its good empirical results on the 3 benchmarks considered. However, given the complex nature of the proposed system, it is difficult to understand which aspects of the proposed technique are truly effective.

For instance, I see from Table 1 that the gain on ReClor and LogiQA "test" sets is only about 1 percentage point, compared to the best existing baseline (DGAN). While this gain may be statistically significant (i.e., not a result of random chance) because the test set is large enough, it is difficult to rely mainly on a 1 point gain to justify a new approach.

The active response of the authors is much appreciated. It would certainly make the paper much stronger once the explanations and new results from the response are included in the next draft of the paper. I'm sorry the NeurIPS reviewing model does not allow uploading a revised version of the paper (similar to many conferences). The changes needed before publication, however, are substantial and would generally merit a fresh round of review.
Decision: Reject