Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a framework, called Focal Reasoner, to perform logical reasoning to answer questions. The proposed approach first extracts the facts from raw text. The authors describe a fact unit as a triple of (argument, predicate, argument). These collections of facts can be viewed as a graph with predicates as undirected edges between arguments. After forming a subgraph, reasoning is performed by a graph convolution module to predict the correct answer. 
The authors show that their approach outperforms existing approaches on two benchmarked datasets.
The authors also perform a detailed analysis that sheds more light on the internal workings of the model.
main_review: Strengths:
1. The approach is interesting boasts great results on two benchmarks compared to previous approaches. 
2. The authors also perform useful analysis on the model which is interesting and would be useful to the community. 
3. The approach is, to some extent, described well. 
4. The authors also plan to open source their code which would be beneficial to the community. 

Concerns, questions, and suggestions:
1. In the related works section, the authors fail to describe and differentiate themselves from the two baselines that they compare against (DAGN and LReasoner).  
2. In the related work section, the authors describe a slew of other methods that the model is not compared against. This makes it hard to place their approach in comparison with prior work thus making it hard to understand the novelty of the approach. 
3. Perhaps, the authors could add a section describing related works that are used for the tasks described. (The authors claim that the baselines are described in the appendix but that is not the case)

summary_of_the_review: The approach is interesting but it is hard to understand what aspect is novel compared to other competitive approaches. Perhaps the authors should describe other methods that have been used to solve this task. This would help make it easier for the reader to understand the novel contribution of the paper and also makes the paper more complete by itself.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model architecture for question answering which takes advantage of shallow proposition structure and coreference links using (relational) graph convolutional networks. Subject-predicate-object triples (dubbed "fact units") are extracted from the text using dependency parsing, extra coreference links are added using a coreference system, and the resulting graph is initialized with representations from a contextualizing encoder, passed through a GCN, and recombined with the contextual representations via multi-headed attention. The resulting representations serve as input to the final classification layer for answer prediction, as well as a "logical facts regularization" step which encourages the final representation of the object in each fact unit to align (via cosine) with the sum of the subject and predicate representations.

In experiments on several datasets designed to test logical reasoning, the proposed architecture scores higher than several baselines.
main_review: ## Strengths

* The model seems fairly intuitive. The idea of representing the (entity + relational) structure of a text directly and propagating information through that structure has been prevalent in the IE / RC space for a while but it seems that many have had difficulty getting it to work well. This paper shows improvements which are big enough to indicate that there's probably something to it (though I have a few concerns about this, described in the weaknesses). It seems to me that the key to the success of the model is probably just in the distribution of data that it's applied on. Such approaches may not have worked for earlier datasets (such as SQuAD) simply because they were not suited to teach or test the kind of information aggregating abilities that this model is designed to do.

* The ablations are nice.

## Weaknesses

* I take issue with some of the terminology — use of the terms "fact" and "logical reasoning" are, in my view, a bit misleading as descriptions of what a model like this one is doing. The tuples being extracted from the text are more like "propositions" than "facts" and the operations in the GCN don't seem to have much in common with "logical reasoning" (which normally involves the application of rules from a formal system). I understand there seems to be a bit of precedent for the terms here, as the datasets being used were designed to capture aspects of logical reasoning, but I think it's less correct to reuse them when describing model components such as "Logical Fact Regularization." Something like "propositional structure regularization" might be more appropriate.

* I'm worried that this model's improvements will end up being specific to the distribution in "logic"-focused datasets like ReClor and LogiQA, because of these datasets' focus on aggregating and combining information about a small set of entities in a paragraph. While I understand that this capability is the focus of this work, I think it's important to know if the proposed architecture is too specialized: does it maintain performance over its RoBERTa or DeBERTa baseline if applied to more naturalistic, extractive reading comprehension datasets like Natural Questions, QuoRef, or HotpotQA? If not, is it because of the distribution of text, extractive format, or other factors? It's fine if the the model doesn't end up improving these cases — I think investigating these issues would greatly improve the paper either way.

* I would suggest more caution when it comes to interpretability. Attention weights cannot be relied on to provide explanations of model behavior. To make the interpretability case study in Section 5.3 more believable, I would suggest modifying the attended element and seeing if the answer changes: for example, change "playing football can improve students' academic performance" to "football players are held to higher academic standards than non-athletes" and see if the attention pattern or the answer changes. If it doesn't, then the attention pattern — while still indicative of the information relevant to answer the question - might not be indicative of the kind of logical reasoning we suspect.

* I'm not sure if the findings are statistically robust. All of the test sets here are very small — less than 1000 items. The numbers seem good but I'm not sure what the uncertainty looks like. Please provide confidence intervals for the Focal Reasoner and at least some of the stronger baselines.

* Are the baselines well-tuned? Time and time again, modeling innovations are proposed which ultimately get beaten by a better-tuned baseline down the line. How many experiments did you run for each model variant? Are you reporting the best of several runs? How did the performance of each model vary over runs? Can you say that the distribution of the new model's performance is appreciably better than the baseline models? By how much, and with what confidence?

## Minor comments

* In equations 4 and 5, wouldn't it be better to use $v_\text{predicate}$ instead of $v_\text{relation}$? It's a bit confusing to me to use the word "relation" here since earlier the word is used to refer to the types of edges between vertices in the graph.
summary_of_the_review: The model seems nice and the results seem positive, but I think the experiments are weak. We don't really learn much from this paper about the strengths and limitations of the proposed model, and it seems plausible that the experimental results are not statistically robust, due to a combination of the small test set size and the limited experimental reporting that is done in the paper.

---

EDIT: Some of my concerns were addressed, particularly the issues regarding reporting more details on the model's performance (number of runs, variance, etc.) which I think were most critical. I am still not satisfied with the discussion of interpretability (which I think should be removed; see discussion below) and I think the paper could have done a better job with experiments demonstrating the relative strengths and weaknesses of the model, but the results that are present seem strong. So I wouldn't say I'm particularly eager to accept the paper but I won't gatekeep it either. Raising my score from a 5 to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper claims that a lot of previous work in QA-based MRC focused only on entity-aware common sense knowledge.  To overcome this, the paper proposes to build a finer-grained local fact, (entity, predicate, entity) triplet, information based on dependency parsing, and then connect the triplet nodes with global information such as coreference, entity information. 
Architecture-wise, a passage, question, and options are passed to RoBERTa (or DeBERTa) and then this information goes through the graph-attention network (GAN), based on the graph structure described above. In the experiment section, the authors show that the proposed FOCAL REASONER outperforms the previous SOTA models and run an ablation study on how each component contributes (in Table 5,6).

The main contribution of this paper, as I understand, is bringing more fine-grained local facts and connecting them with global information such as coreference and entity linking.
main_review: As I served as one of the reviewers for this paper before, I am also sharing a summary or quote of reviews from the last round here as the paper did not go through major changes since the last submission. Content-wise, $LReasoner_{RoBERTa}$ , $LReasoner_{DeBERTa}$ , as well as $FocalReasoner_{DeBERTa}$ were added as new result in Table 1. 

### Strength

- (From the previous review) All reviewers noted that the main strength of the work to be its good empirical results on the 3 benchmarks considered.
- This strength is corroborated with the additional results of competitive baselines and  $FocalReasoner_{DeBERTa}$.
- The paper is well-written overall. I appreciate the authors making an effort to organize and clarify the complex steps that this paper takes. 

### Weakness
-  (From the previous review) The unanimous criticism was that while the results are impressive, given the complex nature of the proposed system, it was difficult to understand which aspects of the proposed techniques are truly effective.
- While the coreference, and entity linking are the major resource in creating the supergraph, I don't think the challenges nor machinery in getting this information is well described.  (It might be in the text, but I am not sure how the authors get dependency parse trees as well.) It would be nice to have  more description on the challenges of identifying the global relations (same entity, coreference)
- As described in the *summary* section, I think the main contribution of this paper is in getting fine-grained facts and connecting them with global information. I am not entirely convinced whether this is a very novel approach. What is the major scientific contribution here? 
- The authors attempt to decompose some parts of the method in Tables 5 and 6. Related to the earlier weakness on the complex nature of this work, I am not sure what role the *logical fact regularization loss* serves in this paper. Is this one of the contributions of this paper? Or is it simply a tool to get a better results? Without it, the performance 66.8 --> 64.2 which is lower than competing models.
 
Some of the comments that other reviewers made from the last round of review (which resonated with me):
- " **Significance**: This is where I am most conflicted about the paper. The results are difficult to interpret. Certainly, there are gains from the paper's technique and the ablations show that the different components of the model all contribute to the performance. But when fairly important elements of the model like coreference edges contribute only around 2% accuracy, it's hard to know how much to trust the narrative here about reasoning."
- "More generally, in a multiple-choice task, it's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines. It's not easy to form an apples-to-apples comparison in terms of number of parameters. The case study in Section 5.3 shows that something is happening in the graph component, but whether this is *causally associated* with the ability to get the right answer as opposed to a byproduct is hard to determine."

### Questions/Suggestions

- I am not sure what each edges really do: (*default-in, default-out, reverse-in, reverse-out, self*). It would be nice if authors can describe the roles of these edges.
- Figure 5's *global edge* and what this paper claims as *global (typical entity or sentence relation)* is confusing. Maybe try to use distinguishable terms?
- Why don't you update 4.3 (1) sentence to match newly updated best result table 1?
    - "FOCAL REASONER also outperforms the prior best system LReasoner, reaching 77:05% on the EASY subset, and 44:64% on the HARD subset."
summary_of_the_review: FocalReasoner exhibits a strong experimental result and it is corroborated in the new submission with added experiment results in $LReasoner_{RoBERTa}$ , $LReasoner_{DeBERTa}$ , as well as $FocalReasoner_{DeBERTa}$ in Table 1. 

Albeit its high-performance, as the previous review pointed out, I am not sure what is the major factor in the improvement when many parts of the neural machinery are used together. I view the main contribution as using the Levi graph from dependency trees as  Beck et al. 2018 and connecting them with coreference and entity linking.

Overall, I'm borderline about this paper. I think this paper tests whether fine-grained pieces of information in the paragraph are useful or not (intuitively they should, and this paper shows that it does). However, I am not sure whether the usage graphs are novel or significantly better than previous work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel graph reasoning model based on supergraphs constructed via fact triplets. The authors define the fact triplets as subject-predicate-object relational path extracted from the dependency parser and use them as local knowledge, compared with the commonly used entity-relation knowledge to enhance the task of logic-driven question answering.  
main_review: Strengths:
The paper is well-written with clear motivations and structure. The concept of fact units is interesting and novel which are easily constructed via dependency trees. Compared with commonsense and entity-relation knowledge, the fact units are more informative to each specific question. By associating nodes across different fact units based on coreferences and mentions, a supergraph is built that connects all related information and conducts graph reasoning for answer predictions. The proposed model is then experimentally verified on three logic-driven datasets which demonstrates some performance gain.

Weaknesses:
1. Some of the details for model description are missing and confusing, e.g., (1) How the representation is initialized for a supernode corresponding to a fact triplet and how does the supernode propagate information to the sub-nodes? (2) Equation 2 and 3 use the same graph encoder $F_{G}$. Are they the same? Since they both use $m$ nodes, does it mean the supergraph also operates on each sub fact node? If so, does the global node $V_g$ connect to all the sub fact nodes? It is then not clear how each fact block (grey squares in Figure 4) functions as a whole. The notations in equation 2 and 3 are also inconsistent with equation 5. What does L, S, F and m represent?  (3) The loss for answer prediction in Eq 6 is not clearly described: do you use an aggregated embedding over all nodes for prediction? (4) How exactly is the interaction module processed?
2. According to the description, the fact units are constructed using the dependency parser. Hence, the model relies on whether the parser accurately discovers the crucial information. I am also wondering if the extracted facts could bring too much noise to the question. As shown in Figure 6, when more than 12 facts are constructed, the performance becomes worse. 
3. More discussions on comparing with symbolic logic reasoner model LReasoner are needed. Indeed, the proposed graph model seems to only implicitly convey knowledge across facts in terms of local reasoning. On the other hand, the symbolic logic rules are able to express global patterns. Hence, to my understanding, it may not be suitable to use the term "global reasoning" in this work.
4. For experiments, Table 1 shows the FOCAL Reasoner (DeBERTa) brings much performance gain, compared to FOCAL Reasoner (RoBERTa) (the performance seems comparable with DAGN and LReasoner using RoBERTa). Can you explain why? Is it possible to run DAGN (DeBERTa)?
summary_of_the_review: Novel concept of fact units to answer logic-driven questions. More discussions are needed.
review_title: null

Meta-review: Strengths:
* Well-written paper
* Strong empirical results on three benchmarks
* Interesting approach of producing semantically augmented LMs using dependency parses to extract svo triples, and finding coreferences between them across multiple sentences

Weaknesses:
* None of the reviewers seem particularly excited about the paper
* Stronger baseline comparisons would have improved the paper
* Authors re-define a lot of terminology, but the novelty of the method is more from the type of graph used to initialize their method, which seems to be a function of OpenIE triplets
Decision: Reject