Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors survey the literature on probing the understanding, reasoning, and commonsense properties of transformer-based models, outlining several limitations of current models. They cover mispriming, negations, the tendency of such models to learn simple heuristics, under-sensitivity to word order, reasoning with patterns resembling Horn clauses, commonsense reasoning, reasoning with temporal knowledge. Furthermore, they also cover several natural language understanding tasks such as natural language inference and mathematical reasoning.

It would be nice to have snippets from the different datasets, to get a better understanding of what's out there for training and evaluating our models, but I understand the space limitations.

There are a few missing references:
- On the Power of Saturated Transformers: A View from Circuit Complexity (https://arxiv.org/abs/2106.16213)
- A couple of systems focus on solving CLUTRR using dedicated architectures: https://arxiv.org/abs/2007.06477 and https://github.com/ML-KULeuven/deepproblog
review_title: Nice survey on some of the limitations of modern transformer-based models

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper surveys the performance of BERT on tasks that requires logical reasoning. The survey first introduces basic building blocks that a model should have in order to perform reasoning on natural language (e.g. handling negations, stability to word-order, etc). Next it describes a series of reasoning tasks and report performance on BERT-based models. BERT-based model performs well on reasoning tasks when all information required to perform deductive reasoning, such as facts and rules, are provided (e.g. logical reasoning). However, when certain information needs to be inferred (background knowledge), these models fail (text understanding, mathematical reasoning). Finally, the survey discusses two tasks (Parity and Dyck-2) that has been theoretically proved that transformers cannot perform and the paper comes up with two example reasoning tasks and shows that transformer models indeed fail on them.

### Strengths:

1. The paper presents a comprehensive list of reasoning tasks and cites many relevant work and can serve as a go-to survey for the BERT-based model's performance on natural language reasoning tasks

### Weaknesses:

1. The survey currently reads like a list of reasoning tasks which BERT mostly fail at and one or two example, demonstrating where it failed. It would be more helpful to have a discussion around each of the failed reasoning tasks. For example, why is BERT not able to perform those reasoning tasks? What % of examples need this reasoning, does the model fail at? Do they fail at all examples?
2. I think the paper would benefit if it clearly lays down what the audience will learn after reading the survey. A discussion about how we can make the transformer model perform reasoning could also be helpful

Missing citations:
Boratko et al 2020 ProtoQA: a question answering dataset for prototypical common-sense reasoning shows another case where pre-trained LMs don't perform well because of the lack of background knowledge.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview: 
This paper provides a survey on recent work in the area of reasoning using transformer based models. It first describes common issues that are faced by transformers such as falling prey to lexical overlap and indifference to word order, and then goes on to describe different kinds of reasoning that can be tackled using transformers and the relevant work tackling each. Finally, the paper gives two examples of types of tasks that are beyond the capabilities of transformers. 

High level comments: 
The paper is well written, easy to read and covers many sub-types of reasoning tasks. It would benefit from being more detailed on the models that are being evaluated on each of the datasets, maybe in some form of aggregated table so the reader has a sense of which models perform better on which types of reasoning tasks. Since the different models - GPT-3/BERT/etc. are all trained on different amounts of data as well as different types of data, it would be very useful to know what scale of data and size of model is required for good performance on these various tasks. I believe this would be a more useful addition to the paper than section 3.5.

Detailed comments: 
I would rename the first section as “basic building blocks” doesn’t seem appropriate when one of the subtitles is “mispriming”. This section seems to cover the artifacts that the model needs to learn to be resilient to, such as not falling prey to mispriming, not overly relying on pattern heuristics, not only taking into account words without using order information etc. Maybe a better section title would be “Commonly observed pitfalls and ways to avoid them.” ?
Similarly, I feel the section 2 title would be more informative if you named it “Types of reasoning with transformers” since just “Reasoning with transformers” is not very informative given that it’s the topic of the whole paper. 

Section 3.5, It would be useful to the reader if you give some intuitions on *why* it’s not possible to solve tasks such as parity using transformers in addition to stating that a certain paper shows that it cannot. For instance, in parity tasks, it requires the model to aggregate information from each segment and since all the tokens are processed in parallel by self attention, it would need the network depth to increase proportionally to the length of the sequence, to be able to solve the task making it not possible for non-recurrent models like transformers solve the task. 

In the lights example, it is mentioned that the model is trained for 50 iterations. Did the model learn the task in these 50 iterations? Did the training converge? It’s hard to make conclusions about why the test precision was 0.5 - whether it is because the model was unable to learn the task during training or if it has to do with the train test distribution shift which causes the model to not do well on the task, without this information.  

Same issue with the cake task: missing details about model convergence and performance on the training data. Further it would be interesting to know if changing the type of positional encodings (learnt embeddings/sinusoidal embeddings/CAPE [1] ) can change the results for generalizing to longer length.

Missing citation: 
Transformers for symbolic math : Lample, Guillaume, and François Charton. "Deep learning for symbolic mathematics.", ICLR 2020.

Minor typos: 
Intro last paragraph, make transformer-based reason -> make transformer-based models reason
Section 3.4 Did you mean to say performer based models?
Section 4.2 contining -> containing

[1] CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings


review_title: Survey paper on recent work applying transformers for a variety of reasoning tasks.

final_score: rating: 7: Good paper, accept
review_content: **Summary:** The paper surveys the ability of pretrained transformer-based models on logical reasoning tasks.

* Section 2 explores common reasoning abilities required to solve logical reasoning tasks.
* Section 3 considers different logical reasoning tasks, including both standard and adversarial variants. It also outlines the performance on such tasks of different model variants (i.e., pre-training only, fine-tuned on the target task, and fine-tuned with additional training objectives or symbolic components).
* Finally, Section 4 touches on tasks that are theoretically impossible for a transformer-based model to completely solve. It also presents natural language versions of such theoretical tasks.

**Strengths:**

The content of this survey is comprehensive and appropriate for the page limit. It gives a good overview of the major logical reasoning tasks and how well the models perform on them. Section 3.6 in particular provides a good summary: the models perform well when all information is explicit, perform reasonably on commonsense reasoning, and struggle on adversarial tasks but can be remedied by additional techniques.

**Weaknesses:**

1. The **scope of the models** explored is not well-defined.
    - "Transformer-based models" is a very wide scope. Even when considering only BERT-like models (BERT, RoBERTa, T5, GPT-3, etc.) without additional modifications, the model performance still depends on many factors such as the model architecture, model size, pre-training data, pre-training task (mask filling, generation, sentence-level objective), fine-tuning data, and fine-tuning methods.
    - Most works cited in this paper use BERT or its close variants such as RoBERTa. It is unclear if the results carry on the models trained on generative objectives such as T5 and GPT-3.
    - The paper often makes claims that can be considered too general (e.g., "Pretrained transformers-based models **cannot** differentiate between positive and negative statements" in Section 2.1, or "mispriming works only when ..." in Section 2.2) while they might only apply to certain model variants.
    - The paper also mixes the results of models with and without fine-tuning, especially in Section 2. While these can be discerned from careful reading, it would be great to make this more clear to prevent readers from drawing incorrect conclusions.

2. Likewise, the **scope of the tasks** is not well-defined.
    - It is not explained what "logical reasoning" means, and the paper contains many examples that are not "logical reasoning" in the traditional sense. Examples include having world knowledge (e.g., that the iPhone was produced by Apple, as mentioned in the introduction), having commonsense knowledge (e.g., that ravens can fly, as mentioned in Section 3.2), and doing arithmetic (e.g., adding two numbers in Section 3.5). While these skills are useful for some logical reasoning tasks, I wouldn't call them logical reasoning on their own.
    - Mispriming (Section 2.2) does not seem to be a building block for any logical reasoning task. Though one could argue that the ability to distinguish useful from distracting contexts in general is an essential building block.
    - "Text understanding" (Section 3.4) also sounds too broad and encompasses the other types of reasoning in prior sections.

3. (minor) The natural language version of impossible reasoning tasks (Sections 4.2 and 4.3) is not very informative.
    - BERT is known to not generalize well to long inputs, especially when it's longer than what the model have seen during training. When the model performs poorly on the proposed light switch and cake tasks, it is unclear whether it's due to the length problem above or the theoretical impossibility of the task.

**Other comments:**

1. Title: What does "shallow reasoning" mean? Some of the presented tasks, such as the Horn clause one, require quite deep reasoning.
2. Section 2: The term "building block" is a bit ambiguous. I thought it meant model building blocks (e.g., a self-attention layer) during the first read. Maybe something like "building blocks for logical reasoning tasks", "subtasks", or "logical phenomena the models need to handle" would be clearer?
3. Section 2.4: "they will still reply" --- reply correctly or reply randomly?
4. Section 3.1: It would be nice to mention that all information is present in the task and no external knowledge is needed.
5. Section 3.2: While the listing of datasets is helpful as references, it would be nice to give some descriptions of what each dataset aims to do, possibly in the appendix. This comment also applies to other sections.
6. Section 3.4: "e.g.," --> "for example" (should be spelled out here)
7. Section 3.4: The provided example from LogiQA looks similar to the Horn clause examples in Section 3.1, yet the models struggle here. Why is this the case?
review_title: A good overview of how transformer models perform on logical reasoning tasks, though the scopes of "transformer models" and "logical reasoning" are not well-defined.

Meta-review: The paper surveys the ability of pretrained transformer-based models on logical reasoning tasks. This is a blooming area of research which makes this paper particularly attractive to AKBC community. Authors introduce basic capabilities in order to do reasoning (e.g. handling negations, stability to word-order, etc). Later, they survey a series of reasoning tasks and report performance on BERT-based models. The main finding is that BERT-based model performs well when all information (rules, facts) required to perform deductive reasoning is given as input. When any of this information is missing, models fail to infer those which is necessary to be successful at reasoning. Reviewers have given excellent set of comments to improve the draft in terms of scope of models and tasks, and main takeaways for the reader. We urge authors to incorporate missing details in the next draft of this paper.
Decision: Accept