Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a synthetic task that aims to capture aspects of multi-hop reasoning, and uses it to characterize the behavior of Transformer models. It finds that pre-trained transformers learn the task more rapidly, and attributes this to specific patterns of attention in the network. Introducing mechanisms that emulate these specific attention patterns helps close this gap in performance without the need for expensive pre-training. Models struggle with extrapolating to longer chains of reasoning than required during training. This deterioration can be mitigated by repeating layers with the same parameters as well as training with stochastic depth. 
strength_and_weaknesses: Strengths:
- The proposed task is useful as a tool for understanding some ways in which a Transformer can perform multi-hop reasoning
- The proposed operationalization of requiring length extrapolation is effective at disentangling issues with positional encoding from the question of whether the model can extrapolate to longer chains of reasoning 
- Experiments help provide an understanding for how pre-training, as well as stochastic depth, help improve certain aspects of the models' capabilities
- Hardcoded attention patterns are shown to be just as effective for solving this specific task
- The writing of the paper is clear and well-structured

Weaknesses:
- One of the strengths of the attention mechanism is that it can learn any attention pattern a task may require, and the multi-head version can learn when (at what layer) and how much to use each pattern. The hardcoded attention patterns seem sufficient for the proposed synthetic task -- almost by construction of the LEGO task. There are some preliminary findings that they may also be effective for masked language modeling and at least SQuAD, but this direction is underexplored and perhaps not all of the necessary components for real NLP tasks are captured.
- The proposed task is perhaps not the best test-bed of *pre-training* specifically, because the approach used is to pre-train on natural language but then test on synthetic expressions. It would not be surprising for language-specific reasoning to not undergo transfer in this case, potentially leaving only the most general of structural patterns, like the attention patterns revealed in this work.
- Overall, it's not clear how much of the Transformers' capabilities are stressed by the proposed synthetic task. It could be that understanding multi-hop computation in the style of the LEGO task is the key for explaining current behaviors and proposing improvements. Alternatively, multi-hop reasoning may just not be a crucial component of many NLP tasks as currently formulated, or LEGO might not be capturing the actual character of chain-based reasoning as required for real-world task.
clarity,_quality,_novelty_and_reproducibility: See above

Minor:
- Regarding Transformers that repeat layers: in addition to ALBERT, it would be helpful to cite other instances such as Universal Transformers (Dehghani et al)
summary_of_the_review: My overall recommendation is based on weighing the thorough and well-written exposition/investigation of the LEGO task with the potential downsides of any synthetic benchmark, namely that it may or may not capture the key elements of actual real-world tasks, and can therefore be limited in its impact.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: LEGO is a synthetic reasoning task that targets chain reasoning. Think Markov chain, where the current state only depends on the previous state. They specifically evaluate BERT and ALBERT. ALBERT is particularly interesting since weights are shared across all layers in ALBERT and is hypothesized that it may be particularly suited for this type of reasoning. Indeed, it outperforms BERT. Various ablations are carried out to understand why.
strength_and_weaknesses: The task and setup are framed well. Ablations such as stochastic depth length extrapolation, pretraining, and probing classifier are creative ways at understanding the inductive biases of the networks.

The task seems not that specific to transformers. I do wonder how other RNNs would fare.
clarity,_quality,_novelty_and_reproducibility: I found the paper clear. I appreciated the language of abstract algebra, but I am not sure if all readers will. I thought the figures were good. The experiments are well done and should be simple to reproduce. There is not much originality in the modeling, but the task and analysis are novel
summary_of_the_review: I think the paper adds to the understanding of the various inductive biases that different transformer architectures have. There is much interest int the mathematical ability of transformers. I would like to see more models benchmarked on this task.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a synthetic task, LEGO, and uses it to understand how Transformers operate. The task is in following a chain of simple reasoning, in particular for simple mathematical operations with variables valued either +1 or -1 and expressions of the kind a=+b or a=-b. With this task, the authors consider BERT’s and ALBERT’s abilities to generalize to longer inputs. They find specific attention heads (e.g., local attention patterns) and propose modification with some of the heads fixed to these patterns.
strength_and_weaknesses: While the paper introduces the new task to help understanding Transformers, the task does not lead to new observations not widely known before. Specifically, it’s well-known that the most important Transformer attention heads play interpretable roles, including positional ([1], [2], [3], etc). This was used extensively to improve/simplify models: [4] fixes attention window size for some of the heads and gets improvement for low-resource settings (similar to this work), [5] develops hard-coded attention without learned parameters, to name a few. From this perspective, (1) neither finding these attention patterns nor proposing fixed attention is novel; (2) doing it on a synthetic task when this has already been done before for different real-world tasks does not make much sense.


[1] ACL 2019: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.

[2] EMNLP 2019: ADaptively Sparse Transformers.

[3] BlackBoxNLP 2019: What does BERT Look at? An Analysis of BERT’s Attention

[4] EMNLP findings 2020: Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

[5] ACL 2020: Hard-Coded Gaussian Attention for Neural Machine Translation

clarity,_quality,_novelty_and_reproducibility: See comments above.
summary_of_the_review: The paper introduces a synthetic task to analyse Transformers. In the analysis, finds specific attention patterns and proposes model modification with some of the heads being fixed. Most of the observations done in the paper and model modification have already been done before without resorting to synthetic tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides empirical investigation of transformer by constructing a synthetic reasoning task -- LEGO (Learning Equality and Group Operations). The LEGO tasks consist of group operations and a chain of reasoning. The major purpose of the synthetic experiment is to provide a controlled setting and unveil what transformer learns. To do so, the paper conduct a series of experiments on the both classic setting (training and testing distribution are the same) and the OOD setting (out of distribution) and they found

(1) Classical generalization happens all the architecture.

(2) OOD generalization depends on the architecture and data preparation.

(3) The non-extrapolating models (architecture fails for OOD) learns short-cut solution.

The author also propose hardcode attention layer that incorporates knowledge over the dataset that seems to work well on LEGO. 
strength_and_weaknesses: Strength. The paper provides solid contribution towards understanding the Transformer architecture. By constructing a synthetic benchmark, it provides a series experiment that provide new insight and confirm existing observations. 

Weakness. There is no major weakness.

Minor comment. The related work section can be improved, in particular, there is a previous line of work "unveils" transformer over regular language ( in particular dyck language) that should be cited and discussed. Examples include [1] (and see the reference therein), which studies the dyck language and provide some theory and empirical investigation on what transformer learns.

[1] Self-Attention Networks Can Process Bounded Hierarchical Languages. Shunyu Yao, Binghui Peng, Christos Papadimitriou, Karthik Narasimhan. ACL'2021






clarity,_quality,_novelty_and_reproducibility: Yes
summary_of_the_review: The paper provides solid contribution to the literature and I vote for acceptance.


Some minor questions over "shortcut":
The paper finds transformer sometime learns short-cut solution to the reasoning tasks. However, it is unclear to me why this is a bad thing. I feel it is not equivalent to spurious feature because some reasoning tasks involves short-cut solution or parallelizable solution, e.g., it is possible that a linear size circuit only has depth $O(\log n)$ (i.e., do not follow the chain of reasoning). Some further discussions should be a good addition on the finding!



------------

Post rebuttal

I have read the rebuttal and my evaluation remains.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new toy task, called LEGO, to investigate how transformer networks perform iterative reasoning. Using this task, the authors then perform an empirical analysis of different architecture variants of transformers, with and without pre-training, and show that different roles are played by different attention heads.

There is a wide range of scores for this paper, and I thus had a thorough discussion with the reviewers. On the positive side, the reviewers enjoyed the direction of this paper, and the analysis provided on the transformer models regarding their abilities to perform reasoning, to generalise to longer sequence than seen during training, and the role of pre-training. On the other hand, some reviewers raised concerns regarding the discussion of previous works that also study how transformers works. More precisely, one of the main concerns around what one could learn by using this new toy task, compared to existing work. I tend to agree with this concern: while the analysis of this paper is well made, I also feel that I did not learn anything new by reading the paper, since most of the observations were already made in other papers. I also believe that the proposed task is a bit naive (eg compared to previous considered ones), and I am not sure what it can add to the field. Moreover, the extensive literature on the subject is not well discussed. For this reason, I recommend to reject this paper.
justification_for_why_not_higher_score: I agree with the concerns raised by reviewers, about the limited impact of the proposed toy task. I think that the paper does not do a good job at situating its contributions compared to existing work, and in particular, what was, or could be, learned by using the proposed LEGO task about reasoning abilities of transformer networks. Moreover, I think that this toy task is not well motivated compared to other toy tasks that were previously considered in the field of ML and NLP.
justification_for_why_not_lower_score: n/a
Decision: Reject