Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies input length extrapolation for Transformer language models; i.e., how Transformer LMs perform on test sequences that are longer than training sequences. The paper finds that how positions are encoded plays a crucial role for input length extrapolation. Models with sinusoidal and rotary position embeddings do not extrapolate well, while T5’s position-dependent attention mechanism (dubbed T5 bias) enables better extrapolation. The paper then proposes ALiBi, another attention mechanism that also allows extrapolation while being computationally more efficient than T5 bias. These results are empirically confirmed on two datasets.
main_review: Strengths:
- To my knowledge, the paper is the first to study length extrapolation in Transformer language models. This is an important open problem for language modeling.
- The proposed ALiBi mechanism is simple to implement and computationally efficient.
- Experiments confirm that the proposed method enables length extrapolation for language modeling.
- The paper is well-written and easy to follow.

Weaknesses:
- Experiments can be expanded. I am curious if the findings also apply to other tasks, such as text classification, sequence labeling, and sequence-to-sequence generation. The proposed method is simple to implement, so I imagine it would not be hard to add a few more tasks.

Missing related work: 
- Xu et al., 2021. How neural networks extrapolate: from feedforward to graph neural networks. This paper studies a similar kind of input length/size extrapolation for graph neural networks.
summary_of_the_review: The paper studies a novel problem, input length extrapolation in language modeling, and proposes a simple solution with good empirical results. The paper is also well-written. One way to further improve the paper is to add experiments on other tasks. Overall, I recommend acceptance of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the extrapolation problem where a test sequence longer than training sequences is given and proposes Attention with Linear Biases (ALiBi) that adds a penalty linear to the distance between a query and a key to the attention scores. ALiBi shows remarkable input length extrapolation ability while computationally efficient with almost marginal overhead compared to the standard transformer. Moreover, ALiBi does not induce any additional parameters and generalizes well to a billion scale language model.
main_review: The method is simple and quite effective. The paper addresses an important research problem of input length extrapolation. ALiBi developed on Wikitext-103 generalizes to 1.3B parameter model. ALiBi’s inductive bias also improves the accuracy.

Previous works did not rigorously evaluate the extrapolation of a transformer and simply assumed the possibility of extrapolation. On the other hand, this paper carefully measured extrapolation compared with other works (Rotary and T5 Bias) and devised their own method to overcome the limitations of previous works. The method itself might look less novel or incremental because previous works inspire its many parts. It would be much better to provide theoretical explanations more than empirical proof on why ALiBi enables better extrapolation and higher final accuracy.

ALiBi is only evaluated on language modeling in this paper. A transformer is a widely used neural architecture for many different tasks and domains. They also mentioned in the related work section that other works studied extrapolation on other tasks. I wonder about the authors’ thoughts whether their ALiBi could be helpful to other tasks as well. Of course, the importance of the longer context and extrapolation ability may vary depending on the task.

One minor question is that the dot products of queries and keys are usually divided by the square root of the dimension, and it is maybe abbreviated in the equation. I am curious this division is performed after or before adding a bias.

Each head has a different slope for the linear bias, so I expect that heads learn different patterns. An analysis of that would be interesting. The authors argue that the method is robust to slope choice, but they found that other alternatives underperform, such as learning these slopes. Because many other design choices are possible, I am curious how they found the final solution and what they tested.

ALiBi was tested on two different model sizes. According to their results, extrapolation on a billion language model (improving until ~2x) is relatively inferior to that on a Wikitext-103 scale language model (improving until ~6x). I worry whether extrapolation ability reduces as the model becomes bigger (or with more training data).
summary_of_the_review: The paper is well written and easy to follow. The contribution is concrete and practically useful since a transformer is a building block of many machine learning models. More importantly, the size of language models becomes bigger, so their training cost is prohibitive. ALiBi improves the efficiency of language model (or transformers in general) training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission proposed an effective approach to allow pre-trained transformer-based language models to extrapolate beyond the maximum length used in training, which potentially reduces the training time as extrapolation is empirically guaranteed. The proposed method adds fixed biases to the dot-product values between queries and keys that linearly decays w.r.t. the gap between two positions. Empirically, the proposed method indeed successfully allows pre-trained models to be evaluated on sequences that are multiple times longer than the training ones without significant loss.
main_review: At a very high level, I did enjoy the paper as the method is simple and it indeed helps a pretrained transformer-based models to extrapolate to much longer sequences. Some of my concerns were addressed in the authors' response, and the others do require extensive exploration. Therefore, I would like to see this submittion at ICLR2022.


====end of the update====

1. When the dimension of a transformer module is roughly the same as or significantly larger than the number of tokens, the dimension becomes the main contributing factor to the time complexity, which explains why, with the linear bias, the model only achieved ~10% speedup.

2. I was wondering if we could directly manipulate the probability after the softmax layer, it probably would achieve a similar effect. For example, one can multiply the probability map with a matrix with 1s in the diagonal terms and with linearly decaying off-diagonal terms towards 0, which also effectively biased the model to learn from nearby tokens. 

My point here is that the submission could have been more generalised in a way that, say, as long as the bias terms are fixed before training and they have an impact on the attention scores or the probability maps, the model will extrapolate to very long sequences. This would've been a stronger and more generalised message.

3. The title and the intro gave me the impression that it was designed for transformers, but I was wondering whether it would hinder transformers' capability in modelling images or biological sequences where tokens that are far from the current one would still play an important role.

For images, the current approach of serialising an image is either at pixel-level or at patch-level, which means that tokens surrounding the current one in 2-dimensional space will be the context, however, the proposed approach would potentially worsen the situation.  

summary_of_the_review: The submission proposed a simple yet effective method that helps pre-trained language models to extrapolate beyond the sequence length used in the training, but I think the paper could've delivered a stronger message. I am open to discussions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the extrapolation capability of transformer-based language models. The authors observed that existing positional encoding methods (e.g., sinusoidal embedding, relative positional embedding) fail to generalize to longer sequences in language modeling tasks. Therefore, they introduce a new positional encoding method called ALiBi, which adds temporal bias to the multi-head attention to penalize attention score proportional to token distances. Experimental results show that ALiBi has significantly stronger extrapolation capability compared to other positional encoding methods.
main_review: Pros:
- Injecting temporal bias to attention is a neat idea for the language model extrapolation problems.
- This paper presents comprehensive experiments on comparing the proposed method with existing positional encoding approaches.
- The paper is well written and easy to understand.

Cons:
- It would be helpful to discuss the potential applications of the proposed method other than language modeling.
- I am curious about comparing the transformer+ALiBi with LSTM in extrapolation tasks. The idea of adding temporal bias to attention is similar to the forget gate in LSTMs. Therefore, adding LSTM as a reference will make the paper stronger.

**Update:** The additional results are convincing. I raised my rating to acceptance.
summary_of_the_review: This paper proposes an interesting and novel idea for enhancing the extrapolation capability of transformer-based language models. A few additional experiments and discussions will make the paper stronger.
review_title: null

Meta-review: This submission proposes a simple, efficient, and effective position representation method for the Transformer architecture called ALiBi. ALiBi enables better extrapolation and performance (in terms of efficiency and task performance). The submission also includes careful analysis and extensive experiments, and notably suggests that the gains of ALiBi may be less pronounced in more scaled-up settings. All reviewers agreed the paper should be accepted. I think it's reasonably likely that ALiBi will become a common choice in future Transformer models, or at the very least that this work will prompt further work on developing improved position representations for Transformer models. I therefore recommend acceptance.
Decision: Accept (Poster)