DAPE V2: Process Attention Score as Feature Map for Length Extrapolation

Authors: Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li
Primary area: foundation or frontier models, including LLMs
Keywords: Transformers, data-adaptive positional encoding, long context, length generalization

Abstract:
The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. 
In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, **the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem.** 
The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution.  Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.

Decision: reject
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730607520901

thread_type:
official_review

<thread_object>
creator: reviewer_7aat

<content>
summary:
This paper extends DAPE and proposes a new approach to address the Transformer long context extrapolation problem by treating attention scores as feature maps. The authors conceptualize attention mechanisms akin to image processing techniques, utilizing convolutional operations on attention scores across different heads. This methodology, inspired by methods in computer vision, enhances Transformer performance on extrapolation tasks across multiple lengths, both in theoretical underpinnings and through empirical validation. It has outperformed some popular position embedding methods such as RoPE and NoPE.

strengths:
+ The application of convolution on attention maps to improve relative position encoding in large language models is both novel and inspiring. The use of convolution, a fast and efficient operation, allows for seamless integration into existing frameworks.

+ The proposed method demonstrates strong performance on the length extrapolation task, outperforming established techniques such as RoPE and NoPE, which underscores its effectiveness.

weaknesses:
- The paper suffers from poor writing and organizational structure. Basic variables such as X, W_Q, and W_K​ are not adequately explained as the context, despite that Transformers are quite popular.

- Confusing Arguments: 
1) Line 181-182 states: "The result of DAPE-NoPE (the Zheng et al. (2024) only combine DAPE with ALiBi, Kerple and FIRE but not with NoPE or RoPE)." This sentence is confusing and seems disconnected from the preceding context. 
2) Line 191-192 mentions: "potentially hindering the evolution of next-generation Transformer models," which lacks clarity and context.
3) Line 198 states: "RoPE first computes the classic attention scores of key-query multiplication with RoPE." This description is unclear and requires further elaboration.

- The authors fail to adequately explain the rationale and motivation for applying convolution to embed position information, abruptly transitioning to technical details without sufficient context.

- The current popular solution for long-context extrapolation is to fine-tune RoPE-based LLMs on long-context data, which is not addressed in the baseline comparisons.

- There is no discussion of computational efficiency metrics such as FLOPS, which would be valuable for assessing the proposed method's practicality.

- The benchmarks employed in the study are limited, reducing the generalizability of the findings.

questions:
Could the authors elaborate on "Proposition 1: Transformers incorporating convolution operations can perform associative recall tasks without the need for positional encoding"? The rationale behind this proposition is unclear and requires further explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616326907

thread_type:
official_review

<thread_object>
creator: reviewer_uyhr

<content>
summary:
This paper introduces an incremental change over the prior work DAPE (Zheng et al., 2024), by extending the MLP used in the attention to 1x3 convolution. This small change achieves improvement over multiple experiment settings.

strengths:
- The paper is clearly written.
- Extending the MLP in DAPE’s attention model to 1x3 convolution achieves improvement over multiple experiments.

weaknesses:
- The major concern is this paper only introduces an incremental change over DAPE, I.e. extending the MLP in attention model to 1x3 convolution. In addition, compared to the gap between DAPE and other baselines, the gap between this paper and DAPE is relatively small.

- This paper could be written in a more straightforward way, by directly showing the difference between it and DAPE, and highlighting why it is crucial. Readers may have confusion about the contribution of this paper and DAPE.

- Line 126: It is hard to buy the insight: *Transformer’s length extrapolation ability is limited by the expressiveness of the naive query-key dot product.* This conclusion is drawn by showing DAPE without position encoding still achieves improvement. But there exists another explanation as follows. Transformer’s length extrapolation ability is limited due to the lack of accurate position encoding. MLP in DAPE implicitly learns the spatial information from the dot product of query and key, thus improving the performance. And extending MLP to 1x3 convolution can further improve encoding the spatial information.

- Discussion about an important reference is missing. “On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location” (in CVPR 2020), by Osman Semih Kayhan and Jan C. van Gemert. It found that the boundary effects operate even far from the image boundary, allowing the network to exploit absolute spatial location all over the image. This may help explain why convolution introduces more gains.

questions:
Please see weaknesses (especially the third one).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684086152

thread_type:
official_review

<thread_object>
creator: reviewer_i54z

<content>
summary:
The authors propose to improve the extrapolation abilities of Transformer models beyond their training sequence length by building upon the previously introduced method of data-adaptive positional encodings (DAPE). The authors find that replacing DAPE’s standard MLP through a convolutional MLP further improves performance.

strengths:
**Originality & Significance:**   
- The authors slightly expand on the insights of the original DAPE paper and provide new results on the original and their improved variant 

**Quality:**   
- Experiments conducted across two datasets in comparison with multiple popular ‘positional embedding’ methods, including NoPE, RoPE, CoPE, ALiBi, Kerple and FiRE
- Insights into how the computational complexity is affected are provided, as well as results for three model sizes

**Clarity:**  
- The paper is mostly easy to read; 
- Graphs and tables are clearly labeled and easy to interpret

weaknesses:
_TL;DR: While I appreciate the work the authors have put into the manuscript and their experiments, the main ‘methodological’ novelty facilitating the approach has already been presented in the original DAPE paper. The authors’ addition of using a convolution instead of an MLP (i.e. replacing a 1x1 conv with a 1x3 conv, combined with inconsistent improvements) combined with the manuscript in its current state is in my opinion not enough to pass the bar for ICLR;_

-	Minor ‘methodological’ addition to existing DAPE, with results varying from ‘improvement’ to ‘decrease in performance’ – see questions. 
-	Insufficient (no) discussion of limitations, although inconsistencies can be easily seen from the presented results – see questions.
-	Interpretation on an in-sight level of results obtained with different method (FIRE, ALiBi, etc.) could be significantly extended
-	Minor: Quality of Manuscript in terms of wording/preciseness of statements

questions:
**Main concerns, questions & potential improvements:**  
- Most results (in fact, almost all) are reported with for ‘Kerple’, which seems to work well (e.g. Figure 2, Figure 3, and Figure 3) in combination with DAPEv2;  
However, when looking at the ‘broader’ applicability in Figure 5, it quickly becomes clear that results across the board are much more inconsistent!   
-> e.g. ALiBi: ALiBi performs well on its own for training seq-len 128, is improved by DAPE-ALiBi – but significantly worse for DAPEv2; 
The manuscript however states that DAPE-1x3 ‘consistently improves performance’, which is incorrect and should be discussed (including insights)
- Appendix E / Section 4.8 shows results for DAPE with kernel-size 1 and 3 – I assume ‘1’ is the classic DAPE, and ‘3’ the v2?   
If so, again – results vary a lot in terms of which one is better for which task and combined with which ‘pe-method’, and I don’t see this discussed in the manuscript appropriately. 
- General: A wider discussion of the limitations would significantly help any reader/user, and I’d suggest the authors consider being upfront about these and provide the reader with helpful guidance (Similarly when using DAPEv2 with FIRE, while there is some improvement, it still ‘diverges’ quickly)

- I’d like the authors to include actual insights based on their experiences and the background knowledge of working with these different approaches (FIRE, ALiBi, Kerple, etc.) – e.g. is one generally preferable? If not, what are the situations you would recommend combining DAPEv2 with any particular one of these?
- In Figure 6, although the model can cheat, I’d be curious why the authors think that the DAPEv2-ALiBi becomes significantly less stable (than both non-cheating and original-non-cheating) 

Additional comments:
- I’d suggest the authors replace some of the references through the seminal works in their introduction in terms of how Transformers have made an impact (e.g. noting CV but not citing ViT/DeiT isn’t good research practice, as these authors should be acknowledged)
- I’d like to suggest the authors to check and potentially slightly rework the manuscript in terms of preciseness of their wording; While I am aware this might be due to language barrier, there are multiple instances where statements are misleading/confusing/too general, e.g. 
  - Abstract: ‘[…] contributing to interactions among distinct tokens, in contrast to earlier feed-forward NNs’ -> This is not really true/correct, as any FFN can establish interactions between elements of data – e.g. a CNN establishes the same over a local window in a sequence, etc.; 
  - L 49: ‘rendering the outputs non-sensical’ -> In the context of NLP, the output will still be a valid word and hence ‘sensical’, the architecture simply loses its ability to learn relationships over a sequence and reverts back to sets/bag-of-words; Also note: The authors discuss “Transformers” in general, and there actually are multiple use cases where Transformers are used on set-based problems
  - …


---
## Update post-rebuttal: 
Some of my concerns have been addressed, and I am therefore increasing my score slightly from 5 to 6 -- but it still remains a borderline case to me.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731482337257

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to All Reviewers

comment:
Dear all reviewers:

We sincerely appreciate the reviewers for their time and effort in the review. We first address some common questions, followed by detailed responses to each reviewer separately. We hope our responses clarify existing doubts. We will really appreciate it if reviewers could kindly reconsider the decision, provided that the main comments are well addressed.

**Q1: The key difference between DAPE and this work. (Reviewer i54z, Reviewer UyHR, Reviewer 7aAt)**

A1: We have highlighted the difference between DAPE and this work at the beginning of Section 3 Method. We directly copy it below.

**The two key differences between DAPE and this work are**: 
* **1) Insight:** DAPE attributes length extrapolation performance gains to adaptive position encoding and DAPE believes that the Bias Matrix Is Necessary, while this work finds DAPE could still improve performance without position encoding so that **we take a broader view, explaining that the Transformer's length extrapolation ability is limited by the expressiveness of the naive query-key dot product, which can be enhanced using image processing techniques;**
* **2) Performance:** DAPE is designed for additive RPE and may underperform with non-additive RPE (e.g., RoPE), whereas this work suggests that increasing kernel size (e.g., with $DAPE_{1x3}) may improve RoPE's performance**.

**Q2: The improvement compared to DAPE(Reviewer  i54z, Reviewer UyHR)**


A2: **The improvement of $DAPE_{1x3}$ is significant, compared to the original DAPE**. The following are the perplexity results on the Arxiv dataset with training length 512 and model size 125M.

| Method   |512       | 1024       | 2048       | 4096   | 8192 | 
|-----|-----------|-----------|-----------|------------|------------|
| NoPE   | 4.68  | 31.79  | 1867.46  | 4666.60   | 5334.85  |
| DAPE-NoPE   | 4.63  | 12.72  | 751.78  | 2033.33   | 2618.13   | 
| $DAPE_{1x3}-NoPE$   | 4.47  | 6.31  | 56.93  | 196.80  | 259.81   | 
| RoPE  | 4.57  | 43.62  | 144.05  | 278.87   | 297.06   | 
| DAPE-RoPE   | 4.53  | 73.31  | 174.48 | 316.84   | 306.78   | 
| $DAPE_{1x3}-RoPE$   | 4.48  | 13.10  | 29.41  | 53.97   | 68.32   |

* For NoPE: reduce the perplexity **from 2618.13 to 259.81.**
* For RoPE: reduce the perplexity **from 306.78 to 68.32.** 

**Q3: The major contribution of the work, compared to previous work related to length extrapolation  (Reviewer i54z, Reviewer UyHR, Reviewer 7aAt)**

A3: The major contribution of the work is the interpretation of the length extrapolation problem.

**The Importance of Perspective of long-context problem and the Corresponding Research Directions**:
* **Transformer [1] suggests that position encoding is important** so that we have better position encoding methods such as RoPE, ALiBi, Kerple, FIRE, CoPEand so on.
* **Position Interpolation [2] suggests Transformer performs badly because of unseen position ID** so that it down-scales the input position indices so that we have better methods including YaRN, CLEX, and so on.
* Now, **we suggest that the Length Extrapolation is caused by the limited expressiveness of the query and key dot product. Though the current method is straightforward, our community could develop better methods in the future based on this perspective.**

In this work, we first point out that the length extrapolation problem is caused by the limited expressiveness of the query and key dot product. Therefore, from now on, We Do Not have to employ complex methods for length extrapolation/long-context but directly enhance the attention score via image processing techniques.

Before this work, There is NO Paper interpreting long-context/length extrapolation on such perceptive. Following the direction, there are many potential works in the future.
* How about utilizing more powerful image processing methods to further process attention scores?
* If the attention score is just a feature map, then why not first resize it to a lower resolution to process it to reduce the cost?
* Besides regarding the attention score as feature maps, what else could the attention be regarded?

**This work is NOT a simple extension related to DAPE, but contributes an important perspective of how we think about the length extrapolation/long context problem.**  Based on such interpretation, better attention score processing methods will be developed in the future, and long-context solutions WILL  NOT be limited to position encoding design or position interpolation any more.

Reference:

[1] Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems.

[2] Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732264674181

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update Summary and the Core Contribution

comment:
Dear AC and Reviewers,

Thank you very much for your arrangement and comments. We have updated the paper and highlighted the changes in red color.
* We add the results of new results, including $\Delta P$.
* We further discuss the related works, including ViT/DeiT, and On Translation Invariance in CNNs.
* Various writing improvements.

Moreover, we would like to highlight the core contribution of the work: the methodology is NOT the most important part of the contribution, but **the core contribution is: We Provide a Totally New Perspective of Long-Context/Length Extrapolation Problem**. **The experiment parts are all used to prove such perspective**.

Next, we will discuss how such a perspective could have an impact on the following works and the community.
* **Before this work, we mainly solved long-context/length extrapolation from two directions/perspectives:**
   * **Research Direction 1: Better Position Encoding (proposed by Vaswani, A. (2017) ).** With such perspectives, our community develop different position encodings, including RoPE, ALiBi, Kerple, FIRE and so on.
   * **Research Direction 2: Position Interpolation (proposed by Chen, S. (2023) ).** With such perspectives, there are extensive following papers, including YaRN, CLEX and so on.
* **In this work, we propose the Third Research Direction: the length extrapolation is difficult because the query and key dot product have limited expressiveness, and direct convolution operation on attention score could significantly improve the length extrapolation performance, whatever the position encoding is  (e.g. reduce NoPE perplexity from 2618.13 to 259.81)**. 

Following this direction, we could have extensive potential works in the future. Here, we leave several questions that we are also interested to share:
   * Besides regarding the attention score as a feature map, what else could the attention score be regarded as?
   * Besides the convolution operation, any other operations that can also improve the expressiveness of attention score?
   * What is the essence of attention score? If the attention score is Just Feature Map, then could we use a better way to construct the attention score?
   * ... ... 

**It is the work that first suggests and proves the length extrapolation/long-context is limited by naive query and key dot product.**
 
**Besides developing better position encoding or better position interpolation method, now we have the third direction: improve the expressiveness of query and key dot product (attention score expressiveness)**.

Again, thank you all for your attention to this work.

Best regards,

Paper 4210 authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733296854130

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Period Summary

comment:
**After the rebuttal, we addressed the Reviewer's concerns, the Reviewers became more positive about the paper and  there is no more clearly presented concerns after the rebuttal:**
* **The Reviewer i54z increased the score to 6, and there are no more concerns mentioned.**
* **The Reviewer UyHR claims that we have addressed the initial concerns and thanks for addressing the previous concerns**. And the last question is also well-addressed.
* **The Reviewer 7aAt also increased the score, and there are no more concerns mentioned.**

**The Core Contributions** of this work: besides position encoding and position interpolation for length extrapolation, we propose the third research direction: improve the expressiveness of query and key dot product for attention score.
* **Before this work, we mainly solved long-context/length extrapolation from two directions/perspectives:**
   * **Research Direction 1: Better Position Encoding (proposed by Vaswani, A. (2017) ).** With such perspectives, our community develops different position encodings, including RoPE, ALiBi, Kerple, FIR,E and so on.
   * **Research Direction 2: Position Interpolation (proposed by Chen, S. (2023) ).** With such perspectives, there are extensive following papers, including YaRN, CLE,X and so on.
* **In this work, we propose the Third Research Direction: the length extrapolation is difficult because the query and key dot product have limited expressiveness, and direct convolution operation on attention score could significantly improve the length extrapolation performance, whatever the position encoding is.

**The Significant Performance Improvement**
*Table: Perplexity Performance with Different Kernel Sizes (on Arxiv Datasets, Training Length 512, Evaluation from Length 512 to 8192)*
| Method   |512       | 1024       | 2048       | 4096   | 8192 | 
|-----|-----------|-----------|-----------|------------|------------|
| NoPE   | 4.68  | 31.79  | 1867.46  | 4666.60   | 5334.85  |
| DAPE-NoPE   | 4.63  | 12.72  | 751.78  | 2033.33   | 2618.13   | 
| $DAPE_{1x3}-NoPE$   | 4.47  | 6.31  | 56.93  | 196.80  | 259.81   | 
| RoPE  | 4.57  | 43.62  | 144.05  | 278.87   | 297.06   | 
| DAPE-RoPE   | 4.53  | 73.31  | 174.48 | 316.84   | 306.78   | 
| $DAPE_{1x3}-RoPE$   | 4.48  | 13.10  | 29.41  | 53.97   | 68.32   |

* For NoPE: reduce the perplexity **from 2618.13 to 259.81, which reduces 90.07% perplexity.**
* For RoPE: reduce the perplexity **from 306.78 to 68.32, which reduces 77.72% perplexity.** 

The Differential Transformer is a popular work and received an average score of 8 in the ICLR 2025 submission, and we copy its reported loss directly for comparison.
Here, we compare the loss reduction within training length to prove that this work actually achieves significant loss reduction.
*Table: Loss Performance with Different Kernel Sizes (on Books dataset with training length 512)*
| Method   |Loss       | Loss Reduction |Loss Reduction Ratio | 
|-----|-----------|-----------|-----------|
| Differential Transformer Baseline   | 3.086  | ---  | ---  | 
| Differential Transformer   | 3.062  | 0.024  | 0.77%|
| Kerple (This Work)   | 2.987  |  ---  | ---  | 
| DAPE-Kerple (This Work)  | 2.957  | 0.030  | 0.99%|
| $DAPE_{1x3}-Kerple$ (This Work)   | 2.942  | 0.015  | 0.51%|

For the performance within the training length, even reducing 0.01 loss is still very difficult and challenging. The $DAPE_{1x3}-Kerple$ reduces the 0.045 loss compared to Kerple, while Differential Transformer reduces 0.024 loss with a reduction ratio of 0.77%. Even compared with DAPE-Kerple, $DAPE_{1x3}-Kerple$ still reduces 0.015 loss and the reduction ratio is 0.51%. This has proved that the performance gap reduction is actually significant

**The work not only brings significant performance improvement but also helps the community find a New Research Direction and better understand the essence of the long-context/length extrapolation problem. The community could follow such research direction for further analysis, including but not limited to better attention score construction, better feature map processing methods for length extrapolation,and so on.**

Best regards,

Paper 4210 authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The proposed method replaces the MLP layer of DAPE with a 1x3 convolutional layer. While the author rebuttal successfully addressed most clarification requests from the reviewers, a critical concern regarding the limited technical novelty (raised by Reviewers i54z and UyHR) remains unresolved. Overall, the contributions of the paper fall short of the ICLR standard, and the AC, therefore, recommends rejecting the paper.

Additionally, the paper must differentiate itself from a significantly related study on correlation-structure-based attention:
[A] Kim et al., Learning Correlation Structures for Vision Transformers, CVPR 2024.