Title: DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transformers

context-adaptive positional encoding

long context

length generalization
Abstract: Positional encoding plays a crucial role in transformers, significantly impact- ing model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of input data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and can be dynamically adjusted with the given attention. In this paper, we propose a Data-Adaptive Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances in terms of trained length and length generalization, where the improvements are statistically significant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.


Peer reviews:
comments: SUMMARY:
This paper proposes a simple learnable positional encoding called CAPE that boosts the length extrapolation performance of Transformer language models.

STRENGTHS:
The empirical performance of CAPE is substantially better than previous positional encodings. I also think Figure 1 nicely demonstrates the flexibility of CAPE, i.e., CAPE can learn both local and anti-local attention heads.

WEAKNESSES:
The speed is unbearably slow in my opinion. While I appreciate the honesty of reporting speed differences in Table 1, the authors really need to figure out a way to improve CAPE's training and inference efficiency.

QUESTIONS:
1. What are the potential ways to improve the speed of CAPE? Please be as concrete as possible.
2. Can the authors test CAPE using needle in a haystack? This way the readers can have a clearer picture of how CAPE is using long context information.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsidering the fixed parameters of RoPE may lead to the generalization issue, this paper introduce a dynamic position embedding method named CAPE, where position encoding is depend on the input context. Specifically, CAPE enables testing-time adaptation to input context by using a two-layer LeakyReLU neural network to parameterize the positional bias added to the vanilla attention module. By meticulously selecting the hyper-parameters, CAPE can achieve better performance compared with previous methods.\n\nSTRENGTHS:\n1. The concept is intriguing. Making position embeddings data-independent can potentially enhance the model's performance.\n2. The authors conducted numerous experiments on small language models and offered valuable insights.\n3. This paper includes several additional experiments shown in the appendix, which provide valuable insights.\n\nWEAKNESSES:\n1. CAPE can introduce extra computation, as well as lower the training and inference efficiency\n2. The choice of hyperparameters is significant for CAPE, as it is correlated with the model performance and the efficiency.\n3. There is no experiments on LLMs.\n\nQUESTIONS:\n1. Can CAPE adapt to LLMs, for example, Llama3-8B, with few post-training steps? \n2. What's the context length boundary of CAPE? In this paper, experiments just demonstrate the maximum context length with 8192.\n3. The authors claim that \"CAPE is semantically dependent and adaptive.\" What does \"semantically dependent\" mean? If CAPE is used for a summarization task with different context lengths (perhaps 8192 or 4096), what does \"semantic\" refer to in the summarization task? Providing a few examples could help clarify the writing.\n4. About motivation: Even though positional encoding (PE) is fixed during inference, attention still relies on the input context. What if we consider a scenario where PE simply provides the context index to help the language model distinguish the positions of each token (static), while the attention module dynamically selects key information and provides semantic information to assist the language model? Why does CAPE need to incorporate such a dynamic selection mechanism, which is already learned by LLMs, into the PE?\n\nLIMITATIONS:\nSee weakness and question part.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes context-adaptive positional encoding. The paper proposes a 2-layered MLP to non-linearly integrate positional bias information (can be computed using prior methods like Alibi, FIRE, and Kerple) and query-key content-based dot product values representing semantic relations across different heads to dynamically create a contextual position-informed matrix that can additively modulate the self-attention matrix.\n\nSTRENGTHS:\n1. The idea is reasonably motivated.\n2. The empirical results are promising for length extrapolation in language modeling. The experiments on Chomsky's hierarchy tasks are a nice touch.\n\nWEAKNESSES:\n1. If I understand correctly, the original relative encoding (from Shaw et al. and then the one in Transformer XL) could also count as contextual. There, the position-based additive values are computed based on the dot product of queries and distance encodings (treated as keys). So, the context can influence the distance-related bias through the query representations. This could be better discussed in the paper. I would also be curious how Transformer XL style relative encoding would perform in the language modeling datasets. \n\n2. There is some hit in the computation expense. Although it is manageable, the authors don't treat each head independently but use the concatenation of heads with a shared hidden state.\n\nQUESTIONS:\nMinor suggestions:\n\n* It Would be good to refer to the appendix implementation section G when discussing function f and the multi-head CAPE in the main paper. \n* Perhaps it is better to present the pseudocode with einops-style syntax.\n\nLIMITATIONS:\nThere isn't an explicit section purely for limitations, but there is a section for computational cost analysis, which may illuminate some limitations (and authors treat the section like a limitation section in the checklist). Overall, it's mostly adequate.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a doable solution for long context tasks. The paper introduces the Context-Adaptive Positional Encoding (CAPE) method to enhance transformer model adaptability and flexibility in processing long input lengths and contexts. To overcome the limitations of static positional encodings such as Absolute Positional Encoding and Relative Positional Encoding, the proposed method adjusts the positional encodings based on input context and learned fixed priors. Experimental evaluation on Arxiv, Books3, and CHE shows that the proposed method significantly improves model performance in length generalization.\n\nSTRENGTHS:\n1. The long-context ability of LLMs is crucial and fundamental for advancements in the LLM domain, significantly impacting downstream tasks.\n2. The proposed method is intuitive and straightforward to implement and understand.\n3. The performance of the proposed method, as indicated by PPL, appears promising and is well-demonstrated in Figures 2, 3, and 4.\n\nWEAKNESSES:\n1. The major concern is in the experimental evaluation part. The experiment is limited to PPL, which is insufficient for verifying long-context ability.\n2. The experiments are limited to a very small LLM model - a 124M transformer, which is not enough to verify the proposed method, as large LLMs and small LLMs behave quite differently.\n3. The proposed method is not convincing to me. The proposed method is mainly in Equations (2) and (3). In Equation (2), the proposed method does not seem promising. Considering the next step ( $h(*)$ ) in the neural network, it will be $ h(A_{\\text{CAPE}}(X)) = h(XW_Q(XW_K)^\\top + f(XW_Q(XW_K)^\\top, B)) = g(XW_Q(XW_K)^\\top, B) $. Could the author please elaborate on this?\n4. Similarly, for Equation (3), $ A_{\\text{CAPE}}(X) = XW_Q(XW_K)^\\top + B + f(XW_Q(XW_K)^\\top, B) $, it has a similar problem to Equation (2).\nDue to the major weakness in the experiment and the unreasonable proposal of the bias term, I would like to reject this paper.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}