DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

Authors: chuanyang  zheng
yihang  gao
han  shi
minbin  huang
jingyao  li
jing  xiong
xiaozhe  ren
michael  ng
xin  jiang
zhenguo  li
yu  li
  noah's ark lab
  hkbu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:53:39

Abstract: Positional encoding plays a crucial role in transformers, significantly impact- ing model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of input data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and can be dynamically adjusted with the given attention. In this paper, we propose a Data-Adaptive Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances in terms of trained length and length generalization, where the improvements are statistically significant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.

Main paper: 
 introduction: 
 Transformer-based models have shown state-of-the-art performances in many language processing tasks, including translation [6] , question-and-answer [82, 29, 3] , and commonsense reasoning [65] . The transformer mainly consists of attention block, feed-forward block, and positional encoding. Recent works [8] have proved that quadratic-cost attention from the softmax is necessary for better performance, especially in long-context processing. The attention block was originally designed by applying softmax to the key-query multiplication, which requires quadratic computational cost. To address such challenges, some efficient transformers were proposed, including sliding window transformers (e.g., Streaming LLMs [77] ), linear transformers (e.g., Performer [17] ), and sparse transformers (e.g., Reformer and sparse Sinkhorn transformer [66, 25] ), etc. However, some negative results exist regarding efficient transformers' performances [80] .It has been noticed recently that well-designed positional encoding significantly improves the model performances, especially in the long-context tasks [33] . While transformer-based models exhibit satisfying performances in tasks of consistent length and distribution, their effectiveness tends to diminish sharply when the input length exceeds the training length, e.g., long document summarization, "needle in a haystack" search, and long text generation. To avoid the expensive computation in training, the training length is usually preferred to be relatively small due to the quadratic cost of softmax-based transformers. However, real-world applications often require processing longer input sequences, posing a significant challenge. Therefore, there is a growing interest in evaluating model performance by training on shorter sequences while testing on longer inputs. Standard transformers may not distinguish the ordering of tokens without external assistance. In practice, they depend on positional encoding to incorporate positional information, enabling the model to make meaningful token predictions. Without these encodings, token generation would lack the necessary contextual order, rendering the outputs nonsensical. The RoPE [62] positional encoding method demonstrated a notable performance degradation, failing entirely when the input length is double that of the training length [51, 10, 24] . A common characteristic among these positional encodings is their pre-defined and static nature. Specifically, they are fixed across various tasks and models, which may lead to their inability to adapt to varying input lengths and contexts effectively. To address this issue, recent works have introduced Functional Interpolation for Relative Positional Encoding (FIRE) [41] , which utilizes a neural network to learn an implicit mapping from input positions to positional bias.A functional approach to positional encoding that dynamically adjusts positional biases based on semantic information (input context) allows the model to empower adaptability beyond the fixed inductive bias as adopted in previous studies (such as RoPE [62] and Alibi [52] ). Although FIRE utilizes MLPs to learn positional embeddings, these embeddings remain fixed across different tasks once the training is completed. Intuitively, the learned static positional encoding (such as Kerple and FIRE) is an average optimal solution across all training samples. Consequently, while they might be generally effective, they are inherently suboptimal for any specific instance. This static nature limits their flexibility and applicability in various real-world scenarios that deviate from the training context. In this paper, we introduce a data-adaptive positional encoding (DAPE) method, inspired by the limitations of static PEs. DAPE dynamically adjusts the PE based on the semantic information (e.g., the current attention value) a and the positional indicator b. The proposed PE is represented by MLPs due to their universal approximatability, i.e., MLPs(a, b). We note that DAPE is compatible with all additive relative PEs and offers advantages in terms of interpretability and ease of implementation. The proposed DAPE incorporates both the semantic and the positional information, making the PE adaptive with the input data. The adaptivity allows DAPE to overcome the inflexibility and achieve relatively optimal performance for each individual instance by dynamically adjusting on each specific input data. To the best of our knowledge, this is the first semantically dependent and adaptive positional encoding method introduced in transformer architectures.The paper is organized as follows. In Section 2, we review some related works on positional encoding methods, including absolute and relative positional encodings as well as the potentially no positional encoding in some transformer models. In Section 3, we introduce the proposed DAPE method with implementation on multi-head attention and analysis on computational costs. We conduct comprehensive experiments on DAPE, validating its effectiveness and performances on various language tasks and datasets, as reported in Section 4. In Section 6, some concluding remarks and potential future works are presented. 
related works: 
 No positional encoding. Haviv et al. [30] show that decoder-only Transformers with causal attention masks can learn positional information even without any explicit positional encoding. Recently, Kazemnejad et al. [33] proved the effectiveness of no positional encoding (NoPE) [71] . Although the NoPE can implicitly catch the positional information, it performs poorly compared with some explicit positional encoding methods [41] .Absolute positional encoding. Vaswani et al. [69] proposed Absolute positional encoding (APE) to endow transformers with positional information. In particular, in the first layer, a (learnable or fixed sinusoidal) real-valued encoding [69, 35, 42, 70, 47] e i ∈ R d is assigned to each position i, leading to an APE matrix E = [e 1 , • • • , e n ] ⊤ , which will be added to the input sequence. Though simple and straightforward, APE-based Transformers usually generalize poorly to longer sequences [52] .Relative positional encoding. Relative Positional Encoding (RPE) is another popular way to encode positional information [58, 56, 52] , One popular RPE method in large language models is rotary positional encoding (RoPE) [62, 18, 67] . RoPE rotates the query and key vectors with an angle proportional to their absolute positions before the dot product attention, which results in attention being a function of the relative distance between the tokens, capturing the relative positional information. Press et al. [52] and Kazemnejad et al. [33] found that RoPE-based language models have poor length generalization. To address this, positional interpolation (PI) [11] is proposed to extend the context window. Following the direction, there are LongLora [12] , LongRope [24] , YaRN [51] and CLEX [10] . Another popular direction is additive positional encoding. For most of these additive RPE methods, the computation of the (pre-softmax) attention logits can be unified using the following formula:A RPE (X) = XW Q (XW K ) ⊤ + B,where the bias matrix B ∈ R n×n is induced by the position encoding function b : N 2 → R and the (i, j)-th entry of B is defined as b(i, j). Different formulations and parameterizations of b lead to various RPE variants. Several methodologies that facilitate arbitrary sequence lengths include T5's RPE [56] , Alibi [52] , Kerple [13] , Sandwich [14] , and FIRE [41] . Currently, additive RPE delivers relatively robust performance in length extrapolation without necessitating additional operations. Alibi constructs the bias matrix B utilizing prior knowledge, resulting in a basis matrix devoid of learnable parameters [52] . Conversely, both Kerple [13] and Sandwich [14] incorporate two learnable parameters to facilitate the learning of a bias matrix while retaining some elements of priors. FIRE suggests adopting a learnable continuous function, such as MLPs, to convert input positions to biases [41] . Observing these developments, it becomes evident that the next generation of bias matrices will likely incorporate adaptivity and flexibility. Based on this insight, we propose our method DAPE, a semantically adaptive method. 
method: 
 In this section, we formally introduce DAPE (data-adaptive positional encoding), a new relative positional encoding approach that further enhances transformer performance. Compared with previous works on static positional encoding methods, DAPE adopts semantically adaptive positional bias matrices depending on input context. We first demonstrate that most of the popular positional bias matrices are fixed once the training is finished, independent of the input sequences. To address this limitation, we then accordingly develop DAPE that captures the implicit relationships by MLPs and adjusts the bias matrices based on input context. Furthermore, we discuss a variant of DAPE with residual connections and its extensions to multi-head transformers.For most additive RPE methods, the computation of pre-softmax attention logits can be unified under the following formula:EQUATIONwhere the bias matrix B ∈ R n×n is induced by the position encoding function b : N 2 → R and the (i, j)-th entry of B is defined as b(i, j). We observe from the formulation of those additive RPEs that they remain static once the training process is completed and depend solely on the positions, regardless of the input context. This inflexibility and lack of adaptivity can lead to performance degradation, especially in tasks involving long-context generation and reasoning. Intuitively, the learned static RPEs are optimal on average across all training samples. However, this means they are suboptimal when considering each individual instance, as they cannot adapt to specific tasks. To address these challenges and enhance model performance, it is essential to adopt an alternative approach using a semantically adaptive RPE that depends on input context.For simplicity, we first consider the single-head case and the extension to the multi-head transformer will be discussed subsequently. The design of data-adaptive positional encodings in natural language tasks is motivated by the need to capture the intricate relationships between tokens. Arora et al. [5] reveals that associate recall accounts for most of the perplexity difference between transformer, RNNbased, and convolution models. For example, we consider a consistent pairing that "Hakuna" is always followed by "Matata" in a long paragraph. This pattern suggests a reduced reliance on positional information in favor of enhancing token embedding similarity, thus allowing for 'Hakuna' to be effectively linked with a preceding 'Matata'. Similarly, in tasks involving long-context understanding and search, semantic similarity should be prioritized in the attention mechanism rather than being overshadowed by positional encodings, which can be less relevant over long distances. Consequently, the transformer should preserve information without being influenced overly by positional distance. Instead, a satisfactory PE should integrate both semantic and positional information. Therefore, a semantically dependent positional encoding approach is preferable and expected to enhance model performances. Here, we use the attention XW Q (XW K ) ⊤ to represent the semantic information and positional bias matrices B (e.g., Alibi and FIRE) to capture positional details. Then the context-adaptive PE is described by f (XW Q (XW K ) ⊤ , B), where f (•) is an implicit function that integrates both semantic and positional data into the desired positional encodings. Thus, the pre-softmax attention logit incorporated with DAPE is formulated asEQUATIONHere, f : R T ×T × R T ×T → R T ×T is an element-wise function. In practice, we utilize a two-layer LeakyReLU neural network to parameterize f (•) due to its universal approximability [36] . All parameters are learned directly from the data during the training process. This architecture allows f (•) to dynamically adjust positional embeddings based on the input context, ensuring that the encoding method is both adaptive and dependent on the input data.Different from FIRE, which also models the implicit positional encoding by MLPs, our approach additionally integrates semantic information. This integration enables the adaptivity, flexibility, and context-dependency of the positional encodings. Significantly, our method is compatible with most additive RPE techniques, as these commonly involve positional bias matrices B that inherently contain positional relations. Unlike previous RPEs, which rely solely on absolute positional differences, our DAPE method, can be seen as utilizing multi-level positional bias matrices. Here, the bias matrices dynamically adjust based on the input context, offering a more reasonable and responsive encoding mechanism.Expressiveness. Due to the universal approximability of (LeakyReLU) neural networks [36] , f (•) is capable of capturing complex relationships between the desired positional encoding and both semantic and positional information. Regardless of the semantic component, when the relative position i -j is used as input, DAPE can realize classical additive RPEs (e.g., Alibi and Kerple), according to [41] . This demonstrates the versatility of DAPE in accommodating traditional encoding schemes while also offering enhanced capabilities. There exists a fundamental trade-off between the expressiveness and computational costs. Wider hidden layers lead to higher expressiveness but also contribute to more computational costs. In practice, we find that two-layer neural networks with 32 hidden units per layer provide sufficient expressiveness to deliver satisfactory performance, balancing complexity and efficiency effectively.Discussion. We can also interpret the proposed method from an alternative perspective. In the standard transformer architecture, the pre-softmax attention typically involves the key-query similarity and the positional encoding by either addition (in the form of a + b, e.g., Alibi and Kerple) or multiplication (in the form of a * b, e.g., RoPE). Here, we propose a unified approach by replacing them with learnable MLPs, i.e., MLP(a, b). This configuration allows the model to learn the desired relationship between the pre-softmax attention, the key-query similarity and the positional encoding. It can also be regarded as a new transformer architecture that empower the transformer with additional MLPs on pre-softmax attentions.A variant of DAPE with residual connections. It is well-known that deep neural networks may suffer from gradient vanishing. To further enhance the practical performances, we introduce the residual connection for positional information. Consequently, Equation 2 is modified as follows:EQUATIONIn this reformulation, f (•) acts as an adaptive correction term to the traditionally fixed RPE, dynamically adjusting the positional bias matrices B based on both semantic and positional inputs. In Section 4, we empirically explore the impact of residual connections in DAPE. Our observations reveal that for well-behaved bias matrices B, the DAPE model with residual connections, as specified in Equation 3, is preferable. Conversely, if the bias matrix is underperforming but still conveys positional information, the original implementation in Equation 2 is more effective.Multi-head DAPE. In its simplest form, DAPE is considered for a single-head case as described in Equation 2 and Equation 3. However, adopting a multi-head mechanism significantly enhances model capabilities. To effectively combine both semantic and positional information, the DAPE in a multi-head setup processes the key-query similarities and bias matrices from all heads. Specifically, for an h-head layer, the function f (•) inputs a 2h-dimensional concatenation of key-query similarities and positional biases. It then outputs h-dimensional vectors, where each element corresponds to the DAPE for the respective head. We have shown the code implementation in Appendix J. Importantly, semantic and positional information across different heads are processed simultaneously within the same MLPs, rather than sequentially. This approach not only improves computational efficiencies through parallel processing but also capitalizes on the richer semantic information available across all heads. Compared to the key-query similarity derived from a single head, the comprehensive attention from all heads yields more substantial semantic information.Computational costs analysis. Here, we evaluate the additional computational costs introduced by the DAPE method, compared with the classical positional encoding methods (e.g., Alibi and Kerple). We consider a transformer model with h heads and assume a sequence length of N and all hidden dimensions in the attention layer being d.  
experiment: 
 Baselines. We evaluate the proposed DAPE against a range of established baselines, including NoPE [33] , RoPE [62] , YaRN [51] , Randomized RoPE [57, 31] , T5's Bias [56] , Alibi [52] , Kerple [13] , and FIRE [41] . For RoPE, the randomized positional encoding [57, 31] is applied to enhance the model performance, extending the randomized length to four times that of the training length.Datasets. Our analysis involves training language models on the Arxiv and Books3 datasets, which are frequently used benchmarks for evaluating model performance [52, 13, 41, 24] . We start our evaluation by comparing the last 256 tokens' zero-shot perplexity across different input lengths.Besides perplexity as evaluation metrics, we also employ the downstream datasets in randomized positional encoding [57] The performance on longer training length 1024. As shown in Figure 3 , the proposed method consistently delivers state-of-the-art performance for the training length of 1024. When the evaluation extends to 2048, both DAPE-Kerple and DAPE-FIRE achieve notable results, recording performances of 3.91 and 3.93 perplexity scores, respectively. Remarkably, DAPE-FIRE behaves well at the longer evaluation length of 8192, achieving a performance of 3.91 scores and surpassing Alibi's score of 4.28. These findings reveal that DAPE sustains robust performance with a longer training length of 1024.DAPE enhances intra-length performance, indicating that its lower perplexity may come from thorough utilization of entire sentences but not disregarding long-distance information (Also proved in Figure 1 ). Compared to Alibi, Kerple, and FIRE, the adapted versions DAPE-Alibi, DAPE-Kerple, and DAPE-FIRE demonstrate consistently and significantly better intra-length performance. With the growing sequence length, the Alibi tends to transition from full attention to almost local attention, and this is why Alibi is worse than most baselines within training length but better beyond training lengths. The results (as shown in Table 5 ) indicate that the superior intralength performance of DAPE is statistically significant, with a p-value less than 0.05. Therefore, the consistent intra-length performances across various training lengths indicate that the lower perplexity of DAPE results from effectively utilizing the entire sequence, rather than focusing on local parts and neglecting long-distance information.DAPE significantly improves length extrapolation performance, compared to ALibi, Kerple, and FIRE. DAPE-Kerple significantly surpasses competitors like vanilla Kerple when training and evaluating at different lengths. On the Arxiv dataset trained at a length of 128, DAPE-Kerple achieves a remarkably low perplexity of 5.00 at an evaluation length of 8192, in stark contrast to Kerple's 31.93. Similarly, on the Books3 dataset with a training length of 512, DAPE-Kerple records a perplexity of 17.88 at the same extended evaluation length, far outperforming Kerple's 39.31. These results affirm that DAPE, through its semantic adaptivity and flexibility, consistently enhances performance beyond training lengths, eclipsing static positional encoding methods. DAPE enhances performance with increasing model sizes. As the model size grows (as shown in Figure 4 ), DAPE consistently demonstrates an improvement in performance metrics. When the model size is augmented from 125M to 350M, the perplexity at an evaluation sequence length of 8192 (with a training length of 512) for DAPE-Alibi shows a notable decrease from 3.82 to 3.57. These numbers are appreciably smaller than those recorded for original Alibi, which decreases from 4.54 to 4.21 in perplexity, indicating a robust performance improvement. Additionally, DAPE-Kerple significantly reduces the perplexity for Kerple, bringing it down from an initial 22.76 to an impressive 3.43. These results confirm that DAPE retains its efficacy and continues to perform well even as the model size is increased, mainly due to the adoption of semantically adaptive PEs. DAPE methods almost are ranked top-3 with large model size. With the incremental model size, DAPE-FIRE begins to match, and nearly approach, the performance levels of Alibi. Initially, at a model size of 125M and a training length of 512, DAPE-FIRE achieves a perplexity of 5.71 at an evaluation sequence length of 8192, while Alibi stands at a perplexity of 4.54. However, as the model size is increased to 350M, the performance gap significantly narrows. Specifically, DAPE-FIRE outperforms Alibi regarding the perplexity scores when the evaluation length is smaller than 4096, as the model size grows for evaluation. In conclusion, as shown in Figure 4 , we observe that the DAPE methods almost win the top-3 among all positional encoding methods. This trend underlines the scalability and adaptability of DAPE, emphasizing its potential to handle more substantial computation challenges.In this section, we evaluate the performance of DAPE across its various forms. Our analysis focuses on DAPE-Kerple. Notably, as shown in Figure 5 , all variants of DAPE surpass the baseline performance of Kerple. The Addition_Residual variant of DAPE, while requiring less computational effort, delivers relatively inferior results. As illustrated in Figure 5 Add_Residual: XWQ(XWK ) ⊤ + B + f (XWQ(XWK ) ⊤ + B); (2) Concate: XWQ(XWK ) ⊤ + f (XWQ(XWK ) ⊤ , B); (3) Concate_Residual: XWQ(XWK ) ⊤ + B + f (XWQ(XWK ) ⊤ , B).Even small D DAPE can improve the performance. The experiments are conducted with Alibi and DAPE-Alibi. As shown in Appendix Figure 6 , when considering the training length 128 and D DAPE is set as 4, the DAPE-Alibi achieves 8.25 at evaluation length 128 and 5.67 at length 8192, which is better than Alibi's 8.31 and 5.85. Whatever D DAPE is 4, 16 32, or 64, the performance is always better than the original Alibi at all evaluation lengths. This suggests the effectiveness of DAPE, even with smaller D DAPE .The choice of D DAPE . Based on the experiment, overly small values of D DAPE can degrade performance, although they still perform better than the baseline. Conversely, larger values of D DAPE increase computational costs. The function f (•) is implemented as a two-layer MLP, where the input dimension is either the head number or twice the head number, and the output dimension is the head number. Therefore, we recommend setting the hidden dimension to the head number to prevent information loss and ensure the capacity of f (•). In this subsection, we present the visualization of learned positional encoding biases from a DAPE-Kerple model pretrained on Arxiv (training length is 512). We plot the learned positional encoding bias for the query token at the 8192th position, for all the attention heads from selected layers in Figure 1 . We would like to highlight two features of DAPE. First, in different attention heads, the bias matrix of DAPE learns both local and "anti-local" attention patterns that emphasize more on far-away keys (just like FIRE), compared to a fixed local inductive bias (such as Kerple and Alibi).Secondly, the bias matrix can be dynamically adjusted with different attention values, compared to the static bias fixed for all attentions. We have shown more examples, including different layers and different samples, in Appendix I.Besides employing perplexity as an evaluation metric, we also evaluated DAPE on downstream Chomsky Hierarchy Evaluation Benchmark (CHE) [21] (need to utilize the whole sentence information to generate correct answers) to further discuss its effects. The experimental setup follows randomized positional encodings [57] , detailed in Table 4 , with the experiment setting shown in Appendix Section D. Overall, FIRE outperforms Kerple in 9 out of 14 tasks, while Kerple outperforms Alibi in 11 out of 14 tasks. This observation aligns with findings in [41] , suggesting that the experiments in Table 2 are reliable and reflect the performance of positional encoding in downstream tasks.DAPE works better on permutation-variant tasks. DAPE (with Kerple and FIRE) presented the best performance in 10 out of 11 permutation-variant tasks (which require positional information), achieving the second-best performance in the SOLVE EQUATION task. This underscores the efficacy of DAPE with semantic adaptivity in handling permutation-variant challenges.DAPE's performance on permutation-invariant tasks. In tasks that are permutation-invariant, where positional information is non-critical, DAPE demonstrated comparable performance. Notably, DAPE-Alibi achieved scores of 50.30 on PARITY CHECK and 99.38 on BUCKET SORT tasks, compared to the highest scores of 50.97 and 99.57, respectively, demonstrating competitive performances.Comparative performance improvements. DAPE consistently enhanced performance across various tasks, especially on permutation-variant tasks. Specifically, DAPE improved upon Alibi and FIRE's results in all 11 tested permutation-invariant tasks. Similarly, it outperformed Kerple in 10 of these tasks. These results highlight the effectiveness of DAPE over static positional encoding methods like Alibi, Kerple, and FIRE, resulting from its dynamic adaptivity. 
evaluation protocol: 
 In this work, we initially utilize the model to process the entire input sentence and subsequently select the final 256 tokens for perplexity computation. This approach contrasts with a variety of other studies, which employ methods that process the full sequence during perplexity calculations [51, 31, 12, 24, 10, 41] . As a result, our reported baseline perplexity is comparatively higher than the results presented in ALiBi [52] , which adopt a non-overlapping evaluation strategy. This method divides sequences longer than L into multiple segments of length L, thereby yielding lower perplexity figures.Though ALiBi [52] and Kerple all claim that they use non-overlapping evaluations, their reported results are different. For example, in the ALiBi paper Table 2 , the sinusoidal position encoding perplexity increases from 20.05 (evaluation length 512) to 406.01 (evaluation length (evaluation length 15512), while in Kerple paper Table 3 the sinusoidal position encoding perplexity from 33 to 30046. This may be caused by different evaluation protocols and training strategies.Recommended Protocols. We strongly recommend that researchers process the entirety of the sequence before selecting the last K tokens for the purpose of calculating perplexity. The rationale behind processing the whole sentence is that it provides a comprehensive evaluation of the model's capability to handle long-context dependencies, thus offering a more accurate reflection of its performance. Following this step, we advocate for the selection of the last K tokens to compute perplexity, ensuring that the same number of tokens is used across different evaluation lengths, which promotes consistency and comparability in the results.Release this work's code for future work. In light of this methodology, we have made our code publicly available to other researchers in the field. This initiative aims to facilitate a standardized comparison and evaluation of their respective methods, thereby advancing the collective understanding of model performance in relation to perplexity calculations. 
conclusion: 
 In this paper, we propose the data-adaptive positional encoding (DAPE) by incorporating both the semantic and the positional information to improve the model performance. We show that the additional computation introduced by DAPE is not significant under proper choices of hyperparameters. We conduct comprehensive experiments on Arxiv, Books3, and CHE to validate the effectiveness of the proposed method, revealing that the adaptive PE method has advantages over static PEs. We believe that the DAPE could benefit the whole community, especially on length generalization tasks. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, we have cited the related works.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Positive societal impacts. We propose a method for length extrapolation, which will be helpful for transformer-based models to process long context.Negative societal impacts. This method may be abused for other potential long-context applications.All experiments are conducted on 8 x A800 GPUs. The 125M model configuration is the following. Following the framework established by [21, 57] , we conduct evaluations of our DAPE on a suite of tasks derived from the domain of formal language recognition. These tasks include modular arithmetic, reversing and duplicating strings, binary operations and bucket sort. Based on the Chomsky hierarchy [16] , these tasks are categorized into distinct classes: Regular (R), Context-Free, Context-Sensitive (CS), and Recursively Enumerable. Each class aligns with specific computational models: Regular tasks are solvable using Finite-State Automata (FSA); Deterministic Context-Free tasks can be addressed by an FSA equipped with a deterministic stack; and Context-Sensitive tasks require an FSA complemented by access to a bounded tape. Table 4 : The example of different tasks. † † † denotes permutation-invariant tasks, which are expected to be solved without positional information. More explanation of tasks can be found in [21] . Moreover, DAPE-Alibi, DAPE-Kerple, and DAPE-FIRE not only perform better than their respective original methods but also show superiority over RoPE and T5's bias across all validation lengths. This suggests that the modifications implemented in DAPE versions provide a more robust and generalized model, capable of maintaining lower perplexity and thus better performance on the Arxiv dataset.EQUATIONIn conclusion, the statistical analysis confirms that the proposed DAPE variations offer significant improvements in perplexity performance, thereby validating their effectiveness in comparison to their original counterparts and other baseline methods.F Data-Adaptive Related Position Encoding Performance Comparison According to the experiment, the transformer-xl achieves good performance on length extrapolation. Therefore, this also suggests that the position encoding should interact with attention/query/key to further improve the performance. We further conduct an ablation study on the f , proving that f help enhance the bias matrix. DAPE improves the bias matrix for A f inal , while the A f inal is used to calculate A f inal K. For the unseen position, the B partially could handle it (FIRE changes the problem to interpolation), but is not accurate enough so that DAPE helps enhance the bias matrix B via attention score. The experiment suggests two points: 1) The DAPE f (QK T , B) is better than naive f (B), suggesting that the contextadaptive is important.2) The QK T + B + f (B) is better than QK T + B, suggesting that benefiting from improving the expressiveness of the bias matrix.H The result on 2.7B and 6.7B The model is trained with DAPE-Kerple on length 512. Relative Distance Relative Distance Relative Distance Relative Distance Relative Distance Relative Distance In this section, we present the implementation of the proposed DAPE module in PyTorch [49] . • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: **Summary**

Authors propose a new positional embedding which consists of an additional mlp layer learnt on top of existing additive bias (expresses the relative positions) and the attention itself. Authors show that proposed context-aware positional embedding improves generalization over the baselines when applied on top of AliBi, KERPLE and FIRE for the sequences which are longer than training ones. Also authors show that proposed modification is learning both local and anti-local information.

**Justification for decision making**

It was a long and productive discussion among reviewers and authors, and also between AC (me) and authors. Most of the issues, concerns and ablations were resolved during long discussion and e.g. one of the reviewers raised the score from 6 to 8. However, 1/4 reviewers did not comment anything to rebuttal and has the lowest score of 3: I believe authors provided proper and reasonable answers after which the argumentation for rejection based on that review is no longer valid to me. 

I raised several important issues and misalignments for the paper during the discussion period. During rebuttal period authors provided explanations to my points saying that they changed the evaluation protocol compared to prior works and argued that this protocol is more suited. They pointed out that this change in the protocol led to different results for the baselines compared to prior works results for that baselines. During AC-reviewers discussion one of the reviewers agreed to the raised issues, but also confirmed that the protocol authors used is more suitable. However the reviewer also agreed that better apple-to-apple comparison would strengthen the paper. In the end this reviewer is still on the positive side of the paper acceptance and experienced similar observations on many aspects as authors reported.

**Recommendation**

Based on the positive narrative of 3/4 reviewers (6, 7, 8), taking into account rebuttal to the 4th reviewer which did not respond to the rebuttal, also taking into account AC-authors discussion and AC-reviewers discussion, I **recommend acceptance of the paper**.

However, I ask authors for **several critical modifications for the camera-ready paper**:
- **change the title of the paper** due to collision with already existing work on positional embedding called CAPE and published at NeurIPS: Likhomanenko, T., Xu, Q., Synnaeve, G., Collobert, R. and Rogozhnikov, A., 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34, pp.16079-16092. to avoid naming problems in the future.
- include detailed discussion of evaluation protocols and why results for the baselines are different from the prior works to avoid misinterpretation of results and doubting the baselines from the readers, also for the sake of science.
- include all rebuttal discussions and ablations with new results into appendix as they all have merits and contribute to better understanding and practical applications of the proposed method.
- if possible add at least one apple-to-apple comparison and setup with prior works so as to have strong support that the baselines implementation is correct and consistent with prior works. I and reviewers still found some results reported for the baselines to be strange even when 512 training length is used (128 length definitely gives different observations compared to 512 training length and thus different from prior works as they all used larger context length for training).
- if the numbers and behaviour you observe diverges from prior work, please discuss this in the results and point to the source of this discrepancy.

Reviews: review: paper_summary: paper_summary: This paper offers a doable solution for long context tasks. The paper introduces the Context-Adaptive Positional Encoding (CAPE) method to enhance transformer model adaptability and flexibility in processing long input lengths and contexts. To overcome the limitations of static positional encodings such as Absolute Positional Encoding and Relative Positional Encoding, the proposed method adjusts the positional encodings based on input context and learned fixed priors. Experimental evaluation on Arxiv, Books3, and CHE shows that the proposed method significantly improves model performance in length generalization.
strength_weakness: strength_weakness: 1. The major concern is in the experimental evaluation part. The experiment is limited to PPL, which is insufficient for verifying long-context ability.
2. The experiments are limited to a very small LLM model - a 124M transformer, which is not enough to verify the proposed method, as large LLMs and small LLMs behave quite differently.
3. The proposed method is not convincing to me. The proposed method is mainly in Equations (2) and (3). In Equation (2), the proposed method does not seem promising. Considering the next step ( $h(*)$ ) in the neural network, it will be $ h(A_{\text{CAPE}}(X)) = h(XW_Q(XW_K)^\top + f(XW_Q(XW_K)^\top, B)) = g(XW_Q(XW_K)^\top, B) $. Could the author please elaborate on this?
4. Similarly, for Equation (3), $ A_{\text{CAPE}}(X) = XW_Q(XW_K)^\top + B + f(XW_Q(XW_K)^\top, B) $, it has a similar problem to Equation (2).
Due to the major weakness in the experiment and the unreasonable proposal of the bias term, I would like to reject this paper.
questions: questions: See Weaknesses
limitations: limitations: See Weaknesses
paper_summary: paper_summary: The paper proposes context-adaptive positional encoding. The paper proposes a 2-layered MLP to non-linearly integrate positional bias information (can be computed using prior methods like Alibi, FIRE, and Kerple) and query-key content-based dot product values representing semantic relations across different heads to dynamically create a contextual position-informed matrix that can additively modulate the self-attention matrix.
strength_weakness: strength_weakness: 1. If I understand correctly, the original relative encoding (from Shaw et al. and then the one in Transformer XL) could also count as contextual. There, the position-based additive values are computed based on the dot product of queries and distance encodings (treated as keys). So, the context can influence the distance-related bias through the query representations. This could be better discussed in the paper. I would also be curious how Transformer XL style relative encoding would perform in the language modeling datasets. 

2. There is some hit in the computation expense. Although it is manageable, the authors don't treat each head independently but use the concatenation of heads with a shared hidden state.
questions: questions: Minor suggestions:

* It Would be good to refer to the appendix implementation section G when discussing function f and the multi-head CAPE in the main paper. 
* Perhaps it is better to present the pseudocode with einops-style syntax.
limitations: limitations: There isn't an explicit section purely for limitations, but there is a section for computational cost analysis, which may illuminate some limitations (and authors treat the section like a limitation section in the checklist). Overall, it's mostly adequate.
paper_summary: paper_summary: Considering the fixed parameters of RoPE may lead to the generalization issue, this paper introduce a dynamic position embedding method named CAPE, where position encoding is depend on the input context. Specifically, CAPE enables testing-time adaptation to input context by using a two-layer LeakyReLU neural network to parameterize the positional bias added to the vanilla attention module. By meticulously selecting the hyper-parameters, CAPE can achieve better performance compared with previous methods.
strength_weakness: strength_weakness: 1. CAPE can introduce extra computation, as well as lower the training and inference efficiency
2. The choice of hyperparameters is significant for CAPE, as it is correlated with the model performance and the efficiency.
3. There is no experiments on LLMs.
questions: questions: 1. Can CAPE adapt to LLMs, for example, Llama3-8B, with few post-training steps? 
2. What's the context length boundary of CAPE? In this paper, experiments just demonstrate the maximum context length with 8192.
3. The authors claim that "CAPE is semantically dependent and adaptive." What does "semantically dependent" mean? If CAPE is used for a summarization task with different context lengths (perhaps 8192 or 4096), what does "semantic" refer to in the summarization task? Providing a few examples could help clarify the writing.
4. About motivation: Even though positional encoding (PE) is fixed during inference, attention still relies on the input context. What if we consider a scenario where PE simply provides the context index to help the language model distinguish the positions of each token (static), while the attention module dynamically selects key information and provides semantic information to assist the language model? Why does CAPE need to incorporate such a dynamic selection mechanism, which is already learned by LLMs, into the PE?
limitations: limitations: See weakness and question part.
paper_summary: paper_summary: This paper proposes a simple learnable positional encoding called CAPE that boosts the length extrapolation performance of Transformer language models.
strength_weakness: strength_weakness: The speed is unbearably slow in my opinion. While I appreciate the honesty of reporting speed differences in Table 1, the authors really need to figure out a way to improve CAPE's training and inference efficiency.
questions: questions: 1. What are the potential ways to improve the speed of CAPE? Please be as concrete as possible.
2. Can the authors test CAPE using needle in a haystack? This way the readers can have a clearer picture of how CAPE is using long context information.
limitations: limitations: None.

Comments: title: Summarizing the Contributions of this Paper
Response to Area Chair a2or (Part 2/2)
Response to Area Chair a2or (Part 1/2)
Reply
Kindly Remind of Discussion Period
Response to Area Chair a2or
Resubmit below
Reply
Response to Area Chair a2or
Reply
Response to Reviewer NzBk
Final Response to Authors
Response to Reviewer NzBk (Part 2/2)
Response to Reviewer NzBk (Part 1/2)
Response to Area Chair a2or and Reviewer NzBk (Part 3/3)
Response to Area Chair a2or and Reviewer NzBk (Part 2/3)
Response to Area Chair a2or and Reviewer NzBk (Part 1/3)
Have a similar question with AC
Re-Response to Authors
Question about learnt additional bias
Response to Reviewer NzBk
Response to Authors
Response to Reviewer CfZF
Response to Reviewer QhvW
Thank you for the rebuttal.
Author Response
Rebuttal Response
comment: Dear ACs and Reviewers,

Thank you all for your time and effort in reviewing this paper. We are grateful for the ACs to arrange the discussion. We are grateful for the positive recognition by CfZF, NzBk and QhvW. Our contributions are well recognized by reviewers CfZF, NzBk and QhvW. Our experiment results are endorsed by reviewers CfZF, NzBk and QhvW, and our paper presentation is favored by reviewers tTna, CfZF, NzBk and QhvW.

Additionally, we outline our paper's main contributions, including the additional conclusions during the rebuttal discussion phase:
* We proposed CAPE, a novel approach to length extrapolation, which utilizes the attention score to dynamically adjust the position encoding.
* We conduct extensive experiments to validate the proposed method CAPE. We conduct experiments with different training lengths (128, 512, and 1024), different datasets(Arxiv, Books, and CHE benchmark), and different model sizes (125M, 350M, 2.7B, and 6.7B) and different evaluation metrics (perplexity metrics and accuracy metrics)
* Our proposed method is successfully trained on length 128 or 512, and then keeps good performance on evaluation length 8192.
* We promise that we will incorporate our discussion part into the paper's final revision, including but not limited to different model size experiments, the analysis of the computation cost, the visualization of CAPE, the discussion of context-boundary, the discussion of RoPE, and so on.

**The CAPE implementation is shown in Appendix G. Within less than 10 minutes of modification, CAPE (such as CAPE-Kerple or CAPE-FIRE) can be easily and directly used for your own transformer experiments (CAPE could improve performance for both within training length and beyond training length). We will release all the code for reproducing the experiment results. We promise that our experiment settings are fair and consistent.**

Again, we thank the ACs and all the reviewers for actively engaging in the rebuttal discussion and for their positive recognition of our work.
**Q4: The performance of Kerple**

A4: This may be caused by the evaluation protocol or training strategy. 

For our evaluation protocol, we select the last 256 tokens in the sequence to evaluation so that the evaluation is more critical and could better reflect the model performance with the current sequence length. 

**According to the Kerple paper, you may find that its reported results achieves relatively better performance than our paper, FIRE and BiPE paper[1]**. For example, the RoPE result in Kerple's paper achieves 5.94 ppl with an evaluation length of 2048 on the Github dataset (training length is 512), while BiPE reports that RoPE achieves 15 ppl with an evaluation length of 2048 on the Github dataset (training length is 1024). This may be caused by different evaluation protocols or training strategies.

* **Moreover, even if we directly use previous works' results, our proposed method (with a more critical evaluation protocol and the baseline is lower) could still achieve better performance.**
   * For example, on the arxiv dataset Kerple's paper reports that it could achieve 5.01 ppl on evaluation length 8192 (with training length 512), and our proposed method achieves 3.70 ppl on evaluation length 8192 (with training length 512).
   * This suggests that even with a more critical evaluation protocol, and proposed method could still achieve good performance.

**Q5: how all results on 128 length are helpful for overall conclusions and claims of the paper**

A5: **It suggests that our proposed method has the potential to extrapolate 64 times, compared to the training length**

We would like to know the context boundary. Because of the limitation of computational resources, we would like to know the largest many times that our proposed method could extrapolate. With a training length of 128, we successfully extrapolate to 8192, suggesting the potential that our proposed method could extrapolate 64 times the training length.

**Q6: The question for prior papers and Figure 5**

A6: We will answer the question in two parts: 1) The question for the prior paper; 2) The Figure 5 baseline is Kerple, and thank you very much for your notice.

**The question for the prior paper**
As we mentioned above, the prior paper may use different evaluation protocols and experiment settings so that their performance may be better. For example, the Alibi uses the sliding-window evaluation protocol. 

For the previous paper that claims they could extrapolate to 16K. **We may pay attention to the FIRE paper. According to FIRE figure 6, we can find that it is hard for the prior paper method to keep performance on evaluation length 8192 (when the training length is 512).**

**For Figure 5 caption**

Thank you very much for your notice. The Figure 5 caption should be CAPE-Kerple but not CAPE-Alibi. We will revise it in the final revision.

**Q7: Question about NoPE**

A7: The training setting may be different. As we said above, the NoPE may need more training tokens and length. For our experiments, we train with 50K with length 512 (following Kerple setting) and 8 GPUs on the Arxiv or Books dataset, while FIRE runs 600K steps with length 2048 and 128 GPUs.

Also, as we said above, the NoPE implicitly learns the position information so that we may sufficiently train the model so that the NoPE can present its ability to outperform other methods. 

**Q8: Not clear why the function will be able to do better generalization in general.**

A8: We further conduct an ablation study on the $f$, proving that **$f$ help enhance the bias matrix.** CAPE improves the bias matrix for $A_{final}$, while the $A_{final}$ is used to calculate $A_{final}K$. **For the unseen position, the $B$ partially could handle it (FIRE changes the problem to interpolation), but is not accurate enough so that CAPE helps enhance the bias matrix $B$ via attention score**
|   Books Dataset (Train Length 512)         | 512   | 1024  | 2048  | 4096  | 8192  |
|------------|-------|-------|-------|-------|-------|
| $QK^T+B$  (baseline) |19.68|19.06|20.44| 28.34|39.31|
| $QK^T+B+f(B)$     | 19.64  | 18.83  | 18.49  | 20.62  | 23.49   |
| $QK^T+B+f(QK^T, B)$     | 19.22| 18.22| 17.15| 17.63| 17.88|
The experiment suggests two points:
* The CAPE $f(QK^T, B)$ is better than naive $f(B)$, suggesting that the context-adaptive is important.
* The $QK^T+B+f(B)$  is better than $QK^T+B$, suggesting that benefiting from improving the expressiveness of the bias matrix.

**We promise that we will release the code. We are sure that the experiment setting is fair and the experiment results are reproducible**

**Thank you very much for your constructive comments. If there is any further question, please let us know.**

Reference

[1] He, Z., Feng, , ... & He, D. Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. ICML, 2024.

[2] Ruoss, A., , ... & Veness, J. (2023). Randomized positional encodings boost length generalization of transformers.
Dear Area Chair a2or,

Thank you very much for your reply. We will answer your question below.

For the question related to experiments: 
* **Our Table 2 CHE results (with 200K training steps) are almost aligned with the baseline[2].**
* Previous work (such as Alibi) uses the sliding-window evaluation protocol, which divides sentences with L tokens and then evaluates. This will make the performance look better.
* Previous works (such as Kerple) also use non-overlapping evaluation, while we choose the last 256 tokens so that the evaluation is more critical. Our Kerple implementation comes from its released code.
   * **Kerple's paper uses non-overlapping evaluation: To train on or evaluate a sequence longer than $L$ tokens (which is usually the training length), it is typical
to segment the sequence into L-length subsequences and train on or evaluate them independently. This will make the performance look better, compared to our evaluation protocol. For details, please refer to Alibi paper**
* **With the different evaluation protocols and training settings, the reported results may be different, such as the Kerple performance difference between Kerple paper and FIRE paper**. In Kerple's paper, its ppl gradually decrease with the evaluation length increases. However, the FIRE paper Figure 6 (train length 512) reports that the Kerple method ppl gradually increases with the evaluation length increases.
* **Moreover, even if we directly use previous works' results, our proposed method (with a more critical evaluation protocol so that the baseline is lower) could still achieve better performance.**
   * For example, on the arxiv dataset Kerple paper reports that it could achieve 5.01 ppl on evaluation length 8192 (with training length 512), and our proposed method achieves 3.70 ppl on evaluation length 8192 (with training length 512).
   * This suggests that even with a more critical evaluation protocol, and proposed method could still achieve good performance.
* All our experiments are fair and consistent. **We directly use the popular GPT-NeoX framework for our experiments (it has Alibi implementation), and we promise that we will release the code. We are sure that the experiment setting is fair and the experiment results are reproducible.**

**Q1: Why then in your comparison in Figures 2, 3, 4 and above Tables FIRE is the worst in the generalization? ? Even worse than RoPE?**

A1: **This may be caused by the different training strategies and settings**. 

For our experiments, we train with 50K with length 512 (following Kerple setting) and 8 GPUs, while FIRE runs 600K steps with length 2048 and 128 GPUs. The FIRE uses MLP to predict the position encoding so it may need more training tokens and other potential training strategies. 

Also, **FIRE performance is actually better than RoPE, with a larger training length**
|   Books Dataset (Train Length 512)         | 512   | 1024  | 2048  | 4096  | 8192  |
|------------|-------|-------|-------|-------|-------|
| RoPE|19.62|170.93|336.76|426.50|559.13|
| FIRE     | 19.67|19.51|34.95| 129.33|278.783|

**Our FIRE implementation**: we directly utilize the FIRE implementation code provided in FIRE's paper.

**Q2: Why did you decide to select 128 context lengths for training?**

A2: We do present the results of training length 512 (Figire 2) and 1024 (Figure 3) in our paper, which follows the setting of Alibi paper and Kerple paper.

**Why choose 128?**
The reason is that we focus on length extrapolation so we would like to know based on the training length $T_{train}$, how long our proposed method could still achieve good performance on evaluation length $T_{eval}$. Basically, there are two ways:
* Train on the longer length and then evaluate on the longer length. However, this will take a lot of computing cost, as we know that the transformer cost is $N^2$, while the N is the sequence length.
* The second way is that we train on a relatively short length (such as 128) and evaluate on a relatively longer length (such as 8192). This could reduce the computational cost.
* Based on our hardware, we additional train on 128.

**Q3: Could you explain the choice of Arxiv and Books3 data as e.g. prior works used WikiText, C4, OpenWebText2, GitHub?**

A3: The Arxiv dataset and Books dataset are also commonly used in previous works, and **their Mean Document Size is relatively bigger.**
* The mean document size: Wikipedia (en) (**1.11 KiB**), OpenWebText2 (**3.85 KiB**), GitHub (**5.25 KiB**)
* The Arxiv dataset is commonly used in previous works, such as Kerple, FIRE and BiPE. The Mean Document Size of Arxiv is **46.61 KiB.**
* The Books dataset is also commonly used in previous works, such as LongRoPE. Also, The Mean Document Size of Books is **538.36 KiB**, which is almost the largest one in Pile dataset.
* As we focus on length extrapolation, we prefer to validate our proposed method on the long-context situations so that Arxiv and Books3 are preferred.
Dear Authors,

Thanks again for involving in the discussion.

I have several more questions regarding overall numbers you shared:
- FIRE proposed a variant of mapping positions into (0, 1) so the problem then becomes interpolation rather than extrapolation and they show in their paper that the method extrapolates to unseen positions better than others. Why then in your comparison in Figures 2, 3, 4 and above Tables FIRE is the worst in the generalization? Even worse than RoPE? This on its own seems to be strange.
- Why did you decide to select 128 context lengths for training? It seems unreasonable, and thus results on generalization you present may not be aligned with prior results as e.g. in FIRE 2048 context length is used, and AliBi e.g. 512 and 1024.
- Could you explain the choice of Arxiv and Books3 data as e.g. prior works used WikiText, C4, OpenWebText2, GitHub?
- In KERPLE paper authors show that being trained on 512 context they are able to extrapolate with even lowering perplexity on Arxiv till 16k context (see in their paper Table 3 and 5), while in your results above KERPLE degrade in performance very quickly, e.g. it doesn't generalize already at 2048 length from Fig 2, while in KERPLE paper Table 3 it shows improvements over increasing the context length at test time?
- Looking at Fig. 3 it seems that extrapolation depends a lot on the training context size and larger size used, better we extrapolate to longer sequences (maybe not surprising), but then it is not clear how all results on 128 length are helpful for overall conclusions and claims of the paper.
- Figures 2, 3 and 4 demonstrate that all prior papers which claimed extrapolation at least till 16k do not hold and only AliBi is able to generalize to that length. This I found contradicting all prior works tbh. But then in Figure 5 you are showing that actually AliBi also does not extrapolate, as baseline red there is AliBi and at 2048 length it degrades significantly.
- For Nopos: I agree it is not uniform, but what I meant - it is kept uniform for some time and then degrades, outperforming others in the generalization. Your results with Nopos are questionable, as the perplexity degrades super quickly to the wildly off values. This is inconsistent with finding in e.g. FIRE, where Nopos is also reported in Figure 1: while performance degrades, it slowly becomes worse, moreover Nopos outperforms in generalization RoPE which is not the case in your Table above. Furthermore, in the paper "The impact of positional encoding on length generalization in transformers." it was shown that while Nopos degrades it outperforms others in the generalization, which is entirely opposite in your observations.
- I still don't understand how CAPE could generalize to unseen positions as $B$ which is input to $f$ is not seen during training, thus again we are in the regime of extrapolation / generalization, and not clear why function $f$ will be able to do better generalization in general. 

Having all above inconsistency with prior works reported results, this questions the validity of all results. Please, clarify your empirical findings which diverge from prior works.

Thanks,

AC
Dear Reviewer tTna,

We would like to thank you again for your detailed reviews. We have updated the experiment results and the explanation of our method in the above response.

As the discussion period will be closed soon, we would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions. We would be happy to do any follow-up discussion or address any additional comments.

Again, thank you very much for your attention to our work.
Dear Area Chair a2or,

Thank you very much for your reply. We will answer your question below. 

**Q1: The attention pattern of Kerple and CAPE-Kerple**

A1: To purely analyze the function $f$, we further analyze the visualization with  $QK^T+f(QK^T,B)$ (The Equation 2 for CAPE in our paper) so that the total bias matrix is $f(QK^T,B)$, without being affected by the residual $B$.

**For CAPE-Kerple bias with CAPE   formulation $QK^T+f(QK^T,B)$**.

According to the visualization, the total bias matrix $f(QK^T,B)$ presents clearly anti-local attention pattern. For example, the layer 1's head 8 and head 2 present clearly anti-local attention pattern, which is relatively similar to the anti-local pattern shown in the Appendix F layer 1's $f(QK^T,B)$ in formulation $QK^T+B+f(QK^T,B)$. Hence, we could have confidence to say that the proposed CAPE bias could present anti-local attention patterns. The function $f$ not only changes the slope of decay but also provides the anti-local attention pattern if needed.

**The final attention pattern comparison between Kerple and CAPE-Kerple**

We also further visualize the Kerple final attention(which is $QK^T+B$) and CAPE-Kerple final attention (which is $QK^T+f(QK^T,B)$) . As the CAPE total bias matrix $f(QK^T,B)$ could clearly provide the anti-local attention pattern, the CAPE-Kerple final attention ($QK^T+f(QK^T,B)$)  successfully clearly presents the anti-local attention pattern on some attention heads so that Kerple final attention pattern and CAPE final attention pattern are different.

**We promise that we will provide the visualization of Kerple and CAPE-Kerple ($QK^T+f(QK^T,B)$) in the final revision**





**Q2: Why you don't have perfect generalization to the longer sequences? Any explanation/intuition?**

A2: **The previous work already proves that the NoPE performance may not be uniform and independent from the sequence length, such as NoPE Paper[1] and FIRE [2].**

* According to the NoPE paper[1], the performance of NoPE is not uniform and independent of the sequence length. For example, for the NoPE Paper [1] Figure 3, the performance of NoPE will be worse if the evaluation length is larger than the training length. The NoPE paper claims that there is potential for NoPE to have better extrapolation performance than other position encodings, but it does not claim that the NoPE performance is uniform and independent from the sequence length.
* The FIRE paper also compares the performance of NoPE in FIRE paper's Figure 1. According to the figure, we can find that the NoPE performance becomes worse (perplexity also increases) when the evaluation length is larger than the training length.
* Our personal opinion.
   * As claimed by NoPE Paper [1], the NoPE implicitly learns the position encoding information, which may not be easy to learn and is inefficient.
   * Therefore, we may need to sufficiently train the model if we use NoPE.
* **Therefore, according to the previous works (such as [1][2]), the NoPE performance may not be uniform and independent of the sequence length.**


**Thank you very much for your constructive comments. If there is any further question, please let us know.**


Reference:

[1] Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., & Reddy, S. (2024). The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36.

[2] Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., ... & Bhojanapalli, S. Functional Interpolation for Relative Positions improves Long Context Transformers. In The Twelfth International Conference on Learning Representations.
After the reply submission, it seems that the openreview system does not send email and notification. Therefore, we resubmit the reply below
Dear authors,

Thanks for additional experiments and clarifications on those I missed from some Tables.

Just to clarify, I am trying to understand if we see something entirely new compared to all prior positional embeddings with the new idea of learning the positional embedding and inductive bias you brought.

Regarding the total attention: if I do in mind sum over all 3 plots, this will lead still to the kind of KERPLE style final attention. What happens is adjusting the proper decay of positions, showing that maybe log heuristic is not ideal, and something better is needed. The latter is what CAPE is doing exactly. Also I would then disagree that CAPE learns anti-local attn: on its own it does it, but then the total bias is $B + f(...)$ and there is no anti-local attn for it (I do (2) + (3)). In my view right now, it is a bit overstated in the paper, as CAPE is about finding better functions of position decay compared to KERPLE e.g. (which is totally fine, but there is no new principled pattern in CAPE in general). And then yep, it is very dependent on the base method, as $f$ is trying to correct a bit the slope of decay, but cannot change it drastically for the unseen positions.

Regarding NoPos - I don't understand why you don't have perfect generalization to the longer sequences - I expect here performance being worse overall than other positional embeddings e.g. for the length same as training, but then it should be uniform and independent from the sequence length as you don't have issues on the position extrapolation at all. Your numbers are not uniform, but they are even significantly worse for generalization to longer than training sequences, which is strange to me. Any explanation / intuition?

AC.
Dear Area Chair a2or,

Thank you very much for your reply. We will answer your questions below.

**Q1: Wanna to exclude inductive bias on $B$ here**

A1: We present the CAPE $QK^T+f(QK^T, B)$ in the following tables (Figure 5).

|   Arxiv Dataset (Train Length 128)         | 128       | 256       | 512       | 1024       | 2048       | 4096       | 8192       |
|------------|---------|---------|---------|---------|---------|---------|---------|
| $QK^T+B$ (baseline)          | 8.28  | 7.09  | 6.29  | 9.93  | 16.97 | 26.14 | 31.91 | 
| $QK^T+B+f(QK^T+ B)$       | 8.21 | 6.99 | 5.41 | 5.27 | 5.46 | 5.40 | 5.17 |
| $QK^T+f(QK^T, B)$ (**w/o inductive bias $B$**)         | 8.18 | 6.95 | 5.35 | 5.18 | 5.34 | 5.25 | **5.03** |
| $QK^T+B+f(QK^T, B)$       | 8.20 | 6.98 | 5.39 | 5.21 | 5.36 | 5.25 | **5.00** |


|   Arxiv Dataset (Train Length 512)         | 512   | 1024  | 2048  | 4096  | 8192  |
|------------|-------|-------|-------|-------|-------|
| $QK^T+B$  (baseline) |4.56  | 4.38  | 5.58  | 8.13  | 10.91  |
| $QK^T+B+f(QK^T+ B)$  | 4.50  | 4.21  | 4.19  | 3.98  | 3.75   |
| $QK^T+f(QK^T, B)$ (**w/o inductive bias $B$**)      | 4.47  | 4.18  | 4.14  | 3.94  | **3.69**   |
| $QK^T+B+f(QK^T, B)$     | 4.50  | 4.20  | 4.18  | 3.95  | **3.70**   |

According to the question, **the CAPE variant $QK^T+f(QK^T, B)$ may be what we need**. If we need to try other variants, please let us know.


**Q2: Looking at the Appendix F if I correctly interpret the leftmost plot - this is the final attention.**

A2: The leftmost plot is $QK^T$. We have described the figure Attention, Figure Kerple Bias and CAPE bias in Figure 1. We directly copy it here.
* (1) The Attention (the leftmost plot) is $QK^T$, which is the naive attention implementation
* (2) The Kerple bias (the middle plot) is $B$; 
* (3) The CAPE (with Kerple) bias (rightmost plot) is $f(QK^T,B)$.
* **Therefore, the final attention is (1)+(2)+(3). We promise that we will provide the final attention visualization in the final revision.**

**Q3: Baseline of NoPE**

A3: We have run the baseline of NoPE, as shown in Figure 1. For convenience, we directly copy our results here.
|   Arxiv Dataset (Train Length 128)         | 128      | 256      | 512        | 1024        | 2048        | 4096        | 8192        |
|------------|--------|--------|----------|----------|----------|----------|----------|
| Nope       | 8.42   | **8.64**   | 122.04   | 549.48   | 880.23   | 1264.52  | 1366.45  |
| RoPE       | 8.29   | **8.55**   | 22.79    | 69.68    | 142.95   | 249.74   | 274.15   |
| T5's bias  | 8.29   | 7.11   | 5.84     | 6.42     | 8.70     | 15.54    | 33.30    |

|    Arxiv Dataset (Train Length 512)        | 512        | 1024        | 2048        | 4096        | 8192        |
|------------|----------|----------|----------|----------|----------|
| Nope       | 4.68   | **31.79**  | 1867.46  | 4666.60  | 5334.85  |
| Rope       | 4.57   | **43.62**  | 144.05   | 278.87   | 297.06   |
| T5's bias  | 4.54   | 4.51   | 13.87    | 121.45   | 338.06   |

**With sufficient training, the NoPE may achieve relatively good performance**
   * When the training length is 128, the NoPE is worse than RoPE, from the evaluation length 128 to 8192
   * When the training length is 512, the NoPE could get better performance than RoPE on evaluation length 1024.
   * Therefore, NoPE actually could potentially have acceptable length extrapolation performance, while it needs sufficient training.

**For the NoPE attention visualization**

According to your suggestion, we also visualize NoPE. 

**On Arxiv Dataset**
* With evaluation length 512:
   * **For the early layer (layer 1 to 2)**: its attention distribution is similar to some leftmost plots, where all heads look in average per position. This may because early layer does not receive sufficient implicit position information.
   * **For the middle layer (layer 3 to 7)**
      *  Most attention heads present long-term decay. This decay may follow either a linear relationship with distance or a logarithmic pattern.
   * **For the latter layer (layer 8 to 12)**
      * The attention distribution gradually becomes relatively similar to the early layer.

* With the increase in the evaluation length, such long-term decay becomes more obvious.


**On Books3 Dataset**
* The Books3's attention distribution in different layers is relatively similar to the Arxiv dataset's head pattern.
* Also, we observed the non-local attention head pattern, such as layer 2's head 9. For example, with the distance increase, the attention score of layer 2's head 9 does not decrease. 
* This may explain why additive RPE and CAPE are better: explicitly help provide the needed local and anti-local attention pattern.
 
Finally, **we promise that we will provide the NoPE attention visualization and discussion in the final revision.**

**Thank you very much for your constructive comments. If there is any further question, please let us know.**
Dear Authors,

Thanks for clarifications and pointing out to the plots I missed in the Appendix.

Some further thoughts:
- Figure 5 actually has different variants of the residual but it doesn't have version where you don't have explicit residual on $B$. By any chance did you run configuration where there is only $f$ w/o residual on $B$? Wanna to exclude inductive bias on $B$ here.
- Looking at the Appendix F if I correctly interpret the leftmost plot - this is the final attention. So, from the rightmost plots it seems that some heads learn $f$ to be uniform across position (so attend to all despite the position index) and some learn kind of windowing (with log decay, or some other dependency) similar to KERPLE. But interesting that resulting attention for all heads looks in average per position (if I do smoothing of the left column plots across x-axis) as uniform. Having this, I wonder if you run standard baseline on the noPos which was shown and proved before can learn positional embedding in decoder only models because of the causal masking.

AC.
Dear Reviewer NzBk,

Thank you very much for your reply, for your encouragement and support. **We promise that our final revision will include some modifications during the discussion phase**, including but not limited to the experiment of large model size, the context length boundary of CAPE, the modification of "semantically dependent", the discussion of RoPE and so on.

Again, thank you very much for your attention and support to this work, and wish you a good day.
Dear Authors

Glad to hear the authors' constructive responses, both the general and individual responses.
 
The authors conduct numerous experiments during the discussion period and offer many insightful opinions and explanations about the work.

I hope this paper can be accepted by NIPs, as it may contribute to the LLM community.

BTW, I hope the authors make sure the final revision includes some modifications during the discussion phase, as it can better help readers understand the work and drive progress in the field of long context models.

I have no further questions and will raise my score from 6 to 8.

Best, 

Reviewer NzBK
**Q3: From the results in Figure 2, it seems that the mainstream RoPE is not as effective as Additive RPE. What do you think about this phenomenon?**

A3: The following is our personal opinion. The mainstream method is RoPE because of three reasons: 1) The timing of RoPE and additive RPE ; 2) Within training length, RoPE has comparable performance with additive RPE; 3) The development of LLaMA.

**The timing of RoPE and additive RPE**
The RoPE paper is on arxiv from 20 April 2021, while there was no Alibi (27 Aug, 2021), Kerple (20 May, 2022), or FIRE (3 Oct, 2023).

**Within training length, RoPE has comparable performance with additive RPE.**
As shown in Figure 2 and Appendix 5, for the performance within training length, the proposed RoPE achieves close or similar performance with Kerple and FIRE. For example, with a training length 512, RoPE achieves 4.5755 ppl, Kerple achieves 4.5817 and FIRE achieves  4.5741. Therefore, for the performance within training length, the different is not very large. The additive RPE presents its superiority for length extrapolation, and the length extrapolation problem has become important recently. 

Therefore, as previously we mainly considered performance within training length, the RoPE is enough.

**The development of LLaMA**
* LLaMA uses RoPE. Therefore, if anyone is working on open-source LLM, then they will use LLaMA so that they will use RoPE.
* Therefore, we could find that the YaRN, CLEX, ChunkLLaMA or other works that focus on length extrapolation or long-context all focus on RoPE.
* Also, currently data is relatively important so the architecture from LLaMA to LLaMA 3 is not changed a lot. Hence, the current mainstream position encoding method is RoPE.

**Q4: Should the mainstream RPE now use additive RPE rather than cumulative RPE like RoPE?**

A4: It is an interesting question, and we are also curious about it. Both additive PE and RoPE-based PE try to incorporate the key-query similarity with the positional information of tokens. However, they adopt different operations, using addition and multiplication operations respectively. In this paper, we developed an additive context-adaptive PE. It is hard to say which kind of PE (additive or RoPE-based) will be the mainstream finally. [RoPE is widely recognized and used in Llamma models.]

Based on the current evidence and the experiment results, additive RPE may be a better choice.
* The FIRE has prove the it could achieve better performance than RoPE, whatever within training length or beyond training length.
* CAPE further helps achieve better performance within training. As proved in our paper (Figure 2 and Table 5), the CAPE could help additive RPE achieves better performance within training length.
* CAPE further helps achieve better length extrapolation performance. Also, as proved in our paper, the CAPE could help additive RPE achieve better performance beyond training length.
* Therefore, based on the current evidence and experiment results, the additive RPE may be better.

**Thank you very much for your constructive comments. If there is any further question, please let us know.**
Dear Reviewer NzBk,

Thank you very much for your notice in Line 233, we have revised it for the final revision.

And thank you very much for your question. We will answer your question below.

**Q1: Can you explain why the additive RPE is better than the RoPE-based model here intuitively**

A1: The additive RPE has two attributes: 1) could have explicitly long-term decay (local position pattern); 2) could have the anti-local pattern. We will explain it below step by step.

**Explicitly long-term decay (local position pattern)**
* The RoPE paper claims that long-term decay is important for long context, and the RoPE achieves it via implicitly long-term decay.  
* As the additive RPE has the formulation $A(x)=XW_Q(XW_K)^T+B$, we could implement the explicitly long-term decay (local position pattern) pattern via a bias matrix that B has a negative value.
   * For example,  $B(i,j) = -r_1\log(1+r_2|i-j|)$ (logarithmic variant) , where $r_1, r_2>0$ are learnable scalars.

**The anti-local pattern: emphasize far away keys more**
* For long-context, we cannot abandon long-distance information, otherwise it will become local attention. Therefore, the anti-local position pattern is important so that the model could pay attention to long-distance information.
* The RoPE has long-term decay so that with distance increase, the long-distance information weight will be smaller. **Therefore, RoPE does not have anti-local position pattern.**
* The additive RPE FIRE successfully achieves anti-local pattern, as shown in its paper Figure 4.
* Our CAPE-Kerple also successfully achieves anti-local position pattern, as shown in Figure 1 and Appendix F. For example, with the distance increases, the bias value is non-decreasing.

Therefore, as the additive RPE could have both local and anti-local position patterns, it may be better.

**Q2: Are all the experiments in Figure 2 fair and consistent? Including model size, training data, etc.**

A2: **We promise that the experiments are fair and consistent**. We follow the training protocol from Kerple. **And we only change the position encoding method for different experiments.** The model size of Figure 2 is 125M, and the model size of Figure 4 is 350M. The experiment setting is shown in Appendix B. And we directly copy it here.

|                             |             | **125M**    |             | **350M**   |
|-----------------------------|-------------|-------------|-------------|------------|
| Training sequence length    |             | 512         |             | 512        |
| Batch size                  |             | 32 × 8      |             | 32 × 8     |
| Number of iterations        |             | 50k         |             | 50k        |
| Dropout prob.               |             | 0.0         |             | 0.0        |
| Attention dropout prob.     |             | 0.0         |             | 0.0        |
| Attention head              |             | 12          |             | 16         |
| Feature dimension           |             | 768         |             | 1024       |
| Layer number                |             | 12          |             | 24         |
| Optimizer                   |             | Adam        |             | Adam       |
| Optimizer parameter betas   |             | [0.9, 0.95] |             | [0.9, 0.95]|
| Learning rate               |             | 6e-4        |             | 3e-4       |
| Precision                   |             | float16     |             | float16    |
**Q6: Why do you need still to have $B+f(QK^T,B)$ and why not just $f(QK^T,B)$**

A6: We thank for pointing out this. We have discussed performance of $B+f(QK^T,B)$ and $f(QK^T,B)$ in Section 4.3 Different Variants of CAPE Figure 5.

In equation (2) at the methodology part of the paper, we introduce the context-adaptive PE (denoted as f(*)) as a surrogate to the previous fixed PE. The original formulation of CAPE is attention score plus the context-adaptive PE f(*). As the residual connection is a popular technique in deep learning, we further introduce a variant of CAPE with the residual connection as shown in equation (3). Experimental results (shown in Section 4.3, Figure 5) on both two formulations of CAPE indicate that they have very minor performance differences.

Specifically, at a training length of 128, $B+f(QK^T,B)$ records a score of 5.00 and  $f(QK^T,B)$
scores 5.03 at an evaluation length of 8192.
With a training length of 512, $B+f(QK^T,B)$ achieves a score of 3.70, and $f(QK^T,B)$ scores 3.69 at
 an evaluation length of 8192. We use $B+f(QK^T,B)$ just to hope that it will be easier for $B+f(QK^T,B)$ to degrade to $B$ if needed, to be sure that CAPE is better than baselines.

**Thank you very much for your constructive comments. If there is any further question, please let us know.**
**Q3: Could you clarify why for CAPE in contrast to all other baselines you observe improvement with the longer context, while other degrade always?**

A3: The CAPE performance is highly related to the baseline method, such as Kerple or FIRE. **Though CAPE could usually improve additive RPE method performance, CAPE may achieve worse performance if the baseline additive RPE method collapses.**

The following are the results of FIRE and CAPE-FIRE on the Books3 dataset. With a training length of 512, CAPE can consistently improve performance.

| Method (train 512)             | 512 | 1024 | 2048 | 4096 | 8192  |
|------------------------|-------------|-------------|-------------|--------------|--------------|
|RoPE|19.62|170.93|336.76|426.50|559.13|
| Kerple             | 19.68       | 19.07       | 20.44       | 28.34        | 39.32        |
| FIRE               | 19.67       | 19.52       | 34.96       | 129.33       | 278.78       |
| CAPE-Kerple (Ours) | 19.23       | 18.23       | 17.15       | 17.63        | 17.88        |
| CAPE-FIRE (Ours)   | 19.14       | 18.39       | 18.22       | 21.45        | 28.86        |

And if we change the training length from 512 to 128, we will have the following results.
| Method (Train 128)           | 128     | 256     | 512     | 1024     | 2048      | 4086       |8192       |
|-------------------|-------|-------|-------|--------|--------|--------|---------|
|RoPE|32.08 | 29.80 | 33.12 | 102.51 | 234.86 | 383.83 | 513.31 |
| Kerple            | 31.96 | 29.03 | 29.71 | 42.75  | 56.25  | 73.59  | 87.03   |
| CAPE-Kerple (Ours)| 31.45 | 28.25 | 24.93 | 24.34  | 23.30  | 24.33  | 24.93   |
| **FIRE**              | **31.91** | **29.06** | **32.61** | **110.29** | **321.53** | **534.54** | **802.76**  |
| **CAPE-FIRE (Ours)**  | **31.42** | **28.40** | **26.81** | **34.03**  | **80.98**  | **333.44** | **1107.74** |

According to the experiments, we can see that the CAPE-FIRE performance could be even worse when the FIRE performance almost collapsed (which is 802.76 ppl at evaluation 8192). 

Therefore, the CAPE performance is highly related to the baseline additive RPE performance. CAPE could improve performance when the baseline method could achieve acceptable performance under the current situation, while the performance may degrade if the baseline method collapses under the current situation.

**Q4: Could it be that with even longer context the trend will continue?**

A4: As we discussed above, the performance of CAPE is highly related to the baseline method performance, such as Kerple and FIRE. For Kerple, our method still works well when the training length 128 and the evaluation length is 16384.
| Method        | 128   | 256   | 512   | 1024  | 2048  | 4096  | 8192  | 16384 |
|---------------|-------|-------|-------|-------|-------|-------|-------|-------|
| Kerple        | 31.96 | 29.02 | 29.70 | 42.74 | 56.24 | 73.59 | 87.03 | 93.38 |
| CAPE-Kerple   | 31.44 | 28.25 | 24.93 | 24.33 | 23.29 | 24.32 | 24.93 | 25.33 |


**Q5: Any intuition from the learned representation why this happens? Maybe compare Fig 1 for position 2k, 4k, 8k to see the trend of change in how the bias is learnt?**

A5: We have presented more examples of the Figure 1 visualization in Appendix F, with length 512, 2048 and 8192. And the training length 512. **Intuitively, our CAPE works better because it has both local and anti-local position patterns, which is achieved by different heads.**
* **The CAPE could really adjust the bias matrix value for different samples**. In Appendix F, for each validation length (512, 2048 or 8196), we present the visualization of two different samples. Let us first focus on the evaluation length 512.
   * For sample 1 with evaluation length 512, its layer 2's $4^th$ head output bias value is mostly smaller than 5.
   * For sample 2 with evaluation length 512, its layer 2's $4^th$ head output bias value could be larger than 5.
   * This suggests that the CAPE can really adjust the bias matrix for different samples.
* **Different heads have different functions (trends), whatever the evaluation length is 512, 2048, or 8192 .**
   * Whatever the evaluation length 512, 2048 or 8192, it seems that the different layers' different heads have their own functions after training, which may help improve the model performance.
   * For the original attention score, it seems that different heads may not have different functions. This may be why NoPE does not work very well.
   * For the Kerple bias, all heads pay attention to local information.
   * For our CAPE, the different heads have different functions
      * The layer 1's head 8 is used to pay attention to long-distance information.
      *  The layer12's head 7 is used for the local information.
Dear Area Chair a2or and Reviewer NzBk,

Thank you very much for your question. We will address them below.

**Q1: Nutshell window attention**

A1: We have discussed the question in our paper Section 4.1 Comparisons with Baselines and Table 2 (you can find the Table 2 in the rebuttal to Reviewer tTna). To prove that our model could sufficiently use the entire sentence information, we conducted the experiments in Table 2.

**CAPE enhances intra-length performance, indicating that its lower perplexity may come
from thorough utilization of entire sentences but not disregarding long-distance information
(Also proved in Figure 1).** Compared to Alibi, Kerple, and FIRE, the adapted versions CAPE-Alibi, CAPE-Kerple, and CAPE-FIRE demonstrate better intra-length
performance. With the growing sequence length, the Alibi tends to transition from full attention
to almost local attention, and this is why Alibi is worse than most baselines within training length
but better beyond training lengths. The results (in Table 5) indicate that the superior intra-length performance of CAPE is statistically significant, with a p-value less than 0.05. Therefore, the
consistent intra-length performances across various training lengths indicate that the lower perplexity
of CAPE results from effectively utilizing the entire sequence, rather than focusing on local parts and
neglecting long-distance information.

**Experiment on tasks that need the entire sequence information (as shown in Table 2)**. As the some methods could cheat perplexity (such as Alibi), we further conduct experiments on benchmarks that need to sufficiently utilize the whole sentence. The results are shown in Table 2 Page 9.
* **CAPE works better on permutation-variant tasks.** CAPE (with Kerple and FIRE) presented the
best performance in 10 out of 11 permutation-variant tasks (which require positional information),
achieving the second-best performance in the SOLVE EQUATION task. This underscores the efficacy
of CAPE with semantic adaptivity in handling permutation-variant challenges.
* **CAPE’s performance on permutation-invariant tasks.** In tasks that are permutation-invariant,
where positional information is non-critical, CAPE demonstrated comparable performance. Notably,
CAPE-Alibi achieved scores of 50.30 on PARITY CHECK and 99.38 on BUCKET SORT tasks, compared to the highest scores of 50.97 and 99.57, respectively, demonstrating competitive performances.


**Q2: what is the intuition how the model solves the generalization problem of not seen positions?**

A2: **Recent works prove that the transformer can be generalized to unseen potions**, such as Kerple and FIRE. CAPE helps to enhance such ability via dynamically adjusting position encodings for different input sequences, and its performance is actually highly related to the baseline method.
| Method        | 128   | 256   | 512   | 1024  | 2048  | 4096  | 8192  | 16384 |
|---------------|-------|-------|-------|-------|-------|-------|-------|-------|
| Kerple        | 31.96 | 29.02 | 29.70 | 42.74 | 56.24 | 73.59 | 87.03 | 93.38 |
| CAPE-Kerple   | 31.44 | 28.25 | 24.93 | 24.33 | 23.29 | 24.32 | 24.93 | 25.33 |

**The CAPE-Kerple achieves 24.33 ppl on evaluation length 1024 and 25.33 on evaluation length 16384. Therefore, the performance of CAPE-Kerple will also degrade as the performance of Kerple becomes worse.**

* As the Area Chair a2or mentioned above, the Kerple uses the log to reduce the distribution gap between seen position id and unseen position id.
* The recent work FIRE even uses the formulation $B(i,j)=f(\frac{i-j}{\max(\{L, i\})})$ to reduce the gap between seen and unseen position id. Whatever how long the context is, the $\frac{i-j}{\max(\{L, i\})} \in [0,1]$ so that the gap between training and validation becomes even smaller.
* There are also other papers that suggest transformers can generalize to unseen positions, such as "Transformers Can Achieve Length Generalization But Not Robustly".
* Therefore, based on the previous work, the bias matrix $B$ already could partly solve the unseen position id problem.
* Hence, the one potential problem becomes: how can we further better utilize the bias matrix $B$ to get better position encodings?
   * One potential solution: we try to design a better position encoding function $N^{*2}\to R$ to get a better bias matrix. There are related works, including Alibi, Kerple and FIRE.
   * Another potential solution is: all we need is a better attention score and position encodings so why not dynamically adjust the bias matrix $B$ to get better position encoding? Following the direction, we developed the CAPE and it actually works well, without additional operations.
* Intuitively, our CAPE may help the additive RPE to be learnable and optimized, as the $B+f(QK^T,B)$ could degrade to $B$ if $f(QK^T,B)$ is not necessary. Also, our CAPE is context-adaptive position encoding so that CAPE could dynamically adjust position encoding for different sequences.
Dear Authors, 

Similar to question#2 raised by AC, I also have this question (see Question #1 and #3 posed in Re-Response to Authors by NzBk).

The authors can combine these questions to respond, thx. 

Reviewer NzBk
Ok, I appreciate the authors' professional response. 

In your paper, on line 233, the authors mention, "As shown in Figure 4 and Table 5, CAPE consistently outperforms established baselines such as RoPE" (I know you mention Figure 2, please correct this in the final revision). It is surprising for me to know that Additive Position Embedding works better than RoPE-based methods.

I also read the general response by the authors, the ``Q2: Result on Large Model Size 2.7B and 6.7B (Reviewer tTna, Reviewer NzBk)'' also surprised me, as it indicates that additive RPE can achieve much better results than RoPE under the context scaling settings.


1. Can you explain why the additive RPE is better than the RoPE-based model here intuitively?

2. Are all the experiments in Figure 2 fair and consistent? Including model size, training data, etc.

3. From the results in Figure 2, it seems that the mainstream RoPE is not as effective as Additive RPE. What do you think about this phenomenon? Should the mainstream RPE now use additive RPE rather than cumulative RPE like RoPE?

I hope the authors answer the above questions, which are very important for my judgment of your work.
Dear Authors,

Thanks for all your responses and extra experiments. I have additional question related to your discussion with reviewers.

It was shown that ALiBi is in the nutshell window attention (https://arxiv.org/pdf/2212.10356). Also there was a discussion in the KERPLE about similar behaviour as well log version of KERPLE is not window attention and far away positions are actually used and the longer context improves overall model performance.

Now, regarding your method: 
- As in Fig.1 you showed that the learned bias is both local and anti-local - what is the intuition how the model solves the generalization problem of not seen positions? As anyway you will have $B$ values which are not seen during training, so we could expect that $f$ will generalize well to the test-time $B$. 
- Could you clarify why for CAPE in contrast to all other baselines you observe improvement with the longer context, while other degrade always? This seems to be a new phenomenon - so you are showing that you are not only generalizing but also the longer context (maybe) is used effectively. Could it be that with even longer context the trend will continue? Any intuition from the learned representation why this happens? Maybe compare Fig 1 for position 2k, 4k, 8k to see the trend of change in how the bias is learnt?
- Why do you need still to have $B + f(QK^T, B)$? Why not just $f(QK^T, B)$?

AC.
Dear Reviewer NzBk,

Thank you very much for your reply. We will answer your questions below.

**Q1: The RoPE and its B(x)**

A1: **We claim "B(x) is set as a constant for all input sequence" under the discussion of additive RPE**. Usually, the RoPE is not considered as an additive relative position encoding, as discussed in FIRE [3] paper Section 2.2. 

We will answer the question about RoPE in two parts: **1) The definition of Additive RPE**; **2) General view of the $B(x)$ and RoPE**

**The definition of Additive RPE, which should have formulation $A(x) = XW_{Q}(XW_{K})^{T} + B$ and B is induced by position encoding function**

According to FIRE paper [3], the additive relative position encoding can be represented by the formulation $A(x) = XW_{Q}(XW_{K})^{T} + B$, while the B $\in R^{N \times N}$ is induced by the **position encoding function** $N^{*2} \to R$. To be more specific, $B=g(D)$, where $g(.)$ is a position encoding function and $D$ is the distance matrix. Usually, the $D$ is the following:

```python 
D= [0 0 0]
   [1 0 0]
   [2 1 0]
```
Therefore, when we are discussing the $A(x) = XW_{Q}(XW_{K})^{T} + B$, usually the $B$ should fulfilled the mentioned requirements. According to FIRE paper [3], the RoPE is not an additive relative position encoding so the RoPE actually does not have such $B$. Therefore, under the original definition, RoPE actually does not have either $B$ or $B(x)$. And for our $B(x)$, we mainly focus how to utilize the additive RPE matrix $B$ and attention score, so that there should be an additive relative position encoding bias matrix $B$.

Therefore, considering the previous definition, we will have the following:
* **The naive query and key implementation:** $XW_{Q}(XW_{K})^{T}$
* **The part that utilizes both bias matrix B and $XW_{Q}(XW_{K})^{T}$:** $B(x)$
* **The part is that fixed after training**: $B$
* For the previous naive additive position encoding method: $B(x)=B$
* For our CAPE: $B(x)=B+f(XW_{Q}(XW_{K})^{T}, B)$.

**General view of the $B(x)$ and RoPE**

If we do not consider the definition from FIRE[3] and make the definition of $B$ becomes more general, then we may have the following.
* **The naive query and key implementation:** $XW_{Q}(XW_{K})^{T}$
* **The part that removes naive query and key implementation:** $B(x)$
* **The part is that fixed after training for additive RPE**: $B$
* For the previous naive additive position encoding method: $B(x)=B$
* For our CAPE: $B(x)=B+f(XW_{Q}(XW_{K})^{T}, B)$.
* For RoPE: $B(x)=Qf(K)^{T} + f(Q)K^{T} + f(Q)f(K)^{T}$. From the general additive RPE perspective, this may be why RoPE is better than its baseline Sinusoidal encodings.

**Q2: Why CAPE is better than other RPE methods e.g.**

A2: Compared to other RPEs e.g. YaRN (YaRN is also a great work), we further improve the performance of the current great method Kerple and FIRE[3] via dynamic position encoding to get better performance. Also, YaRN improves performance via sequence-length-related adjustment.

* **Our baseline is powerful**. We may check the FIRE paper[3] Figure 1 (on page 2). Figure 1 of the FIRE paper compares the performance between Kerple, and FIRE and other RPEs (including YaRN). The results prove that Kerple and FIRE achieve relatively good performance.
* **CAPE solves the baseline Kerple and FIRE's limitation, whose position encoding is fixed after training.** As we claim in our paper, though additive RPE achieves good performance, its position encoding is fixed after training. Hence, we propose to further improve the additive relative position encoding performance by dynamically adjusting the position encoding via attention score.
* Therefore, our method could achieve relatively better performance than other methods.

**Q3: The definition of "better extrapolation than other RPE methods"**.

A3: The better extrapolation means: with the same experiment setting (training length, training tokens, and so on), our proposed method could achieve better performance on evaluation length $T_{eval}$, while $T_{eval}$ is larger than training length $T_{train}$. 

For example, we train CAPE-Kerple with **the maximum training length of 1024**, and we could say that CAPE-Kerple could have better extrapolation performance if CAPE-Kerple achieves better performance on evaluation length 8192. We follow the definition of length extrapolation from previous works [1][2][3].

**Thank you very much for your reply. If there is any other question, please let us know.**

Reference:

[1] Press, O., Smith, N., & Lewis, M. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR, 2021.

[2] Chi, T. C., Fan, T. H., Ramadge, P. J., & Rudnicky, A. (2022). Kerple: Kernelized relative positional embedding for length extrapolation. NIPS, 2022

[3] Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., ... & Bhojanapalli, S. Functional Interpolation for Relative Positions improves Long Context Transformers. ICLR, 2024.
I appreciate the authors' response, and I believe conducting experiments on LLMs is essential for this paper. I hope the authors can incorporate the results of these experiments in the revision.

However, some questions still confuse me.

1. First, the formulation of A(x) = XW_{Q}(XW_{K})^{T} + B(x). Using RoPE as an example, which can be written as: A(x) = (Q+f(Q))(K + f(K))^{T} = QK^{T} + Qf(K)^{T} + f(Q)K^{T} + f(Q)f(K)^{T}. Is B(x) equal to Qf(K)^{T} + f(Q)K^{T} + f(Q)f(K)^{T} ? or B(x) is equal to f(Q)f(K)^{T}? As the authors mentioned ``B(x) is set as a constant for all input sequence'', however, for RoPE, B(x) has Q and K terms (if B(x) is the former formulation), which is input-depend, rather than a fixed term.

2. Can you explain why CAPE has better extrapolation than other RPE methods, e.g., YaRN, from an Intuitive perspective? 

3. Does the ``better extrapolation than other RPE methods'' in the paper mean that CAPE performs better than other RPE methods with the same extrapolation length, e.g., when YaRN and CAPE scale the model context length to both 8192, CAPE has better performance? or does CAPE potentially have a longer context scaling length than YaRN (as you know, YaRN has a limited scaling length)?

I hope the authors can respond to those questions.
Dear Reviewer CfZF,

Thank you very much for your reply, and thank you very much for your support. We promise that we will include the additional details/experiments in the paper/appendix.
Dear Reviewer QhvW,

Thank you very much for your reply, for improving the score, and for your encouragement. We will continue to refine our work, focusing on both efficiency and effectiveness. We sincerely hope that our efforts will contribute to and inspire the entire research community.
Thank you for the additional details. This could be included in the paper/appendix.
I like the additional details provided by the authors. I encourage the authors to continue pushing the efficiency of CAPE. I increased the score to 6.
Dear Reviewer CfZF,

Thank you very much for appreciating our work. We will follow your suggestion to add the new experiment results  and discuss the transformer-xl in our paper, and we will update it immediately when we are allowed to revise the paper. Also, we will answer the two questions mentioned in the following:

**Q1: The Experiment of Alibi**

A1: The experiment of Alibi is the following.

**For model size 2.7B:**
| Model size| Method | 512 | 1024 | 2048 | 4096 
|-------|-------|-------|-------|-------|-------|
|2.7B|RoPE| 21.01|25.00|48.13|160.59 |
| | Alibi|21.23|22.17|22.91|23.22| 
| | T5's bias (RPE)|21.10|21.88|23.59|33.23| 
| | Kerple|21.14|22.08|23.38|27.21|
| | CAPE-Kerple|20.52|21.01|20.23|19.57| 

**For model size 6.7B**:
| Model size| Method | 512 | 1024 | 2048 |
|-------|-------|-------|-------|-------|
|6.7B|RoPE| 20.86|22.27|28.01|
| | Alibi|20.79|21.63|22.45|23.22| 
| | T5's bias (RPE)|20.79|21.60|22.32|
| | Kerple|20.71|21.57|22.07|
| | CAPE-Kerple|20.09|20.54|19.83| 


The reason for missing Alibi experiment result: with the model size increase, we face some engineering challenges:
* Our implementation is based on the framework GPT-NeoX, which already implements the Alibi position encodings.
* You may notice that our maximum evaluation length is 8192 for 125M and 350M, while the maximum evaluation length for 2.7B is 4096 and the maximum evaluation length for 6.7B is 2048. The reason is that the Alibi position encoding faces out-of-memory challenges for the evaluation length 8192 for the 2.7B model size and 4096 for the 6.7B model size.
* Therefore, considering two aspects ( 1. evaluation as long as possible; 2. the most popular position encoding method is RoPE), we mainly present the results of RoPE, T5's bias, Kerple and CAPE-Kerple in rebuttal so that the evaluation length can be longer (evaluation length 4096) for both 2.7B model size and 6.7B model size.
* If you would like to see more experiment results, please let us know. We will try our best to finish the experiment as soon as possible.

**Q2: Discussion with XPos**

A2: The CAPE is designed for Additive Relative Position Encoding, while XPos is an improved version of RoPE (Note: RoPE is Relative Position Encoding, but NOT Additive Relative Position Encoding). Therefore, CAPE and XPos focus on different directions of position encodings. 


**The Difference between XPos and CAPE**

The XPos:
   * The implementation of RoPE: $f_q(q,n)=qe^{i\theta n}$
   * The implementation of XPos: $f_q(q,n)=qe^{\xi n+i\theta n}$
   * Apparently, the XPos is an improved version of RoPE, while XPos degrades to RoPE when $\xi$ becomes zero.

The CAPE:
   * The implementation of Additive Relative Position Encoding: $A(x)=XW_Q(XW_K)^T+B$. The $B$ is the bias matrix, which could be Alibi, Kerple, FIRE, or other potential additive relative position encoding methods.
   * The implementation of CAPE: $A(x)=XW_Q(XW_K)^T+B + f(XW_Q(XW_K)^T, B)$. 
   * Apparently, the CAPE can be applied to any potential additive relative position encoding methods.

**The performance of XPos and CAPE-Kerple**

The previous paper BiPE [1] conducted experiments using XPos on the Arxiv dataset. As shown in the BiPE [1] paper's Figure 4, with the training length 1024, the perplexity of XPos increases quickly from about 6 ppl (at evaluation length 1024) to about 16 ppl (at evaluation length 6144), while our proposed method (CAPE-Kerple) decreases the ppl from 5.21 (evaluation length 1024) to 5.00 ppl (at evaluation length 8192) with training length 128. This suggests that our method should be better than the mentioned XPos.


Therefore, the XPOS is an improved version of RoPE (note that RoPE is not additive relative position encoding), while CAPE generally improves the performance of the additive relative position encodings, including Alibi, Kerple, FIRE and so on. 

**Finally, if you have any questions or would like to discuss anything, please let us know. We will try our best to share our opinions or conduct experiments.**

Reference:

[1] He, Z., Feng, G., Luo, S., Yang, K., He, D., Xu, J., ... & Wang, L. (2024). Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. arXiv preprint arXiv:2401.16421.
Thank you for the rebuttal. It generally addresses my initial concerns. The new experiments should improve the paper. I would encourage adding more discussion in comparison with Transformer XL as well. I also do not see any technical issue unlike tTna and your rebuttal makes sense to me. I maintain my acceptance score for now. 

But two questions: 

1. I noticed in your bigger LLM experiments that you provided in the response that AliBi is missing. Is there a reason for that? Do you have any comments on it? 

2. I may have missed it the first time around - but how does it stack up against something like xPos? Can you comment more on that method. I noticed that the xPos paper is in the references (42), but I couldn't find the context where it is cited (if at all).