DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Authors: Chuanyang Zheng
Yihang Gao
Han Shi
Minbin Huang
Jingyao Li
Jing Xiong
Xiaozhe Ren
Michael Ng
Xin Jiang
Zhenguo Li
Yu Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformers
context-adaptive positional encoding
long context
length generalization
Primary area: natural_language_processing
TL;DR: We propose a novel context-adaptive positional encoding to improve Transformer's capabilities, especially in long context .

Abstract: Positional encoding plays a crucial role in transformers, significantly impact- ing model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of input data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and can be dynamically adjusted with the given attention. In this paper, we propose a Data-Adaptive Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances in terms of trained length and length generalization, where the improvements are statistically significant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Chuanyang Zheng!* Yihang Gao2*
Han Shi
Minbin Huang'
Jingyao Li'
Jing Xiong Xiaozhe Ren"
Michael Ng Xin Jiang
Zhenguo Li SHKBU
Yu Li!
'CUHK NUS
Noah's Ark Lab
*HKU
Abstract
Positional encoding plays crucial role in transformers, significantly impact- ing model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of imput data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and be dynamically adjusted with the given attention. In this paper, we propose a Data- Adaa-ve Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances terms of trained length and length generalization, where the improvements are statistically sigmificant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.
Introduction
Transformer- based models have shown state-of-the-are performances in many language processing andwer [82, 29, 3], and commonsense reasoming [65]. The transformer mainly consists attention block, forwaforward block, and positional encoding. Recent works [8] have proved that quadratic-cost attention from the softmax is necessary for better performance, especially in long-contens processing. The attention block was originally designed by applying softmax to the key-query multiplication, which requires quadratic computational cost. To address such challenges, some efficient transformers were proposed, including sliding window transformers (e.g., Streaming LLMs [77]), linear transformers (e.g., Performer [17]), and sparse transformers (e.g., Reformer and sparse Sinkhorn transformer [66, 25]), etc. However, some negative results exist regarding efficient transformers' performances [80].
tasks, including translation [6].
has been noticed recently that well-designed positional encoding significantly improves the model performances, especially in the long- context tasks [33]. While transformer-ba baasedd--baedd models exhibit satisfying performances in tasks of consistent length and distribution, their effectiveness tends to diminish sharply when the input length exceeds the training length, e.g., long document summariza- tion, "needle in haystack" search, and long text generation. To avoid the expensive computation in training, the training length is usually preferred to be relatively small due to the quadratic cost of
"Equal Contribution
Contact Email: 2yzheng I @ link.cuhk.edu.kk
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Example Attention Hayer 13)
Head-1
5.0
Atleten Value
25-
Cure No Se02tteee
-2
1088-0
3.
x
1008-12
-25
Ralattiv Cistance
Relatire Distance
Rallttive Distance
Figure 1: Visualization of DAPE learned positional biases for the 8192th query position with key positions between and 8192, while the training length is 512. We notice that DAPE learss both local and anti-local position patterns. The model is trained with Equation 2: (1) The Attention is X XWo( k)) : (2) The Kerple bias is B; (3) The DAPE (with Kerple) bias is f(X (XWx) BB). More examples are shown in Appendix
I
transformers. However, real-world applications often require processing longer input sequences, posing a significant challenge. Therefore, there a growing interest in evaluating model performance by training on shorter sequences while testing on longer inputs. Standard transformers may not distinguish the ordering of tokens without external assistance. In practice, they depend on positional encoding to incorporate positional information, enabling the model to make meaningful token predictions. Without these encodings, token generation would lack the necessary contextual order, rendering the outputs nonsensical. The RoPE [62] positional encoding method demonstrated notable performance degradation, failing entirely when the input length is double that of the training length [51, 10, 24). A common characteristic among these positional encodings is their pre-defined and static nature. Specifically, they are fixed across various tasks and models, which may lead to their inability to adapt to varying input lengths and contexts effectively. To address this issue, recent works have introduced Functional Interpolation for Relative Positional Encoding (FIRE) [41], which utilizes a neural network to learn an implicit mapping from input positions to positional bias. A functional approach to positional encoding that dynamically adjusts positional biases based on semantic information (input context) allows the model to empower adaptability beyond the fixed inductive bias as adopted in previous studies (such as RoPE [62] and Alibi [52]). Although FIRE utilizes MI.Ps to learn positional embeddings, these embeddings remain fixed across different tasks once the training is completed. Intuitively, the learned static positional encoding (such as Kerple and FIRE) an average optimal solution across all training samples. Consequently, while they might be generally effective, they are inherently suboptimal for any specific instance. This static nature limits their flexibility and applicability in various real-world scenarios that deviate from the training context. In this paper, we introduce a data-adaptive positional encoding (DAPE) method, inspired by the limitations of static PEs. DAPE dynamically adjusts the PE based on the semantic information (e.g., the current attention value) and the positional indicator b. The proposed PE is represented by MLPs due to their universal approximatability. i.c., MLPs((a b). We note that DAPE is compatible with all additive relative PEs and offers advantages in terms of interpretability and ease of implementation. The proposed DAPE incorporates both the semantic and the positional information, making the PE adaptive with the input data. The adaptivity allows DAPE to overcome the inflexibility and achieve relatively optimal performance for each individual instance by dynamically adjusting on each specific input data. To the best of our knowledge, this is the first semantically dependent and adaptive positional encoding method introduced in transformer architectures.
The paper is organized as follows. In Section 2, we review some related works on positional encoding methods, including absolute and relative positional encodings as well as the potentially no positional encoding in some transformer models. In Section 3, introduce the proposed DAPE method with implementation on multi-head attention and analysis on computational costs. We conduct comprehensive experiments on DAPE, validating its effectiveness and performances on various language tasks and datasets, as reported in Section 4. In Section 6, some concluding remarks and potential future works are presented.
2 Related Works
No positional encoding. Haviv et al. [30] show that decoder-only Transformers with causal attention masks can learn positional information even without any explicit positional encoding. Recently, Kazemnejad et al. [33] proved the effectiveness of no positional encoding (NoPE) [71]. Although

---

the NoPE can implicitly catch the positional information, it performs poorly compared with some explicit positional encoding methods [41].
Absolute positional encoding. Vaswani et al. [69] proposed Absolute positional encoding (APE) to endow transformers with positional information. In particular, in the first layer, (learnable or fixed sinusoidal) real-valued encoding [69, 35, 42, 70, 47] e; E R is assigned to each position i, leading to an APE matrix E _e,,.... en] which will be added to the input sequence. Though simple and straightforward, APE-based Transformers usually generalize poorly to longer sequences [52].
Relative positional encoding. Relative Positional Encoding (RPE) is another popular way to encode positional information [58, 56, 52], One popular RPE method in large language models is rotary positional encoding (RoPE) [62, 18, 67]. RoPE rotates the query and key vectors with an angle proportional to their absolute positions before the dot product attention, which results in attention being a function of the relative distance between the tokens, capturing the relative positional information. Press et al. [52] and Kazemnejad et al. [33] found that RoPE-based language models have poor length generalization. To address this, positional interpolation (PI) [11] is proposed to extend the context window. Following the direction, there are LongLora [12], LongRope [24], YaRN [51] and CLEX [10]. Another popular direction is additive positional encoding. For most of these additive RPE methods, the computation of the (pre-softmax) attention logits can be unified using the following formula:
ARPE(A (x) = XWo(X T + B,
where the bias matrix B E R2 induced by the position encoding function b N2 - IR and the (i,j)-th entry of B is defined as b(i.j). Different formulations and parameterizations of b lead to various RPE variants. Several methodologies that facilitate arbitrary sequence lengths include T5's RPE [56), Alibi [52], Kerple [13], Sandwich [14], and FIRE [41]. Currently, additive RPE delivers relatively robust performance in length extrapolation without necessitating additional operations. Alibi constructs the bias matrix B utilizing prior knowledge, resulting in basis matrix devoid of learnable parameters [52]. Conversely, both Kerple [13] and Sandwich [14] incorporate two learnable parameters to facilitate the learning of bias matrix while retaining some elements of priors. FIRE suggests adopting a learnable continuous function, such as MLPs, to convert input positions to biases [41]. Observing these developments, it becomes evident that the next generation of bias matrices will likely incorporate adaptivity and flexibility. Based on this insight, we propose our method DAPE, a semantically adaptive method.
Method
In this section, we formally introduce DAPE (data-adaptive positional encoding), new relative positional encoding approach that further enhances transformer performance. Compared with previous works on static positional encoding methods, DAPE adopts semantically adaptive positional bias matrices depending on input context. We first demonstrate that most of the popular positional bias matrices are fixed once the training is finished, independent of the input sequences. To address this limitation, we then accordingly develop DAPE that captures the implicit relationships by MLPs and adjusts the bias matrices based on input context. Furthermore, we discuss variant of DAPE with residual connections and its extensions to multi-head transformers.
3.1 Additive Relative Positional Encoding
For most additive RPE methods, the computation of pre-softmax attention logits can be unified under the following formula:
ARPE(X) XXx(XXWX) + BB,
(I)
where the bias matrix B E TRax induced by the position encoding function b N2 - lR and the (i,j)-th entry of B is defined as b(i,j). Various formulations and parameterizations of b give rise to
different variants of RPE. Examples of additive RPE include: (1) Alibi: b(i.)) ji = -rli jl. with
the scaler > 0 as hyper-parameter; (2) Kerple: b(i.j) -rg + r2li - jl) with 1 and 72
are two learnable parameters; (3) FIRE: b(i,j) = fe ((mmxx[L..) where the positional encoding function so parameterized by 0 is learned from data and & a transformation function aimed at assigning more model capacity to local positions.
3

---

We observe from the formulation of those additive RPEs that they remain static once the training process completed and depend solely on the positions, regardless of the input context. This infiexibility and lack of adaptivity can lead to performance degradation, especially in tasks involving long-context generation and reasoning. Intuitively, the learned static RPEs are optimal on average across all training samples. However, this means they are suboptimal when considering each individual instance, as they cannot adapt to specific tasks. To address these challenges and enhance model performance, it is essential to adopt an alternative approach using a semantically adaptive RPE that depends on input context.
3.2 Data-Adaptive Positional Encoding
For simplicity, we first consider the single-head case and the extension to the multi-head transformer will be discussed subsequently. The design of data-adaptive positional encodings in natural language tasks is motivated by the need to capture the intricate relationships between tokens. Arora et al. [5] reveals that associate recall accounts for most of the perplexity difference between transformer, RNN- based, and convolution models. For example, we consider a consistent pairing that "Hakuna" is always followed by "Matata" in a long paragraph. This pattern suggests a reduced reliance on positional information in favor of enhancing token embedding similarity, thus allowing for 'Hakuna' to be effectively linked with a preceding 'Matata'. Similarly, in tasks involving long- long-context understanding and search, semantic similarity should be prioritized in the attention mechanism rather than being overshadowed by positional encodings, which can be less relevant over long distances. Consequently, the transformer should preserve information without being influenced overly by positional distance. Instead, a satisfactory PE should integrate both semantic and positional information. Therefore, a semantically dependent positional encoding approach is preferable and expected to enhance model performances. Here, we use the attention X &Wo (X WK) to represent the semantic information and positional bias matrices B (e.g.. Alibi and FIRE) to capture positional details. Then the context-adaptive PE is described by f(X ( (WK) BB. where f(-) is an implicit function that integrates both semantic and positional data into the desired positional encodings. Thus, the pre-softmax attention logit incorporated with DAPE is formulated as
ADAPE(X) XX + (XWx)( .B).
(2)
Here, RTTT X TTXT + RTXT is an clement-wiss function. In practice, we utilize two-layer LeakyReLU neural network to parameterize f()) due to its universal approximability [36]. All parameters are learned directly from the data during the training process. This architecture allows f(-)
to dynamically adjust positional embeddings based on the input context, ensuring that the encoding method is both adaptive and dependent on the input data.
Different from FIRE, which also models the implicit positional encoding by MLPs, our approach additionally integrates semantic information. This integration enables the adaptivity, flexibility, and context-dependency of the positional encodings. Significantly, our method compatible with most additive RPE techniques, as these commonly involve positional bias matrices B that inherently contain positional relations. Unlike previous RPEs, which rely solely on absolute positional differences, our DAPE method, can be seen as utilizing multi-level positional bias matrices. Here, the bias matrices dynamically adjust based on the input context, offering a more reasonable and responsive encoding mechanism.
Expressiveness. Due to the universal approximability of (LeakyReLU) neural networks [36], f() is capable of capturing complex relationships between the desired positional encoding and both semantic and positional information. Regardless of the semantic component, when the relative position - j is used as input, DAPE can realize classical additive RPEs (e.g., Alibi and Kerple), according to [41]. This demonstrates the versatility of DAPE in accommodating traditional encoding schemes while also offering enhanced capabilities. There exists a fundamental trade-off between the expressiveness and computational costs. Wider hidden layers lead to higher expressiveness but also contribute to more computational costs. In practice, we find that two-layer neural networks with 32 hidden units per layer provide sufficient expressiveness to deliver satisfactory performance, balancing complexity and efficiency effectively.
Discussion. We can also interpret the proposed method from an alternative perspective. In the standard transformer architecture, the pre-softmax attention typically involves the key-query similarity and the positional encoding by either addition (in the form of a + b, e.g., Alibi and Kerple) or multiplication (in the form of b, e.g., RoPE). Here, we propose unified approach by replacing them with

---

learnable MLPs. i.e., MLP(a. b). This configuration allows the model to learn the desired relationship between the pre-softmax attention, the key-query similarity and the positional encoding. It can also be regarded as a new transformer architecture that empower the transformer with additional MLPs on pre-softmax attentions.
A variant of DAPE with residual connections. It is well-known that deep neural networks may suffer from gradient vanishing. To further enhance the practical performances, we introduce the residual connection for positional information. Consequently, Equation 2 is modified as follows:
ADAPR ()= XWo(XXx)) + B B ((Xoo B).
In this reformulation, f()) acts as an adaptive correction term to the traditionally fixed RPE, dy- namically adjusting the positional bias matrices B based on both semantic and positional inputs. In Section 4, we empirically explore the impact of residual connections in DAPE. Our observations reveal that for well-behaved bias matrices B, the DAPE model with residual connections, as specified in Equation 3, is preferable. Conversely, if the bias matrix is underperforming but still conveys positional information, the original implementation in Equation 2 more effective.
Multi-head DAPE. In its simplest form, DAPE is considered for a single-head case as described in Equation 2 and Equation 3. However, adopting a multi-head mechanism significantly enhances model capabilities. To effectively combine both semantic and positional information, the DAPE in multi-head setup processes the key-query similarities and bias matrices from all heads. Specifically, for an h-head layer, the function f(-) imputs a 2h 2h-dimensional concatenation of key-query similarities and positional biases. It then outputs h-dimensional vectors, where each element corresponds to the DAPE for the respective head. We have shown the code implementation in Appendix J. Importantly, semantic and positional information across different heads are processed simultaneously within the same MLPs, rather than sequentially. This approach not only improves computational efficiencies through parallel processing but also capitalizes on the richer semantic information available across all heads. Compared to the key-query similarity derived from single head, the comprehensive attention from all heads yields more substantial semantic information.
Computational costs analysis. Here, we evaluate the additional computational costs introduced by the DAPE method, compared with the classical positional encoding methods (c.g.. Alibi and Kerple). We consider transformer model with fa heads and assume sequence length of N and all hidden dimensions in the attention layer being d. Then the total computational cost for a standard transformer equipped with classical PEs is O (h.2d + hNdd). When incorporating the proposed DAPE, which employs two-layer MLPs with hidden dimension DAAPE, the additional computational costs are O (h.N² DrAPE). If the hidden dimensions DDAPE & d, the incremental computational cost introduced by DAPE is not significant.
4 Experiment
Baselines. We evaluate the proposed DAPE against a range of established baselines, including
NoPE [33], RoPE [62], YaRN [51], Randomized RoPE [57, 31], T5's Bias [56]. Alibi [52]. Ker- ple [13], and FIRE [41]. For RoPE, the randomized positional encoding [57, 31] is applied to enhance the model performance, extending the randomized length to four times that of the training length.
Datasets. Our analysis involves training language models on the Arxiv and Books3 datasets, which frequently used benchmarks for evaluating model performance [52, 13, 41, 24]. We start our evaluation by comparing the last 256 tokens' zero-shot perplexity across different input lengths. Besides perplexity as evaluation metrics, we also employ the downstream datasets in randomized positional encoding [57] to evaluate DAPE, where details are included in Appendix D.
Experiment settings. Initially, we compare DAPE with other baselines at training lengths of 128, 512. and 1024, with model size 125M decoder-only Transformers [9], whose configuration is shown in Appendix B. Subsequently, we evaluate the performane of larger model size 350M, DAPE variants and explore the impact of hidden dimension of MLPs DDAPE- We also examine the computational efficiency of DAPE, focusing on processing times. Additionally, we provide visualizations of the DAPE bias in the Appendix I. Finally, we also evaluate DAPE on algorithmic reasoning datasets via accuracy metrics.
5

---

Arxiv Dataset (Length 128)
Anxiv Dataset (Length 512)
Wilddiion peppacity
10
10.
128
256
512
1024
2008
6096
01992
2
512
1024
2048
4096
8192
Validation sequence length
Validation sequence length
Books3 Dataset (Length 128)
Books3 Dataset (Length 512)
60.0
37.5
Win
35.0
Fandorlzce
Vel perg
Nateston
30.0
27.5
25.0-
20
225
20.0
512
1024
2048
4096
8192
Valication sequence length
Validation sequence length
Figure 2: Comparisons with basclines: performance with training lengths 128 and 512 on Arxiv and Books3 datasets.
4.1 Comparisons with Baselines
DAPE's superior performance within training length and beyond training length, compared to all baselines. As shown in Figure and Table 5, DAPE consistently outperforms established baselines such as RoPE, Alibi, and Kerple across various settings. Notably, DAPF-Kerple (the positional information in DAPE comes from Kerple bias matrices) outstands in both short and long training lengths (128 and 512), compared to previous RoPE, T5's bias, and so on. It demonstrates that the semantic adaptivity of DAPE significantly enhances its state-of-the-ar performance against all other static positional encoding methods.
Dataset (Length 1024)
The performance longer training length 1024. As shown in Figure 3, the proposed method consistently delivers state-of-the-art per- formance for the training length of 1024. When the evaluation extends 2048, both DAPE- Kerple and DAPE-FIRE achieve notable re- sults, recording performances of 3.91 and 3.93
14
Colidation perploxity
12
T2's Mis
Rancemiced
/4hi
10-
Kape
FRE
DAP
perplexity scores, respectively. Remarkably, DAPE-FIRE behaves well at the longer eval- uation length of 8192, achieving performance of 3.91 scores and surpassing Alibi's score of 4.28. These findings reveal that DAPE sustains robust performance with longer training length of 1024.
1024
2048
4096
8192
Valisation sequence length
Figure 3: Results on the training length 1024.
DAPE enhances intra-length performance, indicating that its lower perplexity may come from thorough utilization of entire sentences but not disregarding long-distance information (Also proved in Figure 1). Compared to Alibi, Kerple, and FIRE, the adapted versions DAPE- Alibi, DAPE-Kerple, and DAPE-FIRE demonstrate consistently and significantly better intra-length performance. With the growing sequence length, the Alibi tends to transition from full attention to almost local attention, and this is why Alibi is worse than most baselines within training length but better beyond training lengths. The results (as shown in Table 5) indicate that the superior intra- length performance DAPE is statistically significant, with a p-value less than 0.05. Therefore, consistent lentth performances across various training lengths indicate that the lower perplexity of DAPE results from effectively utilizing the entire sequence, rather than focusing on local parts and neglecting long-dis distance information.

---

DAPE significantly improves length extrapolation performance, compared to ALibi, Kerple, and FIRE. DAPE-Kerple significantly surpasses competitors like vanilla Kerple when training and evaluating at different lengths. On the Arxiv dataset trained at length of 128, DerE- achieves a remarkably low perplexity of 5.00 at an evaluation length of 8192, in stark contrast to Kerple's 31 ,93. Similarly, on the Books3 dataset with a training length of 512, DAPE-Kerple records perplexity of 17.88 at the same extended evaluation length, far outperforming Kerple's 39.31. These results affirm that DAPE, through its semantic adaptivity and flexibility, consistently enhances performance beyond training lengths, eclipsing static positional encoding methods.
4.2 The Effect of Model Size
350M Model (Length 128)
350M Model (Length 512)
14
14.
Randomized RUPE
753 DMs
Validation perpiciity
Validaion perplesity
122
AIbI
12
Ktrple
10
DAPE-AID
FICE
10:
DAPE Karple
6-
4.
128
256
512
1024
2048
4096
8192
512
1024
2048
4096
8192
Validation sequence length
Validation sequence length
Figure 4: The effect of model size: for the 350M model, the performance with training lengths 128 and 512 on the Arxiv datasct.
DAPE enhances performance with increasing model sizes. As the model size grows (as shown in Figure 4), DAPE consistently demonstrates an improvement in performance metrics. When the model size is augmented from 125M to 350M, the perplexity at an evaluation sequence length of 8192 (with training length of 512) for DAPE-Alibi shows notable decrease from 3.82 to 3.57. These numbers are appreciably smaller than those recorded for original Alibi, which decreases from 4.54 to 4.21 perplexity, indicating a robust performance improvement. Additionally, Kerple significantly reduces the perplexity for Kerple, bringing it down from an initial 22.76 76 to an impressive 3.43. These results confirm that DAPE retains its efficacy and continues to perform well even as the model size is increased, mainly due to the adoption of semantically adaptive PEs.
DAPE methods almost are ranked top-3 with large model size. With the incremental model size, DAPE-FIRE begins to match, and nearly approach, the performance levels of Alibi. Initially, at model size of 125M and a training length of 512, DAPE-FIRE achieves a perplexity of 5.71 at an evaluation sequence length of 8192, while Alibi stands at a perplexity of 4.54. However, as the model size is increased to 350M, the performance gap significantly narrows. Specifically, DAPE-FIRE outperforms Alibi regarding the perplexity scores when the evaluation length is smaller than 4096, as the model size grows for evaluation. In conclusion, as shown Figure 4, we observe that the DAPE methods almost win the top-3 among all positional encoding methods. This trend underlines the scalability and adaptability of DAPE, emphasizing its potential to handle more substantial computation challenges.
4.3 Different Variants of DAPE
In this section, we evaluate the performance of DAPE across its various forms. Our analysis focuses on DAPE-Kerple. Notably, as shown Figure 5, all variants of DAPE surpass the baseline performance of Kerple. The Addition Residual Addirion_Residaal variant of DAPE, while requiring less computational effort, delivers relatively inferior results, As illustrated in Figure 5, concatenation methods (either Concat or Concat_ Residual)d outperform the Addition Residual approach, for both the training length of 128 and 512. Furthermore, both Concat and Concat Residual exhibit comparable performance metrics. Specifically, at training length of 128, Concat Residual records score of 5,00 and Concar scores 5.03 at an evaluation length of 8192 whereas Add_ Residual posts a 5.17 perplexity score. With a training length of 512, Concal_Ree Residuessiallll achieves a score of 3.70, and Concul scores 3.69 at an evaluation length of 81 8192 compared to Add_ hiiiddaesdlll's s 3.75. Based on the current observation, the different variants of DAPE show comparable performances, compared to baselines.
7

---

Different Variant (Length 128)
Different Variant (Longth 512)
Beaffe
Wildeiion perpaccity
Are hrstis
Orex, Noskue
5-
128
256
512
1024
2008
6096
0192
532
1024 2048
4096
8192
Validation sequence length
Validation sequence longth
Figure 5: Different variants of DAPE: the DAPE-
DAPE-Kerple Kerple performance under different variants. (1)
Add Residual: XWa(XWx) + + I(XWa(XWa + B): (2) Concate: XWo(XWx) +
f(XWa B); (3) Concate_Residudtt X XWa(XWx) B ((XWo(XWx) B).
4.4 The Effect of the Hidden Dimension DDAPE
Even small DAAPE can improve the performance. The experiments conducted with Alibi and DAPE-Alibi. As shown in Appendix Figure 6, when considering the training length 128 and DDAPE set as 4, the DAPE-Alibi achieves 8.25 evaluation length 128 and 5.67 at length 8192, which is better than Alibi's 8.3 I and 5.85. Whatever DAPE is 4, 16 32, or 64, the performance is always better than the original Alibi all evaluation lengths. This suggests the effectiveness of DAPE, even with smaller DDAPE.
The choice of DDAPE- Based the experiment, overly small values of DAAPE degrade per- formance, although they still perform better than the baseline. Conversely, larger values of DDAPE increase computational costs. The function f(-) is implemented dimension either the head number or twice the head number, and the output dimension is the head number. Therefore, recommend setting the hidden dimension the head number prevent information loss and ensure the capacity of f(-).
two-layer MLP, where the input
4.5 The Time Cost
Table 1: The time cost (millisecond) under different testing lengths, with DDAPE 32 and default batch size 1, with training length 512.
Method
350M Total
Ratio
2.7B Total
Ratio
6.7B Total
Ratio
RoPE [62]
210.01
0.9366
472.63
1.1187
635.57
0.8858
T5's bias [56]
355.16
1.5839
537.62
1.2725
808.85
1.1273
ALiBi [52]
172.60
0.7697
325.95
0.7715
596.77
0.8317
Kerple [13]
189.91
0.8469
370.32
0.8765
661.82
0.9224
FIRE [41]
248.13
1.1066
432.63
1.0240
797.68
1.1118
DePp-Kerple
224.22
1.0000
422.48
1.0000
717.46
1.0000
Practical additional time cost. The additional training ratio will gradually decrease with a larger model size, compared to baseline Kerple.. The cost of Feed-Forward Network is: O (f Faa dh Ch Pridden where is constant, N is the sequence length, dhead the attention head number and dhidden is the dimension for attention calculation. The cost of Attention: 2 dreaddhidden 2 dheaddhiddens where b is constant. The additional dheadd
cost of DAPE: O( DAPE) " where с is a constant. The cost ratio
aacheaddi +bN dasaden teeeeeNNddddd Therefore, with the fixed sequence length and dDAPE, with the model becomes larger (with bigger dhead and drsdden), the additional cost ratio DAPE will greatly become smaller. Also, we have shown in Figure that dPAPE still works well with very small value, such 4.
aNcheaed 2h dhrdden +bN2 dhenddhiaten
ads Paaden toNdhidden
aN de Ladden bN Poreddaaaddan adeaddnedden +cN dhdaddaad

---

Table 2: Train length 40 with 200k steps, and test from lengths 41 500. The random accuracy is 50%, except for MODULAR ARITHMETIC (SIMPLE), CYCLE NAVIGATION, BUCKET SORT, SOLVE EQUATION and MODULAR ARITHMETIC, where it is 20%. tft denotes permutation-invarian tasks, which are expected to be solved without positional information.
Randomized
DAPE (Ours)
Level
Tusk
Learned
RoPE
Relative [19]
ALiBi
Kerple
FIRE
Alibi
Kerple
FIRE
R
PARITY CHECK 11
MODULAR ARITHMETIC (SIMPLE)
50.14
19:95
20.39
50.52
50.05
21.35
20.84
50.09
20.02
50.09
50.07
21.79
21.09
50.97
50.30
24.47
50.07
24.16
50.04
CYCLE NAVIGATION11!
34.97
25.37
27.63
26.95
24.64
29.47
28
22.99
34,53
27 54
MANIPULATION
65.92
64.73
66.42
66.93
69:
REVERSE STRING
67.28
MODULAR ARTHMETTCC
31.00
30.20
31:25
31,74
30:56
24.79
30.92
31.34
32.37
31 31.50
SOLVE EQUATION
20.00
19.97
21.85
22.93
19.92
21.15
22.06
20.03
22.49
22.42
DUPLICATE STRING
52.77
65.44
64,97
67.66
65.13
66.72
69.03
70.84
72.95
72.71
DUPLICATE
79.27
ODDS FIRST
$2.77
58.61
61.00
61.57
59.88
62.59
63.
63.78
67.08
66
CS
BINARY A ADDITION
54.63
55. 78
55.59
$6.96
54.72
56.35
55. 70
59.71
61.88
56.
COMPUTE SQRT
50.47
$1.11
$1.88
51.63
50.63
51.11
50.80
51.64
51.33
52.46
BUCKET SouTHf
98.92
99.38
99.57
98.81
99.37
4.6 The Visualization of DAPE
In this subsection, present the visualization of learned positional encoding biases from a DAPE- Kerple model pretrained on Arxiv (training length is 512). We plot the learned positional encoding bias for the query token at the 8192th position, for all the attention heads from selected layers in Figure 1. We would like to highlight two features DAPE. First, in different attention heads, the bias matrix of DAPE learns both local and "anti-local attention patterns that emphasize more far-away keys (just like FIRE), compared to a fixed local inductive bias (such as Kerple and Alibi). Secondly, the bias matrix can be dynamically adjusted with different attention values, compared to the static bias fixed for all attentions. We have shown more examples, including different layers and different samples, in Appendix
4.7 Experiments CHE Benchmark
Besides employing perplexity as evaluation metric, we also evaluated DAPE on downstream Chom- sky Hierarchy Evaluation Benchmark (CHE) [21] (need to utilize the whole sentence information to generate correct answers) to further discuss its effects. The experimental setup follows randomized positional encodings [57], detailed in Table 4, with the experiment setting shown in Appendix Section D. Overall, FIRE outperforms Kerple in 9 out of 14 tasks, while Kerple outperforms Alibi in 11 out of 14 tasks. This observation aligns with findings [41], suggesting that the experiments in Table reliable and reflect the performance of positional encoding in downstream tasks.
DAPE works better on permutation-varian tasks. DAPE (with Kerple and FIRE) presented the best performance in 10 out of 11 permutation-varitn tasks (which require positional information), achieving the second-best performance in the SOLVE EQUATION task. This underscores the efficacy of DAPE with semantic adaptivity in handling permutation-varian challenges.
DAPE's performance on permutation-invariat tasks. In tasks that are permutation-invarian,, where positional information is non-critical, DAPE demonstrated comparable performance. Notably, DAPE-Alibi achieved scores of 50.30 on PARITY CHECK and 99.38 on BUCKET SORT tasks, com- pared the highest scores of 50.97 and 99.57, respectively, demonstrating competitive performances.
Comparative performance improvements. DAPE consistently enhanced performance across various tasks, especially on permutation-van variann tasks. Specifically, DAPE improved upon Alibi and FIRE's results in all 11 tested permutation-invariie
tasks. Similarly, it outperformed Kerple in 10 of these tasks. These results highlight the effectiveness of DAPE over static positional encoding methods like Alibi, Kerple, and FIRE, resulting from its dynamic adaptivity.

---

Evaluation Protocol
In this work, we initially utilize the model to process the entire input sentence and subsequently select the final 256 tokens for perplexity computation. This approach contrasts with a variety of other studies, which employ methods that process the full sequence during perplexity calculations 31, 12, 24, 10, 41J. As result, our reported baseline perplexity is comparatively higher than the results presented ALiBi [52], which adopt a overllppgg evaluation strategy. This method divides sequences longer than L into multiple segments of length L, thereby yielding lower perplexity figures.
Though ALiBi [52] and Kerple all claim that they use non-overlapping evaluations, their reported results are different. For example, the ALiBi paper Table 2, the sinusoidal position encoding perplexity increases from 20.05 (evaluation length 512) to 406.01 (evaluation length (evaluation length 15512), while in Kerple paper Table 3 the sinusoidal position encoding perplexity from 33 to 30046. This may be caused by different evaluation protocols and training strategies.
Recommended Protocols. We strongly recommend that researchers process the entirety of the sequence before selecting the last K tokens for the purpose of calculating perplexity. The rationale behind processing the whole sentence is that it provides a comprehensive evaluation of the model's capability to handle long-context dependencies. thus offering more accurate reflection of its performance. Following this step, we advocate for the selection of the last K tokens to compute perplexity, ensuring that the same number of tokens used across different evaluation lengths, which promotes consistency and comparability in the results.
Release this work's code for future work. In light of this methodology, we have made our code publicly available to other researchers in the field. This initiative aims to facilitate standardized comparison and evaluation of their respective methods, thereby advancing the collective understanding of model performance in relation to perplexity calculations.
6 Conclusion
In this paper, we propose the data-adaptive positional encoding (DAPE) by incorporating both the semantic and the positional information to improve the model performance. We show that the addi- tional computation introduced by DAPE is not significant under proper choices of hyperparameters. We conduct comprehensive experiments on Arxiv, Books3, and CHE to validate the effectiveness of the proposed method, revealing that the adaptive PE method has advantages over static PEs. We believe that the DAPE could benefit the whole community, especially on length generalization tasks.
References
[1] Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. arXiv preprint arXiv:2403.0000 099544 2024.
[2] Devanshu Agrawal, Shang Gao, and Martin Gajek. Can't 't remember details in long documents? you need some r&r. arXiv preprint 403002400.00004 2024.
[3] Joshua Ainslie, Tao Lei, Michiel de Jong. Santiago Ontanon, Siddhartha Brahma, Yury Zemlyan- skiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. CoLT5: Faster onggrangee transformens with conditional computation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[4] Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Ling- peng Kong. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.1 17663, 2024.
[5] Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint ar arXhv:2312.04927, 2023.
10

---

[6] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2015.
[7] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp. Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024.
[8] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- -ocumentt transformer. arXiv preprint 00t::20044.055000 2020.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
[10] Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. CLEX: Continu- ous length extrapolation for large language models. In International Conference on Learning Representations, 2023.
[11] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprini arXiv:2306. 15599555...55 2023.
[12] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- gLoRA: Efficient fine-tuning of long-context large language models. International Conference on Learning Representations, 2023.
[13] Ta- Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. KERPLE: Kernel- ized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 11:8386-8399 2022.
[14] Ta-Chung Chi, Ting-Hap Ling-HanDu Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13522 13537, 2023.
[15] Ta-Chung Chi, Ting- Fan, and Alexander Rudnicky. Attention alignment and flexible posi- tional embeddings improve transformer length extrapolation. arXiv preprint 1rX0v2231..00684 2023.
[16] Noam Chomsky, Three models for the description of language. IRE Transactions on Information Theory, 2(3):113-122)) 1956.
[17] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representattons, 2021.
[18] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[19] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, 2019.
[20] Soham De, Samuel Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint 2rXiv:2402.19427 2024.
11

---

[21] Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, et al. Neural networks and the chomsky hierarchy. In International Conference on Learning Representations, 2022.
[22] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, Volume I (Long and Short Papers), pages 4171-4186, 2019.
[23] Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Ste- fano Soatto. Fewer truncations improve language modeling. arXiv preprint arXiv:2404. 108300 2024.
[24] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. LongRoPE: Extending IIm context window beyond million tokens. arXiv preprint 24Xi:22402.1355: 2024.
[25] Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context Ilms. arXiv preprint arXiv:2407.0944 2024.
[26] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprins 2402:22102.11771,, 2024.
[27] Yihang Gao, Chuanyang Zheng. Enze Xie, Han Shi, Tianyang Hu, Yu Li, Michael K Ng. Zhenguo Li, and Zhaogiang Liu. On the expressive power of variant of the looped transformer. arXiv preprint vrXiv:2402.13577 2024.
[28] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.0075: 2023.
[29] Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. T5: Efficient text-to-text transformer for long sequences. Findings of the Association for Computational Linguistics: NAACL, 2022.
[30] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382-1390, 2022.
[31] Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang. Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Liwei Wang. Two stones hit one bird: Bilevel positional encoding for better length extrapolation. arXiv preprini arXiv:2401. 16421, 2024.
[32] Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. LLM maybe LongLM: Self-extend LLM context window without tuning. arXiv preprint urXiv:2401.0132 2024.
[33] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
[34] Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In International Conference on Learning Representutions, 2020.
[35] Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 3309-3321, 2021.
[36] Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 666):861 1993.
12

---

[37] Jingyao Li, Pengguang Chen, Zexin He, Shaozuo Yu, Shu Liu, and Jiaya Jia. Rethinking -f--o---ittriutiionn (OOD) detection: Masked image modeling is all you need. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11578-11589 June 2023.
[38] Jingyao Li, Pengguang Chen, and Jiaya Jia. Motcoder: Elevating large language models with modular of thought for challenging programming tasks, 2024.
[39] Jingyao Li, Pengguang Chen, Shengju Qian, and Jiaya Jia. Tagelip: Improving discrimination ability of open-vocabulary semantic segmentation, 2023.
[40] Jingyao Li, Pengguang Chen, Shaozuo Yu, Shu Liu, and Jiaya Jia. Bal: Balancing diversity and novelty for active learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3653-3664, 2024.
[41] Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang. Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. In International Conference on Learning Representations, 2023.
[42] Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozh- nikov. CAPE: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34: 14791699-9 2021.
[43] Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xi- afei Qiu, Shen Li, et al. InfiniteLLLM: Efficient LLM service for long context with distattention and distributed kvcache. arXiv preprint arXiv:2401. 026669, 2024.
[44] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n-gram language models to trillion tokens. arXiv preprint arXi::2401. 17377, 2024.
[45] Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, et al. E' 2-LLM: Efficient and extreme length extension of large language models. arXiv preprint arXiv:2401 06955,, 2024.
[46] Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of RoPE-based extrapolation. In International Conference on Learning Representations, 2023.
[47] Xuanqing Liu, Hsiang-Fu Fu Yu, Inderjit Dhillon, and Cho-Jui Jui Hsich. Learning encode position for transformer with continuous dynamical model. In International Conference on Machine Learning, pages 6327-6335. PMLR, 2020.
[48] Shengjie Luo, Shanda Li, Tianle Cai, Di He. Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, and Yin Liu. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances Neural Information Processing Systems, 34:22795-223 228077, 2021.
[49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[50] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. RWKV: Reinventing RNNs for the transformer era. Findings of the Association for Computational Linguistics: EMNLP, 2023.
[51] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient con- text window extension of large language models. In International Conference on Learning Representations, 2023.
[52] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021.
13

---

[53] Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong, Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401 1.44658, 2024.
[54] Zhen Qin, Yiran Zhong, and Hui Deng. Exploring transformer extrapolation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18897-18905, 2024.
[55] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 (1440): 2020.
[57] Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length gen- eralization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1889-1903, 2023.
[58] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464 468, 2018
[59] Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. International Conference on Learning Representations, 2023.
[60] Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszezur, Henryk Michalewski, Lukasz Kuciński, and Piotr Mitoš. Structured packing in LLM training improves long context utilization. arXiv preprint arXiv:2312. 17296, 2023.
[61] Jianlin Su, Murtadha Ahmed, Luo Ao, Mingren Zhu, Yunfeng Liu, et al. Naive bayes- context extension for large language models. arXiv preprint arXiv:2403.17552. 2024.
[62] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 168:127063, 2024.
[63] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11566 2023.
[64] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatabb transformer. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14590-14604, July 2023.
[65] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4149-4158 4158, 2019.
[66] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438 1438-9447. PMLR, 2020.
[67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint UrXiv:2302.139// 2023.
[68] Szymon Tworkowski, Konrad Staniszewski, Mikotaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoš. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[70] Benyou Wang, Lifeng Shang. Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in BERT. In International Conference on Learning Representations, 2020.
[71] Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404. 22244 2024.
[72] Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance RoPE: Improving context length generalization of large language models. arXiv preprint arXiv:2403.00077 00007 2024.
[73] Y Wang. D Ma, and D Cai. With greater text comes greater necessity: Inference-time training helps long text generation, arXiv preprint arXiv:240l. 2024.
[74] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom:
A 176b-paramen parameter open-access multilingual language model. arXiv preprint arXiv:22/1.05/00 2022.
[75] Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. Long context alignment with short instructions and synthesized positions. arXiv preprint arXiv:2405.003: 00392409 2024.
[76] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. InfLLM: Unveiling the intrinsic capacity of LLMs for under- standing extremely long sequences with training-free memory. arXiv preprint arXiv:2402.04617 2024.
[77] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024.
[78] Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, al. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. arXiv preprint urXiv:2310.09940 2023.
[79] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint urXiv:2309. 16039, 2023.
[80] Kai Yang, Jan Ackermann, Zhenyu He, Guhao Feng, Bohang Zhang, Yunzhen Feng, Qiwei Ye, Di He, and Liwei Wang. Do efficient transformers really save computation? International Conference Machine Learning. 2024.
[81] Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. arXiv preprint 2.Xi::2402.16111 2024.
[82] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 1 11328-11339. PMLR, 2020.
[83] Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found the middle: How language models use long contexts better via plug-and-play positional encoding, arXiv preprint 24Xv:2243304497, 2024.
[84] Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompt- ing improves reasoning in large language models. arXiv preprint 27X99.2304.09797 2023.
15

---

[85] Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers lear? a study in length generalization. In International Conference on Learning Representations, 2024.
[86] Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but not robustly. arXiv preprint a7Xiv:2402.09377, 2024.
[87] Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of Ilms via positional skip-wise training. In International Conference Learning Representations, 2023.
[88] Shiyi Zhu, Jing Ye, Wei Jiang, Qi Zhang, Yifan Wu, and Jianguo Li. CoCA: Fusing position embedding with collinear constrained attention for fine-tum tin-tuning free context window extending. arXiv e-prints, pages arXiv 2309, 2023.
16

---

A Broader impacts
Positive societal impacts. We propose method for length extrapolation, which will be helpful for mansformer-basee models to process long context.
Negative societal impacts. tions.
This method may be abused for other potential congecontext applica-
B Model Configuration
experiments are conducted on A800 GPUs. The 125M model configuration is the following.
Table 3: Model Contigurations.
125M
350M
Training sequence length
512
512
Batch size
32 x 8
32 8
Numer of iterations
50k
50k
Dropout prob.
0.0
0.0
Attention dropout prob.
0.0
0.0
Attention head
12
16
Feature dimension
768
1024
Layer number
24
Optimizer
Adam
Adam
Optimizer parameter betas
[0.9, 0.95]
[0.9, 0.95]
Learning rate
6c - 4
3c 0 4
Precision
f6
float16
17

---

C
Appendix: The Effect of the Hidden Dimension DDAPE
Hidden Size (Length 128)
Hidden Size (Length 512)
Saun Ins (A
Valication per plexity
Valliatioo per
1024 2048
512
1024
4096
8192
Validation sequence length
Valication sequence length
Figure 6: The effect of DDAPE: the performance with training lengths 128 and 512 the Arxiv dataset.
The experiments are conducted with Alibi and DAPE-Alibi.
D Experiments on Chomsky Hierarchy Evaluation Benchmark (CHE)
Following the framework established by [21, 57], we conduct evaluations of our DAPE on a suite of tasks derived from the domain of formal language recognition. These tasks include modular arithmetic, reversing and duplicating strings, binary operations and bucket sort. Based on the Chomsky hierarchy [16], these tasks categorized into distinct classes: Regular (R), Context-Free, Context-Sensitive (CS), and Recursively Enumerable. Each class aligns with specific computational models: Regular tasks are solvable using Finite-State Automata (FSA); Deterministic Context-Free tasks can be addressed by an FSA equipped with a deterministic stack; and Context-Sensitive tasks require an FSA complemented by access to a bounded tape.
Table 4: The example of different tasks. tit denotes permutation- invarriinn tasks, which expected to be solved without positional information. More explanation of tasks can be found in [21].
Level
Task
Example Input
Example Output
EVEN PAIRS
aabba
True
MODULAR ARITHMETIC (SIMPLE)
+ 2 - 4
Regular
PARITY CHECK!ti
anabba
True
CYCLE NAVIGATIONI
011210
2
STACK MANIPULATION
abboa POP PUSH POP aabba
abba
REVERSE STRING
abbaa
DCF
MODULAR ARITHMETIC
2) (4-3 3 (-2))
0
SOLVE EQUATION
2) (4 (-2))
DUPLICATE STRING
abaab
abaababaab 0
MISSING DUPLICATE
10011021
ODDS FIRST
aaabaa
aaaaba
CS
BINARY ADDITION
10010 + 101
10111
COMPUTE SQRT
100010
110
BUCKET SORTH!i
421302214
011222344
Problem Setting Building upon the framework proposed by Ruoss al. (2023) [57], we utilize the encoder-only configuration of the original sequence-to-sequene by Vaswani al. (2017) [69]. In scenarios that necessitate a multi-token output sequence y, such as the task of string duplication, we extend the input sequence by appending (y/ placeholder tokens. Subsequently, we compute the entire Transformer output from this augmented sequence without resorting to autoregressive sampling techniques. Training is conducted on sequences whose lengths are uniformly distributed, sampled from (11 with N set to 40. Evaluation performed on sequences that vary in length from N + 1 to M, where M equals 500. The implemented architecture comprises layers, 8 attention heads, and a feature dimension of 256. The dimension DAApe, is established at 64, while the maximum randomized position L is set to 2048.
Transformer model, as delineated
18

---

E
The Error Bar and Significance Value
Table 5: The perplexity performances on the Arxiv dataset when the training length is 512 and running with three random seeds.
Method
512
1024
2048
4096
8192
mean
4.5755
45.1974
134.1615
222.3333
265,4545
RoPE
std
0.0216
12.2241
21.8522
22.5955
20.3862
mean
4.5547
4.3730
12.1017
163.9289
595.0829
T5's bias
std 0.0204
0.0747
6.5327
136.8294
306.3671
mean 4.6146
4.5475
4.8693
5.0278
4.7679
Alibi
std 0.0278
0.0990
0.1246
0.1845
0,2196
mean 4.5262
4.1921
4.1068
4.1315
4.0013
DAPE-Alibi
std 0.0270
0.0653
0.0885
0.0617
0.2708
Kerple mean 4.5817
4.3504
5,4438
8.7511
13.3524
std 0.0252
0.0526
0.3187
1.1066
2.4109
mean 4.5123
4.1716
4.0505
4.0033
3,8642
Kerp-
std 0.0251
0.0637
0.0973
0.0333
0,2342
FIRE mean 4.5741
4.6953
30.1164
165.5394
308.6173
std 0.0227
0.1125
4.4317
41,2065
78.4652
mean 4.4879
4.1990
4.2826
4.7983
6.1623
DAPE-FIRE
std 0.0206
0.0619
0.0709
0.1701
0.9226
According Table the performances of different methods evaluated based perplexity across various validation lengths ranging from 512 to 8192. The results indicate that DAPE-Alibi consistently outperforms the Alibi method, DAPE-Kerple surpasses Kerple, and DAPE-FIRE shows better performance than FIRE, all with p-values less than 0.05, suggesting significant improvements.
DAPE-Alibi demonstrates lower perplexity values compared Alibi all lengths, indicating more effective learning and generalization. Similarly, HAPE--erple shows significant improvements over Kerple, especially larger lengths, where the gap in performance widens. The DAPE-FIRE method also shows notable enhancements over the FIRE method, particularly at higher lengths where standard FIRE struggles with increased perplexity.
Moreover, DAPE-Alibi, DAPE-Kerple, and DAPE-FIRE not only perform better than their respective original methods but also show superiority over RoPE and T5's bias across all validation lengths. This suggests that the modifications implemented in DAPE versions provide a more robust and generalized model, capable of maintaining lower perplexity and thus better performance the Arxiv dataset.
In conclusion, the statistical analysis confirms that the proposed DAPE variations offer significant improvements in perplexity performance, thereby validating their effectiveness in comparison to their original counterparts and other baseline methods.
F Data-Adaptive Related Position Encoding Performance Comparison
Table 6: The performance comparison between data-related position encoding, with dataset Books3 and training length 128.
Method
128
256
512
1024
2048
4096
8192
Transformer-XI.
31.57
28.49
26.07
26.98
27.90
32.76
41.12
CoPE
31.61
28.41
25.79
27.96
33.80
54.08
90.66
DePP-Kerple
31.49
28.27
24.93
24.31
23.34
24.38
25.01
19

---

According to the experiment, the transformer-xl achieves good performance on length extrapolation. Therefore, this also suggests that the position encoding should interact with attention/query/key to further improve the performance.
G Ablation Study on Bias Matrix
Table 7: Books Dataset Results: Train Length 512
512
1024
2048
4096
8192
QK" + B (baseline)
19.68
19.06
20.44
28.34
39.31
QKT + B f(B) QK" + B B(QK",, B)
19.64
18.83
18.49
20.62
23.49
19.22
18.22
17.15
17.63
17.88
We further conduct an ablation study on the f. proving that ff help enhance the bias matrix. DAPE improves the bias matrix for A finats while the A Afinal used calculate A finat For the unseen position, the partially could handle it (FIRE changes the problem to interpolation), but is not accurate enough that DAPE helps enhance the bias matrix B via attention score. The experiment suggests two points: 1) The DAPE (QK B) is better than naive (B), suggesting that the context- adaptive is important. 2) The QK + (BB better than QK + B, suggesting that benefiting from improving the expressiveness of the bias matrix.
H
The result on 2.7B and 6.7B
Table 8: Model Size and Method Comparison with Training Length 512
Model Size
Method
512
1024
2048
4096
2.7B
RoPE
21.01
25.00
48.13
160.59
RPE
21.10
21.88
23.59
33.23
Alibi
21.23
22.17
22.91
23.22
Kerple
21.14
22.08
23.38
27.21
DAPE-Kerphe
20.52
21.01
20.23
19.57
Table 9: Model Size and Method Comparison with Training Length 512
Model Size
Method
512
1024
2048
6.7B
RoPE
20.86
22.27
28.01
RPE
20.79
21.60
22.32
Alibi
20.79
21.63
22.45
Kerple
20.71
21.57
22.07
DePE-Kerple
20.09
20.54
19.83
According to the result, we can find that the proposed DAPE still works well, whatever the model size is 2.7B 6.7B. With 2.7B model size, RoPE achieves 21.01 evaluation length 512 and 160.50 on evaluation 4096, while our DAPE- DAPE-Kerple achieves 20.52 and 19.57 respectively. Also, Kerple achieves the best performance, whatever the model size is 2.7B 6.7B from evaluation length 512 to 8192. This suggests that our proposed DAPE has great scalability.
20

---

I DAPE Visualization
The model is trained with DAPE-Kerple on length 512.
I.1
Visualization length 512
Exampte Attention
Example Kerple
Dample 21 DAPI tis Hoyyer
DAVR Podltooel fllee
Nivettn Valu
ma.55
-20
Rad
Good
300
200
400
300
100
200
303
402
500
1.0
200
305
500
Enamphe (Layerr
Example Kerple (Layer
Kerpte Soottoons Bit.
Altention Value
DWP Restioonll
.ood 30
200
200
300
Ieelad Disiarice
Delutive Distance
Pedaiiiv Diliance
Example Attention Baayer
Example Korple Blas (Layer 3)
Cuargle HLayer
100
Medd3
DAPE Fooflooal
-t0
Aeterrtinn
Macc
made
-15
madd
Hadd33
Mard
303
Relative
Relative
Rebstive Distance
Exampic 1. Attention LLayer
(Layer
Examett DUPE
10g soconoo 90200
DAYE Rechimal fllme
Attondoon Value
Prepee wwwph May
-10
-ood
385
200
300
480
500
160
200
301
601
5:00
520
Belatrvw
Distance
Esompte 1: Attention (Layer 5)
Example Korple Bios (Layer 5)
Korple notiooo Bibs
Alteritee Value
DAPF Poatiiona
maad
-20
Relotive Distance
Relative Distance
Relalive Distance
Figure Evaluation Length 512 Example Part
21

---

Eexmple Aitection Hayer
Example 1: Karple Biss (Layer
Kariee Nostonal Bios
74.
Altection Valto
Paad
DAPE Poellaan
March
Houd-33
Mord
Redlattvve Distaicce
Relative Distance
MARIMEO Distance
3
Exampte Attention
Example Kerple
PAAINRO DUMPO 9AN
madd3
Kerple Foottoral Bes
mad
Niwnbon Value
Hood
D4V1 Pow
Read
32
200
302
1.00
200
305
50c
(Layer
Example Kerple (Layer
CIaTd DANC (Lawe
Altertion Vnlue
Korple Postonss
CAPE Tokitornl
2
Mard
350
200
300
100
307
402
Iulaaive Disiance
Palutive Distance
Example 1: Attention (Layer
Example 1: Karple Blas LLayrr 9)
Examete
Head-1
Korple Pusitora B8is
madd
mead
DATE Bick
head5
mad--
Head
Head.
Head 2::
300 Refettive
Relattire
Relstive
Gaamaao
Hovi
Positionaa
Hesd-1
Heap-
-20
Warsie
Trad-22
Heso
20-
100
266
300
403
320
303
400
503
164
200
300
460
Rolotive Distance
Rebitrr Distance
Relative Distance
Figure 8: Evaluation Length 512 Example Part
22

---

Example I: Artertiion (Layer 11)
Exampie I: Kerple Bias [Layer 11)
Exampl
Keppl Postiiona Bias
Med33
D4Pt heelteeal Blee
Hood
Ni
Heod
Houd
Hewu
250
Barket Distance
Reletive Bistance
Relation Distance
Exammie Attertion
Examate Bas
Example DANC thi Eaaper
Korplla Puzorra
-18
Hesc
-25
-35
152
130
233
360
460
560
100
2/6
915
410
150
500
Distancc
Diciaree
Figure Evaluation Length 512 Example Part
23

---

Example (Layer
Exarmple Kerple (Layer
0001
Kelple Foottorral Bass
DAVE Foochooaa
Nowbion
30
220
Dhelad Disiartcc
Pelutiso Distance
Example Attention [Layer 21
Examppee Kerple Bies (Layer
Guanpli Elayer
Kerple Postonnl Bas
Med
DAVe Postioma
Aetertion
-10
300
301
Relstive
Example Altientioon (Layee
Example Korple (Layer
Gramato DAPG
Redd
Hadd
Kerpe Rosttioral Biss
Bomeel Pochona fllee
soot
J00d 10
306
400
590
150
200
301
601
5:00
303
400
500
Relativee Distance
Relative Distance
Rebittive Distance
Exampte (Layer
Exemple 2: Kerple Bies (Layer
Duameli fLayer
Kerple Fostrooal 8:35
Altersion Value
DAVE hisiband
V
Maad-12
Dtelation Distamcce
Delucivo Dislance
Btal Dislancc
Exampple Actention Bayyer
Korple (Layer
Gammols 21 DARI Bisn ELRyer
Head3
Kerpe Posittona Bioe
DXP Postiooall Blte
Altonsonn Voloe
0.8-
Radd
V
mad5
Hood 12
384
200
220
400
526
350
200
300
402
530
1.00
310
303
400
550
Example [aayr
Examppe Kerple (Layer
CIaToR (Laard)
Kerple Foottonal
Altantion
2004 32
DAVE
Ptelattio Distance
Peluclio Distance
Raelel Distance
Figure 10: Evaluation Length Example 2: Part
24

---

Example Layer
Examppee Serple (Layer
0001
DArE Dlas
Kempe Foottorral
Altention
100d 30
-10
Faelattiv Distance
Pelutivo Distance
Flad Disiiarie
Example Attention [aayer B)
Example Kerple Bies (Layer
Kerple mostoon Bay
Actertion
mard-3
300
301
Relstive
Example (Layer
Examppe Blas (Layer
Enampiic SLagper
Heads
Heads
Kerpia KarplaPPsitti Posiboral 863
Proc Lenelaaelli
1.0
300
100
500
100
200
106
110
500
166
200
200
Relative Distance
Relotive Distor ce
Refottre Distance
Exsemple 2: Albertier Baayer 10)
Exommle 2: Kerple Bas (Layer 10)
0.6
-25
Hood)
terpl Ruzderra
Head
DAVE Itsitaad
Head
2.1
Head 3
-20
Hiad 12
Dalaticc Disturcce
Dalatiie Dilanne
Deelatice Distarcc
Esoemple ayee
Example Korpte Blas [Layer 11)
Esaypplo DARS Sugge
Head3
10.0
Kerpe Posittora nise
DXP Poddicoal Blax
Altonionn Volue
Had5
mad
mad5
Hood 12
-22
304
200
220
400
520
350
200
300
402
500
1.00
380
303
400
550
Exeempgle BLager
Kerple (Layer
Hesp
Berple Po Booral DILG
Head-
Dirt Pass Stnmlll 6am
Hesc
Hesc3
100
Bolatiiee Dillanee
Balai Dislance
Seletive Distanic
Figure 11: Evaluation Length Example 2: Part
25

---

1.2
Visualization on length 2048
Esample E: Aleetion (Layer 13
Exemple 1: Kerple Bies (Layer
fLayer
Kerie muttonal
DAVE Pomeee
Altenitnn
Deeldd Disience
Delutivn Dislance
Dolalive Diilance
Example Attention [Layer
Korple (Layer
Dasreds 21 DAPG Fis Haayer
He.d3
Kerpee Korple Posttorr mise
Ha.55
DXPF Posdioonl
Attendaa
5 1.
mad
-10
-25
oood 12
-10
540
3000
2500
2006
500
1000
2003
2700
530
2003
1.500
2006
Exampte Atteention (Layer
Example Kerple (Layer
Korple Froittonal
-20
Altendann
-20
100 10
DAY
1000
Selatiiv Distance
Relucive Distance
Relalive Discance
Examplle Allanniion (Layer
Exampic Korple Blas (Layer 4)
Gannds Elayer
Keeple Post miss
DAPE Foodioona
-10
Mitantonn
Hadd
madd
Madd-1
2000
Distance
Relative Distance
Rebstivee Distacce
Example (Layer
Ciemete DAPS FLayer
Redd3
Jarple Pestttonal Bixs
DAPE Recthmal
Nownkoor
-10
30
-2
1600
15.00
3020
620
1000
1090
2000
1003
2000
Divtance
Deletive Distance
Dedaalive
Figure 12: Evaluation Length 2048 Example 1: Part
26

---

200
Example Attention Layer
10013
Danpo 22 BAC Sa Sardey
Korsle Posttorn
DET Rositerel
Aftentoon
mood
Heau
1006
2006
530
1000
1000
2000
3000
2000
Reelativw Sistance
Distance
Exompte Attention (Layer 7)
Example Korple Bias (Layer
korple notioraa Bibs
Altertion Valu
DMPT Foaltoona
madd-2
madd::
-22
2030
Retotive Distance
Relative Distance
Relative Distance
Esample 2: Altention (Layer
Exemple Kerple Bies (Layer
Cuameli 21 DUMPO FLAVVY
Hedd
Korste Fosttonna Bas
DATE Hirnal Blee
-20
1.
Hood 12
2030
1660
Baaeer
Blas (Layer
Enarpie Di Raper
Head
Head2
Karate boctonal Bls
DAPE mosaii 1aa
Value
Heaz5
Head
Head-7
mead0
Trad-12
God
550
2005
1300
2003
500
3000
2008
Relotive D Dictanc
Relotive Diston de
Retottve Distance
Excmpie 1: Attention (Layer 15)
Exampte Kerpie Bias (Layer 10)
Gaver
Korple CupplaPozoe Rusiboral 855
-20
Delatice Ditance
Beleiivw Dislance
Relative Distaric
Figure 13: Evaluation Length 2048 Example 1: Part
27

---

Exampie I: Kerple Bias [aaeer 11)
Example
Kepple Rosttions Biss
Med33
DIPE Treiliieal Blee
Aachicon Value
mood
100
Hood
Hewu
-20.0
Reletive Dishance
Relative Bistance
Relattvv Distence
Exampte Attention
Example
Head
Narple Fosti Blas
Head5
Head0
Head112
-20
35c
2003
1500
2000
560
3200
1300
2000
560
1o00
1000
2000
Figure 14: Evaluation Length 2048 Example 1: Part 3
28

---

Exampptee (Layer
Exarmple Kerple (Layer
1001
Ke Foottoral Bass
DATE Fow.hooalaBli Blas
Riwnioon Value
32
2000
10:0
1060
Ieelltive Disiance
Pelutive Distance
Fldd Disiiance
Example Attention [Layer 21
Example Kerple Bies (Layer
Ganrle
125
Kerpte Rostonnl Bas
Mad
DAPE Rostiom
Actertionn
10
-5.0
2006
5.0
Relative Distance
Esample 2: Altention Layyer
Example Korple (Layer 3)
Gaareto DAPG HLayer
10.0
Ha.
Kerple Foottoonal Bix
Altondon Volue
DXPF Boose
10:04
2000
1.000
2008
5:0
1060
1000
2000
5.0
2006
Relodive Oiitance
Relative Distance
Relative Distance
Exampte (Layer
Exemple 2: Kerple Bies (Layer
FLaVer
Altention Walu
Korple Korple Footroool
DWVE Iistbond
L.
Maad-12
ma
22
-32
Deelation
Delutivn Dislance
Etelaiive Disiancie
Examppee Actention Laayer
Example Korple (Layer
Danrols 21 DANI BiAN ELRyer
10.0
Hea
Kerpl Postttora Bioe
Alc-noon Volue
Had5
DDP Posdtoall
0.8
madd
mad5
-30
-15
-ood 12
504
266
500
1000
2503
2700
530
1003
1.500
2009
Example Kerple (Layer
Exampic 2: Actention LLayer
Merpte Roontoonal 835
Alcomtinn V2U0
DIVT Revermad
1wod 32
-10
1000 Reletive Distance
1000 Pelucivv Distance
Pelalive Dicanne
Figure 15: Evaluation Length 2048 Example 2: Part 1
29

---

Example (Layer
Exampple Serple (Layer
Rombion Value
DATE Froe Frchioeaalll
Ke Foottoral
100
10od 30
15
10:0
2030
1060
Hedal Disiancc
Pelutive Distance
Flad we Disiiarci
Esample Alveniin (Layer
Example Kerple Bies (Layer
Guanpli
Kerste Rostonnll Bas
Altenton Value
DAVE Restioma
mard-3
0.
mard-3
Retotive
Relative Distance
Example (Layer
Blas (Layer
Head
Head2
Karpi Potibraa 863
Ha225
Hest
1003
3500
2000
560
1008
1500
2003
0000
12:00
Retative Distance
Relotive Diston
Rebative Distance
Exummlee 2: Altertior Bayer 10)
Esumple Kerple Bias BLayer 13)
Karsle Pootonna 6.00
Headi1
DATE
Delation Disience
Relakiow Distance
Relattie Distarrcc
Exampte
Example Korpte Blas
Esnyple DAMS Suager 301
Kerpe Postooona mise
DXP Podttooll Blte
Mbeetio Wabue
Had5
RaddE
-20
mad55
ma
Hood 12
500
2000
1500
2030
500
1000
2503
2700
530
1003
1.500
2000
Exeemple Baager
Example Kerple (Layer 12)
Commple
Korpta Resitional Bazz
125
-158
526
15:0
Relative Datance
Relaiive Disiance
Pelai Distence
Figure 16: Evaluation Length 2048 Example 2: Part
30

---

1.3
Visualization on length 8192
Esample 2: Alvetion (Layer 13
Exemple 1: Kerple Bies (Layer
(Layer
Kerple nustonal Bas
DAVE Iostbomdl
Alteniten
Maad-33
Beldd Disiance
Delutivn Dislance
Tolalive Diilance
Exampte Attantion
Korple (Layer
Dasreds 21 DAPG Bis Huayer
He.d3
Korple Frotttons Kerpe Korple Postitora Mise
Miwntan Vakue
Ha.55
Posthool
Rad
ma
1ood 12
2008
4000
5000
5020
2002
6000
6000
8700
3008
4003
5000
5006
Example (Layer
Example Kerple (Layer
Atlention Velue
DAP Rostkoal
0-
100u 10
10000
40.0
Itelal rvwl Disiarice
Relucive Bistance
Redaaiive Diciance
Example
Exampic Korple Blas (Layer 4)
Grands ELaver
Keeple Postoonal Bass
Aboonton 10
DAPE Foattooal
Hedd3
Hadd
-0.6
1
Madd-1
Rotodiv
Distance
Rebstivve Distance
Example Attention Baaer
(Layer
Ciamattc DAPS
Redd3
Jorple Pestttonal Bixss
DAPE Recthmmll
k. L.
10
2200
4000
6200
1000
2000
4000
60b3
8000
2000
3000
1000 5000
Retartio
Deletion Distance
Deelalive Distanc
Figure 17: Evaluation Length 8192 Example 1: Part
31

---

Example I: Attention (Layer
100d3
Korsle Pooitfens
DMP Rosilamm
Attwrdan
mood
2008
6000
6600
5030
2000
4000
6000
8200
2000
8003
Distance
Example Bayer
Example Korple Bias (Layer
10.0
Medd
Karple Notiooma Bibs
Attenaa Value
DAPF Boadhoonal
-10
madd-3
madd
Relative Distbince
Relative Distance
Relottve Distance
Exampte 1: Attenttion (Layer
Exemple Kerple Bies (Layer
Cxamplic 21 DUAIO DAIS FLIVER
Korple Morste Fosttonal Bas
Altendam Vine
swort
DAYE Pre
Hood
5030
2000
4060
2000
Exemple T: Altertion 3 Bayeer
Blas (Layer
Ecorele DAPE
Head
Head2
Karple bockonal Bls
Head5
mead
Head:
mead
Head-1
2002
4005
6000
8003
1203
4000 Rebative Distancs
Del Aeteen sslncce
Relotive Diston de
Attontion
Exampte Karpie Bias (Layer 10)
Gaver
16.0
179 2oA.D midoy
5.0
15-
-20
-30
Relstive
Beleiivw Dislance
Relatice Distarcc
Figure 18: Evaluation Length 8192 Example 1: Part
32

---

Example 1: veen (Layer
Exampie I: Kerple Bias [laeer 11)
Altention Valtow
Kepple Rottiona Biss
1
Med33
DiPt Teeilieell Biee
meod
-10
Hood
Hewd 32
Petattion
Reletive Bistance
Delativv Distance
Attention
Esample
Head
Narple Fostoraa Blus
Head5
16820
-30
Head0
Head112
2000
4030
0000
8000
2003
4200
6000
0000
2003
4000
0000
8000
Figure 19: Evaluation Length 8192 Example 1: Part 3
33

---

Example (Layer
Exarmple Serple (Layer
1001
Ke Foottorral Bas
-00
DATE www.hoallll
Nimnion
-20
32
-00
4000
Dela ve Disiance
Pelutive Distance
Italalive Daciance
Example Attention [aaer 23
Example Kerple Bies (Layer
Ganrls ELayer
200
Korple fostions Bas
DAV Pestimme
Actertion
marb-33
5000
2000
Relative Distance
Example Attient lon (Layyer
Example Korple (Layer
Ganeel 2: DAPG HLayer
Korple Fosttoonl Bixs
DXP Pochonal
10
2008
0000
0050
2000
4060
60b3
8200
2000
4002
6900
5000
Retattive Distance
Relative Distance
Relative Distance
Exampte (Layer
Exemple 2: Kerple Bies (Layer
fLayer
Altertion Walue
Korple Foott Proste
-10
Maad-12
Dellttion Distance
Delutivn Dislance
Todalive Dislanc
Exampte Attontion
Example Korple (Layer
Ganell 21 DAPG Ble Huayer
Kemp Posittonal miae
DXP Podiiooal Blte
Mbeetion Value
Had5
1. 2
10
mad5
sood 12
2008
400D
5000
3000
2302
4000
6000
8000
2000
4003
5000
DO0G
Distance
Exampte Attentioon (Layer
Example Kerple (Layer
Kerple Roottons Ba5
DIPE Reseiomd
-30
4000
Delative Dislance
Delucive Distance
Tollllive Discanc
Figure 20: Evaluation Length 8192 Example 2: Part 1
34

---

Example Layer
Exarppee Serple (Layer
100
0001
Altention Velue
Ke Foottoral
Dirt Fochioaa
-0
1ood 30
Faelattion Dtotance
Pelutive Distance
Flad Disiiarci
Example Alvention (Layer 8)
Example Kerple Bies (Layer
Guampl
Kerple Post tonst Bass
Altantoo Valee
DAP Reshoomd
-10
mard-32
2000
Retotive
Relative Distance
Example (Layer
Blas (Layer
Ecorpbi
Head
Head2
Korpia Pusiboral Blas
D30T Pow.booe
Attortion
Hest
2000
1900
GOCO
1000
2003
6000
6000
8000
2003
4000
oedo
Retative Distancc
Relotive Distor
Rebative Distance
Ecample Bayer
Esumple Kerple Bias BLayer 12)
MardlePoo Postonal
Narsle
Head11
Head-1
Hellatioon
Relakiow Distance
Delative Distaric
Esoemple on Layer
Example Korpte Blas
Esnyple Suager
10.0
Hedd
Kome Posttoora mise
AMcnnonn Volue
Had5
Roditoaal
mad55
Hood 12
-20
-20.0
3300
1060
2308
4000
6000
8000
2008
4003
5000
5006
Exeemple Bager
Exomple Kerple (Layer 12)
Korpta Rositional Bax
10-
-20-
2000
2000
6000
Belative Dittanee
Relaiive Disitane
Deletive Distancc
Figure 21: Evaluation Length 8192 Example 2: Part
35

---

Implementation
In this section, we present the implementation of the proposed DAPE module in PyTorch [49].
import torch
import torch nn
class DAPE nn :odule):
def -iini (self, head number=12, mlp_ mlp_width wowwhihhhh2 =32):
DAPE attention bias module.
Args:
num heads: number of attention heads. mlp_width: Width of MLP. u*n
super (DAPE, self) -nit)__
self. mlp Sequential mn. Linear (2+ humbee width). nn. Le.klyyelLLU nn. Lincar (mlp_width, num_heads)
def forward(self,, attention: torch, Tensor, bias: torch, Tensor):
Args:
attention: input sequence, which is q-
shape [bsz, num_heads, seq_len, seq_len] bias matrix, which can be generated by Alibi, Kerple FIRE or other additive position encodings shape [1, num_heads, seq_len, seq_len]
bias:
Returns:
attention with DAPE, shape [bsz, num_heads, seq_len, seq_len]
bias bile tile rbpas "1 'hTT TTT,, b=attention shape [0])
Concatenate attention and bias intention_bias_cmme
torch. eat ((attention, sile_ dim=l)
Rearrange the dimensions for MLP processing
attention_bias_conee = rearrange (atttntionnniaae (attention_bias_concat,, bb h T
T -> b bTbb)
Apply the MLP
attention bias concat = self. melf.mlp( attention bias sias_concat)
bias_concat
Rearrange back to original dimensions attention_bias_conee conca rearrange cattention_bias_connal h >> bhTTT)
return attention + bias + attention_biase biw.bbaassscaee concat
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have stated the claim in Abstract and Introduction part (Section 1). We also conduct extensive experiments Experiment part (Section 4)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The discuss the limitation in the Time Cost part (Section 4.5) of Experiment.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yes)
37

---

Justification:
We have prove the expressiveness of our method in the Method part.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we have proved the implementation in Method (Section 3) part and model configuration in Appendix Model Configuration.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

reproducible, they Answer: [Yes]
Justification:
We have provided the pytorch implementation code in Appendix Section J.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
mublic/guides Saliidee/CodeSummiis wwwssssssslioooooooooooooiiissil for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /ttppscccppu....///// ctcccc/ppoo wwtpccccppccccccee.ec..ec..ccllt wwwpc//ggg////eeeeeeeeeeeeeeeeee
gguides/CodeSubmi /oodeSubmeeeeeeeeeee ssssinnn aeionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the experiment setting in the Experiment (Section part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the paper, we report the mean perplexity values and standard variance Ap- pendix Section E. Additionally, we also report that the proposed method shows statistically significant improvements over other static PE methods (according to the p-value less than 0.05).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
39

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute resources in the Time Cost part of Experiment (Section 4) and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:s:////uuuii /wwrris.....///.
Answer: [Yes]
Justification:
We have followed the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed it in the Broad Impact section, which is Appendix Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we have cited the related works.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "NEjV3P1L8g", "reviewer": "Reviewer_tTna", "summary": "This paper offers a doable solution for long context tasks. The paper introduces the Context-Adaptive Positional Encoding (CAPE) method to enhance transformer model adaptability and flexibility in processing long input lengths and contexts. To overcome the limitations of static positional encodings such as Absolute Positional Encoding and Relative Positional Encoding, the proposed method adjusts the positional encodings based on input context and learned fixed priors. Experimental evaluation on Arxiv, Books3, and CHE shows that the proposed method significantly improves model performance in length generalization.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The long-context ability of LLMs is crucial and fundamental for advancements in the LLM domain, significantly impacting downstream tasks.\n2. The proposed method is intuitive and straightforward to implement and understand.\n3. The performance of the proposed method, as indicated by PPL, appears promising and is well-demonstrated in Figures 2, 3, and 4.", "weaknesses": "1. The major concern is in the experimental evaluation part. The experiment is limited to PPL, which is insufficient for verifying long-context ability.\n2. The experiments are limited to a very small LLM model - a 124M transformer, which is not enough to verify the proposed method, as large LLMs and small LLMs behave quite differently.\n3. The proposed method is not convincing to me. The proposed method is mainly in Equations (2) and (3). In Equation (2), the proposed method does not seem promising. Considering the next step ( $h(*)$ ) in the neural network, it will be $ h(A_{\\text{CAPE}}(X)) = h(XW_Q(XW_K)^\\top + f(XW_Q(XW_K)^\\top, B)) = g(XW_Q(XW_K)^\\top, B) $. Could the author please elaborate on this?\n4. Similarly, for Equation (3), $ A_{\\text{CAPE}}(X) = XW_Q(XW_K)^\\top + B + f(XW_Q(XW_K)^\\top, B) $, it has a similar problem to Equation (2).\nDue to the major weakness in the experiment and the unreasonable proposal of the bias term, I would like to reject this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tTna,\n\nThank you for the detailed review. We will address your concerns below.\n\n**Q1: The major concern is in the experimental evaluation part.**.\n\nA1: We have presented the experimental results besides PPL, as shown in Page 9 Section 4.7 and Appendix D Experiments on Chomsky Hierarchy Evaluation Benchmark (the benchmark and its variants are used in previous works), which **uses accuracy as evaluation metrics.**\n\nFour convenience, we directly copy the result of CHE Benchmark here. We train with 200K steps and length 40, while we test on length 500.\nThe random accuracy is 50%, except for modular arithmetic simple, cycle navigation, bucket sort, solve equation, and modular arithmetic brackets, where it is 20%. \n$\\dagger$$\\dagger$$\\dagger$ denotes permutation-invariant tasks, which are expected to be solved without positional information.\n\n| Level | Task | Learned | sin/cos | RoPE | Relative | ALiBi | Kerple | FIRE | | Alibi | Kerple | FIRE |\n|-------|------|---------|---------|------|----------|-------|--------|------|-------|--------|-------|------|\n|       |      | **Randomized**  | **Randomized**  | **Randomized**  | **Randomized**  | **Randomized**  | **Randomized**  | **Randomized**  | | **CAPE (Ours)**  | **CAPE (Ours)**  | **CAPE (Ours)**  |\n| R     | even pairs | 50.04   | 91.27 | 99.98 | 96.60   | 73.52  | 57.50  | 73.86 | | 99.99 | 99.58  | **100** |\n|       | modular arithmetic simple | 19.95   | 20.39 | 21.35 | 20.84   | 20.02  | 21.79  | 21.09 | | 23.58 | **24.47** | 24.46 |\n|       | parity check$\\dagger$$\\dagger$$\\dagger$ | 50.14   | 50.52 | 50.05 | 50.09   | 50.09  | 50.07  | **50.97** | | 50.30 | 50.07  | 50.04 |\n|       | cycle navigation$\\dagger$$\\dagger$$\\dagger$ | 24.97   | 25.37 | 27.63 | 26.95   | 24.64  | 29.47  | 28.41 | | 22.99 | **34.53** | 27.54 |\n| DCF   | stack manipulation | 59.92   | 65.92 | 61.49 | 64.73   | 66.42  | 66.93  | 69.33 | | 68.18 | **72.04** | 70.90 |\n|       | reverse string | 52.76   | 67.28 | 65.23 | 65.59   | 71.09  | 71.54  | 65.89 | | 73.37 | 70.74  | **76.40** |\n|       | modular arithmetic brackets | 31.00   | 30.70 | 31.25 | 31.74   | 30.56  | 24.79  | 30.92 | | 31.34 | **32.37** | 31.50 |\n|       | solve equation | 20.00   | 19.97 | 21.85 | **22.93** | 19.92  | 21.15  | 22.06 | | 20.03 | 22.49  | 22.42 |\n| CS    | duplicate string | 52.77   | 65.44 | 64.97 | 67.66   | 65.13  | 66.72  | 69.03 | | 70.84 | **72.95** | 72.71 |\n|       | missing duplicate | 50.38   | 49.78 | 63.37 | 72.34   | 74.21  | 79.06  | 79.27 | | 83.41 | 87.57  | **89.17** |\n|       | odds first | 52.77   | 58.61 | 61.00 | 61.57   | 59.88  | 62.59  | 63.28 | | 63.78 | **67.08** | 66.34 |\n|       | binary addition | 54.63   | 55.78 | 55.59 | 56.96   | 54.72  | 56.35  | 55.70 | | 59.71 | **60.88** | 56.62 |\n|       | compute sqrt | 50.47   | 51.11 | 51.88 | 51.63   | 50.63  | 51.11  | 50.80 | | 51.64 | 51.33  | **52.46** |\n|       | bucket sort$\\dagger$$\\dagger$$\\dagger$ | 98.32   | 98.92 | 98.12 | 99.31   | 98.45  | 99.38  | **99.57** | | 99.38 | 98.81  | 99.37 |\n\n\n**Comparative performance improvements.**\nCAPE consistently enhanced performance across various tasks, especially on permutation-variant tasks. Specifically, CAPE improved upon Alibi and FIRE's results in all 11 tested permutation-invariant tasks. Similarly, it outperformed Kerple in 10 of these tasks. \n\n**Q2: Experiments on large model size**\n\nA2: The following is the result of the experiment on 2.7B and 6.7B, with training length 512 and micro_gpu_batch_size 4. We further discuss the experiment on large model size in **Result on Large Model Size 2.7B and 6.7B** in **Author Rebuttal by Authors**\n\n| Model size| Method | 512 | 1024 | 2048 | 4096 \n|-------|-------|-------|-------|-------|-------|\n|2.7B|RoPE| 21.01|25.00|48.13|160.59 |\n| | RPE|21.10|21.88|23.59|33.23| \n| | Kerple|21.14|22.08|23.38|27.21|\n| | CAPE-Kerple|20.52|21.01|20.23|19.57| \n|6.7B|RoPE| 20.86|22.27|28.01|110.00  \n| | RPE|20.79|21.60|22.32|26.31|\n| | Kerple|20.71|21.57|22.07|24.48|\n| | CAPE-Kerple|20.09|20.54|19.83|19.32|\n\n**Q3: Explanation of Equation 2 and Equation 3**\n\nA3: The next step of our operation is $softmax$. Therefore, $h(A_{CAPE}(x))=softmax(XW_Q(XW_K)^T+f(XW_Q(XW_K)^T, B))!=softmax(XW_Q(XW_K)^T+B)=h(A(x))$, while $A(x)$ is the naive transformer attention score calculation without CAPE.\n\nLet us explain the attention module in Transformer step by step:\n* Step 1 (Calculate attention score via query $XW_Q$ and $XW_K$): \n    *  Previous implementation of $A_{score}=XW_Q(XW_K)^T+B$\n    *  CAPE implementation of     $A_{score}=XW_Q(XW_K)^T+B+f(XW_Q(XW_K)^T, B)$\n* Step 2 (use softmax on attention-score, row-wise. Therefore, the **next step is** $\\textbf{softmax}$):\n    * $A_{scoreSoftmax}=softmax(A_{score})$\n* Step 3 $A_{scoreSoftmax}$ and value $XW_K$ to get embedding of each token:\n    * output=$A_{scoreSoftmax}XW_V $\n   \nIf we understand correctly, the neural network h(∗) mentioned in the comment corresponds to the feedforward network layer (FFN), that follows the attention layer. We would like to clarify that the MLPs introduced in the CAPE model are not duplicated. In CAPE, these MLPs dynamically adjust the positional encodings based on context information (**CAPE: Step 1**). Subsequently, softmax operations are applied across attention score, row-wise. It is important to note that the CAPE's adjustments, applied during the attention phase, do not directly alter the token values but act on the attention computation. However, the FFN layer that follows attention modifies (**FFN: after Step 3**) each token through a nonlinear transformation.\n\nIf there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5647enCFX", "reviewer": "Reviewer_CfZF", "summary": "The paper proposes context-adaptive positional encoding. The paper proposes a 2-layered MLP to non-linearly integrate positional bias information (can be computed using prior methods like Alibi, FIRE, and Kerple) and query-key content-based dot product values representing semantic relations across different heads to dynamically create a contextual position-informed matrix that can additively modulate the self-attention matrix.", "questions": "Minor suggestions:\n\n* It Would be good to refer to the appendix implementation section G when discussing function f and the multi-head CAPE in the main paper. \n* Perhaps it is better to present the pseudocode with einops-style syntax.", "limitations": "There isn't an explicit section purely for limitations, but there is a section for computational cost analysis, which may illuminate some limitations (and authors treat the section like a limitation section in the checklist). Overall, it's mostly adequate.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is reasonably motivated.\n2. The empirical results are promising for length extrapolation in language modeling. The experiments on Chomsky's hierarchy tasks are a nice touch.", "weaknesses": "1. If I understand correctly, the original relative encoding (from Shaw et al. and then the one in Transformer XL) could also count as contextual. There, the position-based additive values are computed based on the dot product of queries and distance encodings (treated as keys). So, the context can influence the distance-related bias through the query representations. This could be better discussed in the paper. I would also be curious how Transformer XL style relative encoding would perform in the language modeling datasets. \n\n2. There is some hit in the computation expense. Although it is manageable, the authors don't treat each head independently but use the concatenation of heads with a shared hidden state.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CfZF,\n\nThank you very much for appreciating our work. We will address your concerns below.\n\n**Q1: The performance of transformer-xl relative encoding**\n\nA1: We have shown the performance of transformer-xl below, which also presents great length extrapolation performance. Also, the Relative in Table 2 is just the transformer-xl  relative encoding. The experiment is conduct on 125M model with training length 512 and batch size 32.\n| Method     | 512 | 1024 | 2048 | 4096 | 8192 |\n|------------|-----------|-----------|-----------|-----------|-----------|\n| Rope       | 19.75     | 261.39    | 411.24    | 635.80    | 762.86    |\n| T5's bias  | 19.67     | 19.45     | 33.41     | 141.94    | 347.36    |\n|Transformer-XL|19.40|19.23|19.17|21.21|23.23|\n| Alibi      | 20.04     | 19.75     | 20.17     | 20.50     | 21.31     |\n| Kerple     | 19.83     | 19.20     | 20.49     | 28.33     | 40.95     |\n| FIRE       | 19.77     | 21.09     | 103.14    | 308.58    | 484.55    |\n| Cape-Kerple| **19.25**     | **18.28**     | **17.20**     | **17.58**     | **17.85**     |\n\nAccording to the experiment, the transformer-xl achieves good performance on length extrapolation. Therefore, this also suggests that the position encoding should interact with attention/query/key to further improve the performance.\n\n**Q2: The computation expense**\n\nA2: Thank you very much for your precious comment on the compuation cost. Yes, we concatenate the attention and bias on the head dimension, and then we use an mlp to process them to dynamically adjust the position encoding values. We have further analyze the CAPE cost in **CAPE computation cost** in **Author Rebuttal by Authors**.\n\n**Q3: Mention appendix implementation section G when discussing function f and the multi-head CAPE in the main paper.**\n\nA3: Thank you very much for your suggestion. We will revise the paper with the following sentence:\n\n**Original:** It then outputs $h$-dimensional vectors, where each element corresponds to the CAPE for the respective head.\n\n**Revised:**   It then outputs $h$-dimensional vectors, where each element corresponds to the CAPE for the respective head. We have shown the code implementation in Appendix G.\n\n**Q4: Present the pseudocode with einops-style syntax**\n\nA4: Thank you very much for your suggestion. We will add the einops-style syntax in Appendix G.\n```python \nimport torch\nimport torch.nn as nn\nfrom einops import rearrange, repeat\nclass CAPE(nn.Module):\n    def __init__(self, head_number=12, mlp_width=12):\n        \"\"\"\n        CAPE attention bias module.\n\n        Args:\n          head_number: number of attention heads.\n          mlp_width: Width of MLP.\n        \"\"\"\n        super(CAPE, self).__init__()\n\n        self.mlp = nn.Sequential(\n            nn.Linear(2 * head_number, mlp_width),\n            nn.LeakyReLU(),\n            nn.Linear(mlp_width, head_number)\n        )\n\n    def forward(self, attention: torch.Tensor, bias: torch.Tensor):\n        \"\"\"\n        Args:\n          attention: input sequence, which is q^T * k,\n             shape [bsz, num_heads, seq_len, seq_len]\n          bias: bias matrix, which can be generated by Alibi, Kerple \n          FIRE or other additive position encodings\n             shape [1, num_heads, seq_len, seq_len]\n\n        Returns:\n          attention with CAPE,\n          shape [bsz, num_heads, seq_len, seq_len]\n        \"\"\"\n        # Repeat the bias for batch size\n        bias_tile = repeat(bias, '1 h l1 l2 -> b h l1 l2', b=attention.shape[0])\n        \n        # Concatenate attention and bias\n        attention_bias_concat = torch.cat((attention, bias_tile), dim=1)\n        \n        # Rearrange the dimensions for MLP processing\n        attention_bias_concat = rearrange(attention_bias_concat, 'b h l1 l2 -> b l1 l2 h')\n        \n        # Apply the MLP\n        attention_bias_concat = self.mlp(attention_bias_concat)\n        \n        # Rearrange back to original dimensions\n        attention_bias_concat = rearrange(attention_bias_concat, 'b l1 l2 h -> b h l1 l2')\n        \n        return attention + bias + attention_bias_concat\n```\n\nIf there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KYKh9AB4jm", "reviewer": "Reviewer_NzBk", "summary": "Considering the fixed parameters of RoPE may lead to the generalization issue, this paper introduce a dynamic position embedding method named CAPE, where position encoding is depend on the input context. Specifically, CAPE enables testing-time adaptation to input context by using a two-layer LeakyReLU neural network to parameterize the positional bias added to the vanilla attention module. By meticulously selecting the hyper-parameters, CAPE can achieve better performance compared with previous methods.", "questions": "1. Can CAPE adapt to LLMs, for example, Llama3-8B, with few post-training steps? \n2. What's the context length boundary of CAPE? In this paper, experiments just demonstrate the maximum context length with 8192.\n3. The authors claim that \"CAPE is semantically dependent and adaptive.\" What does \"semantically dependent\" mean? If CAPE is used for a summarization task with different context lengths (perhaps 8192 or 4096), what does \"semantic\" refer to in the summarization task? Providing a few examples could help clarify the writing.\n4. About motivation: Even though positional encoding (PE) is fixed during inference, attention still relies on the input context. What if we consider a scenario where PE simply provides the context index to help the language model distinguish the positions of each token (static), while the attention module dynamically selects key information and provides semantic information to assist the language model? Why does CAPE need to incorporate such a dynamic selection mechanism, which is already learned by LLMs, into the PE?", "limitations": "See weakness and question part.", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The concept is intriguing. Making position embeddings data-independent can potentially enhance the model's performance.\n2. The authors conducted numerous experiments on small language models and offered valuable insights.\n3. This paper includes several additional experiments shown in the appendix, which provide valuable insights.", "weaknesses": "1. CAPE can introduce extra computation, as well as lower the training and inference efficiency\n2. The choice of hyperparameters is significant for CAPE, as it is correlated with the model performance and the efficiency.\n3. There is no experiments on LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NzBk,\n\nThank you very much for appreciating our work. We will address your concerns below.\n\n**Q1: The efficiency of CAPE**\n\nA1: **With the model size increase, the additional computing cost ratio will decrease, compared to baseline Kerple**. Moreover, the CAPE can even speed up the training because a smaller training length of CAPE can still achieve good performance. We discuss this in **CAPE computation cost** in **Author Rebuttal by Authors**.  \n\n**Q2: The choice of $D_{CAPE}$**\n\nA2: The CAPE is relatively robust to the choice of $D_{CAPE}$, as shown in Figure 6. To achieve satisfactory performances, we find that CAPE with a modest size (not large $D_{CAPE}$) is sufficient. For convenience, we will copy the results (Arxiv dataset) below. \n|      | 512 | 1024 | 2048 | 4096 | 8192 |\n|------|---------|---------|---------|---------|---------|\n| $D_{CAPE}$ 4 |  4.54   |  4.27   |  4.38   |  4.28   |  4.06   |\n| $D_{CAPE}$ 8 |  4.53   |  4.26   |  4.33   |  4.17   |  3.97   |\n| $D_{CAPE}$ 16|  4.52   |  4.24   |  4.26   |  4.08   |  3.86   |\n| $D_{CAPE}$ 32|  4.50   |  4.22   |  4.22   |  4.04   |  3.82   |\n| $D_{CAPE}$ 64|  4.50   |  4.21   |  4.22   |  4.04   |  3.85   |\n**Q3: The experiment on LLM**\n\nA3: We further conduct experiments on 2.7B and 6.7B model sizes, which proves that the CAPE still works well. We further analyze the result of 2.7B and 6.7B in **Result on Large Model Size 2.7B and 6.7B** in **Author Rebuttal by Authors**.\nWe add larger model (2.7 and 6.7BB) experiments in the following, with micr_gpu_batch_size 4 and length 512 (Books Dataset).\n| Model size| Method | 512 | 1024 | 2048 | 4096 \n|-------|-------|-------|-------|-------|-------|\n|2.7B|RoPE| 21.01|25.00|48.13|160.59 |\n| | RPE|21.10|21.88|23.59|33.23| \n| | Kerple|21.14|22.08|23.38|27.21|\n| | CAPE-Kerple|20.52|21.01|20.23|19.57| \n|6.7B|RoPE| 20.86|22.27|28.01|110.00  \n| | RPE|20.79|21.60|22.32|26.31|\n| | Kerple|20.71|21.57|22.07|24.48|\n| | CAPE-Kerple|20.09|20.54|19.83|19.32| \n\n**Q4: Can CAPE adapt to LLMs, for example, Llama3-8B, with few post-training steps?**\n\nA4: The comment is quite helpful. Although we train from scratch for all transformer models with CAPE (that is the main reason that we did not try very large transformer models due to the computation limitation), it is still possible that we merely train on the CAPE part (i.e., the introduced MLP) but freeze (or fine-tune) all other parameters of transformer models. This strategy will definitely accelerate the training and require fewer post-training steps. We will discuss the possibility of adapting CAPE to LLMs with fewer post-training steps in the paper.\n\n**Q5: What's the context length boundary of CAPE? In this paper, experiments just demonstrate the maximum context length of 8192.**\n\nA5: We further validate CAPE on length 16384, which proves that CAPE still works well. For the length of 32768, the GPU reports out-of-memory. Therefore, we believe that the context boundary of CAPE is more than 16484, while the training length is 128.\n|Method| 128|256|512|1024|2048|4096|8192|16384|\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|\nKerple|31.96| 29.02| 29.70| 42.74| 56.24| 73.59| 87.03|93.38|\nCAPE-Kerple|31.44| 28.25| 24.93| 24.33| 23.29| 24.32| 24.93|25.33\n\n\n**Q6: What does \"semantically dependent\" mean?**\n\nA6: Thank you for pointing out. The \"semantically dependent\" indicates that our position encoding value depends on the semantics (which is the attention score in this paper). For clear presentation, we will change such words to ``context adaptive`` or ``contextually dependent``. And we will also change explain that the context in the paper is attention score.\n\n**Q7: Why does CAPE need to incorporate such a dynamic selection mechanism, which is already learned by LLMs, into the PE**\n\nA7: The reason is that the CAPE has more expressiveness.\n\nSuppose that the ``optimal'' attention mechanism is composed of the key-query multiplication (denoted as $A(x)=XW_Q(XW_K)^T$) and the additive positional encoding bias (denoted as $B(x)$), i.e., $A_{optimal}(x)=XW_Q(XW_K)^T+B(x)$. Here, x and X are the input sequence and the corresponding token embeddings. In previous static PE, $B(x)$ is set as a constant for all input sequence, i.e., $B(x) = B$ and the constant $B$ is optimized across all samples $\\{x\\}$ during training. However, our main contribution and claim is that the optimal positional encoding should vary for different sequences. Therefore, we proposed the dynamic context-adaptive PE (CAPE), where $B(x)$ depends on both the sequence and the positional information. In contrast with the static PE, the proposed CAPE can adjust dynamically with the input context, and is optimal for each input.\n\nSpeaking from higher-level, we see that the general fixed and static PE (even though the PE is learned from data) is an averaged optimal positional encoding over all training samples, while the dynamic PE is context-dependent and is optimal for each sample. That is the core motivation of using dynamic PE rather than the static PE.\n\nIf the the optimal solution is $B(x) = B$, the $f(XW_Q(XW_K)^T, B)$ can be reduced to zero as $f(*)$ is a universal approximate function (two-layer mlp with activation). Therefore, our CAPE is at least no worse than the baseline.\n\nIf there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A28muZkgBF", "reviewer": "Reviewer_QhvW", "summary": "This paper proposes a simple learnable positional encoding called CAPE that boosts the length extrapolation performance of Transformer language models.", "questions": "1. What are the potential ways to improve the speed of CAPE? Please be as concrete as possible.\n2. Can the authors test CAPE using needle in a haystack? This way the readers can have a clearer picture of how CAPE is using long context information.", "limitations": "None.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The empirical performance of CAPE is substantially better than previous positional encodings. I also think Figure 1 nicely demonstrates the flexibility of CAPE, i.e., CAPE can learn both local and anti-local attention heads.", "weaknesses": "The speed is unbearably slow in my opinion. While I appreciate the honesty of reporting speed differences in Table 1, the authors really need to figure out a way to improve CAPE's training and inference efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QhvW,\n\nThank you very much for appreciating our work. We will address your concerns below.\n\n**Q1: The speed of training**\n\nA1: We will answer the question in three parts: 1) **The potential way to improve the speed of CAPE**; 2) **The additional training ratio will gradually decrease with a larger model size**.; 3) **Moreover, CAPE indeed can speed up training**. Please refer to the more detailed answer to **CAPE computation cost** in the **Author Rebuttal by Authors**.\n\n**The potential way to improve the speed of CAPE**:\n* **Reduce the size of $D_{CAPE}$**. The computation cost is $O(hN^2D_{CAPE})$. Therefore, by reducing the $D_{CAPE}$ to be half, the CAPE computation cost will be half.\n* **Algorithm that is more efficient than MLP**. The MLP in CAPE can be changed to other more efficient operations, such as sparse-MLP.\n* **Sparsity**: we can make the MLP operation sparse to speed up CAPE. As shown in Figure 6, our CAPE could work well with only $D_{CAPE}=4$, while the Table 1 count with  $D_{CAPE}=32$\n  * Pruning: use pruning on MLP to remove the redundant parameters. \n  * Dynamic sparsity: we could use dynamic sparse training methods, such as Sparse Momentum, Dynamic Sparse Reparameterization (DSR), Dynamic Sparse Training (DST) and so on.\n* **Data Parallel**: The CAPE consists of fully-connected layer. Therefore, better data parallel can significantly reduce the time. \n* **GPU with high memory width**. The CAPE read/write the matrix with size $N^2$. Therefore, a GPU with a higher memory width will help improve the speed of CAPE.\n* **GPU with high computation compatibility**. The CAPE consists of a multiple-layer perception network, which is computationally dense. Therefore, GPU has better computation compatibility can help speed up the \n* Finally, with the increase in hardware, the cost of CAPE will be acceptable. For example, with the development of GPU, the Large Model gradually is accepted, while it is unimaginable to train a 175B model 10 years ago.\n\n**The additional training ratio will gradually decrease with a larger model size, compared to baseline Kerple.**\n\nThe following is the time cost with a training length of 512 with micro_gpu_batch_size 1.\n\n| Method |  350M Total | Ratio(Compared to CAPE-Kerple) | 2.7B Total |Ratio(Compared to CAPE-Kerple) | 6.7B Total | Ratio(Compared to CAPE-Kerple)|\n|------|------|------|------|------|------|------|\n|RoPE|210.01|0.9366| 472.63|1.1187| 635.57|0.8858\n|T5's bias|355.16| 1.5839|537.62|1.2725|808.85|1.1273\n|Alibi|172.60|0.7697|325.95| 0.7715|596.77|0.8317\n|**Kerple**|189.91| **0.8469**|370.32| **0.8765** |661.82|**0.9224**\n|FIRE|248.13|1.1066|432.63| 1.0240|797.68|1.1118\n|**CAPE-Kerple**|224.22|**1.0000** |422.48|**1.0000**|717.46|**1.0000**\n\nApparently, when the model becomes large, the additional computational cost of CAPE gradually decreases. Therefore, the CAPE may be a potential good choice for an extremely large language model.\n\n**Moreover, CAPE indeed can speed up training, compared to current popular RoPE**\n| Evaluation | RoPE Length 4096 & Batch 1  |Kerple Length 512 & Batch 8|CAPE-Kerple Length 128 & Batch 32 |CAPE-Kerple Length 512 & Batch 8 |CAPE-Kerple Length 1024 & Batch 4 | CAPE-Kerple Length 2048 & Batch 2 | CAPE-Kerple Length 4096 & Batch 1 |\n|------|------|------|------|------|------|------|------|\n|128|38.36|33.04|31.49|32.22|33.22|34.71| 36.65|\n|256|33.21|29.11|28.27|28.32|29.02|30.08|31.57|\n|512|27.33|24.68|24.93|23.88|24.14|24.77|25.68|\n|1024|25.49|23.82|24.31|22.62|22.62|23.09|23.80|\n|2048|23.55|24.03|23.34|21.16|21.00|21.30|21.84|\n|4096|**24.58**|30.76|24.38|**21.79**|21.34|21.45|21.83|\n|8192|152.54|36.81|25.01|21.70|21.12|21.24|21.50\n|Time Cost|**265.48** |117.10 |128.94|**192.45**|314.86|547.78|1217.34\n\nWith the same training token, the CAPE with a training length of 512 and batch size of 8 can even with comparable performance with a RoPE training length of 4096 and batch size of 1. Also, the CAPE with a training length of 512 and batch size only takes 192.45ms, while RoPE takes 265.48 ms. Therefore, the CAPE could be a choice for speeding up training in the future.\n\n**Q2: How CAPE is using long context information and haystack test**\n\nA2: We analyze how CAPE uses the long context information via visualization analysis, as shown in Figure 1 and Appendix. According to Figure 1 and Appendix, the CAPE not only helps the model to pay attention to the local information but also helps the model to look at the information far away, with different heads having different functions. \n\nThe haystack test, requires more training (including pretrain and alignment) so that the model can follow the instructions to finish the test, while currently, we do not have such resource to train such a model. However, in our CHE benchmark, there is one task named missing duplicate that may be similar to haystack test.\n* haystack test: \n  * It works by embedding specific, targeted information (the “needle”) within a larger, more complex body of text (the “haystack”). \n  * The goal is to assess an LLM’s ability to identify and utilize this specific piece of information amidst a vast amount of data.\n* CHE Benchmark Missing Duplicate Task: \n  * The input is a binary string of the form $ww$ where w is itself a binary\nstring. **One token in this string has been hidden, and the network must find out which one it is**,\nby looking at the value at the same position but on the other side of the string. For instance, if\n$x = ab$_$aba$ (i.e., w = aba), then $y = a$. \n  * The goal is also to assess a model's ability to identify and utilize this specific piece of information amidst a vast amount of data.\n * As shown in Table 2, Kerple only gets 79.06% accuracy, while CAPE-Kerple gets 87.56%. This suggests the ability of CAPE on the haystack test.\n\nIf there are any questions, please let us know. And if you think that we have addressed your concerns, could you please consider raising the score? Thank you very much for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers:

We sincerely appreciate the reviewers for the time and efforts on the review. We first address some common questions, followed by detailed responses to each reviewer separately. We hope our responses clarify existing doubts. We will really appreciate it if Reviewer tTna can kindly reconsider the decision, provided that the main comments are well addressed.

**Q1: CAPE computation cost (Reviewer CfZF, Reviewer NzBk, Reviewer QhvW)**

**The additional training ratio will gradually decrease with a larger model size, compared to baseline Kerple.**.

Theorically Analysis: 
 * The cost of Feed-Forward Network is: $O(Nd_{head}^2d_{hidden}^2)$=$aNd_{head}^2d_{hidden}^2$, where a is a constant, N is the sequence length, $d_{head}$ is the attention head number and $d_{hidden}$ is the dimension for attention calculation.
* The cost of Attention: $O(N^2d_{head}d_{hidden})$=$bN^2d_{head}d_{hidden}$, where b is a constant.
*  The additional cost of CAPE: $O(N^2d_{head}d_{cape})$=$cN^2d_{head}d_{cape}$, where c is a constant.
* The cost ratio is $\frac{aNd_{head}^2d_{hidden}^2+bN^2d_{head}d_{hidden}}{aNd_{head}^2d_{hidden}^2+bN^2d_{head}d_{hidden}+cN^2d_{head}d_{cape}}$=$\frac{ad_{head}d_{hidden}^2+bNd_{hidden}}{ad_{head}d_{hidden}^2+bNd_{hidden}+cNd_{cape}}$. Therefore, with the fixed sequence length and $d_{CAPE}$, with the model becomes larger (with bigger $d_{head}$ and $d_{hidden}$), the additional cost ratio of CAPE will greatly become smaller. Also, we have shown in Figure 6 that $d_{cape}$ still works well with very small value, such as 4.


The following is the time cost with a training length of 512 with micro_gpu_batch_size 1 on Books3 dataset.

| Method |  350M Total | Ratio(Compared to CAPE-Kerple) | 2.7B Total |Ratio(Compared to CAPE-Kerple) | 6.7B Total | Ratio(Compared to CAPE-Kerple)|
|------|------|------|------|------|------|------|
|RoPE|210.01|0.9366| 472.63|1.1187| 635.57|0.8858
|T5's bias|355.16| 1.5839|537.62|1.2725|808.85|1.1273
|Alibi|172.60|0.7697|325.95| 0.7715|596.77|0.8317
|**Kerple**|189.91| **0.8469**|370.32| **0.8765** |661.82|**0.9224**
|FIRE|248.13|1.1066|432.63| 1.0240|797.68|1.1118
|**CAPE-Kerple**|224.22|**1.0000** |422.48|**1.0000**|717.46|**1.0000**

Apparently, when the model becomes large, the additional computational cost of CAPE gradually decreases. Therefore, the CAPE may be a potential good choice for an extremely large language model.

**Moreover, CAPE indeed can speed up training, Compared to current popular RoPE**
| Evaluation | RoPE Length 4096 & Batch 1  |Kerple Length 512 & Batch 8|CAPE-Kerple Length 128 & Batch 32 |CAPE-Kerple Length 512 & Batch 8 |CAPE-Kerple Length 1024 & Batch 4 | CAPE-Kerple Length 2048 & Batch 2 | CAPE-Kerple Length 4096 & Batch 1 |
|------|------|------|------|------|------|------|------|
|128|38.36|33.04|31.49|32.22|33.22|34.71| 36.65|
|256|33.21|29.11|28.27|28.32|29.02|30.08|31.57|
|512|27.33|24.68|24.93|23.88|24.14|24.77|25.68|
|1024|25.49|23.82|24.31|22.62|22.62|23.09|23.80|
|2048|23.55|24.03|23.34|21.16|21.00|21.30|21.84|
|4096|**24.58**|30.76|24.38|**21.79**|21.34|21.45|21.83|
|8192|152.54|36.81|25.01|21.70|21.12|21.24|21.50
|Time Cost|**265.48** |117.10 |128.94|**192.45**|314.86|547.78|1217.34

With the same training token, the CAPE with a training length of 512 and batch size of 8 can even with comparable performance with a RoPE training length of 4096 and batch size of 1. Also, the CAPE with a training length of 512 and batch size only takes 192.45ms, while RoPE takes 265.48 ms. Therefore, the CAPE could be a choice for speeding up training in the future.

Finally, with developments in hardware, the cost of CAPE will become more manageable. For instance, the development of GPUs has led to the widespread acceptance of large models, something that would have been unimaginable 10 years ago when training a 175-billion parameter model.

**Q2: Result on Large Model Size 2.7B and 6.7B (Reviewer tTna, Reviewer NzBk)**

A2:  We add larger model (2.7 and 6.7BB) experiments in the following, with micr_gpu_batch_size 4 and length 512 on Books3 dataset.
| Model size| Method | 512 | 1024 | 2048 | 4096 
|-------|-------|-------|-------|-------|-------|
|2.7B|RoPE| 21.01|25.00|48.13|160.59 |
| | RPE|21.10|21.88|23.59|33.23| 
| | Kerple|21.14|22.08|23.38|27.21|
| | CAPE-Kerple|20.52|21.01|20.23|19.57| 
|6.7B|RoPE| 20.86|22.27|28.01|110.00  
| | RPE|20.79|21.60|22.32|26.31|
| | Kerple|20.71|21.57|22.07|24.48|
| | CAPE-Kerple|20.09|20.54|19.83|19.32| 

According to the result, we can find that the proposed CAPE still works well, whatever the model size is 2.7B or 6.7B. With 2.7B model size, RoPE achieves 21.01 on evaluation length 512 and 160.50 on evaluation 4096, while our CAPE-Kerple achieves 20.52 and 19.57 respectively. Also, CAPE-Kerple achieves the best performance, whatever the model size is 2.7B or 6.7B from evaluation length 512 to 8192. This suggests that our proposed CAPE has great scalability.

Meta-review: **Summary**

Authors propose a new positional embedding which consists of an additional mlp layer learnt on top of existing additive bias (expresses the relative positions) and the attention itself. Authors show that proposed context-aware positional embedding improves generalization over the baselines when applied on top of AliBi, KERPLE and FIRE for the sequences which are longer than training ones. Also authors show that proposed modification is learning both local and anti-local information.

**Justification for decision making**

It was a long and productive discussion among reviewers and authors, and also between AC (me) and authors. Most of the issues, concerns and ablations were resolved during long discussion and e.g. one of the reviewers raised the score from 6 to 8. However, 1/4 reviewers did not comment anything to rebuttal and has the lowest score of 3: I believe authors provided proper and reasonable answers after which the argumentation for rejection based on that review is no longer valid to me. 

I raised several important issues and misalignments for the paper during the discussion period. During rebuttal period authors provided explanations to my points saying that they changed the evaluation protocol compared to prior works and argued that this protocol is more suited. They pointed out that this change in the protocol led to different results for the baselines compared to prior works results for that baselines. During AC-reviewers discussion one of the reviewers agreed to the raised issues, but also confirmed that the protocol authors used is more suitable. However the reviewer also agreed that better apple-to-apple comparison would strengthen the paper. In the end this reviewer is still on the positive side of the paper acceptance and experienced similar observations on many aspects as authors reported.

**Recommendation**

Based on the positive narrative of 3/4 reviewers (6, 7, 8), taking into account rebuttal to the 4th reviewer which did not respond to the rebuttal, also taking into account AC-authors discussion and AC-reviewers discussion, I **recommend acceptance of the paper**.

However, I ask authors for **several critical modifications for the camera-ready paper**:
- **change the title of the paper** due to collision with already existing work on positional embedding called CAPE and published at NeurIPS: Likhomanenko, T., Xu, Q., Synnaeve, G., Collobert, R. and Rogozhnikov, A., 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34, pp.16079-16092. to avoid naming problems in the future.
- include detailed discussion of evaluation protocols and why results for the baselines are different from the prior works to avoid misinterpretation of results and doubting the baselines from the readers, also for the sake of science.
- include all rebuttal discussions and ablations with new results into appendix as they all have merits and contribute to better understanding and practical applications of the proposed method.
- if possible add at least one apple-to-apple comparison and setup with prior works so as to have strong support that the baselines implementation is correct and consistent with prior works. I and reviewers still found some results reported for the baselines to be strange even when 512 training length is used (128 length definitely gives different observations compared to 512 training length and thus different from prior works as they all used larger context length for training).
- if the numbers and behaviour you observe diverges from prior work, please discuss this in the results and point to the source of this discrepancy.
Decision: Accept (poster)