Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
Authors: Yi Lu
Wanxu Zhao
Xin Zhou
Chenxin An
Chenglong Wang
Shuo Li
Yuming Yang
Jun Zhao
Tao Ji
Tao Gui
Qi Zhang
Xuanjing Huang
Venue: COLM 2025 (2025)
Keywords: Long Context
Extrapolation
Training-Free Framework
TL;DR: Dimensionally manipulating the relative position matrix to extrapolate the context window of LLMs without additional training.

Abstract: Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length.
Recent advancements in long-context extension have significantly expanded the context window of LLMs but require expensive overhead to train the large-scale models with longer context. 
In this work, we propose Dimension-Wise Positional Embeddings Manipulation (DPE), a training-free framework to extrapolate the context window of LLMs by diving into RoPE's different hidden dimensions.
Instead of manipulating all dimensions equally, DPE detects the effective length for every dimension and finds the key dimensions for context extension.
We reuse the original position indices with their embeddings from the pre-trained model and manipulate the key dimensions' position indices to their most effective lengths.
In this way, DPE adjusts the pre-trained models with minimal modifications while ensuring that each dimension reaches its optimal state for extrapolation.
DPE significantly surpasses well-known baselines such as YaRN and Self-Extend.
DPE enables Llama3-8k 8B to support context windows of 128k tokens without continual training and integrates seamlessly with Flash Attention 2.
In addition to its impressive extrapolation capability,
 DPE also dramatically improves the models' performance within training length, such as Llama3.1 70B, by over 18 points on popular long-context benchmarks RULER.
When compared with commercial models, Llama 3.1 70B with DPE even achieves better performance than GPT-4-128K.

Published as a conference paper at COLM 2025
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
Yi Lu", Wanxu Zhao 1, Xin Zhou', Chenxin An², Chenglong Wang",
Shuo Li', Yuming Yang', Jun Zhao ,Too Tao Gui Qi Zhang" Xuanjing Huang
Fudan University The University of Hong Kong Northeastern University
Abstract
Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length. Recent advancements in long-context extension have significantly expanded the context window of LLMs but require expensive overhead train the large-scale models with longer context. In this work, we pro- pose Dimension-Wise Positional Embeddings Manipulation (DPE), a training-fre free framework to extrapolate the context window of LLMs by div- ing into RoPE's different hidden dimensions. Instead of manipulating all dimensions equally, DPE detects the effective length for every dimension and finds the key dimensions for context extension. We reuse the original position indices with their embeddings from the pre-trained model and manipulate the key dimensions' position indices to their most effective lengths. In this way, DPE adjusts the pre-trained models with minimal modifications while ensuring that each dimension reaches its optimal state for extrapolation. DPE significantly surpasses well-known baselines such YaRN and Self-Extend. DPE enables Llama3-8k 8B to support context windows of 128k tokens without continual training and integrates seam- lessly with Flash Attention 2. In addition to its impressive extrapolation capability, DPE also dramatically improves the models' performance within training length, such as Llama3.1 70B, by over 18 points on popular long- context benchmarks RULER. When compared with commercial models, Llama 3.1 70B with DPE even achieves better performance than GPT-4-128K, We release our code at https:/
1 Introduction
Long-context comprehension is fundamental to enable practical implementations of modern large language models (LLMs). This capability powers applications such as PDF document processing (AI, 2025), multimodal understanding and generation (Zhan et al., 2024), and inference-time reasoning (OpenAI, 2024; DeepSeek-Al al., 2025).
Achieving long- contect ability typically requires continued fine-tuning on extended se- quences (Xiong et al., 2023; Qwen et al., 2025; Liu et al., 2025). For example, LLaMA-3 incrementally expanded its context window from 8K to 128K over six stages using 800B tokens (AI@Meta, 2024). This "train long, test long" extension faces significant challenges, including the computational burden imposed by the quadratic complexity of attention mechanisms (Xiong et al., 2023) and the quality constraints of long-context data (Fu et al., 2024; Yen et al., 2025). To unlock efficient long-conted extension, researchers explored the "train short, test long" (e.g., trained on 8K and then test/ geterreerr on 128K) methods and even pushed further into training-free methods for length extrapolation.
This paper focuses on a training-free approach that extrapolates rotary positional encodings (RoPE) by applying tailored modification to specific RoPE frequencies. RoPE applies positional priors via a rotation matrix R(e, m), where & denotes the angular frequency and m
* Equal contribution.
Corresponding authors. Correspondence getaaji.gui.///@huaaeee.cm

---

Published as a conference paper at COLM 2025
represents the relative distance. During length extrapolation, RoPE exhibits significant out- of-distribution (OOD) issues. Existing research addresses these issues by scaling the angular frequency 0 (Xiao et al., 2023; bloc97, 2023a; emozilla, 2023) reassigning the relative position i (e.g., through truncation (Su, 2023) or grouping (Jin et al., 2024) strategies). Taking the buneed ReRoPE as example, when the context exceeds the pre- trained length L, ReRoPE sets a maximum length w (w < L) within the pre-trained length, changing the relative distance sequence (0, 10,1,.., L-,,,,,,,,,, L,t,,,, to (0,1,..., W, w, w....) Some attention frequency analyses (Barbero et al., 2025; Hong et al., 2024) and the empirical results from angular frequency scaling (bloc97, 2023b; Peng et al., 2023) indicate that each attention head exhibits varying sensitivity across different frequency subspaces, thereby necessitating differentiated modifications. However, these approaches uniformly modify the relative positions across all heads and frequency subspaces, which raises the question: Should we reassign the relative positions in differentiated manner?
We address this question from two perspectives: 1) How should relative positions be reas- signed for specific frequencies, particularly regarding the maximum position constraints? 2) Which frequency subspaces require reassigned?
Using the example of extrapolating an 8K-trained LLM to 128K, we initially restrict all frequency subspaces to within the pretraining length (e.g., w set to 4K). Then, we organize similar frequency subspaces into fine-grained groups (e-g., four subspaces per group) and progressively relax the constraints for each 128K. By analyzing downstream task performance variations, we discovered that different frequency subspaces exhibit distinct preferences for maximum position. Specifically, both extremely high-frequency and extremely low-frequency subspaces remain effective even when their w exceeds the pretraining range. In contrast, mid-range frequencies demonstrate almost no extrapolation capability. The performance collapses immediately when their maximum position extends beyond the pretraining length.
from 4K eventually to
Then, inspired by recent works analyzing RoPE frequency subspaces (Barbero et al., 2025; Ji et al., 2025), we use the 2-norm attention contribution metric to investigate whether all the RoPE hidden dimensions are utilized and how they contribute to the attention mechanism. We find that reassigning the relative positions only for the top 50% or 75% of key dimensions is sufficient to restore the model's performance for longer contexts. Surprisingly, this selective approach even outperforms reassigning 100% of the dimensions.
Based these findings, we introduce Dimension-Win Positional Embeddings Manipu- lation (DPE), a new training-free free framework to extrapolate the context window of LLMs. Instead of modifying the relative distance for all dimensions equally (Jin et al., 2024; An et al., 2024a), we identify key dimensions for context extension by 2-norm metric. For each key dimension pair within the same frequency subspace, we set the maximum relative position by detecting the effective relative distance. These treatments adjust the pre-trained model's dimensions with minimal modifications while ensuring that each dimension reaches its optimal state for extrapolation. Our contributions can be summarized as follows:
We propose DPE, a training-free length extrapolation method that dimensionally ma- nipulates position embeddings.
We discover that RoPE's different frequency subspaces exhibit distinct preferences for maximum position, and there exist some key dimensions for length extrapolation.
Experiments on three long-context benchmarks demonstrate that DPE is an SOTA training-rre extrapolation method. DPE extends Llama3-8B-8K to 128K and outper- forms the best baseline by 7.56% scores on RULER. Besides, DPE significantly enhances LLMs' performance within the training length. When integrated with powerful LLMs such as Llama3.1-708-122k, DPE outperforms leading commercial model GPT-4-128K.
2 Background
2.1 Rotary Position Embedding (RoPE)
The attention mechanism (Vaswani et al., 2017; Dai et al., 2019) requires explicit position information to represent the order of input tokens(Su et al., 2021). Recently, the RoPE (Su
2

---

Published as a conference paper at COLM 2025
et al., 2024) has become the mainstream position embedding of LLMs,, such as Llabe
et al., 2024) and Qwen et al., 2025). RoPE encodes positional information by applying phase rotation to query and key vectors. Considering query at position and key at position n, have qu (,, q, K, R 1)),, where qm, Kn R², and Id/2 is the frequency basis. In standard RoPE, the 0 {0] b 2j/d, E
1]}, where the base b is usually set to 10000. The rotary matrix R Rdxd at position is defined as:
cos múo
sin ml
0
sin m0g
cos mll
R (0, m)
(1)
0
cos mid/2 1
sin m0a/ 1 cos m0
0
sin m0
Due to the specific arrangement of frequencies, the matrix ensures that
R m)) R m) R0.
(2)
Based on this property, the dot product of qm and kn can be expressed as follows:
q kn = ())) (R(O,n)k) = R n m)k
(3)
This formulation implicitly encodes the relative positional difference within the query-key interaction, thereby influencing the resulting attention score.
2.2 Manipulate Relative Position Matrix for Length Extrapolation
Recent studies (Chen et al., 2023; Han et al., 2023; Peng et al., 2023; Lu et al., 2024a) have shown that LLMs utilizing the original RoPE exhibit limited robustness in length extrapola- tion. One perspective to explain the cause of this limitation is the presence of previously unseen relative positions in the pretraining phase (Chen et al., 2023; Jin et al., 2024; An et al., 2024a). Given sequence length L, the relative position matrix is created by the R(O, (,, m) and nO,
0
1
0
P
P Prerope
(4)
L-1
w
W
11
where the Pm,, n) = n - m encoding the relative distance between the qu and k, and red color indicates OOD position indices. Based on Eq. 4, once the max relative position I. in P exceeds the pre-trained effective length, the performance tends to degrade. To address this issue, recent studies (An et al., 2024a; Su, 2023; Jin et al., 2024) reuse the original position embeddings and manipulate the relative position matrix to avoid the presence of unseen relative positions. For instance, ReRoPE adjusts relative position indices by scaling:
Premope (m,n) n1) w, if
(5)
where w is the truncated length with pre-trained length. After the modification, all position indices are scaled within the effective length, thereby enhancing extrapolation capabilities.
Method
In this section, we first investigate whether the magnitude the OOD effect differs across different dimensions in Section 3.1. Next, try to identify the key dimensions for context extension in Section 3.2. Finally, we introduce our proposed approach, Dimension- Wise Positional Embeddings Manipulation (DPE) in Section 3.3.
3

---

Published as a conference paper at COLM 2025
3.1 Detecting Effective Relative Distance on Different Dimensions
In RoPE, every two dimensions correspond to trigonometric functions with the same frequency 0j¹_ Each dimension of the vectors and kn contributes to the attention score via the dot product. The query-key product between two positions m and is:
d/2-1 Σ
9m K2p
(9//2/+1| (0 m (2)22/2/+11)
(6)
J
According to Eq. 6, the rotation angle depends only on the relative distance n - in and Oj in the input. Previous works (An et al., 2024a; Jin et al., 2024; Su, 2023) have attempted to mitigate the OOD problem of rotation angle by scaling the relative distance changes in dimensions can also alter the frequency of the trigonometric functions, ultimately affecting the rotation angle. Therefore, across different dimensions. We detect the maximum effective relative distance of different dimensions to demonstrate this.
However,
magnitude of the OOD effect should differ
377inn roste 13)072
131072
880,3
86.01
85.2
91.0
0,0
0.0
19.5
1G:01 327/6 6503
00.0
50.0
91.0
90.0
0.0
0.0
0.0
22.2
Mix Rolative Distance
69.8
89.2
81.6
50.0
0.0
0.0
29.5
92.6
Max Relative Distance
00
10:00
88.33
29.5
91.0
13.9
91.2
-5
83:03
40
40000
4006
2008
88.0
80.2
75.8
13.0
76.5
18.6
87.8
20
2001
3924
89.5
87.8
83.5
82.0
13.0
10.2
89.0
88.0
1024
67
8-15
24-31
12:10
48:55
5643
6.7
8:15
16:23
Dimension
Dimensioo
(a) NIAH Accuracy
(b) Ranked NIAH Accuracy
Figure 1: Detecting the effective relative distance across different dimension groups. We show the NIAH accuracy on Llama3-8b in Figure 1a and rank the results in Figure 1b. When the NIAH accuracy is the same, we prioritize ranking based on larger relative distances.
We only manipulate the relative distance matrix of certain dimensions and observe the impact of this change on the model's performance. Given an model with d dimensions, we divide the dimensions into groups G [gi ......, ..../gc) and C is the number of groups. For the same group of dimensions, we assume that they correspond to the same effective relative distance. To detect the maximum effective relative distance of dimension group gi, we only vary the maximum relative distance of from 1k to 128k, while keeping other groups fixed half of the pre-trained length. This varied distance termed the detecting length f. For each detecting step, we scale the position indices in the corresponding relative position matrix P; to ensure the maximum value does not exceed the detecting length f. Since neighboring tokens are crucial for generating fluent content (Jin et al., 2024; An et al., 2024b), we introduce a small local window value w to ensure the stability of the detecting results. Formally, given sequence of length L, we scale the position indices the relative position matrix (except for the local window) and ensure that the maximum relative distance in the matrix equals the detecting length
Experiment Setup
.......gC.
m m-) + w, if > W,
(y(m,))
m
otherwise.
where I.] is the floor division. We evaluate Meedle-ina-HHaysaaANNAAA (gkamradt, 2023) on Llama3 8B to investigate the impact of different detecting lengths. The detecting experi- ment results are shown in Figure 1. More details can be found in Appendix A.
For simplicity, Dimension in this paper refers pair of dimensions that share the frequency.
4

---

Published as a conference paper at COLM 2025
Findings By ranking the relative distances based the NIAH results(Figure 1b), observe that the ranking is different across dimension groups, which indicates that different dimension groups correspond to different effective relative distances. In Figure la, we observe that the magnitude of NIAH accuracy variation differs significantly across dimension groups. Specifically, the accuracy variation is relatively small in both low and high-dimension groups, while it is substantially larger in the middle dimensions. This indicates that different dimensions contribute differently to length extrapolation.
3.2 Identifying Key Dimensions for Context Extension
Since different dimensions contribute differ- ently to length extrapolation, we hypothesize that some dimensions play crucial role in context extension. Since OOD issue is closely related to the attention mechanism(Han et al., 2023; Xiao et al., 2023), we believe that these key dimensions for context extension also play an important role in the attention mech- anism. To identify these key dimensions, we use 2-norm Attention Contribution et al., 2025; et al., 2025), which quantifies the 2-norm contribution to investigate whether these frequencies are utilized and how they contribute to the model's performance. Based on the Sauchy-Schwane inequality, the impact of the j-th dimension on the attention log- its is upper bounded by the 2-norm of the query at position m and key at position n, i.e. Kgm k!)) < 11 [|Ikk/"") We compute the mean 2-norm score of queries and keys for each dimension. For each attention head h, we rank the dimensions by the 2-norm score and select the top-k dimensions key dimensions:
Dituusion 4
50
40
85
10
20
10
Loovr
Loper LD
Laper 25
Looks
Eloma-33
Figure 2: 2-norm Attention Contribution for different heads and layers of Llama3- 8b. We select top-k dimensions as the key dimensions for extrapolation. For example, the top-1 dimension is selected in the red dashed line.
D {j/top-k(llg q ( 113
(8)
where j is dimension index, q, E R and k, Rlxd indicates all the queries and keys of the sequence with length L. Figure 2 visualizes one of the 2-norm results attention head. We observe that only a few dimensions contribute attention, and the key dimensions vary across different heads and layers.
of Llama3's
Experiment Setup After identifying the key dimensions, we scale the position in- dices only for key dimensions Dj while keeping the position indices of other dimen- sions unchanged. We set the scaled length from 2k to 8k. We also use 100 test samples of NIAH with 128k length and evaluate the accuracy under different values of & to vali- date the importance of these dimensions for context extension.
0.0
0.0
0.9
0.0
35.5
87.0
91.0
10.0
58.2
22
84.0
26.0
Mas
Findings In Figure 3, when k increases to 32, the accuracy improves significantly, and the model performance is restored. Ad- ditionally, we find that scaling only 48 di- mensions yields slightly better performance than scaling all dimensions. These results strongly indicate there exist some key di- mensions for length extrapolation.
2010
-20
2.0
36.2
83.5
52.
52.0
Figure 3: NIAH Accuracy on Llama3-8b. Only
top-k dimensions' position indices are scaled.

---

Published as a conference paper at COLM 2025
3.3
Dimension-Wan Wise Positional Embeddings Manipulation
Detecting the Effective
(b) Identifying the Key Dimensions
(c) Scaling the Position Indices
Relative Distance
Position Matrix of Dimension
(1) Identify Key Dimensions for Each Head
Key
Group gi
Ltention Contribution
Dimension Lodet
2
3
2
(2) Query Effective L ength of Key Dimensions
3
6
2
7
3
2
34-3720 (70-57400000 48-56 56-63
Scale Effective Leeughh
Position Matrix of Dimension
Key Dimecsion 47
Group Sj
Eiteehhe Lagght
0
0
5
1
3
0
000
Effective Length 0 Only Scale Top-k Key Dimensions
Effective Length Ranking Different Dimensions
Figure 4: An illustrative example of DPE with three main procedures. (a) We detect the effective length e; e,e; 5 for dimension group 8i- (b) Dimensions 29 and 47 are identified key dimensions, and the corresponding effective length is obtained from the detection result. (c) We set = 2, and the position indices are scaled within the effective length, thereby avoiding the impact of OOD position indices for every dimension.
In this section, we propose new training-free method for context extension: Dimension- Wise Positional Embeddings Manipulation (DPE). The three main procedures of DPE are shown in Figure 4:
(1)Detecting the effective relative distance for different dimensions: Given model with head dimension d2, we first divide its dimension into groups G = [gi, ,2,.,,...,, gc]] Since different dimension groups correspond to different effective relative distances. We detect the corre- sponding effective relative distance E [e,,e2, ... ece for each group. In Figure 4(a), for dimension groups gi and gú, their effective length e; 4, 5.
Idenntiigi the key dimensions: We use the head-wise 2-norm attention contribution in Section 3.2 to find the key dimensions D; for context extension. After the first two steps, we have the key dimension D,, and its corresponding effective length E. In Figure 4(b), we have key dimensions 29 in group gi and 47 in group 8i- We then query the corresponding effective length from the detection results.
(3) Scaling the position indices: Finally, we calculate the scale size for each dimension group with the max effective relative distance N: S
(2,, .., Sc]. Given key dimension j E 8i, we can scale the position matrix pj with the corresponding scale size s;. We also introduce a small local window value W to capture local relationships. The final position matrix is defined as:
B m
w
w, if - m > w,
(m n)
99)
12
m,
otherwise.
We implement DPE using FlashAttention 2(Dao, 2023) with negligible additional over- head. In practice, we set C 8, 1k, and select top-48 key dimensions for all the models. Detailed implementation of DPE is in Appendix C, and Pseudocode DPE in Appendix B.

---

Published as a conference paper at COLM 2025
4 Main Results of DPE
We evaluate the effectiveness of DPE across two widely used LLMs Llama3-8B-Instructikk (Grattafiori et al., 2024) and
(liang et al., 2023). We chose these two models because DPE focuses on extrapolation, which refers to testing LLMs on sequence lengths beyond their training lengths. We also use the latest long-context models like Llama- 3.1 11-8B--nnstruct(28k (Grattafiori et al., 2024), to test DPE's effectiveness in improving performance within the training context size. We evaluate these models on three widely recognized long-context benchmarks: seedle-in-a-Hayseee
et al., 2025) and
1lama-31-700-Instruu(((
(NIAH) (gkamradt, 2023), RULER (Hsieh et al., 2024), InfiniteBench(Zhang et al., 2024b) and HELMET (Yen et al., 2025).
Baselines We mainly compare DPE with several effective extrapolation baselines. Specif- ically, we compare DPE with the following traiming extrapolation methods: NTK- Dynamic(emoziila, 2023), YaRN(Peng et al., 2023), ReRoPE(Su, 2023), Sel et al., 2024), (AA al., 2024a). Dynamic-NTK and YaRN implement extrapolation by in- creasing the base frequency of RoPE. Dynamic-NTK and YaRN achieve extrapolation by increasing RoPE's base frequency, while ReRoPE, Self-Extend, and DCA adjust the position matrix to avoid unseen positions. We implement these baselines using their official reposito- ries. For baselines that adjust the base frequency (Dynamic-NTK and YaRN), we extrapolate to the target length by searching for the optimal base frequency. For methods that modify the relative distance matrix, we achieve extrapolation by scaling all position indices within the pre-trained length. All other configurations remain the same in their paper. Detailed implementation can be found in Appendix C.
Table 1: Results of the peedle-in-a-haystae (4 needles) evaluation for seven instruct models across various methods. Lirain refers to the pre-trained length, and Liest is the test length used for evaluation, with 100 test cases in total.
Model
Lirann / Lust
RoPE
NTK-Dyn YaRN ReRoPE Self-Extend DCA
DPE
Llama-3-8B-Inst.
8k/128k
0.00
0.75
0.00
78.00
89.50
53.00
92.50
Mistral-7B-Inst.r0..2
32k/128k
0.50
73.50
78.75
87.00
79.50
74.25
84.25
Mistral-7B-Inst.od.3
32k
11.50
94.50
88.50
87.00
93.75
85.75
96.25
Llama-3.1-8B-Inss.
128k/128k
94.25
96.50
92.00
94.50
96.25
92.75
97.25
Llama-3.1-70B-Inst
128k/128k
88.00
95.00
86.50
89.50
90.00
92.50
95.50
Qwen-2.5-7B-Inst.
128k/128k
22.25
68.25
68.75
53.50
62.75
22.25
75.75
Average
36.08
71.42
69.08
81.58
85.29
70.08
90.25
2023) involves identify- ing a specific, relevant piece of information (the "needle") within a large set of irrelevant data (the "haystack"). This task is widely used to assess the precision and recall of large language models (LLMs) in situations where crucial information scarce and embedded in substantial noise. Since single-needle retrieval is no longer challenging task for current LLMs (Hsieh et al., 2024; Yen et al., 2025), and we adopt the multi-needle setting following Llama 3(Grattafiori et al., 2024). We evaluate DPE on six widely used models, the results are shown in Table 1. DPE achieves significantly higher average performance across six models, reaching 90.25%, compared to only 36.08% for the original RoPE.
Needle-in-a-Hasstcck Needlleinna-HaarthccMMMAAAAAiii
RULER RULER (Hsieh et al., 2024) enhances the standard NIAH test by incorporating variations with eight different types and quantities of needles. RULER also introduces new task categories, such as multi-hop tracing and aggregation, such as word extraction and question answering (QA). We use 100 test cases for each task and the result are shown in Table 2. For extrapolation on Llama3-8B and Mistral-v0.2-7B, DPE significantly enhances the models' extrapolation capabilities, surpassing all baseline methods. On Llama3-8B, it achieves 56 point, outperforming the next best method Self-Extend by 8 points. Applying our method to the latest long-context models yields remarkable improvements: an 8-point improvement on Llama-3.1 3.1---1--8 3.1-8B and over 40-point improvement on Qwen-2.5-7B More baseline
7

---

Published as a conference paper at COLM 2025
Table 2: We evaluate the performance of various models and methods on RULER using tested sequence length of 128K. The RULER benchmark comprises 13 tasks, grouped into four categories: N
(NIAH), Variable Tracing (VT), Aggregation, and Question Answering (QA). We present the average scores for each category across all 13 tasks. Ltain represents the pre-trained length, and Ltest denotes the length for evaluation.
Models
Lirain/ Liest
NIAH
VT
Aggregation
QA
Avg. (13 tasks)
Llama2-chat
4K 4K
97.63
61.20
88.52
62.50
88.02
GPT-4-1106-
128K 128K
84.8
99.6
79.7
59.0
81.2
Llama3 (8B)
SK 128K
0.00
0.00
0.00
0.00
0.00
+ NTK--Dnamic
8K 128K
15.09
19.60
38.17
0.00
16.67
+ YaRN
8K 128K
10.00
2.80
21.92
0.00
9.74
+ ReRoPE
SK 128K
51.84
78.60
38.17
30.50
48.51
+ Sell-Extend
8K 128K
54.69
34.60
44.84
34.50
48.52
+ DCA
8K / 128K
47.03
43.40
44.52
34.50
44.44
DPE
8K 128K
64.72
49.60
42.34
38.50
56.08
Mistral-v0.2 (7B)
32K / 128K
9.44
0.00
32.34
10.50
12.40
+ NTK-DDnamic
32K / 128K
58.81
80.20
46.49
44.50
56.36
+ YaRN
32K / 128K
70.94
83.80
45.10
34.50
62.35
+ ReRoPE
32K / 128K
69.56
56.20
44.32
27.00
58.10
+ Self-Extend
32K / 128K
76.44
57.00
45.50
43.00
65.04
+ DCA
32K / 128K
64.47
84.20
47.44
45.50
60.45
+ DPE
32K / 128K
77.63
52.00
46.32
53.50
67.13
GradientAI/LLane Alammma3 (8B)
1M / 128K
89.22
56.80
36.20
54.50
73.23
Phi3-medium (14B)
128K 128K
53.75
6.80
45.80
47.50
47.95
Llama3.1 (8B)
128K
128K
89.47
60.00
36.89
56.50
74.04
+ DPE
128K / 128K
96.97
92.40
38.00
60.50
81.93
Qwen2.5 (7B)
128K 128K
31.38
29.20
28.07
21.00
29.10
+ DPE
128K 128K
82.72
80.00
43.22
46.00
70.78
Llama3.1 (70B)
128K 128K
76.38
58.00
41.14
56.00
66.41
DPE
128K 128K
95.94
98.00
55.77
73.00
86.39
results on 1lama-3.1-8B and Qwen-2.5-7B can be found in Appendix D. We also validate its effectiveness on 70B-scale models. Notably, Llama3.1 1-70B with DPE surpasses GPT-4-128K in average performance. The significant performance improvement demonstrates the importance of precise modifications across different dimensions for model performance.
Table 3: Comparison of DPE with three leading commercial long-context models on In- finiteBench. Each model is evaluated using maximum context length of 128K.
Commercial Models
Llama3 8B
Llama3.1 8B
Qwen2.5 7B
Tasks
GPT-4
Claude2
Kimi-chat
RoPEnsigio
DPE
RoPEnsegee)
RoPEEsege
En.QA
22.22
11.97
16.52
0
6.93
13.45
12.88
9.19
10.07
En.MC
67.25
62.88
72.49
0
47.16
65.50
68.99
45.85
68.12
En.Dia
8.50
46.50
9.50
20.50
13.50
Retr.PassKey
100.00
97.80
98.14
0
100.00
100.00
100.00
100.00
99.32
Retr.Number
100.00
98.14
94.42
0
99.49
99.32
100.00
93.39
98,98
Retr.KV
89.00
65.40
53.60
0
0.20
56.20
85.80
0.00
29.00
Code.debug
39.59
2.28
18.02
23.10
23.35
26.40
27.16
27.92
Math.find
60.00
32.29
12.57
30.29
32.18
34.57
37.71
37.71
Avg.
60.82
52.16
47.16
39.58
51.31
55.71
41.41
48.08
InfiniteBench InfiniteBench (Zhang et al., 2024b) consists of both synthetic and realistic tasks covering a wide range of domains. It is specifically designed to assess the under- standing of long-range dependencies in context, making insufficient to solve these tasks by merely retrieving a limited number passages. Table 3 compares DPE (on Llama3 8B, Llama3.1 8B, Qwen2.5 7B) against original RoPE baselines and leading commercial
8

---

Published as a conference paper at COLM 2025
Table 4: Results of in-context learning (ICL) and summarization (Summ) tasks on 128k
context length from the HELMET benchmark.
Model
train/ Leval
ICL
Summ
Avg.
Llama-3-8B
8K / 128K
0.0
7.9
3.9
DPE
8K / 128K
85.9
21.3
53.6
Llama-3.1-8B
128K / 128K
83.9
24.3
54.1
DPE
128K / 128K
83.6
26.3
54.9
Qwen2.5-7B
128K / 128K
72.0
19.6
45.8
+ DPE
128K / 128K
71.9
22.2
47.0
models (GPT-4, Claude2, Kimi-chat). Results show DPE significantly enhances open-source model capabilities. Notably, DPE enables Llama3.1 8B and Qwen2.5 7B achieve scores comparable to commercial models like Claude2 and Kimi-chat.
HELMET HELMET (Yen et al., 2025) is a comprehensive benchmark encompassing seven diverse categories long context tasks. To better demonstrate the effectiveness of DPE in long-context understanding and reasoning capabilities, in-context learning (ICL) and summarization tasks (Summ) on 128k context length from the HELMET benchmark. Results are shown in the table 4.
evaluate it the many
Analysis
5.1
Analysis on DPE's Dimensions
Usma3-80
Lama7 Ro-k
0.8
Owen2 5-70
Nistral78--022
NIAH Accuracy
NIAH Accuracy
Owen2.5-77
0.6
Owen225 5. Ladhis
0.4
0.2
6.0
16
24
32
40
40
50
Rank of Effective Length
Top-k Key Dimensions
(a) Ablation on the rank of effective length
(b) Ablation on the key dimensions selection
Figure 5: Ablation study on the rank of effective length and key dimension selection.
We conduct ablation study on the Keedle-i--a-Haysteee (4 needles) task to examine the impact two main hyperparameters: effective length and key dimensions in DPE.
Effective Lengths Across Dimensions In Figure 1b, we rank all the detected lengths for each dimension group. We use the most effective length (rank-1) for each dimension group. Here, we use rank-n for the ablation study, where 1,2,..., ., In Figure 5a, when и > the model performance begins to decline rapidly. This demonstrates that selecting the most effective relative distance for all dimensions is crucial for extrapolation.
Key Dimensions for Extrapolation DPE only manipulates the relative position matrix of the top-k key dimensions for Extrapolation. We gradually increase k from 0 to 64 and scale the maximum relative distance of all dimensions to their most effective length. We also conduct comparison by selecting the last-k key dimensions. In Figure 5b, when selecting the top-k dimensions, the model performance reaches its peak at k 40, outperforming modifications applied to all dimensions. In contrast, when selecting the last-k dimensions, all models' performance does not restore until all dimensions selected.

---

Published as a conference paper at COLM 2025
5.2
Efficiency Analysis
3.4
m
Average time per token (s)
Flash-Attentinn
Peak GPU Memory (GiB)
Flash-A Aiiss--Attettion
2.5
wm
Teaash-Attentiona
2.404
207
34.96231.907
1.4-
25. 25.66825..00 060
1.0-
20
0.746 0182
6.-
0.40
64k
99
126h.
22
64h
129A
Sequence Length
Sequence Length
Figure 6: Efficiency Test of DPE and the standard Flash Attention based on Llama3.1 8B.
We evaluated the efficiency DPE on Llama3.1-8B on an NVIDIA H800 GPU, comparing inference time and peak memory usage. As shown in Figure 6, DPE introduces negligible overhead: inference speed per token remains highly comparable (2.404s vs. 2.346s at 128k), and peak memory usage consistently similar (53.60 GB vs. 53.64 GB at 128k).
Related Work
Modeling long context has consistently been a challenging task. Recent advancements in large language models (LLMs) have stimulated researchers to explore various ways to extend the context window these models.
Attention Mechanism One way to model long context is to limit the number of attended tokens during inference. Representative works such as StreamingLLM(XXaa et al., 2024b) and LM-Infinite(Han et al., 2023) have shown that LLMs can generate infinite context length. Lu et al. (2024b); Xiao et al. (2024a); Jiang et al. (2024) divide long sequences into context chunks and select relevant chunks for inference. Further improvements, such as NSA( et al., 2025) and MoBA(Lu et al., 2025), demons trate that the performance and efficiency of the chunk selective method can be optimized by continual training and self-designed kernels. However, these methods typically cannot maintain full KV cache, resulting in weakened long-context capabilities.
Modify RoPE's Frequency Researchers have proposed various methods to scale the base of the frequency basis mitigate the OOD issue, with representative works including (Chen et al., 2023; bloc97, 2023a; emozilla, 2023; Chen et al., 2024). NTK-by-parts (bloc97, 2023b), YaRN (Peng al., 2023), and LongRope (Ding et al., 2024) apply different interpolation and extrapolation strategies to different dimensions based on the properties of RoPE. However, since these methods modify the frequency RoPE, they generally require additional training to achieve optimal performance.
Manipulate Position Embeddings An et al. (2024a); Su (2023); Jin et al. (2024); Zhang et (2024a) reuse the original position embeddings and manipulate the relative position matrix to avoid the presence of unseen relative positions, thereby enhancing extrapolation capabilities. An et al. (2024b) shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths.
Conclusion
In this paper, we present DPE as a novel and efficient approach to overcoming the context length limitations in LLMs. By selectively manipulating the relative position matrix across RoPE dimensions, DPE ensures that only the most critical dimensions' relative position indices are adjusted to the most effective lengths while preserving the integrity of others. This targeted adjustment enables effective context extension without the need for costly continual training, significantly enhancing the long-context capabilities of LLMss.
10

---

Published as a conference paper at COLM 2025
References
Mistral AI. Mistral OCR, 2025. URL
Al@Meta. Llama 3 model card. 2024. URL htt.httss///itchu..cc////eeeeett main/MODEL_CARD. md.
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free free long-context scaling of large language models, 2024a.
Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong- Why does the effective context length of llms fall short?, 2024b. URL https://arxiv.orgg htrrg/bbs/2110 18745.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?, org/abs/2418 06205.
2025. URL httpp:///axxiii wwtpeewwwwww...r/eereeeerreere/e
bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degrada- tion, 2023a. URL https: /twtpp::/www..rte reddit. www.://ooaaaaLaMa///aaaaacc/ hwtpl//www..ccooo///////////eee Bagilee_oppaaaalloss_aaaaaaaeeee
bloc97. Add SwwTK--AAwaareiteepplaaatti "by parts" correction, 2023b. URL https:/s://tthhtt com.://uussnelllesscccccrrrrrrpp
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Con- tinuous length extrapolation for large language models, 2024. URL abs/2310. 6450.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, Jan 2019. doi: 10.18653/vin 11/6663//1/119122// URL: //xx doi org/10. 18653/v1/p19-1285.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https::/arxizs wrg./bs//3///// https::/amma..orr/bb///000. 2866.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongüe Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiahhi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao
11

---

Published as a conference paper at COLM 2025
Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang, Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning, 2025. URL https:t//atiiii http:////aariiooorga///////// orgg 129448
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens, 2024. URL https://arxiv, www.vv.rrr///bae httrgg/ab////.... 13753.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar arv:: 217833 2024.
emozilla. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning, 2023. URL https::/www.reeddit.c
reddit. http:////////LaaaaLa///////o
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and
Hao Peng. Data engineering for scaling language models to 128k context, 2024.
gkamradt. Needle in haystack - pressure testing lims, 2023. URL https: //gi://b.........////gg com/ SGGamaattt/LLLTTsstteeeeeeLEAAls
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chem, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Isimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean
12

---

Published as a conference paper at COLM 2025
Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara
13

---

Published as a conference paper at COLM 2025
Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama 3 herd of models, 2024. URL httpp:://aree org/ 21783.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models, 2023.
Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng, Mo Yu, Bowen Zhou, and Jie Zhou. On the token distance modeling ability of higher rope attention dimension, 2024. URL https://arxiv.orr wttp://wwwwr///////////// 08703.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024.
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's s multi-head latent attention in any transformer-based Ilms, 2025. URL https://arxiv.. org/ htt.v/.orrgabb//2..... abs/2502. 14837.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.. orgg httg:/abs2231...0555 26822068........
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention, 2024. URL pattps//aarxit wwriii.orr//ae htrg/abs/2401 02490.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huïyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend Ilm context window without tuning, 2024.
Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, and Xipeng Qiu. Thus spake long-context large language model, 2025. URL // /ttp////xvvvv./////// wwwlv.oorg//aaoo arg/abs/2502. 17129.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang. Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture of block attention for long-context Iims, 2025. URL
wwtrrxiv..r///... org/abs 2502. 13189.
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. A controlled study on long context extension and generalization in Ilms, 2024a. URL https:
wrw. 12181.
Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Longheads: Multi-head attention is secretly a long context processor. arXiv preprint arXiv:2402.10685, 2024b.
OpenAl. Learning to reason with lims, 2024. Boariing-to-reason-i th- wimmml
URL httpp://openni..i wwwpinaai..omiiiiiaaaeea index/
14

---

Published as a conference paper at COLM 2025
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: window extension of large language models, 2023.
Efficient context
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 5wen2.5 technical report, 2025. URL httpps://arxiv. org abs//4/2.....22///// 2412. 15115.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compres- sive transformers for long-range sequence modelling, 2019.
Jianlin Su. Rectified rotary position embeddings. http:://g..tbb....mmoooooorrrrrp 2023.
com/bojone/rerope,
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer:
Enhanced trans- former with rotary position embedding. Cornell University ar arXiv,Cormen University arxiv, Apr 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL /ttpoee//pocceeinggss neuri 3f5ee243547deee1
neurips ipsrrpps..oo
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of lims for understanding extremely long sequences with training-free memory. Xiv preprint xrXiv:2402.04611, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks, 2024b.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023. URL https: / //tp///aavvvv//////a//eee/eaaavl http///ai....gabs////////r org/abs/2309. 110999 16039.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly, 2025. URL httt::::/aaxx..or/abb/////////// orgg/bs/////// 02694.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL https: / //axxva.avvv////////aa/eeeaaaiv wwwprxiv..oogaaaa argg/bss/50a 11089.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal 11m with discrete sequence modeling, 2024. URL http:::/arrx..orr/abb/2222.2 12226.
org/
Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. Hirope: Length extrapolation for code models using hierarchical position, 2024a. URL https://arxit wwwp:ivvv.orgargeooe org. abs//////// 191 15.
15

---

Published as a conference paper at COLM 2025
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extend- ing long context evaluation beyond 100k tokens, 2024b. URL https://arxix: org/ / 2402. 13718.
A Details about Detecting Effective Relative Distance on Different Dimensions
Max Relatio Distance
Max RobIoe Dis PP00OO000
113
500
2040
14.3
264
181
10.3
102
350
Dimension
611
1633
Dimension
36.30
30:30
#047
5045
911
19:21
26:1
2208
69-7
48:00
3642
Figure 7: NIAH Accuracy ranking(left), NTAH Accuracy(middle) and perplexity value(right) of different dimension groups and different detecting distance.
We use Llama3-8B-Instruee to detect the effective relative distance. We evaluate the perplexity using (Rae al., 2019) and assess the model's performance on Needle-in-a- Haystack(NIAHi) (gkamradt, 2023) with 100 test samples. We divide head dimensions into 8 groups. For each group, we detect different lengths from other groups, also introduce local window W = 1k and scale the maximum relative distance to half of the pre-trained length. In Figure 7(right), we observe that the variation in perplexity differs significantly across dimensions, with the largest fluctuations occurring in dimensions 32-56. This also indicates that different dimensions contribute differently to length extrapolation.
128k and set W 1k. For
B Pseudocode for DPE
We present the pseudocode for the three main procedures of DPE: Detecting Effective Relative Distance for Different Dimensions (Algorithm 1), Tdentifying Key Dimensions (Algorithm 2), and DPE FlashAttention Forward (Algorithm 3). Algorithms 1 and 2 serve as preparatory steps for Algorithm Once the effective relative distances for different dimen- sion groups and the key dimensions for RoPE have been identified, DPE FlashAttention Forward can be performed accordingly.
Algorithm 1 Detecting Effective Relative Distance for Different Dimensions
Input: dinensian_groupp grrups. group_num, target_lengta detecting_lengths
Cutput: mest_scaleefactorr
L_rrttriin / detect seaeett_sceeeeaaatttoo [s for in range range(group_nug
best_scale_factors = [s for in (ango((grup__un)
for group_id in range
for detecting_length detectingg detect__cale__ "actorrgrruu seteetdddeeettsale_.feeee
10
target_length letectiin_
11
accuracy accuracy best_accuracy: seteet_ cccaleffactorggrou..iie
13
sest_scale.accooss[greee
16

---

Published as a conference paper at COLM 2025
Algorithm 2 Identifying Key Dimensions
Input: Q, tensors with shape [L. d], topk topk_dimensions
Cutput:
Q_norn
K_narn = rorm(oOreehhapeL
2). din=-1)
d//2, 2) dim=-1)
OK_norn mean (Q_nern * K_norn, din=0)
topk_dimensions topk nOk_norr,, k=topk)
Algorithm DPE FlashAttention Forward
Input: Q, K, [L, d1, window size w, scale_factors, topk_din
Output: attention_output
<--- Calculating position for dimension groups ---> L-1]
positien [e, 1,
For group_id in (grou__pr___))))
ficale_factorssgrouu
position
ccale_K.positioo(@frauu
position //
11 <--- Calculating frequency for dinession groups --->
12 Cas Sin ratary_enb pS,iitiio)))
group_id range (group_po___))))
14
Scale_Q_Cos [group_idd, Scale___Sinngroup..te rotary_enb Scale_Q.pasitiint
group_id] id])
15
Scale_K.Sin(Eroup.lk rotary_enb Scale_K.posittinn
group_id])
16
17 <--- Get final frequency --->
Scale_Q_Cos, Scale_Q_Sin, Scale_K_Cos, Scale_K_Sin din
19 Scale_Q_tos, Scale_Q.Sin, scale_K_Cos, Scale_K_Sin
20
21 <--- Sliding window attention with normal positions --->
22 Q.
roply_rottrr_pos_.mme K. Cos, Sin)
0_sliding, LSE_sliding flash_attn(Q,
sliding.windoweN)
24
25
Attention left-botton triangle with scaled positions
26 N = LL-
27 Q_scale, K_scale applyrrotery_po__em0Q K. Scale_Q_Cos, Scale_Q_Sin, Scale_K_Cos, Scale_K_Sin)
38 0_scale, LSF_scale = flash_attn(Q_scaae[--i K_scale[:N], V[:N])
30 <--- Nerge the attention outputs --->
sttention_output merge_attentions (O_sliding 0_scale, LSE_sliding, LSE_scale)
C Implementation Details
C.1 Implementation of DPE
We divide the head dimension into 8 groups. The final effective length for Llama3-8B- Instruct was determined based on the ranked NIAH accuracy results shown in Figure 1b. For each 8-dimension segment, selected the configuration yielding Rank performance. In cases where multiple configurations achieved Rank 1, we chose the one corresponding to the largest relative distance. We set the detected effective length E: 65536 (dimensions 0-7), 16384 (8-15), 65536 (16-23), 16384 (24-31), 4096 (32-39), 4096 (40-47), 8192 (48-55), and 32768 (56-63), After detection, we select top-48 the key dimension for length extrapolation.
C.2 Implementation of Baselines
We report the hyperparameters of baselines on Llama3-8B-Instrude in our implementation. For NTK-Dynamic, we set the scale factor s to 16. For YaRN, we set beta fast to 32, beta slow to 1, scale factor to 16, and attention factor to log 4. For Belf-Extend, we set the local window size to 1024 and Group size g to 32. For Rerope, we set the trucated length to 2048. For DCA, set the chunk size to ] of the pre-trained length 8k.
17

---

Published as a conference paper at COLM 2025
D More Results on RULER
Supplementary experiments provide RULER benchmark results for Llama3.1-8B and Qwen2.5-7B, evaluating baselines such as NTK-Dynamic, YaRN, ReRoPE, Self-Extend, and DCA.
Table 5: We evaluate the performance of various models and methods RULER using tested sequence length of 128K. The RULER benchmark comprises 13 tasks, grouped into four categories: Needle-in---Haasttad (NIAH), Variable Tracing (VT), Aggregation, and Question Answering (QA). We present the average scores for each category, along with
the overall average across all 13 tasks. Ltrain represents the pre-trained length, while Liest
denotes the test length used for evaluation.
Models
Lirain / Liest
NIAH
VT
Aggregation
QA
Avg. (13 tasks)
Llama2-chat
4K 4K
97.63
61.20
88.52
62.50
88.02
GPT-4-1106-previee
128K 128K
84.8
99.6
79.7
59.0
81.2
Llama3 (8B)
8K 128K
0.00
0.00
0.00
0.00
0.00
+ NIK-DDnamic
8K 128K
15.09
19.60
38.17
0.00
16.67
YaRN
8K 128K
10.00
2.80
21.92
0.00
9.74
+ ReRoPE
8K 128K
51.84
78.60
38.17
30,50
48.51
+ Self-Extend
8K / 128K
54.69
34.60
44.84
34.50
48.52
DCA
SK 128K
47.03
43.40
44.52
34.50
44.44
+ DPE
8K 128K
64.72
49.60
42.34
38.50
56.08
Mistral-v0.2 (7B)
32K / 128K
9.44
0.00
32.34
10.50
12.40
+ NTK-Dynamic
32K / 128K
58.81
80.20
46.49
44.50
56.36
YaRN
32K / 128K
70.94
83.80
45.10
34.50
62.35
+ ReRoPE
32K / 128K
69.56
56.20
44.32
27.00
58.10
+ Self-Extend
32K / 128K
76.44
57.00
45.50
43.00
65.04
+ DCA
32K 128K
64.47
84.20
47.44
45.50
60.45
+ DPE
32K / 128K
77.63
52.00
46.32
53.50
67.13
GradientAI/Llame (8B)
1M 128K
89.22
56.80
36.20
54.50
73.23
Phi3-medium (14B)
128K 128K
53.75
6.80
45.80
47.50
47.95
Ilama3.1 (8B)
128K 128K
89.47
60.00
36.89
56.50
74.04
NTK-Dynamic
128K 128K
95.94
77.00
38.34
61.50
80.32
+ YaRN
128K 128K
83.09
91.80
35,39
54.50
72.02
+ DCA
128K 128K
90.69
85.40
38.17
61.00
77.63
+ Self-Extend
128K 128K
97.03
92.80
37.50
62.50
82.23
+ ReRoPE
128K / 128K
89.22
65.20
38.00
55.50
74.30
+ DPE
128K 128K
96.97
92.40
38.00
60.50
81.93
Qwen2.5 (7B)
128K 128K
31.38
29.20
28.07
21.00
29.10
+ NTK-DDnamic
128K / 128K
72.75
2.60
38.34
43.00
57.48
YaRN
128K 128K
70.59
83.40
44.27
49.50
64.28
+ DCA
128K 128K
53.69
21.00
9.40
21.00
39.33
+ Self-Extend
128K 128K
69.72
86.20
37.49
38.00
61.15
ReRoPE
128K 128K
63.50
28.60
37.82
29.50
51.63
+ DPE
128K 128K
82.72
80.00
43.22
46.00
70.78
1lama3.1 (70B)
128K 128K
76.38
58.00
41.14
56.00
66.41
DPE
128K 128K
95.94
98.00
55.77
73.00
86.39
18


Reviews:
[{"review_id": "3I5TJtalBL", "reviewer": "Reviewer_DyvB", "summary": "This paper proposes a novel method, DPE, for LLM length extrapolation. Different from previous work which manipulates all feature vector dimensions uniformly, the authors propose to identify the effective maximum range for different dimension (group)s, and then identify dimensions that hurt performance if not manipulated by filtering dimensions that contribute most to the logit range. Finally, the authors manipulated those dimensions by scaling the distances longer than 1k to the effective maximum range.\n\nExperiments demonstrate improved performance on a standard and challenging retrieval task: needle in a haystack (4 needles), in both train-short-test-long and train-long-test-long scenarios compared with other length extrapolation methods. On more comprehensive benchmarks such as RULER and InfiniteBench, sometimes compared to vanilla models and sometimes compared with length extrapolation methods. The implementation is also shown to be similarly efficient when integrated into flash attention.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel method for length extrapolation\n1. Insights into the barriers in extrapolating length: not all dimensions are equally important for length representation, and different dimensions have different effective maximum length range to scale to.\n1. Experiments suggest improved scores on retrieval-focused tasks (needle in a haystack) compared with other length. On more tasks combining other natural capabilities, such as VT and QA, score are comparable to best methods or just normal. The average score is, however, superior\n1. Ablation study was included to provide more support for the studied insights", "reasons_to_reject": "1. unclear technical choices: even though top-k key dimensions peak their performance around 40 in Fig 5b, why selecting top 48 would be better than 64 (i.e., all). In other words, in Figure 5b, would using a top k = 64 make DPE equivalent to any existing baseline?\n1. can authors provide more rationales in what baselines to include and not in tables 2 and 3 (computation budgets or?)\n1. performance comparisons among methods are not consistent across tasks, especially when comparing VT with other tasks. Why is it so? Any analysis, explanation, or breakdown of errors?", "questions_to_authors": "line241 - 242 repeated sentences\n\ngrammar in lines 261 - 262\n\nAny mechanical explanations behind the phenomena observed in Figures 1 and 2, or any related work making those efforts? These discussions, even if not directly supporting your method, would provide a more comprehensive view for readers.", "extra_scores": {}}, {"review_id": "IAYE6ghYMm", "reviewer": "Reviewer_HAao", "summary": "This paper aims to design a new position encoding algorithm for extending the context window for LLM.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed algorithm is train-free and the author give the strong mitivation of how to reassign the relative position in a differentiated way.\n\n2. The performnace of this work is good, compared to the existing algorithm.", "reasons_to_reject": "1. The training free strategy is incremental, very similar to \n\n1) Jin, Hongye, et al. \"Llm maybe longlm: Self-extend llm context window without tuning.\" arXiv preprint arXiv:2401.01325 (2024).\n2) Xiao, Chaojun, et al. \"Infllm: Training-free long-context extrapolation for llms with an efficient context memory.\" arXiv preprint arXiv:2402.04617 (2024).\n3) An, Chenxin, et al. \"Training-free long-context scaling of large language models.\" arXiv preprint arXiv:2402.17463 (2024).\n\n\n2. lack the implemented details.", "questions_to_authors": "1. The training free strategy is incremental, very similar to \n\n1) Jin, Hongye, et al. \"Llm maybe longlm: Self-extend llm context window without tuning.\" arXiv preprint arXiv:2401.01325 (2024).\n2) Xiao, Chaojun, et al. \"Infllm: Training-free long-context extrapolation for llms with an efficient context memory.\" arXiv preprint arXiv:2402.04617 (2024).\n3) An, Chenxin, et al. \"Training-free long-context scaling of large language models.\" arXiv preprint arXiv:2402.17463 (2024).\n\n\n2. lack the implemented details. Can you give some presudo/code for how to implement this idea.", "extra_scores": {}}, {"review_id": "ZzAyj8ushS", "reviewer": "Reviewer_Cdu5", "summary": "This work proposes a training-free length extrapolation method based on RoPE, named as DPE. Different from previous method of ReRoPE, DPE proposes dimensional-sensitive frequency adjustment. The overall idea is like DPE differentiates the most effective key dimension for each attention head and only adjust the frequency of these top key dimensions. The authors demonstrate such strong hypothesis via empirical studies in Figure 2 and 3. The performance is remarkable compared with existing training-free length extrapolation methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is training-free and easily-implemented based on FlashAttention. It introduces almost no computational overhead and achieves great inference efficiency.\n\n2. The performance gain on RULER, NIAH, and InfiniteBench are very significant.", "reasons_to_reject": "I am new to the positional embedding area since I worked on data-centric methods for length extension earlier and cannot find some key drawbacks of this work. But through my research experience on long-context models, almost all long-context benchmarks reveal nothing about the long-context understanding capabilities across models. It feels like the metric scores are very random across different models. For two models evaluated on InfiniteBench, one model will win some of tasks while another one will win the remaining tasks. I would like to see the results on many-shot in-context learning as a strong proof on the long-context understanding and reasoning capabilities.", "questions_to_authors": "1. For qwen-2.5-7B in Table 1, is the RoPE column the direct evaluation of the original model on NIAH task since the original model adopts RoPE? It feels like it is lower than expected.\n\n2. Will the DPE downgrade back to normal RoPE when the evaluation context length is smaller the pre-training length?\n\n3. Why Llama-3-8b is worse than llama-2-chat on RULER in Table 2? It seems like for Llama-2-chat model, you only evaluate the model on 4k test length. Then the performance is even better than the Llama-3.1-70b-dpe model on average. In this case, why we need length extrapolation for models based on the results on RULER?", "extra_scores": {}}, {"review_id": "tR3MAWb67i", "reviewer": "Reviewer_X83u", "summary": "This paper introduces Dimension‑Wise Positional Embeddings Manipulation (DPE), a training‑free framework that extends the context window of pre‑trained rotary positional encoding (RoPE) models by selectively rescaling relative position indices in key hidden dimensions. By detecting per‑dimension \"effective lengths\" and identifying the most contributive dimensions via a 2‑norm attention metric, DPE adjusts only a subset of RoPE dimensions to their optimal extrapolation ranges while leaving others untouched. Empirical evaluation on multiple long‑context benchmarks demonstrates that DPE outperforms existing training‑free methods by substantial margins, enabling, for instance, Llama3‑8B‑8K to handle 128K tokens with no additional training and improving performance on in‑distribution contexts by over 18 points on RULER. The approach integrates seamlessly with FlashAttention 2 and incurs negligible overhead.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles a critical bottleneck in deploying large language models for tasks requiring extensive context, offering a novel angle on RoPE extrapolation that departs from uniform scaling or full‑matrix truncation. Its dimension‑wise strategy is both conceptually elegant and practically efficient, as it leverages pre‑existing embeddings without any fine‑tuning. The thorough experimental suite—covering multiple architectures (Llama3, Mistral, Qwen) and challenging benchmarks—provides compelling evidence of DPE’s superiority over state‑of‑the‑art baselines, demonstrating not only extrapolation gains but also in‑distribution improvements. Moreover, the ablation studies convincingly justify the two core design decisions (effective‑length detection and key‑dimension selection), underscoring the method’s soundness. Given the method’s low barrier to adoption and significant empirical impact, publishing this work would provide immediate value to both academic researchers and practitioners seeking scalable long‑context solutions.", "reasons_to_reject": "Despite its strengths, the paper has a few shortcomings that might limit its immediate impact. The clarity of presentation suffers in places from heavyweight notation and dense descriptions of the detection procedure; a more intuitive visual or pseudocode of the full pipeline could aid comprehension. While the benchmarks chosen are appropriate for retrieval‑style tasks, the work does not explore the effect of DPE on generation quality for downstream applications such as summarization or code synthesis, leaving open questions about potential trade‑offs in fluency or factual consistency. Additionally, although the authors integrate with FlashAttention 2, the paper lacks a detailed runtime and memory‑usage analysis comparing DPE to baselines in production‑like settings; this data is crucial for practitioners evaluating the cost–benefit of adoption. Addressing these concerns would strengthen the case for publication", "questions_to_authors": "* Could you include a concise algorithmic summary or pseudocode for DPE, outlining the three stages (detection, selection, scaling) in a single place?\n* Have you measured how DPE‑adjusted models perform on generative tasks (e.g., long‑document summarization)? Does selective dimension scaling affect coherence or factual accuracy when producing free‑form text?\n* What is the wall‑clock overhead (in milliseconds per token) and additional memory usage of DPE at inference compared to vanilla RoPE and other baselines? Including a table of these metrics would clarify real‑world applicability.\n* The effective‑length curves vary widely across dimensions. Can you offer more theoretical intuition or proofs about why mid‑range frequencies collapse while extremes extrapolate well?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a training-free framework to extrapolate the context window of LLMs. All reviewers agree the novelty of the paper, and find the experiments promising. One reviewer mention some limitation of the proposed method, please ensure to include in the camera-ready version for the completeness and allowing community able to working on the follow ups.
Decision: Accept