MrRoPE: Mixed-radix Rotary Position Embedding
Authors: Qingyuan Tian
Wenhong Zhu
Xiaoran Liu
Xiaofeng Wang
Rui Wang
Venue: ICLR 2026 Oral (2026)
Keywords: transformers
nlp
llms
context window extension
attention
rotary embedding
Primary area: generative models
TL;DR: We present a unified theory MrRoPE linking major RoPE-extension methods to radix conversion. Based on this, we propose MrRoPE-Pro, a training-free context window extension method..

Abstract: Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose $\textbf{\textit{MrRoPE (Mixed-radix RoPE)}}$, a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, $\textbf{\textit{MrRoPE-Uni}}$ and $\textbf{\textit{MrRoPE-Pro}}$, which leverage uniform and progressive radix conversion strategies, respectively, to achieve “train short, test long” generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN’s accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.

Published as a conference paper at ICLR 2026
MRROPE: MIXED-RADIX ROTARY POSITION EMBED- DING
Qingyuan Tian', Wenhong Zhu', Xiaoran Liu², Xiaofeng Wang', Rui Wang** Shanghai Jiao Tong University, Fudan University {qy.tian, wangrui12] @sjtu.edu.cn
ABSTRACT
Rotary Position Embedding -xtenssn refers to modifying or generaliz- ing the Rotary Position Embedding scheme handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose Mr- RoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we intro- duce two training-free free extensions, MrRoPE-Uni and MrRoPE-Pro, which lever- age uniform and progressive radix conversion strategies, respectively, to achieve "train short, test long" generalization. Without fine-tuning, MrRoPE:Prr sustains over 85% recall in the 128K-context freedle-in---ayystae test and achieves more than double YaRN's accuracy on Müinite-Bencl retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pree effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.
INTRODUCTION
Effective understanding of long-text contexts is a cornerstone for advanced NLP tasks (Liu et al. 2023a). For Large Language Models (LLMs). this capability is fundamentally underpinned by the Rotary Position Embedding (RoPE) (Su et al. 2024), which provides a robust foundation for mod- eling long-range dependencies (Grattafiori et 2024 Yang al. (0225)). In RoPE, the positional information of tokens is encoded through rotation angles, where each embedding dimension is as- sociated with a distinct rotational frequency. Higher dimensions rotate more slowly, which implies that during training, these dimensions often fail to experience a complete cycle. As result, once the context length exceeds the training window, those nigh-dimeessioal features encounter unseen rotation angles, leading to generalization failure (Liu et al. 2023b). To address this issue, early studies proposed continuing training with larger base frequency, thereby compressing the rotation angles of higher dimensions into the previously observed range and mitigating the occurrence of out- of-domain (OOD) positions (Kazemnejad et al. (003)). However, this strategy requires substantial additional computation (e.g.. 57740 GPU hours for Llama2-70B to 32K context window), making
larger context extension prohibitively expensive (Touvron et al. 2023 Xiong et al. 2023).
Consequently, recent research has shifted toward training- free approaches extend the context win- dow of LLMs. [Chn et al. (2023) first proposed Position Interpolation (PI), which uniformly scales down the rotation angles across all dimensions
modify RoPE, though the method still requires light fine-tuning on a small amount of data. Alternatively. NTK-aware Interpolation introduced by bloc97 (2023) reduces the distortion in low-dimensional spaces, thereby enabling a substantial ex- pansion of the context window without any extra training. Building upon these insights, |eng et al. (2023) introduced YaRN, an NTK-by by-part strategy that applies extrapolation in low dimensions and interpolation in high dimensions, ultimately achieving efficient long-conter extension.
Despite the efffectiveness of YaRN, we identify potential limitation in YaRN's approach: Is linear interpolation used for middle dimensions truly the best choice for maximizing model performance
Corresponding author

---

Published as a conference paper at ICLR 2026
Distinct Frequency Transformation
Radix Conversion Methodology
NTK
YaRN
MiRoPE-UUni
MrRoPPE-Pro
RoPE
Regresstip
Progrexsive
Special Cases
NTK
MrRoPE Theory
B
YaRN
1
B
B
B
B
B
B
Unity by
Intermediate
Lower dins
Higher dims
dimensious
The overall framework of our work. Our key contributions are: (1) unified theoretical framework for major (2) a progressive radix conversion method MrRoPE-Pro, which outperforms other SoTA methods across various tasks.
Figure 1:
reflecting them into a specific radix conversion behavior;
on long sequences? To answer this question, we first introduce MrRoPE (Mixed-radix Rotary Position Embedding), unified theoretical framework to generalize existing RoPE-based extension methods (PI, NTK, YaRN ...) and reflect to a specific radix conversion approach. Guided by this framework, we identify that YaRN's conservative strategy in lower frequency dimensions may excessively disrupt the high-frequency information inherent the original RoPE encoding. Hence, further propose two novel RoPE extension schemes, MrRoPE-Uni and MrRoPE-Pro. which exhibit fundamentally different radix conversion behaviors from YaRN. Their design allows us to systematically compare the efficacy of different radix conversion strategies.
By employing progressive radix conversion to RoPE, MrRoPE-Pro achieves substantial and con- sistent gains across both synthetic and real-world long-context evaluations. On the RULER bench- mark, it maintains stable accuracy up to 128K tokens, whereas YaRN experiences sharp degra- dation beyond 64K. On infiniteBBench MrRoPE-Pro but also approaches even exceeds the performance of specialized fine-tuned long-context mod- els-all without additional training. Complementing these empirical findings, our theoretical anal- ysis demonstrates that MrRoPE-Pro significantly enlarges the effective context window predicted by RoPE Bound Theory and stabilizes attention score distributions in intermediate dimensions. Together, these results establish MrRoPE-Pro as theoretically grounded and empirically robust method for extending the context window RoPE-based LLMs.
only surpasses YaRN by large margins
ROPE AND RADIX THEORY
2.1 PRELIMINARY: ROTARY POSITION EMBEDDINGS (RoPE)
Our research is grounded in the RoPE introduced by Su al. (2024), position embedding scheme that underpins numerous state-of-the-ar LLMs. Consider one specific attention head in given layer of a Transformer-based model, the set of its hidden neurons is denoted by D. Given an input sequence of vectors 21...., æt E TR. DI, it's essential to maintain the relative position information between each (mm Fr,) in the result of the self-attention calculation. To address this issue, RoPE first uses rotation operation that converts vector &m into its corresponding query vector Om and key vector km (Vaswani et al. 2017):
{q, k)m f(g,k) (Im,, m) e im0 W (g.k) Tm:
(1)
where diag(O),.... (p11))))), is a diagonal matrix with 0; = b 2 1)/1P1 b is a predefined value, which is typically set to 10000. In this way, RoPE establishes injective relation between token's 's absolute position and the rotation steps of its embedding vector. Crucially, in the subsequent self-attention computation, the interaction between queries and keys inherently encodes relative positional information as follows:
(gm, kn)) (Sg (Mm, m), (æ,,, n)) Re (2: W W W. (m)-n))
where denotes the conjugate transpose operation. For this reason, the attention score depends solely the relative position rather than the absolute position, which facilitates L.MSs in
2

---

Published as a conference paper at ICLR 2026
more effectively capturing the relationships between tokens. In the implementation stage, e/m in Eq. can be further expressed as a block diagonal matrix as follows:
cos md, sin mtd sin m8j
cos mtj )
imm = diag( AI,, A A ); A
D dimensional complex vector, and applies block- wise rotations (i.e., each component will be rotated in its corresponding frequency domain where higher dimension gets slower frequency).
In conclusion, RoPE treats q or vector 2
2.2 RETHINKING RoPE UNDER THE RADIX THEORY
As we showed above, the key idea of RoPE is to divide the whole qm (or km) vector into D, parts (i.e., D, DI/2), and rotate each sub-vector by a corresponding angle md, (i.e., j 1,2, D,). Formally, given non-rotated complex vector W (g E CD., its one- dimensional position m is expanded to D,- dimensional vector mf E R.Dr by RoPE. The rotation angle m0, for j-th part can be calculated as follows:
m0, (m - mod 2T.
(4)
Most notably, Eq. 4 strikingly similar to what radix (base) conversion does: representing number as a sequence of right to left, starting at j = 1) of m(s) is given by:
digits. Specifically, given decimal number m(10), the j-th digit (counted from
(m())); = m g 1- mod B.
(5)
Then, the relative position m can be calculated as:
m f(m())) = (0-11) (0))))(())))
(6)
(--11 with Eq. Remarkably, both the modulo and trigonometric functions contribute to the periodicity. Based on this observation, Su (2023) hypothesizes that if we temporarily disregard the influence of the flooring operation as well as the period size of the modulo operation, RoPE essentially performs radix conversion from the decimal system to a gr/Dx-radix representation.
Eq. indicates that when b2/D., it shares a common
To prove this hypothesis, reintroduce the ignored components (the floor function and modulo function) to recover biased positional estimate m from the RoPE as follows:
Dr
Dr z0 1) (m0,) = Σ
in = g(m)) = Σ
(m b
+ mod 27
As shown Figure 2 the biased function recovered from RoPE exhibits linear trend analogous to that in Eq 6 This linearity is confined shorter distances for smaller bases, but becomes markedly more pronounced and ex- pands encompass the full context window as the base increases, suggesting that larger bases possess greater po- tential for representing positional information over long sequences, which is in accordance with previous findings (Liu et al. 2023b; Men et al. 2024)..
X10F
2.0
Extradedd Detunee
Overall, this section established RoPE biased 3- radix encoding by combining theoretical derivation with empirical linearity analysis. In the next section, we leverage this radix perspective to unify prior context extension methods.
0.0
00000
50000
100000
120020
Real Distance
Figure 2: The biased positional estimate in of RoPE across different base.

---

Published as a conference paper at ICLR 2026
3 MRROPE: A UNIFY FRAMEWORK FOR ROPE EXTENSION
3.1 MRROPE THEORY
Why do need RoPE extension? In RoPE, generalization failure of test-length position infor- mation stems primarily from the OOD problem of incomplete-cyck dimensions (i.e., dimensions j such that L/0, < 2 (Liu al. 2023b; Ding et al. 2024 Huang et al., (023). From the radix theory introduced above, this phenomenon directly parallels the behavior of high-digit truncation in a radix encoding system: Consider a 3-radix encoding system, when the input is restricted to
the range [o, L]. all digit positions starting from the d-th and higher never experience a complete carry-over cycle:
L 1- mod B 1, for = d, 1.
natural solution to ucale-up such a unbalanced radix system, to scale the radix base for digits before dimension d: Given a 3-radix, if the j-th digit's base is expanded by factor Xj. the repre- sentable range of the system is rescaled by a factor of [ >3. then the j-th digit of this expanded system be expressed as follows:
D.
8-0-1) 1)
(m(an))) m
mod (BXy),
T Aa
where m is the target position for embedding. By rescaling the radix base, the extended mixed radix system is able to prolong the rotational period of each dimension, thereby addressing the high-digit OOD challenge as mentioned above.
Extending RoPE via Mixed Radix Rotary Embedding (MrRoPE). In RoPE, similar mixed-
radix conversion can also be implemented. By assigning each dimension an independent frequency scaling factor, a mixed-radix RoPE extension defined as:
mf² = (m
mod 2T. {q, kmm = e W kn. Cm.
(10)
Ad
The above formulation establishes the Mixed- -adix RoPE (MrRoPE) framework: any RoPE- based extension method conforming to Eq. 10 instantiates a mixed-radix conversion on po- sitional encoding. The framework's 's explanatory power stems from its parameterization: the vec- tor A = {A1, A2,..., } defines the radix conversion factors for each dimension. Consequently, MrRoPE posits that the choice of length extension method is fundamentally equivalent choos- ing a specific policy for redistributing positional information across the frequency spectrum via A. This provides a unifying lens through which existing methods can be systematically analyzed as constrained instantiations of this general conversion process. We next demonstrate how both NTK- aware scaling and YaRN are naturally recovered under this framework.
NTK-awre Interpolation under EarRoPE..nspiree by the Neural Tangle Kernel (NTK) theory Ja- cot et al. 2018) Tancik et al.) 22220) awK-a Interpolation (bloc97 (203) essentially implements uniform radix scaling across all dimensions, which can be denoted as:
A STALT.
(1I)
By uniformly setting the radix conversion factor across dimensions, NTK avoids the abrupt OOD collapse of the highest dimension key to its initial success in training-fiee free context extension.
WTK-by-partssYaRD) under MrRoPE, To further address the OOD challenge, YaRN (Peng al. 2023 first categorizes all D, dimensions into high- mid- and low-frequency groups based on their rotation progress and applies differing conversion scheme to them: For both high- and low- frequency dimensions, YaRN develops a non-conversion strategy, setting A = 1 to better preserve corresponding position information. Besides, for mid-frequency dimensions (i.e., dj S j < dj), YaRN uses a linear interpolation strategy such that the expansion factor of each radix satisfies:
j-1
(12)
d=d.
where ry E (0, 1) is a linear scale factor determined by YaRN. More details about YaRN are pre- sented in Appendix A.2
4

---

Published as a conference paper at ICLR 2026
3.2 MRROPE METHODOLOGY
Based on the empirical evidence from YaRN, we conjecture that an effective radix conversion al- gorithm should adhere the following principle: lower dimensions extrapolate, higher dimensions interpolate, while intermediate dimensions achieve the desired extension factor S.
Although YaRN's formulation does not explicitly define radix conversion strategy for its inter- mediate dimensions, we find it implicitly fulfillls the above principle through regressive scaling conversion (i.e., >3 Aj+1), as proved Appendix AA...1 This observation raises fundamental question: is this the optimal conversion strategies for intermediate dimensions?
Within the MrRoPE framework, we can systematically discuss this question by explicitly designing the À vector. We design two distinct strategies: Uniform Conversion, which applies a constant radix factor (i.e., >, = 1)+1);; and Progressive Conversion, which features monotonically increasing scaling factor (i.e., Xj < Aj+1).
Based on these strategies, we are able to implement two additional radix transformation methods distinct from YaRN: MrRoPE-Uni and MrRoPE-Pro, and find better conversion for intermediate dimensions through comparison.
3.2. MRROPE-UNI
NTK pioneered the concept of uniform radix conversion for context extension. However, its failure to account for spectral distinctions leads to -egufrreueency distortion and persistent OOD errors. Accordingly, MrRoPE-Unt adopts a segmentation-basee NTK approach in which constant scaling factor is applied for uniform radix conversion.
Let A for all j E [dr, dr), where is constant. To achieve the total scale factor S, the radix expansion factor for intermediate dimension can be calculated as:
ス S 44 -4
(13)
In summary, MrRoPE-Uni applies uniform scaling all middle dimensions, independent of their original frequencies, effectively expanding the positional encoding range by factor of S.
3.2.2 MRROPE-PRO
Motivated by high- high-frequency extrapolation, MrRoPE-Pro employs a progressive radix conversion with dimension- wise scaling. In this scheme. lower (igh-firequeny) dimensions undergo smaller dimensions experience proportionally larger expan-
radix expansions, while higher (low-frequenc) sions. Hence, MrRoPE-Pro better preserves the fine- the-grained structure of high-frequency dimensions while enabling effective extension of the representable positional range, thus providing a more faith- ful extrapolation in the intermediate regions.
Let AÚ S To obtain progressive sequence, we assume that the sequence e follows an arithmetic progression. Let éd 0 and e straightforward to obtain that E; = (j - di 1)c for middle- dimeele-ddmessooss.. According to Eq. 12 which implies the constraint Ee the following expression for e, in the middle dimensions can be derived:
6--1 c, where is constant for all j E (d4,dy ), it is
2(1 - -
Ej
(1+ ( (dn di) A, = S
(14)
In summary, MrRoPE-Pro applies progressive scaling to all middle dimensions, resulting in slow-to-steep radix conversion that better preserves the positional information encoded in the lower (high-frequency) dimensions.
3.2.3 GENERAL FORMULATION
Overall Formulation of MrRoPE can be formulized as:
G
b
Dr
m0, (m
mod 2
(15)
PT Xa
where D, |D|/2. For the radix expansion factor Xa d-th dimension, define:
5

---

Published as a conference paper at ICLR 2026
For low- and high- dimensions, the scale factor Aa (i.e., d [1, di) U Udd,,D,) For diddiiodimenssons, the scale factor X values are determined by the method used as follows:
S 4
if MrRoPE-Uni
Ad
2014
(d S d < dn..
(16)
if MrRoPE-Pro Surd 4OC
Additional tricks for implementations are consistent with YaRN, such the value of di and da. which are provided in Appendix[D B Evidently, the fundamental distinction between our method and YaRN resides exclusively in the extrapolation strategy employed for the intermediate dimensions. Figure 3 shows shows the cumulative scaling factor for each dimension across various context window extension methods, where 5d j_ '3. Notably, MrRoPE-Uni and MrRoPE-Pro adopt distinct strategies in the middle dimensions compared to YaRN. In the following experiments, we treat YaRN as special case of MrRoPE- Reegressive, using baseline for performance comparison.
MriRoPF-bask
F-basi--basic
Cumulative Scale Scitop scaledi
NrRoPE pro NTK
MrRoFE pro NTK
Cumulstive
2--
10
20
30
40
60
10
20
30
50
60
RoPE dimension Indext
RoPE dimenssoon Index
(a) Llama 3.1-8B (8K)
(b) Qwen-2.5-7B (32K)
Figure 3: The cumulative scaling factor 88 of different RoPE extension methods across varying dimension (ScaleeS to 16x and 4x)
4 EXPERIMENTS
4.1 SETTINGS
We compare MrRoPE with SOTA test-time RoPE-extension methods, including YaRN, NTK. All experiments were conducted in inference setting (no fine-tuning), and thus, compar- isons were conducted with other context extension methods that require training (Shang et al. 2025 Wang et al. 2024 Hua et al., 2024).
Baselines.
Base Models and Tasks. We evaluate MrRoPE on the following RoPE-based LLMs: LLaMA2-
7B (Touvron al. (023). LLaMA3-SB (Grattafiori al., (024), and Qwen2.5-3B (Yang al.)
2025a). Our evaluation mainly addresses three aspects: (1) perplexity curves on a pre-training
test set assess performance under extended context; (2) long-context stress tasks, including RULER (Hsich et al. (024)) and Needle-In---Haysttae (Kamradt) 2023), to evaluate the handling of long-range dependencies: and (3) real-world benchmarks, Intinite-Bende (Zhang et al. (024) and
Longbench-v2 (Bai et al. 2024 to examine model performance in practical long-conters scenarios.
Theoretical Analysis. Besides the regular tests introduced above, we also provide theoretical ev- idence supporting the reliability of MrRoPE. Inspired by prior work (Liu et al. [023b] Barbero et al. (024), we first investigate the attention scores, with a particular focus on their performance improvement in the middle dimensions. Moreover, based on the theory of [Men et al. (2024), we employ the cosine similarity metric to assess the enhancement in the theoretical maximum encoding length achieved by MrRoPE.
4.2 LONG SEQUENCE LANGUAGE MODELING
Long-sequence modeling serves as the most direct and fundamental evaluation of positional en- coding methods. To assess this ability, compute perplexity scores ten randomly sampled

---

Published as a conference paper at ICLR 2026
sequences exceeding 128K tokens from the Proofpile dataset (Azerbayev et al. 2022),, using float32 precision. Table T reports results across different base models and extension strategies; due to the space limitation, perplexity results of LLaMA2-7B are provided in Appendix C.1
Table I:
Perplexity scores on proofpile dataset across different models. The best and second-best results boldfaced and underlined, respectively.
(a) LL Instruct
(b) Qwen2.5.3B
Context Scale
Extension Method
Evaluation Context Length
Context Window
Scale
Extension Method
Evaluation Context Length
Window (S)
8K
16K
32K
64K
128K
(S)
8K
16K
32K
64K
128K
NTK
3.67
3.72
>5
21
>10
NTK
3,548
3.778
7.782
>10
>10
YaRN
3.68
3.08
2.75
2.49
2.38
YaRN
3.449
2.894
2,589
2.333
2.222
8K
16
MrRoPE-Uni
2.74
2.41
32K
MrRoPE-Uni
3.440
2.224
MrRoPE-Pro
3,63
3,03
2.71
2.45
2.34
MrRoPE-Pro
3.433
2.881
3.579
2.328
2.221
YaRN does not consistently achieve superior performance across models and context lengths. On the Proofpile test set, YaRN yields markedly higher perplexity in the initial half of extended contexts, where both MrRoPF-Uni and MrRoPE-Pro consistently outperform it across model fam- ilies. Moreover, as the context approaches the end of the extended window, YaRN begins pass MrRoPE-Uni, yet remains inferior to MrRoPE-Pro. For instance, at a 128K context length, LLaMA3-8B records perplexity of 2.38, surpassing MrRoPE-Uni (2.41) but remaining substan- tially weaker than MrRoPE-Pro (2.34). Since similar trends are also observed in both Qwen2.5 and LLaMA2, we attribute this behavior to the regressive radix conversion strategy employed by YaRN: radix conversion in lower-dinensional spaces disrupts local positional information, degrading per- formance shorter contexts while partially preserving longer-range coherence. These findings indi- cate that although YaRN alleviates certain challenges in long- ueg-eeeuence modeling, it incurs trade-offs that diminish its overall robustness across the full context window.
MrRoPE-Pro demonstrates superior and consistent performance across the entire extended context window. On both LLaMA3-8B and Qwen2.5-3B, MrRoPE-Pro steadily achieves the lowest perplexity across all evaluated lengths, ranging from 8K 128K. The merit is particularly evident the shorter-range context: for instance, on LLaMA2-7B at 4K, MrRoPE-Pro attains a perplexity of 5.72, outperforming YaRN (6.02) and MrRoPE-Uni (5.84). This superior performance can be attributed to the progressive extension strategy, which avoids out-of-distribution positional embed- dings in higher dimensions while retaining high-frequency details in the original RoPE structure. By mitigating distortions in both local and global positional information, MrRoPE-Pro provides a more balanced and effective approach context window extension.
Considering the weaker long-conteds performance of MrRoPE-Uni than the progressive one, we mainly focus MrRoPE-Pro and YaRN in the following experiments.
4.3 LONG CONTEXT BENCHMARKS
To further evaluate real-world long-contex, reasoning capabilities, we assess model performance on standardized benchmarks. We mainly focus on two key capabilities: (1) the ability to retrieve relevant information, and (2) the ability to utilize such information for real-world tasks.
1874
1m7
475
50%
37%
27%
-0%
17%
IKC
JOK 1INK
VNE
BO VK
(a) ( (-116)
MrRaPE-Pro S-16)
Figure 4: In the Maale-IN-A-Hassttie test, MrRoPE-Pro (right) effectively extends LLaMA3-8B's context window to nearly 96K, which is much longer than the performance of YaRN (left).

---

Published as a conference paper at ICLR 2026
MrRoPE-Pro enables robust context window extension in long-context stress task. To assess the retrieval ability mentioned above, we apply the NIAH test (Kamradt 2023) with varying context lengths and insertion depths. We compare MrRoPE-Pro. our best-performing method, against YaRN on LLLMA3-88, measuring recall with ROUGE-1 [in 2004). Figure 4 shows that MrRoPEE-Pro extends the effective context window
least 96K tokens, exhibiting markedly more stable sealing. Within shorter contexts (8K-56K), it consistently achieves higher scores, approaching near- peerect accuracy at insertion depths of 80-100%. Beyond 96K, the performance gap widens: MrRoPE- Pro maintains clear superiority across the 30-70% range, demonstrating robustness in mid-context retrieval under ultra-long settings. Even at 120K tokens (15x the pre-training length), sustains over 85% recall at most depths, establishing strong upper bound for practical long-context applications.
We further evaluate MrRoPE on the RULER benchmark, with results presented in Table Notably, MrRoPE demonstrates substantially superior performance across the extended context window. This advantage is particularly pronounced from 64K to 128K tokens, where YaRN's performance sharply declines from 89.5 to 79.9, while MrRoPE maintains relatively stable level of performance.
Table 2: Retrieval scores on RULER benchmark across all 13 subtasks.
(a) LLaMA3-8B-Imstee
(b) Qwen2.5-3B-Innst
Scale
Extension
Evaluation Context Length
Scale Setting
Extension Method
Evaluation Context Length
Setting
Method
8K 16K 32K 64K 128K
8K 16K 32K 64K 128K
YaRN
92.1 92.7 89.5 79.9
YaRN
78.1 77.7 75.6
63.2
50.1
8K 128K
32K
122KK
MrRoPE-Pro
96.2 94.2 94.3 91.3 86.6
MrRoPE-Pro
82.3 82.9 78.5
70.4
53.2
MrRoPE-Pro effectively utilizes the inherent extrapolation capabilities the base model. To assess MrRoPE-Pro's ability to integrate retrieved context, we first evaluate it on the Infinite- Bench with sub-tasks under 128K tokens, sampling 100 examples per subset. On simpler tasks, MrRoPE-Pro performs competitively with closed-source models, achieving 100% Passkey Re- trieval (matching GPT-4) and 89% on Number Retrieval. In more complex tasks, such as KV Re- trieval and QA Dialogue, significantly outperforms YaRN (27% 9% and 22% 10%. respectively) and even surpasses specialized long context models like Yi-34B-200K (Young et al., 2024) and Kimi-Chat (Team al., (025) on several subsets. These results show that MrRoPE- Pro not only outperforms open-ended methods like YaRN but also narrows the gap with fine-tuned long-context models- without requiring additional training. In addition, we also report the perfor- mance of MrRoPE-Pro and YaRN on LongBench- V2, another real-world long-context benchmark with more complex test cases, where MrRoPE-PP also outperforms YaRN, the results are shown in Appendix C.2
Table 3: Long context performance comparison on Infinite- In each subset, we randomly choose 100 samples with lengths ranging from 100K to 128K. MrRoPF.PPr outperforms YaRN under the same settings while approaching GPT-4 in some tasks (e.g., QA Dialogue, Math Find).
Extensioin
Retrieve Retrieve Retrieve Code
Math
QA
Models
Avg.
Method
KV
PassKey Number Debug
Find Dialogue
GPT-4
89%
100%
100%
35%
60%
26%
68.3 %
Yi-34B-2008
2%
100%
100%
12%
23%
2%
39.8%
Kimi-Chat
54%
98%
95%
18%
13%
12%
48.3%
LLaMA3-8B
YaRN
9%
100%
85%
3%
57%
10%
44%
LLaMA3-8B
MrRoPE-Pro
27%
100%
89 %
3%
58%
22%
49.8%
4.4 THEORETICAL ANALYSIS
This section establishes the theoretical basis for PERRoPE-Pro'ss extrapolation capabilities by analyz- ing its merits for context window extension. The analysis proceeds in two parts: first, examine the mechanism responsible for its superior extrapolation limit: second, we demonstrate how this design enlarges the effective context window in the self-attention perspective.
8

---

Published as a conference paper at ICLR 2026
Sum cosines on base 10,000
Sum cosines base 10,000 (Nlodle part)
MMC
tork
50
Sum Cosines
Sum Cosines
-10
-20
20000
30000
60200
20000
30000
40000
Sum Cos(olla)
Sum of middle Costati)
Figure 5: The cosine sum of the rotation angles in each dimension, measuring the ability give more attention to similar tokens than random one. The base value and original context length are consistent with the settings of LLaMA2-7B.
MrRoPE-Pro significantly extends the theoretical context window upper-bound of the base model. The RoPE Bound Theory establishes a method for analyzing the potential context length of LLMs by relating it the rotational base b. This theory demonstrates that the upper bound of the effective context window is defined by the root of the function Bp(m) = Σinn cos (m01). shows the evolution of (m) with relative distance for various extension methods, with the vertical dashed line indicating this root. Under base of 10,000, MrRoPE-Pro extends the theoretical context window upper bound from 1K to 28K-neart five times the 6K bound achieved by YaRN. As the key difference between methods lies in their middle-dimeenional strategies, Figure 5p isolates the contribution of these dimensions to (mmm). Evidently, MrRoPE exhibits significantly smaller oscillations and a more stable trend than alternative methods.
MrRoPE-Pro operates by refining intermediate-dimeesioone score distributions. large language models, attention score variation key metric for evalu- ating RoPF-based extension methods. Therefore, to compare MrRoPE-Prr with YaRN, we analyze changes in attention scores across middle dimensions with respect to relative distance. As shown in Figure 6 MrRoPE-Pro extends the periodicity of the original RoPE frequency signals in mid- dle dimensions, maintaining monotonic attention variation over a broader range. This improves the consistency of positional information encoding. Moreover, MrRoPE-Pro produces attention distri- butions that better match pre-training patterns compared to YaRN, enhancing training- free extrapo- lation.
features to stabilize attention
Aitertion Score Middle Part Dinensions
Altentinn Score Middle Part Dimrnnsors
1000
Albertion Saor
500
Ateminn Score
300
0-
0-
-500
10000
20000
30000
40000
50000
63000
-300
20000
400on
R3000
100000
120000
Distance
(a) LLaMA2-7h: 4K-6
(b) LLLMMG--B: 12-K
Figure 6: Middle partial attention score extended context windows. For each relative position, we randomly selected 50 token pairs' corresponding attention score calculated by the middle dimen- sions.
CONCLUSION
We present unified theory, MrRoPE, linking major RoPE-extension methods to radix conversion, where existing methods emerge as special cases. Based on this, we propose MrRoPE-Pi Pro, a training- free method that effectively mitigates the high-dimensional OOD problem by progressively scaling

---

Published as a conference paper at ICLR 2026
the radix base. Empirical results on real-world benchmarks show that MrRoPE-Pro nearly dou- bles the practical context window and surpasses strong baselines, including closed-source models. Theoretically, our improvement stabilizes the attention scores in intermediate dimensions and maxi- mally restores high-frequecy information, thereby pushing the upper bound the effective context window to its maximum.
ETHICS STATEMENT
This work presents a context window extension theory and methodology for Large Language Models via positional encoding- Our research is based on publicly available, pre-trained base models (e.g., LLaMA series) and standard, open-source benchmarks. All data used for evaluation are in the public domain and contain no private or sensitive information. Our work does not introduce new data collection practices or target specific, sensitive applications.
REPRODUCIBILLITY STATEMENT
To facilitate reproducibility, the source code for MrRoPE-Uni/Pro, along with the baseline method YaRN and NTK, has been anonymized and submitted as supplementary materials with this paper. Detailed descriptions of the experimental configuration and all hyperparameters are provided in the Appendix. Our experiments utilize standard, open-souree benchmarks and their corresponding official test scripts to ensure all results are easily verifiable.
REFERENCES
Z. Azerbayov, E. Ayers. and B. Piotrowski. Proof-pile. 2022.
https://github. com/
blaagi=-aeeroayv///roo-----
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longbench Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. 152044 2024.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205, 2024.
bloc97. Ntk- scaled rope allows llama models to have longer context windows.
https:
scal 1 ced_ropp___ ows 11 ama alllamaammeee sotteelssttt_aateeeeee s_tt__aveee///aee 2023.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint (rXii:2306.1559) 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens, arXiv preprint arXiv:2402.13753 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint arXiv:2404. 246.:2644400654, 2024.
Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding. Youbang Sun, Biqing Qi, Yuchen Fan, Xuckai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention's 's periodic extension for length generalization. arXiv preprint arXiv:2412.17739, 2024.
Yumpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang. Hao Chen, et al. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.1255 2023.
10

---

Published as a conference paper at ICLR 2026
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
G Kamradt. Needle in a haystack pressure testing llms. htttp:://github.com// wtt../////aaaa///////a
LLMTestNeedle Saeareeetteeeleeemlaaattaaaaaa InAHaystack. 2023.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892 249222882,,,, 2023.
Chin-Yes Yew Lin, ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona. Spain, July 2004, Association for Computational Linguis- tics. URL https:
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023a.
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An. Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023b.
Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. arXiv preprint arXiv:2405. 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0000 2023.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2/08. / /2409, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-67, 2020.
Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. Longrope2: Near-lossless Ilm context window scaling. arXiv preprint arXiv:2502.20082. 2025.
Jianlin Su. Transformer upgrade path: 10. rope a beta-based encoding. /ttpss:/spacaee Jul 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolltable transformer. arXiv preprint arXiv:2212.10554 10554, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Kimi Team, Angang Du, Bofci Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2026
Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance rope: Improving context length generalization of large language models. arXiv preprint arXiv:24033.0071, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.166 16039, 2023.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.093888, 2025b.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint 2xXiv:2403.0465, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. infty bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.1377 2024.
A RELATED WORKS
A.1 PI
In the study of RoPE-extension, earlier works seek to mitigate incomplete-cycke effects by re-scaling or normalizing position indices before applying sinusoidal transformations. From the radix perspec- tive, these methods can be understood as normalizing positions to the original trained range so that higher-order digits are less likely terminate prematurely. A representative example is the Position Interpolation (PI) method (Chen et al. 2023),, whose attention score can be calculated as:
m' = S (q. k) = S(any (2,m, m²) = im' W km
m
(17)
where S = Liest/ Ltrain- Remarkably, because PI method can not be expressed in the form of Eq. [6] we cannot interpret PI as any form of base conversion behavior. This also provides evidence that Pí constitutes normalization of positional values, allowing its underlying logic to still be explained within the framework MrRoPE. Nevertheless, PI uniformly compresses the digit repre- sentation of positions, causing collisions in lower-order digits and thus disproportionately distorting fine-grained positional information.
A.2 DETAILS OF YARN
YaRN can be considered as further version of NTK, which applies a segmented NTK-aware In- terpolation into different parts. More specifically, YaRN define λa as the wavelength of the RoPE embedding at d-th hidden dimension:
2T
2xbin 24
(18)
0,
It's easy to classify all dimensions into two categories: 1) Ltrain > Ta and 2) Ltrain S Ta. Liu cet al. (2023b) observed that in the first type of dimensions, each position goes through one or more full rotation cycles during training, so the model sees all positional patterns and learns them well. These dimensions remain stable at inference and do not cause antoo--distrbuttion issues. In contrast, the second type of dimensions lacks sufficient training exposure, making them the main cause of ex- trapolation errors and abnormal attention scores. Therefore, YaRN fully accounts for the differences in wavelengths across dimensions and applies different extrapolation strategies accordingly: it uses
12

---

Published as a conference paper at ICLR 2026
direct extrapolation for well-traine' dimensions, periodic interpolation (PI) for those trained for less than one full cycle, and linear approximation for the remaining dimensions as follows:
g (m) = (1 - (--))))) Th + +
(19)
h (Oa) = 0.
where ra L/Ta and
if < if 8 otherwise
(ra)
(20)
r-a
In real coordinates, good values for and 3 are = and 3 = 32. addition, YaRN introduces further scale factor i to modify the computation of attention weights into
4m kn
softmax
WDD
where f can be calculated as V = (n()) +1.
A.2.1 YARN REGRESSIVE RADIX CONVERSION
Considered middle-dimension j E [dz, da), based on YaRN's scaling equation, we have
G-1 mf6; (m b D,
+ (s 1)r; sa mod 2 {q, km s(B
emm W (g.44
(21)
According to the MrRoPE theory, the radix expansion factor Xj satisfies
3-a
Aa = 8 B (8 - 1)r; SC
(22)
d=1
Hence, Ny satisfies
da
8 1)r, - 80
Ad
3 1)
(23)
c (s 1)ry
c (s - 1)rz+1
where c c= 3 3-- sa. Then divide Aú by A--1,
(c (s - 1)r,)² (s (05-1)),aa 1)r (s 1)r5-1)
A--1
(24)
2 + (8 - l)c- 2r, + (s - 1)3r3
2 (8 1)c (ri+1 + +r)-)) + (s- (1-
1)2
Notably, according to Eq. fied as follows:
18 we have 13 -1 Therefore, the Eq. 25 can be further simpli-
+ (s- + (s 1)?r3
Aj-1
2² + (s (8- 18
1)
0² (s 2r;
2 + (8 18-1
(ri+1 + +z-1))
(25)
2/r, + (s-
1)3r, + 1)c 2
L/T3 + (8 1) + (s l)c (b + b
3
13

---

Published as a conference paper at ICLR 2026
In practice, RoPE's original base b is not allowed to be setted as 1. Therefore, for all rope-theta settings where > or < < 1, YaRN's scale factors always satisfy A--1 > X,. which characterizes YaRN as regressive conversion.
B DETAILS OF MRROPE METHODOLOGY
implementation, the calculation of d₃ and d, is the same that of YaRN's. More specifically, dj and d, represent the boundaries that distinguish different types of dimensions, and they be calculated follows:
In
d = max {il Lirain 0 2m d2 min {il Ltrain 0 < a 2n)
(26)
where 0: = b- and A are hyper-parametert In real coordinates, good values for them are a and / 32 (Details are presented in the following section). Another trick used YaRN, the rescaleing factor / is also encouraged to unlock the full potential of MrRoPE.
B.I HYPERPARAMETERS GUIDELINE
sAs we introduced above, changing (a,,8) essentially equivalent to selecting a different dimension partition, i.e., modifying (drd)), and there is a one-to-one correspondence between the two pairs. Therefore, in the following experiments, we treat di and d₁ as the variables and analyze how the positional encoding performance changes accordingly.
First, the purpose of defining dj is to isolate dimensions whose rotations do not complete a full cycle during training. which YaRN identifies as the main source of OOD failures. From this view, d, should be highly sensitive, and increasing it should sharply degrade long-context extrapolation. To test this, we evaluated how PPL changes with d,, on Mamma-88-Inntruuee using d, 35 (i.e., B = 1) as the reference point.
As shown in Figure 77, we observe that the effect of position embedding is not such highly sensi- tive to dx; the model maintains low PPL within the range [34,39], and MrRoPE-Pro consistently outperforms YaRN. similar trend appears when varying d,, as shown in Fig pb
2.50
2.50
2.37
237
2.38
2.36
Y6TH HA0RE
2.33
2:50
(a) PPL score across various da (beta)
(b) PPL various (alpha)
Figure 7: Mamma--B--Instuuee PPL score tested on 128K context window across different a/B set- tings.
In addition, we also conducted experiments on Qwenn.5-3B-Innttee Instruct to verify whether the same hyperparameters can be applied models with different architectures. The results shown in Fig We find that for Soven2.5-3B-Insteee the best values of (di, dr) are (23,40). corresponding to 32 and =
In summary, these results show that different models may prefer different optimal hyperparame- ter settings, but MrRoPE-Pro is relatively insensitive to these choices. In particular, 32 and
14

---

Published as a conference paper at ICLR 2026
22750
1938
2.245
22243
22240
23235
22230
3.232
22223
22230
2.220
22215
(a) PPL score actoss various du (bela)
(b) PPL score actoss various di (ulpha)
Figure 8: Qwent.e-3B-Instruee PPL score tested on 128K context window across different a/B settings.
= consistently yield near-optimal performance across models and can serve as a strong default configuration.
C ADDITIONAL EXPERIMENTS RESULTS
C.1 PPL RESULTTS OF LLAMA2-7B-CHAN
LLaMA2-7b-chat-hrss perplexity score is as follows:
Table 4:
Perplexity scores of LLaMA2-7B-chatt on proofpile dataset. The best and second best
results are boldfaced and underlined respectively.
(a)
Context Scale Window
Extension Method
Evaluation
Context Length
(S)
4K
8K
16K
32K
64K
NTK
6.73
6.76
6.97
7.11
>10
YaRN
6.02
5.59
4.64
4.07
4.03
4K
MrRoPE-Uni
5.84
5.42
4.52
4.01
4.15
MrRoPE-Pro
5.72
5.39
4.57
4.11
3.88
C.2 LONGBENCH- V2 RESULTS
To test the RoPE-extension method on LongBench-Ve V2, we first limit the length of the test sample to less than 128K match the extended window length. The performance results are shown in Table
D USE OF LLMS
To better present our work, the authors used ChatGPT solely for grammatical checking of the manuscript. The initial draft of the manuscript and all code were created without Al assistance. Its use was strictly limited to identifying and correcting grammatical errors in pre-written text. The prompts were exclusively in the form of: "As a senior grammar editor, you are tasked with con- ducting thorough grammatical and syntactical review of the manuscript, ensuring accuracy in all aspects of grammar, including tenses, voices, subject-verb agreement, and clause structures. Pay special attention to the correct of complex sentences and passive voice common in academic
15

---

Published as a conference paper at ICLR 2026
Table 5: Resultes of LLaMAA-SB-Insttn and Owen2.5-33BIInstrutt LongBench dataset. SD: Single-Doeument QA, LD: Long-dialogue History Understanding, MD: Multi-Document QA, LICL: Long In-context Learning, LSD: Long Structured Data Understanding, CU: Code Repository Understanding.
(a) LLaMA2-7B-Instruut
Context Scale
Extension
Tasks
Window
(S)
Method
SD
LD
MD
LICL LSD CU
YaRN
16.7 12.8 17.8
26.8
30.0 6.67
4K
16
MrRoPE-Pro
16.7
15,4
18.9
26.8
40.0 6.67
(bJQwenn.5-3B-IInttuue
Context Scale
Extension
Tasks
Window
(S)
Method
SD
LD
MD
LICL
LSD
CU
YaRN
15.8
10.2
17.6
25
50.0
6.67
32K
MrRoPE-Pro
16.5
12.8
17.6
27.5
50.0 6.67
writing, well the standardization of punctuation enhance the manuscript's academic rigor and authority."
E LIMITATION
Our RoPE-extension method primarily focuses on a training-free extension of the context window. While effective, the absence of fine-tuning experiments limits direct comparisons with other ex- tension methods like xPOS and LongRoPE (Sun et al. [0222] Hua et al. 2024), which are often evaluated fine-tuming setting. Incorporating such experiments in the future would more compre- hensively demonstrate the superiority and integration potential of our method.
Furthermore, the current theoretical framework and its advantages are intrinsically tied to the RoPE mechanism. The generalizability of the core "radix transformation" concept to other positional encoding schemes (e.g.. T5's relative bias, ALibi) (Raffel et al., 2020; Press al.) (202T) remains an open question, which may limit the broader applicability of the approach beyond transformer architectures employing RoPE
16


Reviews:
[{"review_id": "wzcNNoHtw7", "reviewer": "Reviewer_oZP9", "summary": "This paper aims to provide a unified theoretical foundation for context window extension of Rotary Position Embedding (RoPE) in LLMs. It proposes the MrRoPE (Mixed-radix RoPE) framework, which reinterprets RoPE and its extensions as a mixed-radix number system conversion. Based on this, the paper introduces a new training-free method, MrRoPE-Pro (progressive radix conversion), and demonstrates its significant advantages over state-of-the-art methods like YaRN across multiple benchmarks.", "questions": "1.  Could you provide a more rigorous mathematical derivation, starting from YaRN's original interpolation formulas, to derive its corresponding scaling factors `λj`?\n2.  Have you conducted any sensitivity analysis or ablation studies on the hyperparameters `α` and `β` inherited from YaRN?\n3.  Could you elaborate on the fundamental trade-offs among the regressive (YaRN), uniform (Uni), and progressive (Pro) strategies?\n4.  Does your \"progressive\" strategy inherently maintain attention score stability better than YaRN, reducing the reliance on heuristics like temperature scaling `t`?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The work addresses the critical problem of training-free context extension for LLMs, and its unifying framework could have a profound impact on future research. The connection between RoPE extension and radix conversion theory is a highly novel and insightful perspective, elevating heuristic designs to a theoretical level.", "weaknesses": "1. The characterization of YaRN as \"regressive\" is based on observation rather than rigorous derivation. The paper fails to mathematically prove this from YaRN's original equations.\n2. The method heavily relies on hyperparameters (`α`, `β`) inherited from YaRN, yet lacks a sensitivity analysis or ablation study to verify their optimality for the new method.\n3. The comparison is limited to training-free RoPE methods, failing to adequately position the work relative to fine-tuning-based approaches or alternative schemes like ALiBi.\n4. Experiments are confined to small- to medium-scale models (3B-8B). The generalizability of its conclusions to larger models (e.g., 70B scale) remains unproven.\n5. The analysis of downstream performance gains is superficial. It reports *what* improved but lacks qualitative or error analysis to explain *why* MrRoPE-Pro is better.\n6. The method's potential as a better initialization for long-context fine-tuning is untested. A crucial experiment is missing to verify if its advantage persists after fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6Pqt3bXZH", "reviewer": "Reviewer_1qVM", "summary": "This paper focuses on the context window extension problem of rotational position encoding (RoPE). Aiming at the pain point that the existing RoPE extension strategies are diverse and lack a unified theoretical foundation, the paper proposes a mixed-base rotational position encoding (MrRoPE) framework, and develops an efficient training-independent extension method based on it, and the related results are verified by quantitative experiments.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- MrRoPE constructs a universal framework from the radix conversion perspective, and unifies mainstream RoPE extension methods such as NTK and YaRN into different base conversion strategies, which solves the problem of the lack of a unified theoretical basis for the existing extension schemes, and facilitates system analysis and optimization.\n- The proposed MrRoPE-Pro requires no additional fine-tuning and performs outstandingly in long context tasks: over 85% recall in Needle-in-a-Haystack tests with 128K contexts, more than twice the accuracy of YaRN in Infinite-Bench retrieval and dialog tasks, and in benchmarks such as RULER, LongBench-V2, and others. and consistently outperforms existing methods in benchmarks such as RULER, LongBench-V2, etc.\n- The upper bound on the effective coding length of RoPE is theoretically boosted, and the model maintains excellent performance over the full context range of 8K to 128K through a progressive base conversion strategy that preserves the fine-grained information in the high-frequency dimensions and stabilizes the distribution of the attention scores in the intermediate dimensions.", "weaknesses": "- The RoPE extension method (MrRoPE-Uni/Pro) proposed in this paper only focuses on the “no-training” context-window extension mode, and no fine-tuning experiments have been conducted. This makes it impossible to make a direct comparison with extension methods such as xPOS and LongRoPE, which need to be evaluated in fine-tuning scenarios, and it is difficult to comprehensively prove its superiority in different application scenarios and its integration potential with other methods.\n- It is well known that the training process of LLM model is complex and cumbersome, and the variation of hyperparameters may substantially affect the training results, so the authors should explore in detail whether the choice of hyperparameters affects the model performance, and whether the same hyperparameters proposed in the paper can be used for LLMs with different architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f1q706mTuH", "reviewer": "Reviewer_on7t", "summary": "This paper reframes the problem of extending the context window of LLMs with RoPE, to view position encoding through the lens of mixed radix conversion. This allows to unify previous methods as different choices within the proposed framework and outperform YaRN on 13 tasks.", "questions": "Would you explain why the progressive strategy in the proposed method is better than YaRN's way? Is there any inference latency penalty for using the proposed method or YaRN? If the radix concept could be used for other types of positional encodings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The unifying theory is good and the results are compelling across different tasks. Good to see a training free method has consistent gains.", "weaknesses": "It is not clear if this radix idea can work for other positional encoding method, which limits the impact to RoPE.  Also not clear if the progressive scaling in the proposed method introduces any latency compared to YaRN? \n\nWould also be helpful to have sensitive analysis, about the parameters inherited from YaRN in appendix B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQRdN8eSXd", "reviewer": "Reviewer_papb", "summary": "This paper introduces MrRoPE (Mixed-Radix Rotary Position Embedding), a training-free method for extending the context length of transformer models equipped with RoPE (Rotary Position Embeddings). The key insight is to reinterpret RoPE as a form of mixed-radix number system, where each rotational frequency corresponds to a different positional base. The authors propose two practical schemes: 1) MrRoPE-Uni: uniformly redistributes intermediate rotation frequencies; 2)MrRoPE-Pro: progressively adjusts them across dimensions. The authors show that these schemes unify and generalize several prior test-time scaling methods such as NTK and YaRN. Extensive experiments on LLaMA-2/3 and Qwen models demonstrate consistent improvements across multiple long-context benchmarks (RULER, InfiniteBench, LongBench-v2, Needle-in-a-Haystack) without any fine-tuning. MrRoPE-Pro, in particular, exhibits stronger stability and better RoPE-bound behavior at very long sequence lengths (up to 128K tokens).", "questions": "1. How accurate is the approximation of RoPE as a mixed-radix system when floor/mod terms are **non-negligible**? Could you provide error bounds or an empirical validation of this assumption?\n2. How sensitive is MrRoPE-Pro to the choice of scaling parameters and dimensional splits? Would the same configuration generalize to very different models or hidden sizes?\n3. Could unified radix conversion be extended beyond RoPE? For instance, to ALiBi or other relative positional encodings? Is the mixed-radix perspective specific to sinusoidal embeddings?\n4. What is the inference-time cost or memory overhead compared to baseline RoPE and YaRN implementations?\n5. Could you provide results for fine-tuned long-context methods to better contextualize MrRoPE’s efficiency vs. performance trade-off?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Novel theoretical perspective**: The mixed-radix interpretation provides a new and elegant way to understand RoPE scaling. It explains existing methods (NTK, YaRN) under a unified mathematical framework and motivates new variants.\n2. **Practical and lightweight**: The proposed method is training-free and only modifies the RoPE computation at inference time. It can be easily applied to off-the-shelf models, offering immediate practical value.\n3. **Comprehensive empirical evaluation**: Experiments are extensive, covering perplexity, synthetic retrieval, and multiple real-world long-context benchmarks. The improvements of MrRoPE-Pro over prior inference-only methods are consistent and significant.\n4. **Interpretability and diagnostics**: The paper connects the improvements to theoretical properties such as the “RoPE bound” and attention distribution behavior, providing an interpretable explanation of why progressive scaling helps.\n5. **Strong empirical gains without fine-tuning**: MrRoPE-Pro achieves competitive or superior performance compared to other test-time extensions and even approaches the performance of some fine-tuned long-context models.", "weaknesses": "1. **Lack of strictly rigorous theoretical analysis**: The “RoPE as radix conversion” formulation relies on approximations (ignoring floor/mod operations), but the paper does not formally analyze the approximation error or its effect on attention stability. The derivations remain mostly intuitive rather than strictly formal proven.\n2. **Missing fine-tuned or stronger baselines**: The comparison is limited to test-time methods (YaRN, NTK). Including light fine-tuning baselines (e.g., LongRoPE, xPOS, or other retrained RoPE variants) would help position the method more clearly in terms of cost–performance trade-off.\n3. **Hyperparameter sensitivity not analyzed**: MrRoPE-Pro depends on specific tuning of hyper-parameters. The paper does not present an ablation or sensitivity study, leaving unclear whether performance is robust across models or sequence lengths.\n4. **No evaluation of runtime or stability**: Since MrRoPE modifies rotation frequency computation, the paper should report inference-time latency, GPU overhead, and numerical stability (especially under FP16 inference, quantized inference is also welcomed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)