Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
Authors: Xiaoran Liu
Yuerong Song
Zhigeng Liu
Zengfeng Huang
Qipeng Guo
Zhaoxiang Liu
Shiguo Lian
Ziwei He
Xipeng Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Long-Context LLM
Position Embedding
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose RoPE++, an imaginary extension of Rotary Position Embeddings for long-context LLMs.

Abstract: Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.

Published as a conference paper at ICLR 2026
BEYOND REAL: IMAGINARY EXTENSION OF ROTARY POSITION EMBEDDINGS FOR LONG- CONG-CONTEXT LLMS
Xiaoran Li Yuerong Song! Zhigeng Liu Zengfeng Huang!
Qipeng Guo??, Zhaoxiang Liu', Shiguo Lian", Ziwei He?: Xipeng Qiu'.2
FFudan University. "Shanghai Innovation Institute, 'Shanghai AI Lab, *China Unicom xrliu26 @m. fudan. edu. cn, ziwei he@ siliiii... edu. en, xpqiu fudan edu. on
ABSTRACT
Rotary Position Embeddings (RoPE) have become standard for encoding quence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score caleulation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re- re-incorporates this discarded imaginary component. Our method leverages the full complex-valle valued representation to create dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling long-context dependencies by preserving more positional information. Further- more, evaluations on suite of long- long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant context length increases. The code is available at httpp:::s http://wwb.ccoooeeeoo/////roppp/
INTRODUCTION
Large Language Model (LLM) based attention mechanism Vaswani et al. 2017 now dominates
Natural Language Processing (NLP) (OpenAl 2023 Sun et al.) 2024 OpenAl| 2024; Yang et al.
2025a), particularly in the long-context arena Hassabii & Kavukcuoglu 2024 Young al. 2024
2). where attention overcomes the long-dependency bottlenecks of carlier architectures (LeCun et al.
1995; Schmidhuber al. 1997). Recent work extends their context length to the million-token
scale Liu et al., 2024b I 2025). and the key driver is position-embedding design (Su et al.
2024 Press et al. 2022] Peng et al., (024). Among current LLMs, Rotary Position Embedding
(RoPE) (Su et al. 2024) has become the canonical choice (Dubey et al.) 2024 Meta 2024a b). It
encodes the absolute position of every query and key vector go, k., namely token indices 8, / with
rotary matrix or complex multiplication, and when the two vectors make a dot product, it injects their relative position t s, namely the relative distance, into the attention scores, thus combining the merits of traditional absolute and relative position embeddings (Vaswani et al. 2017 Dai et al. 2019
Yan et al.
2019) and securing widespread adoption.
Nevertheless, RoPE also has notable shortcomings, including poor length extrapolation (Press et al. 2022 Chen et al.) 2023 bloc97] (023). lack of data-sensitivity (Golovneva et al., [024]] Yang al. 2025b), and no design for heterogeneous multi-modal input [Su]|2044] prompting extensive research into its improvement. Most efforts concentrate on refining RoPE through interpolation designs (Peng al. 2024 Liu al. 2024d Su 2023). data- awareness Zheng et al., 2024a b), and nadture-dimension partitioning (Wang et al. 2024; Wei et al. 2025). However, few work revisits the intrinsic computation of RoPE or analyze its inherent limitations (Hua al. 2024 Dai et al. (0225). Re-examining RoPE in its complex-multilicatton
form reveals that the standard implementation keeps only the real part of the resulting complex attention score and discards the imaginary part outright (Su et al. 2024). Although taking the real part preserves the direct equivalence between complex multiplication and vector rotation, it incurs an irreversible information loss.
Equal contribution.
Corresponding Author.

---

Published as a conference paper at ICLR 2026
Real Attention
cosine integral function
Stronger Semantic Locality
Efficiency Design
050
d/2-1
Limited Position Information
Eg k
1
K
8n cos(10 a -0-5) s))
0.0
1==0
RoPE directly discard the imaginary information
d/2-1
Relutivd Distance
100
159
200
q.K,
[
kokze (okke-10((50)) 10((5-1)
a Ra - (R #q:) R
==
0.25
d/2-1
RoPE incorporate Imag.
E q.
A
lm 5
= K
17
8R sin $10 a (t s))
At s
0.30
7L=0
Capture Longer Dependency
Cache & Parametric Efficiency
Imaginary Attention
sine integral function
Length Extrapolation
130
Figure I: Overview of RoPE++. RoPE retains only the real part of the complex-valued attention score, whereas RoPE++ exploits the full complex representation to produce both real and imaginary attention. The real attention exhibits stronger semantic locality, while the imaginary attention preferentially captures long-context dependencies. RoPE++ combines the two, yielding multiple advantages.
closer look at the imaginary attention, strictly, the negative imaginary part of attention, shows that, compared with the real attention exhibiting stronger semantic locality, the imaginary heads attend more to long-context information as shown in Figure promising gains on long-context tasks. Moreover, adding imaginary attention also exposes qr, k, to wider positional information range, implicitly improving length extrapolation. Therefore, we propose RoPE++, as illustrated in Figure [] which re-injects the discarded imaginary component as a new group of attention heads computed in parallel with the real attentions. Particularly. we introduce RoPE+tEH that keeps equal attention head number while halving QKV parameters as well KV cache, and RoPE++EC that keeps equal cache size and doubles the number of attention heads. Theoretical analysis and pre-training experiments validate the above advantages, Both RoPE+tHU and RoPE++EC outperform vanilla RoPE and other position embeddings general tasks. On long-context benchmarks, RoPE++EH achieves comparable results with vanilla RoPE with half the cache, whereas RoPE++EC outperforms significantly at the same cache cost. Our contributions can be summarized follows:
We first identify the loss of imaginary information in standard RoPE and find it advantageous for capturing long-context dependencies by analyzing the properties of imaginary attention.
Building on this, we propose RoPE++, which reintroduces the imaginary computation into attention in two configurations, RoPE++EH with equal head number and halved KV cache, and RoPE++Ec with equal cache size and doubled attention heads. Both preserve the unified absolute-relative position-embeddingg format.
Pre-training and evaluation at 376M and 776M sizes show that RoPE++HI and RoPE++EC outperform vanilla RoPE and other position embeddings on average across short- and long- context benchmarks. Further analysis reveals that the imaginary attentions play dominant role in modeling long-context dependencies, confirming the effectiveness of introducing imaginary attention for improved long-context capability.
RELATED WORK
Rotary Position Embedding (RoPE) is the dominant position embedding in current LLMs (Dubey et al. 2024 Meta 2024a b. Yang al. 2025a). We analyze its good properties in Appendix B including unifying relative and absolute information via rotation matrices and complex multiplication, and semantic aggregation well as long-context decay. Yet it still faces many other challenges,
2

---

Published as a conference paper at ICLR 2026
attracting a great deal of effort to its improvement mentioned above. A large body of work targets
length extrapolation, scaling the rotary base (bloc97 2023; Liu et al. 2024d Xiong et al. 2024p.
interpolating or compressing index ranges (Press et al. [022 Peng et al. [024: Jin et al. 2024), or
coupling RoPE with sparse attention (Lu al. [024) Xiao et al.
2024a] Liu et al.
2024c) to let models
process contexts far longer than the training window. Other efforts extend RoPE to heterogeneous,
cross-modal inputs (Su) 2024a), especially text-video sequences (Wang et al. [024 Wei et al. 2025).
Parallel lines design parametric schemes that encode contextual cues (Golovneva al., 2024 Zheng
et al.
2024a
Lin et al. 2025), refining or replacing RoPE to yield data data-dependency.
However, few works revisit RoPE's intrinsic computation or analyze its inherent limitations (Hual et al.] 2024 Yang et al.) (025b] [Dai et al. (025). Particularly, the imaginary information loss of RoPE rotation format compared with the complex multiplication format remains overlooked. Although prior work has tried to incorporate the full complex computation into the self-attention mechanism
or neural networks (Wang et al. 2025 Lee et al., 2022), the characteristics and functionality of the imaginary component in position embedding remain unexplored. Therefore, propose RoPE++ and close this gap through deep analysis of the mathematical properties of imaginary attention and extensive validation both short- and long-context downstream tasks.
3 METHODOLOGY
We begin our method by revisiting the complex form of RoPE. Only the real part of the complex product retained, and the imaginary part is discarded, as shown in Equation [ Although current LLMs perform well with this real-only attention, omitting the imaginary component may remove physical information. LLM no longer sees the full magnitude and phase of the complex attention result. This raises the question: can the imaginary part be re- incorpporrteed into the attention computation?
d/2-1
d/2-1 Σ
A, Re
Σ
[(n)=
(1-(s))))
Re
9t
c (i(()) c-0h0)
(n)
(((m)c-(0n0))
n=0
n=0
(I)
d/2-1
St
(2m)
((2m) + gt
(2++1) ((2++) cos 0, s)+
Σ
=
(2n)
(2m+1) o (2 ((2n) sin On s)
0=0
4t
In this section, we will first propose our RoPE++ by re-introducing the imaginary information, in Section as new group of attention heads, namely imaginary attentions, compared with original real attentions. We then analyze the strengths from three aspects, the imaginary heads' stronger capture of long-context dependencies in Section 3.2, the cache and parameter reduction by combining imaginary and real heads in Section 3.3 and the impact on length extrapolation in Section! 3.4
3.1 IMAGINARY EXTENSION OF ROPE
We first recover the imaginary part that is discarded Equation I The resulting expression given in Equation 2 Strictly speaking, it is the negative imaginary part, and the reason will be detailed in Section 3.2 Similar to the real part, the imaginary part carries relative position information between It, k., so the formula can be rearranged into vector form as shown in Equation 2
d/2-1
8/2-1
Al
Im
Pi
[(n)= (00n))=)-0..(-s))
-m
Σ
le
c-tnt)) (K -180))
d/2-1
(2m) (22m) + at
(2n+1) ( ((2++1)) sin On (t s)-
= Σ
(2m) (22+1))
d: (2n+1) ((2)) cos cos Bn, s)
==0
qi
We observe that the imaginary attention still follows a rotation form and can be decomposed into absolute position embeddings gr. : k,, shown in Equation 3 Specifically, the embedding applied
3

---

Published as a conference paper at ICLR 2026
to K2 is identical that used in the real attention in Equation Appendix B For qt. the embedding equivalent to rotating the vector by -T/ / before applying the same embedding in the real case.
A 1m
d/2-1
dt
(2n+1)
cos On sin On s)
sin O2 (t - cos On 8)
(2m) ke
(2n)
kg 122++1)
n=0
Relative PE
(3)
d/2-1
cos 0,t sin Ont
sin ox cos Ont
qi [2m+1)] 4t
T
cos Ons sin Ons
sin 6ns cos S
kg (2n)) k ((2m+1))
(2m)
n=0
Absolute PE
We thus obtain an expression for the imaginary attention, strictly speaking, the negative imaginary attention. If we denote the rotation matrix as R. and Re... The latter parameterized with 0o.- 81/2-1- The computation of real and imaginary attention can be summarized in Equation 4
d/2-1
A
t,a
Re Re
Σ
4t
(m) (()) e 20 (o-0)) (o-1)
(Ro,
Ro,ks - qi Re.
12
d/2-1
Im
Im
Σ
R 3qi) Ro,sk -R 3R) Ro (m) al c0
===
Notably, the newly introduced imaginary component retains the key property of the original RoPE, that it can still be formulated either as relative position or an absolute position embedding. The only required adjustment is to rotate q: by x /2 and then apply the standard position embedding to obtain the imaginary term. We refer to RoPE augmented with this imaginary extension as RoPE++ This augmentation raises further questions: what semantics does the imaginary attention convey, does introduce additional overhead, and can enhance model performance?
3.2 CAPTURE LONGER DEPENDENCY
As stated in Preliminary Appendix B the original RoPE-based attention real attention exhibits semantic aggregation and long-context decay, both governed by its characteristic curve, as shown in Equation and Figure Similarly, we can derive the characteristic curve for the imaginary attention in RoPE++. It is the average of sin (6 At) over the same frequency distribution, approximating sine integral function as shown Equation and Figure
2 4/2-1
En At). Σ sin (1 d n=
sin 0t
At 10
Clm (At)
Gmm
0 10 do = Si(At) Si
(5)
Although modeling distance with sin(OAt) is counter-intuitive since sin(OAt) is zero at relative distance, rises, then falls, unlike cos()At) monotonic drop in the first half-period, the characteristic curve of the imaginary attention still shares the semantic-aggreattion property of the real part. For At > O, when gr, k2 are similar, their attention is on average larger regardless of relative distance, which is the reason why take the negative imaginary part imaginary attention. Moreover, on average, this component attends more to distant positions. As shown in Figure [ its characteristic curve declines very slowly beyond certain distance. Consequently, the imaginary part assigns more weight to the long-context region than the real part, helping LLM retrieve long-context information.
3.3 CACHE AND PARAMETRIC EFFICIENCY
As described earlier, computing the imaginary attention requires only rotating the qi by -n/2, while every other operation is identical to the original RoPE. Because the positional embedding of k, is unchanged, we can interleave the - 2222-rtttted q, with the original q, and perform the real and imaginary attention in single pass in FlashAttention (Dao 2024). Consequently, extra KV
4

---

Published as a conference paper at ICLR 2026
Mikee
Quato
Queice
8 20/"
GQA with RoPE
(b) GQA with RoPE++pC
(c) GQA with RoPE++a
Visualization of GQA with different RoPE schema. RoPE++Ec shares equal cache and twice the attention head with RoPE, while RoPE+tин has equal attention head and half the KV cache.
Figure 2:
VMM
wMM
Tape
200
100 1026
2006
HIM
(a) Position embedding of (b) Position embedding (c) Position embedding of (d) Position embedding of o (2n) ((2n-1) in RoPE (22n) ((2++1) in RoPE++ q (2m+1)) (22n)) in RoPE (2n+1) ((2) in RoPE++
Figure 3: Comparison of trained position embedding interval between RoPE and RoPE++ The area within the dashed line represents trained relative position, and that beyond is in length extrapolation, with learned position embedding values colored in yellow and the opposite in gray.
cache is introduced, and the method plugs directly into MHA GQA (Ainslie et al. 2023), merely doubling the attention head group size, as shown in Figure 2b We refer to this configuration as RoPE+tEC, namely RoPE++ with equal cache size. The only cost of RoPE++TE is additional imaginary attention computed alongside the real one under the fixed QKV parameter budget.
Conversely, if the total head number is kept fixed, both QKV parameters and KV cache sizes halved. We refer to this configuration as RoPE++EH. namely RoPE++ with equal attention head number, shown in Figure 2c] In long-context scenarios, RoPE++TH halves the cache and raises throughput. Because the imaginary attention doubles the number of output heads, W, must be twice as large Wg. Therefore, W. in RoPE++EH equals the original RoPE size, whereas W, in RoPE++EC is double-sized. Experiments in Section 4 show that RoPE++Ec outperforms the original RoPE, especially on long-context tasks, and RoPE++Eu delivers comparable or even superior results.
Importantly, the imaginary and real attention, though computed independently and treated as separate heads, must share the same parameter. Both RoPE++TH and RoPE++EC share W, between the real and imaginary attention. Allocating distinct subsets of heads to imaginary and real attention would effectively collapse back to standard RoPE, since rotating qi in imaginary attention by T/2 yields real attention, with no architecture modification. In other words, imaginary attention is defined relative to real attention and cannot exist independently. Therefore, configurations such as 75% imaginary vs. 25% real or 100% imaginary (applying only the imaginary part) impossible under RoPE++.
3.4 IMPACT ON LENGTH EXTRAPOLATION
A closer inspection of the real and imaginary attention computations reveals an interesting discovery. In vanilla RoPE-based attention, or real attention, shown in Equation even-index query dimen- sions (2)) and odd-index key dimensions multiplied only by cos O,, s) and sin O, (t s)
5

---

Published as a conference paper at ICLR 2026
whose values are always non-negative when 9, is small. Once the imput length exceeds the pre- training context length, these dimensions encounter out-of-distribution (OOD) negative embeddings as shown in Figure Si and thus extrapolate poorly (Liu et al.) 2022dd Peng et al. 2024). In RoPE++ as shown in Equation these dimensions are multiplied by -cos -cos0n (L (1- 8) and sin on, (t 8) in the
imaginary attention, so during pre-training, they have already observed both negative and positive po- sition embedding as well as their maximum and minimum value I1. Consequenty,, these dimensions no longer suffer from the length extrapolation problem in longer contexts (Liu et al. 2025b).
Likewise, odd-index query dimensions (2 and even-index key dimensions k(2m) encounter only cos (t - s) and sin 0n (4 8) in the real attention, and the imaginary attention further exposes them to cos ot, 8) and sin Oz, (t. -)) Yet this alone does not expand the position embedding range trained in pre-training, as shown in Figure[5hJand Figure However, when real and imaginary atten- tion are combined, qt, k2 in RoPE++ attains the full cos and sin value range, once the training length exceeds half the sinusoidal period, whereas the vanilla RoPE requires a full period. Consequently, more dimensions in RoPE++ observe complete positional information. Therefore, perplexity grows more slowly beyond the maximum supported context length (Liu et al. [024d]] Men et al. (024)).
4 EXPERIMENT
4.1 SETUP
We validate RoPE++ at both 776M and 376M model sizes, with architectural details Appendix C Both models are pre-trained on DCLM-Baselined. 1.0 corpus (Li et al. 2024) by HuggingFace Trans- formers (Wolf et al. 2020) on 8 NVIDIA H200 160 GB GPUs. For each size, we use a batch size of 0.5M tokens and pre- pre-train for 50B tokens. We use Adam W (Loshchilov et al. (017) optimizer with weight decay O. .. a maximum learning rate of 5e-4, and a warmup-stable-decay scheduler. We use the first 0.5B tokens for warmup, and the final 5B tokens for decay, and the learning rate ends at O.
We compare our RoPE++ with standard RoPE (Su et al. 2024) and other well-known position embedding designs, including FoPE (Hua al., 2024), Pythia (namely, partial RoPE with only last 1/4 dimensions being rotated) (Biderman et al.] (023), ), as well as ALiBi (Press et al. 2022). We pre-train all methods 4k context length with initial rotary base of 10000. For RoPE and RoPE++, we conduct continuous long-contex: pre-training, Following Xiong et al. (2024); Lv et al. (2024). we scale the rotary base from 10000 500000 and train for 10B tokens from DCLM on 32k context length, using a aosinl-annealing learning rate scheduler and keeping all other settings.
4.2 SHORT-CONE CONTEXT EVALUATION
We evaluate both short-context and long-context tasks based on OpenCompass (Contributors, 2023). For short-context evaluation, we measure perplexity WikiText (Merity et al. 2017) and LAM-
BADArPa et al. (016) and assess downstream tasks mainly in Open LLM Leaderboard (Hug
gingFace 2023), including TruthfulQA (Lin et al.] (022), PIQA (Bisk et al. 2020). HellaSwag (Zellers
et al. 2019), WinoGrande (Sakaguchi et al. 2020), ARC-e (Clark et al. 2018), GPQA (Rein et al.
2023). SocialIQA (Sap et 2019). OpenBook (Mihaylov et al. 2018), and SuperGLUE (Wang
et al. 2019). All models are tested within 4k context length.
The results are shown in Table [ Our RoPE++EC and RoPE++EH achieve the best average scores on short-context tasks compared with RoPE and every other position embedding design. Notably, RoPE++HI surpasses standard RoPE with only half the KV-cache and QKV parameters. After further long-context pre-training, RoPE++ still retains this edge over RoPE on short-text benchmarks.
4.3 LONG-CONTEXT EVALUATION
For long-context evaluation, we evaluate downstream performance at varying lengths with the classical synthetic benchmarks, RULER (Hsieh et al. 2024) and BABILong (Kuratov et al.) 2024). The results are shown in Table and Figure 6 We highlight the comparison with RoPE in long- long-context training because RoPE is the position embedding currently most widely used by long-context LLMs.
On RULER and BABILong up to 64k context, our RoPE+ again acquires the highest scores. Particularly, RoPE++H achieves comparable performance with vanilla RoPE using half the KV-

---

Published as a conference paper at ICLR 2026
Wiki
LMB
TQA
PIQA
Hella
Wino ace
ARC-e
GPQA
SIQA
OBQA
SG
Avg.
ppl 4
ppl
acc f
acc 1
acc
ace
acc 1
ace
acc
acc 1
376M Short
RoPE
19.9
32.7
35,5
66.3
34.8
50.9
39.3
24.8
38.6
27.4
43.7
40.1
FoPE
19.3
33.0
33.8
65.9
34.5
53.0
37.0
28.8
39.5
24.2
43.6
40.0
Pythia
19.2
32.9
34.7
65.8
34.9
51.5
41.3
21.2
39.7
25.6
42.5
39.7
ALiBi
21.2
34.6
33.8
66.1
34.2
51.1
44.4
24.8
38.7
27.4
43.9
40.5
RoPE++En
20.8
33.6
36.3
66,4
34.5
52.5
40.9
23.7
40.5
24.8
43.2
40.3
RoPE++EC
19.4
32.6
37.3
68.0
35.6
53.0
41.3
25.8
40.3
23.2
44.8
41.0
376M Long
RoPE
20.4
33.8
35.4
64.9
34.1
50.6
40.4
21.2
39.4
27.4
43.5
39.6
RoPE++oD
21.7
34.8
35.2
64.5
34.3
49.9
41.5
22.7
40.0
27.0
43.1
39.8
RoPE++EC
20.0
33.9
37.1
66.1
34.1
53.4
38.1
21.2
39.2
28.4
43.7
40.1
776M Short
RoPE
14.8
27.3
35.5
70.1
43.7
52.3
43.4
25.8
41.3
21.8
43.6
42.0
FoPE
14.7
27.1
33.6
68.7
43.4
52.9
45.0
24.8
39.7
24.8
45.4
42.0
Pythia
14.8
26.9
35.8
68.8
42.9
52.1
39.5
22.2
42.0
21.2
43.6
40.9
ALiBi
15.2
28.3
35.2
70.2
43.7
53.6
43.2
23.7
40.6
27.6
45.9
42.6
RoPE++EH
15.6
28.1
35.4
69.6
42.7
53.5
45.0
15.8
41.6
26.8
42.4
42.5
RoPE+tEc
14.8
27.3
36.1
69.3
43.6
52.3
43.7
28.3
40.1
27.6
44.4
42.8
776M Long
RoPE
27.3
68.9
43.1
51.5
47.6
21.7
40.7
20.2
42.6
41.3
RoPE++H
15.3
28.1
35.4
69.9
41.9
52.6
43.2
28.3
41.0
22.2
43.4
42.0
RoPE+tec
14.4
27.1
35.2
70.4
43.7
52.6
44.8
31.8
40.8
27.6
44.3
43.5
Table Results on short-context tasks for 776M and 376M models pre-trained in 4k context length and further trained on 32k. Best results are highlighted in bold, with the second best underlined for broader comparison. Our RoPE++ achieves the best average performance on different model sizes.
RULER
BA BABILong
4k
8k
16k
32k
64k
Avg.
2k
4k
8k
16k
32k
64k
Avg.
376M Long
RoPE
31.6
25.6
22.0
9.5
5.5
18.8
17.7
16.1
9.1
9.4
5.9
7.8
11.0
RoPE++EH
29.9
28.4
17.6
9.4
5.9
18.2
14.1
15.6
12.2
9.9
8.3
9.7
11.6
RoPE+tEC
36.1
33.0
29.1
17.7
9.0
25.0
19.8
19.8
15.8
12.3
12.8
776M Long
RoPE
37.4
35.1
33.0
21.2
10.4
27.4
33.5
30.7
23.6
22.0
15.1
12.1
22.8
RoPE++EH
38.7
35,4
33.8
24.6
10.7
28.6
31.9
26.5
18.6
16.2
11.0
12.2
19.4
RoPE++EC
42.7
38.6
33.4
21.7
10.9
29.4
32.4
29.9
24.4
24.5
18.6
14.8
24.1
Table 2: Results on long-context tasks, including RULER and BABILong for 776M and 376M models further trained with 5B tokens in 32k context length. Best results are highlighted in bold. Our RoPE+ achieves the best performance on average, especially in long-context scenarios.
cache and QKV parameters, while RoPE++EC delivers significant gains at the same cache size. Although RoPE occasionally edges ahead at few shorter context lengths, RoPE++, including both RoPE++ge and RoPE+taI. maintains more stable performance as context length grows and achieves best performance in 64k context length extrapolation consistently.
DISCUSSION
5.1 RoPE+ AS CACHE OPTIMIZATION
As mentioned in Section 3.3 RoPE+tEH halves KV cache and QKV parameters while keeping the
attention head number equal, yielding evident efficiency gains. We validate this efficiency strength by assessing the memory well as Seme-Per-Quuput-TToon (TPOT) of 376M and 776M models,

---

Published as a conference paper at ICLR 2026
Raffira
Riftinn
RaPfic
Cintro CiniceLorph
CentreLoggh
(a) Memory Cost of 376M
(b) TPOT of 376M
(c) Memory Cost of 776M
(d) TPOT of 776M
Figure 4: Efficiency comparison between RoPE and RoPE++EH in 376M and 776M model. RoPE+tH lowers memory cost and accelerates decoding, and the margin widens as context grows.
Quar Inder
Indea
(a) Layer 2 Head 10 (b) Layer 2 Head 11 (c) Layer 6 Head 10 (d) Layer 6 Head 11 (e) Average RULER-
(Real) in 376M
(Imag) in 376M
(Real) in 376M
(Imag) in 376M
4k score 376M
Ral
Kay I3dea
Kay Eodex
Key Sndes
Sancard
(I) Layer Head (g) Layer Head (h) Layer Head (i) Layer Head (j) Average RULER- (Real) in 776M (Imag) in 776M
(Real) in 776M
(Imag) in 776M
4k curve in 776M
Figure Attention-score patterns and long-context performance in 376M and 776M RoPE++ models. Imaginary heads attend markedly to global information, whereas real heads focus more on local context. Adding Gaussian noise to imaginary attention degrades long-context performance more severely, over 8 points, than the same perturbation applied to real attention.
from 2k to 32k context length. We conduct the efficiency evaluation single NVIDIA H200 160BG GPU, with a batch size of 8 samples. The results are shown Figure 4 At both 376M and 776M, RoPE+tEi consistently reduces memory cost and speeds up decoding, with the margin widening as context length increases.
5.2 ATTENTION PATTERN OF RoPE++
To verify how imaginary attention captures long-context dependencies and contrast it with real attention in RoPE++, we inspect the attention patterns of chort-context--rainn RoPE++ge at 376M and 776M shown in Figure Odd-index imaginary attention highlights the initial positions more strongly than even-index real heads, indicating stronger global focus. Since prior work (Liu et al.) 2025a Wei et al. 2025) shows that dimensions attending globally are more critical for long-context semantics, imaginary attention may play the dominant role in long-context tasks.
For further verification, we design the following validation experiment. add Gaussian noise with equal standard deviation to the imaginary and real attention components separately, and monitor the change in RoPE++ performance on long-come cong-context tasks, such as the average score of RULER-44 Curves for RULER-4k versus standard deviation are plotted for both real and imaginary attention. When the standard deviation is small (a < 0.2), scores with corrupted real or imaginary attentions stay close the baseline; when it is large enough (a 1.5), both drop sharply. Importantly, in the
8

---

Published as a conference paper at ICLR 2026
Short
RULER
BABILong
ppl
score
4k
8k
16k
32k
Avg
2k
4k
8k
16k
32k
Avg
376M Long Pl
RoPE
33.4
42.0
36.5
33.6
19.7
10.6
25.1
19.3
12.3
10.9
12.7
RoPE++uH
34.7
41.7
28.0
27.6
15.8
6.9
19.6
13.3
12.4
12.8
8.9
10.4
11.6
RoPE++Ec
33.7
42.8
37.0
32.4
28.3
10.6
27.1
24.0
20.7
15.9
14.3
12.3
17.4
376M Long YaRN
RoPE
32.8
42.2
36.4
32.9
28.4
15.0
28.2
22.4
16.4
11.4
10.7
11.1
14.4
RoPE++EH
33,9
42.2
32.7
30.2
24.9
10.7
24.7
8.7
9.3
12.1
11.3
10,9
10.5
RoPE++Ec
32.9
43.4
36.0
33,9
31.7
17.8
29.8
27.4
23.6
18.0
16.9
12.3
19.6
776M Long
RoPE
27.8
40.4
37.8
34.4
30.5
13.4
29.0
15.3
16.9
12.7
11.8
9.3
13.2
RoPE++EH
28.8
40.4
37.9
35.0
27.5
14.6
28.8
21.0
22,4
17.1
13.7
11.1
17.1
RoPE++EC
27.8
40.5
43.0
28.8
13.6
31.0
25.7
23.4
16.4
9.4
16.6
776M Long YaRN
RoPE
27.3
40.9
37.6
35.0
33.9
27.5
33.5
26.9
25.6
19.5
16.4
12.2
20.1
RoPE++EH
28.3
40.6
37.9
34,9
32.2
26.1
32.8
28.0
23.9
18.6
17.8
11.7
20.0
RoPE++EC
27.3
41.5
42.9
36.5
36.3
22.2
34.4
26.3
24.1
21.1
19.8
16.9
21.6
Table 3: Results of 776M and 376M models further trained with 5B tokens in 32k context length with YaRN and Linear Pl. Our RoPE++ still achieves the best performance on average.
intermediate range, adding noise to the imaginary attention always performs worse than corrupting the real part. When = 1.0. for example, the real-noised RoPE++ outperforms the imaginary-noisse one by points at 376M and 8 points at 776M, which demonstrates significant gap. Thus, impairing the imaginary heads degrades long-come long-context performance more, confirming that imaginary attention plays a more dominant role in long context modeling.
5.3 COMBINATION WITH OTHER LONG-CONTENT TECHNIQUES
RoPE++ can not only be combined with NTK for context extension during long-context training, but can also combined with other long-context techniques such Linear (Chen al. (002)) and YaRN (Peng al. 2024). Across 376M and 776M model sizes, conduct extensive experiments of long-context further pre-training in 32k context length, with the interpolation coefficient s = for Linear PI and 32 for YaRN, the default values in the original paper. The results are shown Table We report the perplexity on WikiText and the average score of tasks we have presented in Table as the summary of short-context performance, with the full results in Table T0 Results show that RoPE++ consistently achieves the highest scores on RULER, BABILong, and short-context average score, confirming its advantage and generalization. More analysis larger model scale and training convergence is detailed in Appendix C and limitation of RoPE++ can be found Appendix
More discussion on the extrapolation performance
CONCLUSION
We introduce RoPE++. which employs both real and imaginary attentions. Mathematical analysis first reveals the imaginary attention's 's potential for modeling long-context dependencies. Building upon this, we re-incorporate the originally discarded imaginary attention as a new group of heads while preserving the unified absolute-relativ position embedding format. Particularly, we introduce RoPE++EH, with equal head as well as halved cache, and RoPE++Ec with equal cache and doubled heads. Pre-training and evaluation 376M and 776M model sizes show that both RoPE++EH and RoPE++ge outperform vanilla RoPE and other position embeddings on average across short- context tasks and acquire that imaginary attentions are more dominant in long-context modeling compared with original real attention, validating their effectiveness in enhancing long-context I.I.Ms.
larger gains in long-context scenarios. Further analysis confirms

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (No. U24B20181) and Shanghai Pilot Program for Basic Research- Fudan University 21TQ1400100 (22TQ018). We greatly appreciate all reviewers for their constructive reviews, and thanks to Jiasheng Ye for the discussion on scaling verification of model architecture.
ETHICAL STATEMENT
This research follows established ethical standards and practice principles. our knowledge, our study processes no sensitive personal data, involves no human subjects, and targets no ethically risky applications. All experiments and analyses comply with recognized guidelines, ensuring integrity, transparency, and reliability.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of and support the open-source community, have publicly released RoPE+, its trained checkpoints, and the complete training and evaluation code. We expect these as reference for future work long-context LLMs, facilitating progress in this field.
REFERENCES
Joshua Ainslie, James Lee- thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205. 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning about physical commonsense natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 609 aaaii,aaaa..///aaaaaaaaaaaaaa//a v34105. 10.1609/AAAL
AAAI Press, 2020. doi:
URL htLhhttps:/
org
att./aaaa..........a.. 6239
bloc97. Dynamically scaled rope further increases performance of long context llama with zero www.:/aaaa./oomtttt..... htt.ffortther_icceeeeeeee intieeerrncreeeeese/e 14mrgpr/dynam ca cally wws.ccalllo hcc.eleeee_eee_oootteeeeeeeeeeee
fine-tuning, July 2023. htt.hh.ppp::/ww..rdd..... http://wwwwwdeddeoo/////aaaaaaaa
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of
large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge.
CoRR, abs/ abs/1803.05457, 2018. URL httttt:://axxi..err//eeeae
1803 3. 05457
OpenCompass Contributors, Opencompass: A universal evaluation platform for foundation models. 2023.
atttps://githun http://tttthc..ooppaaaaaaaa
com/ petppencooaass/pppncooooo
Chang Dai, Hongyu Shan, Mingyang Song, and Di Liang. Hope: Hyperbolic rotary positional
encoding for stable long-range dependency modeling in large language models. arXiv preprint 1wXii:2509.052/8.. 2025.
10

---

Published as a conference paper at ICLR 2026
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Tralsformer-x:: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position encoding: Learning to count what's important. arXiv preprint arXiv:2405. 18719, 2024.
Demis Hassabis and Koray Kavukcuoglu. Introducing gemini 2.0: our new ai model for the agentic era, 2024. URL httphtt/p/:bblgtt. www.goooooooooooooo le/technol ogy/ googl google-deeeeee.... e-deepm nd/ google-gemin ni -ai waaitpddae--ooe -ewpetteeeddeeeeee--eee0--
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.066544 2024.
Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Youbang Sun, Yuchen Fan, Xuekai Zhu, Biqing Qi. Ning Ding, and Bowen Zhou. Fourier position embedding: Enhancing attention's 's periodic extension for length generalization. arXiv preprint arXiv:2412.17739) 2024.
HuggingFace. Open Ilm leaderboard. 2023. URL http::://hoggggggfacc..e////ecee
#boginngaacet///per_rr leaderboard
InternMM. Internl3-8b. January 2025. URL https://huggianfacc.c//eeeetrr
Internim3 8b- instruct]
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Yew Lin, al. Minference 1.0: Accelerating pre-filling for long-context Iims via dynamic sparse attention. arXiv preprint arXiv:2407.024490 2024.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu, Lim maybe longlm: Self-extend 11m context window without tuning. arXiv preprint ar arXiv:2401.01325, 2024.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Iims with long context reasoning-in- haya-haystack. arXiv preprint arXiv:2406.1011 2024.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 110) 1995.
ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao. Complex-valued neural networks: A compre- hensive survey. IEEE/CAA Journal of Automatica Sinica, 98):1406-1426. 2022.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances Neural Information Processing Systems, 37:14200: 14282, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. -2522, Association for Computational Linguistics, 2022. doi: 10.18653/V 12022 229.. URL 18653/v1/2022 aclo22accll-1 1ong. 229
Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. arXiv preprint arXiv:2503.02130, 2025.

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.0044 004444440444 2024a.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints, pp. arXiv-2402, 2024b.
Xiaoran Liu, Ruixiao Li, Qipeng Guo, Zhigeng Liu, Yuerong Song, Kai Lv, Hang Yan, Linlin Li,
Qun Liu, and Xipeng Qiu. Reattention: Training-free infinite context with finite attention scope. arXiv preprint arXiv:2407.15177 2024c.
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, 2024d.
Xiaoran Liu, Siyang He, Qiqi Wang, Ruixiao Li, Yuerong Song, Zhigeng Liu, Linlin Li, Qun Liu, Zengfeng Huang. Qipeng Guo, et al. Beyond homogeneous attention: Memory-efficient llms via aproxir-ae kv cache. arXiv preprint 2506::22006//888. 2025a.
Xiaoran Liu, Zhigeng Liu, Zengfeng Huang, Qipeng Guo. Ziwei He, and Xipeng Qiu. Longllada: Unlocking long context capabilities diffusion llms. arXiv preprint arXiv:2506. 14429, 2025b.
Hya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint 1rXiv: 5(5):5, 2017.
Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. Longheads: Multi-head attention is secretly long context processor. arXiv preprint 2rx.v:2402.10685, 2024.
Kai Lv, Xiaoran Liu, Qipeng Guo. Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwanjuan: Towards systematic measurement for long text quality. arXiv preprint arXiv:2402.13355 2024.
Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. arXiv preprint arXiv:2405. 14591, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL https
/ //oenreview ew. net wwt/ffornnte htt///orrmmmddee By
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI., 2024a.
AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Mela Al., 2024b.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 23911. Association for Computational Linguistics. 2018. doi: 10.18653/V1/D18- 1260 URL
/ttps::/Aa..orr/000...
1260
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
OpenAI. Gpt-4 technical report. arXiv preprint arX1::2303.08774,,000 2023.
OpenAL. O1: Openai's first model, 2024. URL 12-25.
com/ o1/ Accessed: 2024-
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 18.13////V P16-1144. htRpsttts: //ooi org/10. wtt..011115///1112101 /
12

---

Published as a conference paper at ICLR 2026
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam:: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a
benchmark. CoRR, abs/2311.12022 2023. doi: 10.48550/RRXV.2111..20 7/doi.org/// 48444555/aaa 48554/8825vv..... 2311. 12022
URL https?
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 8732 8740. AAAI Press, 2020. doi: 10.1609/AAAI M.A.1334400.699
URL hwwp::ttss///cc.t. / o org 10. 509/ aaai. v34105. 6399
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4462-4472. As- sociation for Computational Linguistics, 2019. doi: 10.18653/V1/DD19 1454. URL https? //doi. 1ttp:///0111533////111111- 1454
Jürgen Schmidhuber, Sepp Hochreiter, et al. Long short-term memory. Neural Comput, 9(8): 1735-1780, 1997.
Jianlin Su. Rerope: Rectified rotary position embeddings, July 2023. URL hEtps://github
com/bo ham/bojone//eer rerope
Jianlin Su. Transformer upgrade path: 17. insights into multimodal positional embedding, March
2024a. URL /tRppttts////pacss spaces. ac. cn/ archives/ioi000
Jianlin Su. Transformer upgrade path: 18. rope base selection principle, March 2024b. URL
fm. Bttpos//eeexue..haaarhhhrrrrre/e
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Tianxiang Sun, Xiaotian Zhang. Zhengfu Hc, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, Yu-Gang Jiang, and Xipeng Qiu. Moss: An open conversational large language model. Machine Intelligence Research, 2024. ISSN 2731-5398, doi:
URL https: /ttp:://ittttt com/ www.eoeeee//oee.eecee. /MSSS
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., fi F1les/paper/2017 http://ppoocceeiis.c.eeeeeeee Paaerr.
2017. URL #ile:/ffo22454//7eeee11555535333
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garntt (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 3261 3261-3275, 2019. URL htpp:ttt////ppceddnngge nww.eiinsiinnnnnnn.o..e.m.m..eee http://s..e.eoooo/////////////// 9/ahh/hh// a
Abst html
13

---

Published as a conference paper at ICLR 2026
Feiyu Wang, Guoan Wang, Yihao Zhang, Shengfan Wang. Weitao Li, Bokai Huang. Shimao Chen, Zihan Jiang, Rui Xu, and Tong Yang. ifairy: the first 2-bit complex Ilm with all parameters in {Li}. arXiv preprint wrxiv:250.005571 2025.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint wwXiv:2409.12191,, 2024.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding? arXiv preprint arXiv:2502.05 2025.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38 38-45, 2020.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-con congcct Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024b.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4643 4663, 2024.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. Tener: adapting transformer encoder for named entity recognition. arXiv preprint 191 1.04474, 2019.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09380 2025a.
Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transformations. arXiv preprint arXiv:2505. 253.288,05.1.81,,, 2025b.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint 2.Xi::2508.15877 2025.
Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04653 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Mårquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. Florence, Italy, July 28- August 2019. Volume 1: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/P19-11 1472. URL https! 10. www.11866//ppppt 1472
Chuanyang Zheng, Yihang Gao. Han Shi, Minbin Huang. Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, et al. Dape: Data-adaptive positional encoding for length extrapolation. In The ehirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, et al. Dape v2: Process attention score as feature map for length extrapolation. arXiv preprint S4V:22410.047000 2024b.
14

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
We use Large Language Models solely for language-centric assistance, including checking grammar, style, and clarity. No aspect of research, including ideation, experimental design, or scientific contribution, is influenced generated by the output of LLMs.
B PRELIMINARY: KEY PROPERTIES OF ROPE
Rotary Position Embedding (RoPE) encodes absolute positions by splitting the feature dimensions of query and key vectors qt. k, into 2-D pairs and rotating each pair (Su al. 2024). The rotation angle is the product of the token index t or and 0,. Owing to the properties of rotation matrices, the independently applied absolute position embedding on g:, k2 fuse into relative position embedding, namely cos On
s), sin 0n s), of the attention matrix, shown in Equation 6
4/2
(2n)
cos ( s) sin fn. -)
sin fx s) cos (t s)
k
(2n)
A
(2a+1)
ks
12n+1)
Relative PF:
(6)
d/2-1
coo t,t sin 8,,
sin f cos t
(2n) 2n+ qi
1)
cos 8,, sin Sns
sin sns cos S
ke (2n)
((2++1)
Absolute
By default, the rotary angles On 10000 2n/d 0,...
d/2 1
Equation 6 presents RoPE in vector form. Since any 2-D vector corresponds to complex number, the rotation of such a vector is equivalent to complex multiplication.
(2n) + qe (2n+1),
[((m) = k(2m) + f 1(2n+11
= q:
Building on this equivalence, RoPE can be expressed in complex form as shown in Equation !
Besides unifying relative and absolute position embeddings, RoPE exhibits semantic aggregation and long-context decay (Su et al., 2024). On one hand, when k vectors are semantically close, their attention score remains large on average, regardless of relative distance At. This property is detailed in Su (2024b). If we have vector & that is independent and identically distributed with respect to q. with average y and variance o² for every feature dimension, and vector that is only the expected attention score difference can be calculated as
slightly perturbed with respect
follows and proved to be positive.
Eq.k.e [q R
R-silg
+ q TRRRa R-sik]
-Fag T R -
K. 1 R kik
-Eg R_ [
Eg|q] R R.
-EE q R -ig
1²2 R R-sel
d/2-1 1
4/2-1
=Eg
(2n)) + + ((2 cos (-)x
Σ
2p cos (-0,,
n=0
n=0
d/2-1
d/2-1
Σ n=0
2 2122
2 + 0²) cas cas At
Σ
2μ² cos 0,
n=ū
d/2-1
2 cos 110000- 2m At 0 Σ
On the other hand, as increases, the attention between any q, k decreases on average. We can similarly derive this by showing that the expectation of the attention score, shown below, is almost
15

---

Published as a conference paper at ICLR 2026
monotonically decaying with the increase of At.
d/2-1
E. [q R Riakk] = Eg
g (22m) + ((
cos (6,, At)
m=0
d/2-1
Σ n=0
201² + 0²) cos (10000 3" A1)
Both properties arise from averaging cos() over frequency sampled based on On 10000 n = 0, ... d/2 - 1. It a discrete approximation CRe (At) to a cosine integral function CRe( AA), shown in Equation 7 We refer to this as the characteristic curve of RoPE, shown in Figure It is positive and decaying. conferring these two mathematical properties of RoPE.
d/2-1
CRc (At)
CR
Σ 7=0
Cos (10 3 At). CRo (At)
cos Ot
do = Ci(At) - Ci
8 10
10a
10
For the imaginary part lost in RoPE's complex representation, we derive a sine integral function, and that is the characteristic curve for the imaginary attention in RoPE+t, as shown in Equation
d/2-1
K. [q R R 3--r + e) q R
R --a
Σ m=0
20 sin (10000- A1)) 0
d/2-1
Kg [q R R- = Σ
2(1²² + +
sin (10000-at 2n At)
n=0
d/2-1 sin
4m 9m (at)
2
10 sm At),
cmm
d0 Si(At) Si Ol 10
sin Ot
At
104
Finally, we should also clarify that our analysis is expectation- i.e., the average fluctuation of the real or imaginary attention, which is different from the case-study-level discussion in RoPE (Barbero et al. 2024). The analysis of p-RoPE on long-context decay orthogonal to our contribution. We likewise drop the rotary angles for the low-frequency dimensions in both the real and imaginary attention of RoPE++.
C MORE EXPERIMENT
The configuration of our 376M, 776M and 1.5B models can be summarized following table. Dubey al. 2024
Our models use the same tokenizer as the Llama 3 Series (Meta 2024b
376M
776M
1.5B
Hidden Size
1024
1536
2048
Intermediate Size
3584
5376
7168
Num Layer
8
12
Num Attn Head
8
12
16
Num KV Head
4
4
Vocab Size
128256
128256
128256
Table 4: The
of different model sizes.
C.I VALIDATION ON LARGER SCALE
Scaling validation essential for architectural research, though concurrent or earlier work still only performs pre-training validation models smaller than 1B and data volumes smaller than
16

---

Published as a conference paper at ICLR 2026
Wiki
LMB
TQA
PIQA
Hella
Wino ace
ARC-e
GPQA
SIQA
OBQA
SG
Avg.
ppl 4
ppl 4
acc f
acc 1
acc
ace
acc 1
ace
acc
acc 1
1.5 Short
RoPE
27.9
15.3
36.4
70.8
46.2
53.2
44.8
25.8
39.4
24.6
44.8
42.9
RoPE+ta
28.0
16.0
37.1
71.2
45.6
53.6
45.9
25.8
40.8
27.8
44.3
43.6
RoPE++Ec
27.4
15.5
35.1
69.5
46.3
53.3
43.6
28.3
40.5
23.6
45.6
42.9
1.5B Long
RoPE
24.8
12.5
36.3
71.1
49.3
56.7
48.9
27.3
40.8
24.2
46.1
44.5
RoPE+TEH
25.1
13.0
36.1
71.3
48.0
55.9
48.0
23.2
41.2
29.2
44.1
44.1
RoPE++EC
24.4
12.4
36.6
71.3
49.6
55.3
44.8
20.7
41.1
26.2
45.1
43.4
Table 5: Results on short-context tasks for 1.5B models.
RULER
BABILong
8k
16k
32k
64k
Avg.
4k
16k
32k
64k
Avg.
RoPE
50.5
39.9
34.6
31.9
18.3
35.1
25.9
29.7
31.8
30,5
18.5
13.5
29.5
RoPE+tиh
42.5
33.1
12.7
31.0
35,9
30.5
25.1
15.8
32,9
RoPE++EC
53.1
45.7
39.0
30.6
18.9
37.5
25.1
28.3
21.4
16.6
12.6
12.3
22.9
Table 6: Results on long-context tasks for 1.5B models further trained with 5B tokens in 32k.
5B
10B
15B
20B
30B
40B
50B
Training Loss
RoPE
3,4698
3.3576
3.2249
3.3459
3.2945
3,2946
3.1665
RoPE++HH
3,4904
3.3821
3,2511
3.3708
3.3202
3,3194
3.1931
RoPE++Ec
3.4567
3.3473
3.2203
3.3373
3.2848
3.2904
3.1612
Validation Loss
RoPE
3,5509
3.4358
3.3933
3.3629
3.3299
3.3177
3.1881
RoPE++u
3.5772
3.4618
3.4217
3.3907
3.3567
3.3430
3,2141
RoPE+tEc 3,5362
3.4254
3,3870
3.3569
3.3251
3,3140
3.2683
Average Score
RoPE
39.2
39.6
40.4
39,9
39.5
39.8
40.1
RoPE++u 38.2
39.6
40.3
38.9
39.5
39.8
40.3
RoPE++Ec 38.8
38.8
39.3
39.6
40.3
40.2
41.0
Table 7: Comparison of training loss, validation loss, and the average score short-context tasks between RoPE and RoPE++ on the 376M model size under different training tokens.
50B tokens (Dai et al. 2025 Hua et al. 2024). Unfortunately, our available resources limit us scales below 7B. After an extended effort, we have completed 1.5B model trained on 50B tokens in 4k context length, followed by 5B tokens in 32k context length, using the same training hyperparameters used for 776M and 376M. The results in Table 5 and Table demonstrate that RoPE+ still outperforms RoPE. To sum up, within the limits of our available computing resources, we have successfully validated the effectiveness of RoPE++ across all three model scales.
Concerning the data scales, based on the experiments 776M and 376M, 50B tokens are already sufficient for convergence at this scale, as evidenced by plateaued training loss, validation loss, and average short-context scores in Table 7 We verify this judgment by evaluating 776M and 1.5B checkpoints pre-trained on more tokens while the learning rate remains constant. Beyond 50B tokens, the model shows no significant further gain. These results also demonstrate that RoPE++ exhibits loss curves that almost overlap with those of RoPE, showing no training stability issues. Although RoPE may show advantage in the early checkpoints, RoPE++ continues to improve and ultimately surpasses RoPE in average scores. Note that these 776M results at 50B tokens are obtained without learning-rate annealing, they differ slightly from the scores reported above.
17

---

Published as a conference paper at ICLR 2026
5B
10B
15B
20B
30B
40B
50B
60B
70B
Training Loss
RoPE
3.2826
3.1621
3.0279
3.1418
3.0889
3.0913
3.0502
3.0208
3.0307
RoPE+tu
3.3104
3,1955
3.0622
3.1701
3.1236
3.1237
3.0798
3.0496
3.0596
RoPE+tee
3.2797
3.1620
3.0294
3.1377
3.0890
3.0889
3.0489
3.0200
3.0294
Validation Loss
RoPE
3.3480
3.2279
3.1842
3,1552
3.1214
3.1083
3.0912
3.0817
3,0751
RoPE+tEH 3.3865
3.2648
3.2184
3.1877
3.1545
3.1401
3.1232
3.1168
3.1050
RoPE+tEc 3.3541
3,2305
3,1872
3.1550
3.1205
3.1061
3.0903
3.0816
3.0733
Average Score
RoPE
39.3
40.8
41.3
41.5
41.4
41.5
41.7
42.3
41.8
RoPE+tER
40.0
40.9
41.2
40.6
41.7
42.2
41.3
41.9
42.1
RoPE++EC
38.7
40.4
41.1
41.0
41.8
41.7
41.7
41.8
41.9
Table 8: Comparison of training loss, validation loss, and the average score of short-context tasks between RoPE and RoPE++ on the 776M model size under different training tokens.
5B
10B
15B
20B
30B
40B
50B
60B
70B
80B
90B
100B
Training Loss
RoPE
3.1798 3.046 2,9492
2.9048
2,9835
2.8967
2.9447
2.8384
2.9855
2.8224
2.9697
2.9775
RoPE++EH 3.2051 3.0718 2.9724 2,9333
3.0104
2,9244
2.9688
2.8619
3.0110
2.8491
2,9980
3.0018
RoPE++EC
3.1708 3.0384
2.9418
2.9025
2.9824
2.8916
2.9404
2.8307
2.9775
2.8177
2.9664
2.9736
Validation Loss
RoPE
3.2498
3.1297
3,0754
3.0492
3.0112
2.9911
2.8477
2.9711 2.9641
2,9526
2.9503
2.9464
RoPE++E 3.2768
3.1569
3.1021
3.0785
3.0406
3.0167
2.8744
2.9949
2.9904
2,9799
2.9780
2.9681
RoPE++EC 3.2420
3.1236
3.0709
3.0466
3.0086
2.9857
2.8456
2.9669
2.9577
2,9515
2.9488
2.9433
Average Score
RoPE 40.9
41.5
42.2
42.7
43.0
43.2
42.9
42.7
43.2
43.8
43.4
43.0
RoPE++E 39.6
40.8
41.8
41.9
42.1
42.5
43.6
43.3
42.4
43.4
42.8
42.7
RoPE++EC 39.6
41.2
41.4
42.0
42.7
42.4
42.9
42.7
42.7
43.0
43.8
43.1
Table 9: Comparison of training loss, validation loss, and the average score short-context tasks between RoPE and RoPE++ the 1.5B model size under different training tokens.
C.2 MORE DETAILED RESULTS
The detailed results on short-context tasks for 776M and 376M models further trained with 5B tokens in 32k context length with YaRN and Linear PI are shown in Table TT) We also add the comparison of the training throughput (TGS, tokens per GPU per second) at 4k and 32k training context lengths, well the model storage (GB), shown in Table II] Notably, though RoPEge keeps the cache size fixed, it still increases computation and the size of W.. Nevertheless, long-context inference is primarily 1O-bounded rather than cempuutaiion-bounde and dominated by KV cache memory cost. Therefore, the absence of additional cache overhead remains acceptable. We will continue to develop more elegant computation optimizations tailored to RoPE++ reduce this additional cost.
D MORE DISCUSSION
D.I POTENTIAL REDUNDANCY AND CONFLICT
Regarding possible redundancy and conflict among attention heads, we first clarify that these issues already exist in vanilla RoPE-based LLMs, motivating works that directly compress KV cache (such as MLA (Liu et al. 2024a), GQA (Ainslie et al., 2023)) or distinguish head types (such as DuoAttention (Xiao et al. 2024b). MInference (Jiag et al. 2024)). Therefore, this redundancy and conflict likewise remain present in RoPE++. Concerning redundancy, the imaginary and real attentions exhibit distinct biases and, as shown in Figure [S] show different functional patterns. Because RoPE+teu outperforms vanilla RoPE while using half the cache, redundancy between the
18

---

Published as a conference paper at ICLR 2026
Wiki
LMB
TQA
PIQA
Hella
Wino ace
ARC-e
GPQA
SIQA
OBQA
SG
Avg.
ppl 4
ppl
acc f
acc 1
acc
ace
acc 1
ace
acc
acc 1
376M Long Pl
RoPE
33.4
20.1
34,9
69,4
43.1
52.5
42.9
29.3
40.0
21.6
44.3
42.0
RoPE+tиH RoPE++EC
34.7
20.7
35.5
68.4
41.1
53.7
44.6
22.7
40.0
26.0
42.9
41.7
33.7
19.4
36.6
70.1
43.1
51.9
44.8
27.3
40.7
27.0
44.1
42.8
376M Long YaRN
RoPE
32.8
19.3
36.1
69.9
43.5
52.6
44.1
25.3
41.2
24.0
43.4
42.2
RoPE++EH
33.9
20.1
35.7
68.8
41.8
55.0
45.2
25.8
40.4
24.6
42.2
42.2
RoPE++EC
32.9
18.8
36.7
69.5
43.6
52.6
44.1
29.3
41.0
30.2
44.0
43.4
776M Long Pf
RoPE
27.8
14.8
35.8
65.4
33.8
53.1
38.8
25.3
39.6
27.2
44.7
40.4
RoPE++01
28.8
15.6
37.0
66.0
33.1
52.6
40.2
25.8
38.1
26.4
44.0
40.4
RoPE++Ec
27.8
14.9
37.0
65.4
33.9
51.7
39.5
23.2
39.4
28.4
45.8
40.5
776M Long YaRN
RoPE
27.3
14.5
36.7
65.6
33.9
51.1
37.7
29.3
39.7
27.6
46.3
40.9
RoPE++01
28.3
15.1
38.0
66.1
33.6
51.6
40.9
26.3
39.3
26.0
43.9
40.6
RoPE+tpc
27.3
14.5
65.8
34.8
53.4
41.8
27.8
40.1
27.2
45.1
41.5
Table 10: Results on short-context tasks for 776M and 376M models further trained with 5B tokens 32k context length with YaRN and Linear PI.
376M
776M
1.5B
TGS-4k
TGS-32k
Storage
TGS-4k
TGS-32k
Storage
TGS-4k
TGS-32k
Storage
RoPE
80248.2
53317.4
0.8
49617.2
29019.6
1.6
32497.2
17040.8
2.7
RoPE+tEH
80248.2
53498.8
50574.4
29399.3
33752.4
17672.6
RoPE++EC
70457.5
37271.7
0.8
44431.2
22631.1
1.6
26479.2
10922.7
2.9
Table 11: The comparison of the throughput (TGS, tokens per GPU per second) at 4k and 32k training context lengths, as well as the model storage (GB).
two components is lower than that among standard heads. Regarding conflict, although each query vector in RoPE++ participates both real and imaginary attention, the superior results of RoPE++gc over vanilla RoPE under identical cache size shown in Table Jnd Table Zindicate that the benefits of this potential conflict outweigh its possible drawbacks.
D.2 PERPLEXITY CURVE OF RoPE++
Length extrapolation is central issue for long-context LLMs. We have already shown that RoPE++ outperforms RoPE on long-context downstream tasks in training-based length extrapolation. However, RoPE++ cannot directly extrapolate like FoPE (Hua et al., 2024) or PaTH (Yang et al., 2025b). Once the inference exceeds the maximum supported context length, perplexity begins rise. Interestingly, as discussed in Section 3.4 every even-index dimension in query vectors and odd-index dimension in key vectors are trained with full value range of position embeddings, and every dimension has seen both positive and negative positions during training. Consequently, the perplexity curve of RoPE++ climbs more gradually (Liu et al. 2024d).
This is verified in Figure 6 where compare the perplexity of short-context-trainee RoPE and RoPE++ on 376M and 776M model sizes. With or without fixed-NTK interpolation based on scaling factor À 4, both curves rise at the same context length with RoPE, indicating identical stable context upper bound. Beyond that point, however, RoPE++'s perplexity increases more slowly, confirming the earlier prediction about its extrapolation behavior.
19

---

Published as a conference paper at ICLR 2026
RePE Roffio Roffec
RIPE Reflow RuPEec
RoFE Rolfiw RudFRc
ReeB Roffle Reffic
110
140
Terphcry
Terplody
Peplosty
Parplccty
2016
3000
1000
2000
150 Edboo Diw 2006 2000 1946
2000
-30
10.0
20r0
Fibto 13300 2000 2000 aod 10.0
Cookox/Loggl Lenglh
Ouscod Lengl
Lorgli
Cookos Logili
(a) Perplexity curve of pre- (b) Perplexity curve of (c) Perplexity curve of pre- (d) Perplexity curve of trained 376M.
NTK- 376M.
trained 776M.
NTK-scaled 776M.
Figure 6: Perplexity comparison between RoPE and RoPE++ in 376M and 776M models.
LIMITATION
As noted above, RoPE++ markedly boosts performance on both short- and long-context tasks, yet it needs training from scratch and fails to deliver plug-and-play length extrapolation, falling behind such extrapolation designs as FoPE and PaTH. Nevertheless, a method that reintroduces imaginary attention, raising performance under fixed memory or improving efficiency while preserving accuracy, RoPE++ be integrated with part of those designs. Additionally, thanks to the oddity of the sine function, the imaginary component also shows promise for pnteectionnllaatenttiin based diffusion language models (Nie et 2025 Ye al. 2025) as well its extrapolation (Liu al. 2025b). and we will provide experiments on these aspects in follow-up.
20


Reviews:
[{"review_id": "35dPXBy5Gm", "reviewer": "Reviewer_aL3q", "summary": "This paper proposes RoPE++, an extension of Rotary Position Embeddings (RoPE) that reintroduces the imaginary component of complex-valued attention scores, which standard RoPE discards. The authors argue that this imaginary component contains valuable phase information for long-range dependencies. They propose two variants and the method is evaluated different sized parameter models using short-context and long-context benchmarks .", "questions": "1. Have you tested or do you have plans to validate RoPE++ at 7b or larger scales? Are there theoretical or practical reasons to expect different behavior?\n2. Can RoPE++ be combined with other long-context techniques? (YaRN, sparse attention, etc.) Have you explored this?\n3. Does RoPE++ converge at similar rates to standard RoPE? Are there any training stability issues? Could you show pretraining loss curves?\n4. What happens if we only apply the imaginary part of the RoPE? What would be the effects of doing this?\n5. For Figure 5's noise experiment, what are the results with noise added to both components simultaneously?\n6. What happens if you use different rotation angles (not -π/2) for the imaginary component?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel perspective: Identifying and addressing the discarded imaginary component in RoPE is creative and theoretically motivated. The observation that imaginary attention captures longer-range dependencies is interesting.\n2. Theoretical justification: The paper provides mathematical grounding (Equations 2-5) showing that imaginary attention follows a sine integral characteristic curve, complementing the cosine integral of real attention.\n3. Generalization: Method generalizes to diffusion/bidirectional attention models.\n3. Dual efficiency benefits: RoPE++_EH achieving comparable performance with half the KV cache is valuable for long-context scenarios.\n4. Comprehensive evaluation: Testing across multiple model sizes, benchmarks, and ablations (attention pattern analysis, noise injection experiments) strengthens the empirical validation.", "weaknesses": "1. Limited scale: Experiments only go to 700m parameters, which is quite small by modern LLM standards. It's unclear if benefits hold at 7b+ scale where most practical long-context work happens.\n2. Modest improvements: Performance gains are often marginal. In Table 2, RoPE++_EC only outperforms RoPE by ~1-2 points on average.\n3. No plug-and-play extrapolation: The authors acknowledge (Section 5.3, Limitation) that RoPE++ doesn't provide direct length extrapolation like other methods, limiting practical applicability.\n4. Incomplete comparisons: No comparison with recent long-context methods (e.g., YaRN, which is cited but not compared against after long-context training) ALiBi and NoPE trained at 1k context vs. 4k for others makes comparisons less fair.\n5. Incomplete ablations: No ablation study of the contribution of the imaginary component. For example, we could use 75% imaginary and 25% real.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XdNwqWooCf", "reviewer": "Reviewer_ADoG", "summary": "The paper proposes to extend RoPE by including the imaginary component when interpreting the computation of attention weights as multiplications on complex vectors. The claims are mainly the following:\n- \"... the negative imaginary part of attention, shows that, compared with the real attention exhibiting stronger semantic locality, the imaginary heads attend more to long-context information as shown in Figure 1, promising gains on long-context tasks\"\n- RoPE++ outperforms RoPE in the ablations with models of sizes 776M and 376M.\n- one of the variant of RoPE++ reduces the KV cache by half but delivers on par performance.", "questions": "1. In the experiment section, you mentioned \"Since ALiBi and NoPE train unstably at 4k, as we have tried, we train\nthem on 1k context length while keeping the batch size the same.\". This made me very skeptical about whether the parameters you fix is reasonable, and your experiment setup is correct. In my experiences, with reasonable effort, all these positional encoding can be trained without problems up to tens of billions of parameters for trillions of tokens. Have you carefully debugged your experiments? Have you tracked some basic training health metrics across layers? Have you considered redoing and comparing with some reproducible pre-training recipes such as OLMo/Pythia/... to ensure no regression due to bugs and mistakes? Have you sweeped hyperparameters for the architectures you chose?\n2. Do you have experiment to support the \"real part having better locality and imaginary part having better dependency capture\", rather than the hand-wavy point-wise argument by only looking at magnitudes? This could become a very interesting data point, regardless whether the experiment supports or rejects the claim.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea is novel. The authors had the good observation that RoPE only captures the real part when interpreted as complex multiplications.\n2. Good theoretic motivations and explanations.\n3. The evaluation uses a good set of benchmarks and looks convincing given their training horizon (50B tokens) and their scale, though I'm skeptical how good it will continue to be if we train for longer for reasonable amount of tokens.", "weaknesses": "1. Experiment setup doesn't seem to have enough scale to convincingly show the gain in model pre-training. 50B token is unfortunately sometimes too small to have confidence about certain pre-training signals, though I understand that there is typically a budget issue in academia.\n2. I'm skeptical about whether the setup is bug-free and uses reasonable hyperparameters, as the training for ALiBi and NoPE seem to have to take compromises.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Kk0R9oP69", "reviewer": "Reviewer_RADi", "summary": "The paper proposes RoPE++, an extension of Rotary Position Embeddings (RoPE) designed to enhance the modeling of long-context dependencies in Large Language Models (LLMs) utilizing imaginary component of RoPE. It analyses how it affects cache and parameters efficiency, impacts length extrapolation.", "questions": "Could you please clarify the changes (if any)  in the number of parameters, activations and FLOPs (estimate) for RoPE++ methods? Am I understanding correctly that Wq parameters are shared between imaginary and real heads? What about Wo size?\n\nConsidering that RoPE++ performs well on long-context task, would be great to have baseline with one of the derivate RoPE methods, which target long context (i.e one of Linear Position Interpolation (PI), NTK Scaling, YARN, LongRoPE, p-RoPE)\n\nIn section 5.1, is analysing inference time only? Could you add some information how RoPE++ affects training?\n\nline 193:  \"long context decay\". Could you please clarify what distribution of queries and keys you are considering?  As shown in https://arxiv.org/abs/2410.06205v1 decay is clear if RoPE is applied to constant queries and keys, but not so much if queries and keys are Gaussian. \n\nnit: line 256.  \"share the same parameter\" same parameters?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- proposes new method for positional encoding which shows promise, especially for long context\n- performs extensive experimentations on different datasets\n- includes several positional encoding methods as baseline", "weaknesses": "- would be great to have baseline for  RoPE method, which target long context\n- theoretical justification of method could be improved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUdEWxE75j", "reviewer": "Reviewer_mcxi", "summary": "This paper identifies that standard Rotary Position Embeddings (RoPE) discard the imaginary component of complex-valued dot products during attention score calculation(this component potentially containing valuable phase information relevant to long-range dependency modeling). To address this limitation, the authors propose **RoPE++**, an extension that re-incorporates the imaginary component to enable a full complex-valued representation of attention scores, forming dual-component (real + imaginary) attention.", "questions": "1. In Section 3.2, the authors argue that the imaginary component of RoPE++ attends to more distant positions, and Figure 1 supports this by showing that the imaginary attention’s magnitude gradually becomes more prominent for Δt > 25 (facilitating long-range dependency modeling). However, the curve exhibits unexpected behavior near Δt = 0 (a sharp transition) and for Δt < 25 (magnitude first decreases then increases). Could the authors elaborate on whether this short-range fluctuation affects RoPE++’s ability to preserve semantic aggregation ? For example, does it disrupt local semantic coherence when modeling short-range token relationships?  \n2. For RoPE++$_{EC}$: A single query vector (q) undergoes two positional encodings (one for real attention, one for imaginary attention) before computing dot products with keys (k). Do the authors observe any information representation conflicts between the two sets of dot products? If so, how does the model resolve such conflicts to avoid degrading attention quality? If not, what mechanisms (e.g., parameter sharing, attention head specialization) ensure complementary rather than redundant information from the two encodings?  \n3. Existing RoPE extension methods (e.g., NTK-PI, YaRN, LongRoPE) modify RoPE’s behavior via dimension-wise adjustments (e.g., scaling rotary bases, interpolating index ranges, partitioning feature dimensions) to enhance length extrapolation. Could the authors clarify whether these dimension-level modifications are compatible with RoPE++’s imaginary component extension? For instance, do such methods introduce conflicts in how positional information is encoded across real/imaginary components, or can they be combined to further improve long-context performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed **RoPE++$_{EH}$** configuration achieves remarkable efficiency: it maintains the same number of attention heads as vanilla RoPE while halving KV cache and QKV parameters, which is highly valuable for long-context LLMs where memory constraints are critical.  \n2. Empirically, RoPE++ demonstrates consistent advantages over vanilla RoPE across both short-context tasks (e.g., WikiText perplexity, Open LLM Leaderboard benchmarks) and long-context tasks (e.g., RULER, BABILong up to 64k context), validating its generality and effectiveness.", "weaknesses": "1. RoPE++ requires full pre-training (or continuous long-context pre-training) to realize its performance gains and cannot be applied as a \"plug-and-play\" module—this limits its flexibility for scenarios where re-training is computationally prohibitive or unavailable.  \n2. The **RoPE++$_{EC}$** configuration, while delivering superior long-context performance, doubles the number of attention heads (relative to vanilla RoPE) under fixed KV cache size, introducing additional computational overhead that may offset its performance benefits in resource-constrained settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)