Selective Rotary Position Embedding
Authors: Sajad Movahedi
Timur Carstensen
Arshia Afzal
Frank Hutter
Antonio Orvieto
Volkan Cevher
Venue: ICLR 2026 Poster (2026)
Keywords: RoPE
Linear Transformer
Attention
State Space Models
Forget Gate
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce Selective RoPE, an input-dependent rotary embedding that enhances gated linear transformers.

Abstract: Positional information is essential for language modeling. Softmax Transformers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely on input-dependent gates that only decay past key-value norms. We provide a theoretical argument for the necessity of a rotation and decay component in well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax attention performs implicitly. We introduce Selective Rotary Position Embedding (*Selective RoPE*), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with *Selective RoPE* yields a complex-valued recurrent layer that can be implemented efficiently with the “RoPE trick”. On synthetic benchmarks (MQAR, copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source our implementation [here](https://github.com/timurcarstensen/selective-rope).

Published as a conference paper at ICLR 2026
SELECTIVE ROTARY POSITION EMBEDDING
Sajad Movahedi*1.44 Timur Carstensen +1.3, Arshia Afzal*?
Frank Hutter Antonio Orvieto f1.4, Volkan Cevher Equal contribution", Equal supervision!
ELLIS Institute Tübingen', LIONS, 2 University of Freiburg Max-Planck- Institute for Intelligent Systems*, Prior Labs sa sajad movahedi @tuueelllee. ellis.eu cimurcarstensent @mma com arshia. @epfl.ch
ABSTRACT
Positional information is essential for language modeling. Softmax Transform- ers with Rotary Position Embeddings (RoPE) encode it with fixed-angle rotations, while linear Transformers rely input-dependent gates that only decay past key- value norms. We provide a theoretical argument for the necessity of a rotation and decay component well-performing sequence models, and observe that the missing ingredient in linear models is precisely the rotation that softmax atten- tion performs implicitly. We introduce Selective Rotary Position Embedding (Se- lective RoPE), an input-dependent, learnable rotary embedding that generalizes RoPE to arbitrary angles and composes seamlessly with decay gates. Equipping gated linear attention with Selective RoPE yields complex-valued recurrent layer that can be implemented efficiently with the RoPE trick. On synthetic bench- marks (MQAR. copying, state tracking) and 370M-parameter language-model pre-training, the method improves recall, downstream accuracy, and expressivity while adding minimal architectural overhead. We open-source implementa- tion here.
INTRODUCTION
Transformers with softmax attention (Vaswani et al., 2017) form the foundation of state-of-the-art language models, largely because every token can attend to all past tokens without decay, yield- ing strong in-context recall. This expressivity comes a computational cost: even with memory- efficient kernels, the arithmetic complexity of softmax attention is quadratic in the sequence length. To mitigate this, a parallel line of work has developed sub-quadratic sequence models. These mod- els recurrent architectures that in linear time and require only constant memory per step at inference (Katharopoulos et al., 2020; Yang et al., 2024b; Gu & Dao, 2023; Dao & Gu, 2024). Their main bottleneck the fixed state size: information must be selectively retained or overwritten, which often harms long-horizon retrieval. Consequently, recent progress has focused on improving state management in these models.
Theoretical investigations have proven the benefits of complex gating mechanisms for this pur- pose (Orvieto et al., 2023b; Ran-Milo al., 2024; François et al., 2025). However, in practice selective gating (Yang et al., 2024a; Gu & Dao, 2023; Dao & Gu, 2024), more expressive state up- dates (Yang et al., 2024b; Siems et al., 2025; Peng al., 2025) and readouts (Peng al., 2025; Hu et al., 2025) usually play the main role in memory management. As these mechanisms largely act by modulating the norms key-value associations (i.e., how quickly they decay) and do not directly provide the complementary capability of rotating query-key representations to encode relative posi- tion, they fail to materialize the full benefits of the gating mechanism as done in carlier variants of recurrent sequence models (Gu al., 2022c; Orvieto al., 2023a; Sun et al., 2023b).
Our view: recall needs rotation and decay. We posit that strong recall requires two complemen- tary mechanisms: (i) rotation, to encode relative position while preserving norms, and (ii) decay, to selectively discard past keyvalue associations. By relying on the relationship between the softmax function and Random Fourier Features (RFF), reveal that softmax attention can be interpreted as performing input- dependennnddnt selective rotations query-key pairs, thus motivating the use of like-like rotation matrices in recurrent models.

---

Published as a conference paper at ICLR 2026
10
R
R2
q
qt
q1
qt
Aus
A
kir
kr
A
Ave
krr
R
ki
(a) GLA/Maahh22 gating
(b) RoPE
(c) Selective RoPE
Figure I: Scaling vs. rotation on the unit circle. Left: GLA/Mamba-style forgetting encodes history primarily through
scaling: cumulative gates A1 and A attenuate a and k; to de and k, without changing direction. Middle: RoPE encodes position via fixed index-dependent rotations (RC R that preserve norms. Right: Selective RoPE + decay composes inpul-dependent rolations with norm scaling, capturing both phase (relative position) and forgetting.
Why rotation alone is insufficient. purely complex (rotation-only) linear recurrent model be- haves like spectral analyzer with fixed state size. When applied to the input sequence, which can be seen as a finite-length signal, such a model suffers from spectral leakage, leading to a poorer approximation of the underlying signal. This can be mitigated by adding exponentially decaying component to the recurrence. In linear recurrent architectures, the analog of spectral leakage the suboptimal compression of keyvalue associations into the fixed-size hidden state, which remedied by adding selective gating to the state transition.
Based on our recipe, we introduce a complex version of Gated Linear Attention (GLA) (Yang al., 2024a) and demonstrate its superior performance and expressivity, In practice, we show that by using the RoPE trick (Su et al., 2021), we are able efficiently compute complex GLA by applying a learned, input-dependent rotary position embedding to the queries and keys. Selective RoPE is easily incorporated into most gated linear Transformers by applying input-dependent and learned rotations to queries and keys. In Figure 1, we provide visual comparison.
Contributions.
Unifying view. We show that effective recall needs both rotation and decay. Softmax implic- itly implements deputddependent rotations. Complex-oni linear models suffer from spectral leakage, motivating explicit decay. Real parts forget; imaginary parts encode position.
Theory. (1) Using RFF approximation of the exponential kemel,, we expose selective rotations softmax and derive an optimal temperature distribution that matches exponentially decaying frequencies used in RoPE. (2) We investigate diagonal SSMs spectral analysers, showing that they suffer from spectral leakage which can be suppressed through forget gates.
Method:
We propose Selective RoPE. An deput-dependent rotary embedding that generalizes RoPE to learned angles and composes with gates; we make the implementation efficient by incorporating the RoPE trick in both linear and softmax attention.
Empirics. We integrate Selective RoPE with GLA, significantly boosting performance on recall-centric synthetic tasks (MQAR, copying, state tracking) and improving downstream language modeling.
BACKGROUND
In this section, we provide some mathematical background, beginning with introduction of the Transformer architecture, its relevant variants and the RoPE trick (Su et al., 2021). The results of this section also establish relationship between linear Transformer architecture with RoPE and recurrent sequence model with rotations in the state transition. This relationship later aid us through more efficient computation of the latter.
2

---

Published as a conference paper at ICLR 2026
Transformers. Standard causal softmax attention (Vaswani al., 2017) transforms sequence of inputs (2r)/=1 into the sequence of outputs (ot) with Tt, 8t, O2 E R and 24 E R:
St 01
Σ exp
9: krr) .r: 21 vi
exp
va kr),
(1)
21
where qt. kt, 01 = W WLT W and Wg. We. We Raxd are the projection matrices and 2, is the normalization factor. Linear attention (Katharopoulos et al., 2020) replaces the exponential kernel in softmax attention with a kernel with positive feature map (b-) IR." - +R+) which gives rise to the following model:
S.d(g)
S. S v. 2, ():)
21 Σ o okk,)
(2)
Here St E Rdxd and 2 E F are the state and the normalization factor. Due to the linear relationship, one can write the hidden state and the normalization factor in recurrent form S: S vko(k,) and 2; = 2t-1 + Φ(ka). Moving forward, we subsume the feature map o(-) into query-key vectors to simplify notation and drop the normalization factor z, following Sun al. (2023a). Equation (2) was enhanced with a forget gate, A,, to manage the finite-sized hidden state better when processing long sequences:
S. S v.k, or = =. = Σ Ur (
II
A
qi
(3)
Alle.t
which is either diagonal (Yang et al., 2024a; Gu & Dao, 2023) or scalar-valued (Dao & Gu, 2024). In both, the channels of the hidden state evolve independently. In Equation (3), Att,,r denotes the attention score between qi and k,, while the factor IT== 11 A, attenuates this inner product according to the cumulative gate between positions and t. This term can therefore be interpreted as a position encoding (Yang et al., 2025b), since it depends on the relative distance between t and r. For Equation (3) to be stable, the gates must be contractive, which is typically enforced by bounding the spectral norm of A, (Gu & Dao, 2023). More recently, the simple forget gate has been generalized more expressive state transition matrices that also mix channels across time. These are often parameterized in diagonal-plus-tow-rrnk (DPLR) form (Yang et al., 2025a; Peng et al., 2025), which admits memory-efficient representation of products of such matrices.
RoPE and Complex Linear Attention. Rotary Position Embeddings (RoPE) are used to add relative positional information through rotations of the query-key pairs (Su et al., 2021). For queries and keys qı, k E 2 RoPE applies relative positional encoding using the rotation matrix R.:
[cos -sin sinw
A = exp (k. R q:) = (R (R',q:)) R_ =
(4)
cosw
with being the frequency of rotation. The query at time t and key at time т are rotated via (R R... For d-dimensional queries and keys, qr, k, are split into d/2 vectors R2, each rotated independently with their own frequency. This yields a block-diagonal rotation matrix R E ddd where each 2 real matrix R., parameterized by a frequency wk-
The RoPE trick allows expressing a complex parametrization of linear Transformer in the real domain. Consider the real part of the following complex attention score:
Atter = R(K diag ([elwi(t-r)
(l (1-7)) g.h. with q.,k,, k, Cd/2,
(5)
Re 24/2x4/2
where is a unitary diagonal state transition matrix. This expression can be understood as applying RoPE to queries and keys k in Rd, where we interleave the real and imaginary part in the odd and even indices of queries and keys:
d/2
Alle,r
Σ
k 2n-1
cos (t 7) sin (1 7) cos wn -
sin W,,, (t -7)]
7)
44,2n-1
(6)
k
2t.2m
==
R
3

---

Published as a conference paper at ICLR 2026
When unroll the recurrence in Equation (3) and replace the forget gate, An, with the block- diagonal rotation matrix R E Raxd in RoPE, we get:
0, Σ U+ Uo R' qi)
with R" blockdiag ([R R
Note that due to the block-diagonal structure of R. we can write R = (IRT) R², from which follows that & R. (R )* R' q: This allows to express the rotation matrix as applying RoPE to queries and keys, similar to Equation (6).
In summary, linear Transformer with RoPE is equivalent the same model with a unitary, diag- onal and non-selective transition in half the dimensions. The RoPE trick allows to implement this complex parameterization by applying RoPE to queries and keys, effectively staying in the real domain which, allows us to re-use existing (linear) attention kernels. A full derivation is shown in Appendix A.
A UNIFYING VIEW: DECAY AND ROTATION
In this section, we motivate our method, Selective RoPE, by first observing that pure softmax at- tention implicitly performs selective rotations when viewed through the lens of Random Fourier Features (RFFs) (Section 3.1). These rotations are missing in linear attention. In Section 3.2, explain why rotations do not suffice and why selective gating is necessary, building on the com- plementary roles that real (gating) and imaginary (rotation) parts play in diagonal SSMs. Finally, in Section 3.3 we combine the previous insights and present our proposed method.
3.1 SOFTMAX ATTENTION IMPLICITLY PERFORMS ROTATIONS
We begin with the connection between RFFs and softmax attention, and illustrate that rotation is an integral component in softmax attention. Specifically, we start from the definition of the softmax attention in Equation (1) (omitting temperature for simplicity). Following Peng et al. (2021) and Rahimi & Recht (2007, Theorem 1), we define the RFF kernel as (x) exp az / + iw x) with W',, I). When applying the kernel to the dot-product of queries and keys (qt, kr), whose expected real component equivalent to the attention score Att,,7, we obtain:
R (Ew [d (q:) &w (k+)]} exp(q. k.).
(8)
By the law of large numbers, with iid. wy // (2) for j {1.-.., D} and a 1 we can approximate the output of softmax attention before normalizing the attention scores as:
{ (17
R E
Σ Sw (a. Vr
lim R
St.
D-+00
j=1
where $1_ = Pw, (q:) H dwe, (kr) v, is the j-th contribution the attention score Att With some algebraic manipulations and mild assumptions (full derivation in Appendix A.2) and using the definition of ow;, we can re-express 8j as recurrence. Stacking D of these recurrences horizontally gives us matrix-valued recurrence over S, e Rd
S, S.=S. + vikh R = diag diag(exp (ini - 9))) = O o(k,),
(9) Crucially, R2 is diagonal input-dependent rotation matrix parametrized by random Gaussian fea- tures Ω, conditioned on the input via q: -:-1. Furthermore, due to the corresponding telescoping product, the induced rotation is path independent. Consequently, the rotation introduced by the soft- max attention itself lacks an explicit positional inductive bias encoding. Therefore, the relevance of this expression is structural, allowing us to introduce an efficient way to compute a general form of R,, the significance of which will become clear.
Recalling the RoPE trick in Section 2, it should become clear that we can re-express R. as a block-diagonal matrix where each 2 x 2 rotation matrix on its diagonal rotates by the angle 6, = (w,, (q. 9)-))). This stable recurrence, since R, is norm preserving, which also means that it does not forget past key-value associations. Interestingly, the shift over the queries can be expressed by 1d short-convolution, which is a component that is already frequently used in recurrent architectures (Yang et al., 2025a; Dao & Gu, 2024). We can follow similar derivation
4

---

Published as a conference paper at ICLR 2026
Ioput signal s(t)
Windowed signal
Magnitude Spectrom
Window FFT Comparison
1.0
1.0
10"
1.00
Rectangçalo:
0.00
0.5
109
0.76
Hann
10-2
0.00
0.50
-0.5
-1.0
10-9
Leaked abectrum
Windowsed apectrum
0.00
0:25
0:0
0..5
1.0
Time
Time
Proquency [Ha]]
Prequency [HR]
(4)
(b)
(c)
(d)
Figure 3: The effects of windowing on the spectrogram of a finite sample of a sequence.
as in Equation (9) for the normalizer 2t. The read-out proceeds slightly differently than in normal linear attention: since each column of the recurrent state represents the contribution of the j-th random feature to the approximation of 8t, we sum over the columns: $,1.
The equivalence of the RFF kernel in Equation (8) only holds for an unlimited number of samples, i.e., D + co. However, as shown in Theorem (Appendix A.3), in the presence of limited samples, one can have an optimal variance for the RFFs for a single query- key pair, proportional to the angle between the two vectors. As- suming a uniform distribution for the angle hetween the query-key pairs, we obtain spectrum of optimal variances.
0.1
0.01
0.001
0.0001
1.0
RoPE
Ours
0.5
0.0
Extending this perspective, we define the rotation matrix as R. exp(i (t-1)), where θ is a diagonal matrix of temper- atures. Assuming the angle between the queries and keys are uniformly distributed in [0, 2T], the optimal temperatures follow (an () with 0 ~~ [[00, 2T] (ref. Theorem 1). Interestingly, this dis- tribution closely resembles the exponentially decaying frequencies used in RoPE, with a slightly faster decline, as shown in Figure 2.
Channel Index
Figure 2: Distribution of phase temperatures in RoPE vs. Se- lective RoPE. is the inverse of the RoPE base frequency and the soper-bound of query-key&de an- gle in our temperature (details in App. A.3.1).
In summary, we have shown that softmax attention implicitly performs random input-dependent rotations to encode relative positional information between tokens. A comparison to other kernelized linear-attention variants (e.g., Performer, RoFormer, and CosFormer) is provided in Appendix A.6.
3.2 NECESSITY OF GATING: SPECTRAL LEAKAGE IN DIAGONAL. SSMS
By analyzing the role of real and imaginary parts in complex diagonal SSMs, in this section we show that rotations alone are not enough to close the gap between linear and softmax attention. Inspired by the findings of Section 3.1, we analyze a model that is related to GLA, where the diagonal gate A, is instead replaced by the rotation matrix R2 introduced in Equation (9):
S. = S R. + v,k
= R(S.q.).
(10)
By unrolling the recurrence, we can write the output as:
R E e e E hin in n. ue (T))dr}
This equation contains a convolution of the value and key (i.e., the input) with an exponential func- tion with pure imaginary exponents (i.c., e s)..) This is spectral analysis (discrete Fourier trans- form, DFT) of the value signal in the presence of the step-window function (T) (definition in Ap- pendix A.4), which is visualized in Figure 3a. When naively performing DFT over finite sample, the resulting discontinuities at the margins of the sample cause spectral leakage the spectrogram as shown in Figure 3c. To avoid spectral leakage, one usually places non-rectangular window which tapers off towards the margins. The convolution of the signal with a Hann window (Op- penheim, 1999) function is shown in Figure 3b and the resulting magnitude spectrum in Figure 3c. In Figure 3d, we show that we are able to recover the correct frequency after a window FFT when applying Hann window to our input signal. The window function chosen here acts like an expo- nential decay towards the margins, which is analogous to using gate in our model in Equation (10). The use of gates in sequence models has a long history. Starting from the gating mechanism in LSTMs (Hochreiter & Schmidhuber, 1997), it is also widely used in linear attention, linear RNNs and SSMs (Yang et al., 2024a; Gu & Dao, 2023), and even softmax Transformers (Lin et al., 2025). Our results in this section provide theoretical motivation for the use of gating mechanisms.
5

---

Published as a conference paper at ICLR 2026
3.3 DESIGN PRINCIPLES FOR LINEAR ATTENTION
In this section we combine the insights gained in Sections 3.1 and 3.2 to formulate general design principles that are required to narrow the gap between linear and softmax attention. For this, we analyze a general form of linear attention, which encompasses both models in Equation (3) and Equation (10):
S viki o, R Or - Σ o R
A, Ax)a
(11)
In Section 3.1 we showed that softmax attention implicitly performs input-dependent rotations, a capability that is absent in standard linear attention. We can inject rotations into the general form in eq. (11) by setting A, = R,. Since R₂ is a rotation matrix, this choice is stable and recovers the model in Equation (10). However, a purely rotational (complex) linear recurrent model behaves like a spectral analyzer: positional information is encoded through phase, but without any decay it cannot effectively control the contribution of distant tokens and suffers from spectral leakage. To remedy this, we also need decaying (window) component, for which we choose an exponentially decaying gate. Setting A, A, yields the gated model in Equation (3). In summary, a well well-performing linear Transformer requires both (a) rotation and (b) gating.
Both components can be combined by factorizing the transition as A, R, Interestingly, in DeltaNet the rotational component is already present in the form of an input-dependent Householder transform, which performs rotations along the key dimension. Adding an explicit forget gate on top of this, as done by Yang et al. (2025a), improves performance in line with our design principle. For softmax Transformers, the rotational component is already provided implicitly along random axes (ef. Section 3.1); thus, only forget gate is needed to fully satisfy the rotation decay recipe, as empirically validated by the Forgetting Transformer (Lin et al., 2025).
In summary, as the main contribution of the paper, introduce Selective RoPE, which define as Linear Attention with an input-dependent rotation matrix R2 as its state transition:
S. R, + v,kk O2 S.q.
(12)
Recalling the RoPE trick in Equation (7) and defining Raj = II Rs for the input-dependent rotation matrix R., we can equivalently write this as:
Selective RoPE: O2
Σ ur k R r=1
Σ o RT R1
(13)
Here, RS should be understood as an input- input-dependent state transition. Although is parameterized similar to Equation (9), this does mean that a query is being written into memory; rather, it should be seen as a selective modulation of the state transition. This can easily be applied to both queries and keys, allowing us to reuse many parts of existing RoPE kernels. Considering the extensive research done on the forget gate, we rely on the built-in forgetting functionality of the baseline architectures.
In summary, in this section we provide theoretical results that motivate the use of complex rota- tion and exponential decay in a linear attention model. The resulting design principle argues that both these components are required for well-performing sequence model. This design principle also provides fresh perspective on the success variants of softmax Transformers (Press et al., 2021; Lin et al., 2025) and DeltaNet (Yang et al., 2024b; 2025a), which we further elaborate on in Appendix A.7 and Appendix A.5.
EXPERIMENTS
In the following section test proposed model synthetic and real-world language modeling tasks. For this we first provide our implementation details and then explain the specific experimental setup for each task and discuss the accompanying results. We primarily apply Selective RoPE to Gated Linear Attention (GLA) (Yang et al., 2024a) and compare with other linear and softmax attention variants. We sweep learning rates (reported in Appendix B) unless otherwise specified.
4.1 IMPLEMENTATION

---

Published as a conference paper at ICLR 2026
GLA:: String copying
1.0
Table I: MAD benchmark results. We ablate the effectiveness of each extra component introduced Selective RoPE on GLA. The best results are marked in bold and the second best in underline.
Accuracy
0.5
Model
Compress Fuzzy In-Context Memorize Recall
Noisy Recall
Selective Copy
Average
Recall
0.0
GLA NoPE
50
100
120
82.0
8.5
87.3
38.7
87.6
91.1
65.9
Sequencee Length
RoPE
85.2
7.5
92.6
61.4
91.9
96.4
72.5
Sedection
RRPP. NoPR
Selecrive RoPE
85.2
9.0
94.0
57.1
91.7
94,9
72.0
phase gate
85.1
7.5
96.6
56.9
94.3
93.5
72.3
Figure 5: Copying accuracy of GLA with Cls. Dashed line is the training sequence length.
85.0
95.0
61.3
95.4
72.7
+ phase gate & bias
85.4
7.2
95.9
60.4
95.0
95.6
73.2
s_rope (d, k, N_omega, teemp]
In the implementation of Selective RoPE make several design choices that go beyond the architec- ture described in Section 3.3: Following Zhang et al.
onega 1 RLomegally
emega convid (emega)
onega Lemp. comp-cummunn tomega)
sin_o, cos_o sinces (omega)
(2024), where learning the random features intro- duced by Choromanski et al. (2021a) was shown to be
return rope (q, k, cos_o, $in_o)
Figure 4: Pseudocode of Selective RoPE.
more effective, we make the parameters w in Selec- tive RoPE learnable. This makes the rotations input-
dependent and learnable. Following Yang et al. (2025b), we place sigmoid gate on the rotation angles to allow the model to control whether to rotate or not. We also add a learnable bias term, which is not dependent on relative token positions (Li et al., 2024). Finally, we place weight norm (Kingma, 2016) on the input projection. We ablate our architectural choices on the MAD dataset and language modeling experiments.
We implement Selective RoPE in PyTorch and integrate (Yang & Zhang, 2024) for our experiments. Using the RoPE
into
flash-1i iiinear-attntt trick, we are able to implement our method as a prelude to RoPE where we determine the sin and cos from the input shown in Figure 4. To optimize the throughput of our implementation, follow the GPT-NeoX (Black et al., 2022) style of applying rotations to allow for coalesced memory access. This is equivalent
derivations up index permutation.
4.2 SYNTHETIC LANGUAGE TASKS
To investigate which capabilities of linear attention improved when using Selective RoPE, run experiments on synthetic tasks. For this, we mostly focus on recall, since it is essential for language modeling (Arora al., 2024a;b) and good proxy for performance scale.
We evaluate GLA Selective RoPE on Multi- Associative Recall, following the same experimental setup as in Arora al. (2024a, Figure with a finer learning rate grid, as this has been shown to improve performance (Okpekpe &
MQAR.
Soq. 512, KV pairs 64
GLA Selectiv RoPF
Accuracy
Raiti
NuPS
Orvieto, 2025) (cf. Appendix B.2). The results Figure show that GLA improves with extra positional information and that Selective RoPE achieves the greatest improvement over the base model with positional embedding.
RWKK
Model diunension
125
256
R12
Transfurnce
Figure 6: MQAR results.
MAD and Copying. We also evaluate our method on the MAD benchmark suite (Poli et al., 2024) which tests model's ability store and recall information within its context. Across MAD, Selective RoPE improves over NoPE and matches or improves upon RoPE on several recall-oriented tasks; the phase-gated variant achieves the strongest overall average. This task differs from Selective Copy in MAD in that the entire input sequence has to be copied token-by-token after the model is presented with a <copy> token. The results in Figure 5 show that Selective RoPE again improves over the alternatives and learns to length extrapolate very robustly. The poor result of RoPE is reported in prior works (Jelassi et al., 2024; Li al., 2024) and attributable to its generally poor length extrapolation performance without fine-tuning on longer sequence lengths.
State Tracking. A common way to evaluate the expressivity of a model is state tracking on per- mutation composition (Liu al., 2023). Recently, it has been shown that SSMs and linear RNNs are not capable of learning parity (Merrill et al., 2024), which amounts to permutation composition on the symmetric group of elements, S2, and that needs extend the eigenvalue range of

---

Published as a conference paper at ICLR 2026
Group S2
Group As DeltaNet
GLA
DeltaNet
Transformer
1.0
Accuracy
0.5
0.0
128
512
128
512
128
512 122
512
Sequence Length
Selective RoPE
RoPE
NoPE
Figure 7: State tracking peformance of GLA, Transformer, and DeltaNet with different positional embeddings on S2 and Ag. The models on S2 were trained with one layer whereas DeltaNet was trained with two layers on Ag. Vertical dashed line indicates training sequence length.
the state transition A, from [0, 1] to [-1.1] (Grazzi et al., 2025). In Figure we see that GLA with Selective RoPE able to learn and length-extrapolate on S2. This is line with our expectations since the input dependent rotations allow it to model "flips" depending O or 1, while GLA with NoPE and RoPE does not even learn the training context length. Simi- larly,
the input either being
that Selective RoPE also improves the state tracking abilities in Transformers (i.e., softmax attention) allowing them to solve the parity problem up to, and slightly more, than the train sequence length. To the best of our knowledge, Transformer with Selective RoPE is the only variant of Transformers capable of solving the parity task with a single layer up to this sequence length (Liu al., 2023). We also experiment permutation composition on the symmetric group of three elements, limited to even permutations. As we can observe, Selective RoPE improves the expressivity of the model up to a point where is capable of solving A3 up to the training sequence length. To the best of our knowledge, this is the first time these results have been presented for our choice model on this task.
A3 with 2-layer DeltaNet (Yang et al., 2024b), which is the
4.3 LANGUAGE MODELING
For our language modeling experiments train 370M and 1.3B parameter versions of GLA (Yang et al., 2024a) and the Forgetting Transformer (FoX) (Lin et al., 2025) using AdamW (Loshchilov & Hutter, 2019) and warmup and cosine-decay schedule (Loshchilov & Hutter, 2017). All models are trained on 1OB and 26B tokens respectively. We use Fine (Penedo et al., 2024) at context length of 4096 and use the Mistral 7B tokenizer (Jiang et al., 2023) with vocabulary size of 32 000. All remaining architectural and optimizer hyperparameters (batch size, learning rate schedule, gra- dient clipping. weight decay) follow Siems al. (2025) and are detailed in Appendix B. To account for differences in optimal learning rates for the considered positional embedding schemes, we sweep learning rates following Orvieto Gower (2025). To select the best learning rate for each model and position embedding combination, we the perplexity The best models are then evaluated on downstream tasks from tit-eval-harrne 2024), the results of which are shown in Table 3. We follow the default zero-shot evaluation setup in 1m-eva -ha rness, using its standard prompting and report the macro-average accuracy over the core multiple-choice tasks in the Avg. column. We select the same set of tasks as in GLA (Yang et al., 2024a).
million tokens not seen during training.
(Gao et al.,
Initial experiments have shown that Selective RoPE dis- plays training instabilities at higher learning rates, earlier than models trained with RoPE or NoPE. We find that this can be remedied by using the f-normalized input, ar, instead of the queries, Qt. to parametrize the rotation angles w. Further stability improvements were achieved by either placing weight norm (Kingma, 2016) on the projection using low-rank MLP with rank 8 or 16. We also test adding a learnable bias term (Li et al., 2024) but find that this does not significantly improve perfor- mance. We find that the most significant performance im- provements come from adding SiLU activation after the
Table 2: Selective RoPE ablations. GLA 370M trained on 10B tokens of Fine Web.
Model
Avg-PPL
Avg-Ace 1
NoPE RoPE
20.78
46.8
21.42
46.5
Selective RoPE
Baseline
21.27
45.8
SiLU
21.14
46.7
phase gate
21.16
47.1
8

---

Published as a conference paper at ICLR 2026
Model
Wiki.
LMB,
LMB.
PIQA
Hella.
Wino, acc
ARC-e
ARC-c
Avg.
FDA
SQuAD
SWDE
ppl.
ppl
acc 1
acc
acc_n
acc 1 acc_n
acc
acc
acc
FoX 370M params tokens
NoPE
25,27
17.15
42.4
68.9
42.3
51.0
46.8
25.5
46.2
69.6
19.2
65,2
RoPE
24.98
17.31
41.2
68.9
42,6
51.6
49.2
23.8
46.2
49.9
29.9
63.5
Selective RoPE
27.54
16.92
42.7
68.2
42.5
51.1
47.5
23.8
46.0
57.8
29.9
58.9
GLA 370M paroms/ / 10B tokens
NoPE
25.72
15.83
43.1
67.8
43.1
52.2
49.4
24.9
46.8
9.2
30.0
33.0
RoPE
26.51
16.32
42.5
43.3
50.7
48.2
46.5
30,9
Selective RoPE
25.98
16.33
42.2
43.8
51.3
49.9
26.6
47.1
8.7
29.6
35.9
GLA 1.38 params / 26B tokens
NoPE
18.18
8.59
53.8
56.5
58.2
59.0
30.0
55.2
34.8
46.3
RoPE
18.50
8.88
53.0
72.4
56.6
57.1
58.5
28.5
54.4
13.3
32.4
39.6
Selective RoPE
17.87
8.50
53.8
23.1
56.9
56.0
59.3
28.8
54.6
23.2
34.8
54.3
Table 3: Evaluation results tasks from 1m--val-harne
(Gao et al., 2024) for GLA (370M
& 1.3B) and FoX (370M) trained on 10B and 26B tokens of Fene (Penedo et al., 2024). The best results for each model architecture are marked in bold and the second best in underline.
convolution that follows the w projection and from adding a gate on the rotation (phase) Yang et al. (2025b) as described in Table
The results in Table use the best configuration of Selective RoPE for both FoX and GLA at 370M and 370 and 1.3B params with 10B and 26B tokens, respectively. We find that Selective RoPE changes the trade-off between perplexity and downstream accuracy in model-dependent way: it often improves perplexity and/or specific downstream tasks, while the macro-average accuracy may stay similar decrease slightly depending architecture and scale.
RELATED WORK
There have been several attempts at reducing the quadratic complexity of softmax attention (Dao, 2024), one of which is linearization (Katharopoulos et al., 2020), which results in recurrent model with sub-quadratic cost (Martin & Cundy, 2018; Gu et al., 2020). However, the reduced com- plexity comes at the cost of lower performance, especially in recall-intensive tasks (Waleffe et al., 2024; Peng et al., 2021; Choromanski et al., 2021a; Zhang et al., 2024). This led to the develop- ment of architectures which used gating to increase their expressivity. Non-selective state-space models (SSMs) made use of apput-independent gating mechanisms and vector-valued states to per- form sequence modeling (Orvicto et al., 2023a; Gu et al., 2022b;a; Sun et al., 2023a). Later, these architectures were improved by adding selective gating (De et al., 2024; Qin et al., 2023) and matrix- valued states (Gu & Dao, 2023; Dao & Gu, 2024; Yang et al., 2024a; Beck al., 2024; Qin al., 2024). Concurrently, DeltaNet (Schlag et al., 2021; Yang et al., 2024b) extended the notion of gate state transition matrix by using an input-dependent generalized Householder matrix, which implements the coror-corecttin delta-rule (Widrow et al., 1988). A byproduct of our theoretical analysis are further insights into the functionality of the gating mechanism and forget gate in Sec- tion 3. Another line of work has improved sub-quadratic sequence models through better kernel approximations of sofumax attention (Katharopoulos et al., 2020). This approach led the use of random features (Choromanski et al., 2021a; 2022), which was extended to learning the features directly (Zhang et al., 2024). Interestingly, a polynomial kernel inspired by the Taylor expansion of the exponential function has proved effective in closing the performance gap, while being less efficient in terms of computational complexity (Zhang et al., 2024; Kacham et al., 2023). We base theoretical investigation the work of Peng et al. (2021), deriving a linear attention variant as an approximation of the softmax Transformer.
RoPE and complex parameterizations of RNNs. The primary method of encoding positional information in sub-quadratic attention variants is exponential decay (Lin et al., 2025). How- ever, in softmax Transformers, rotary position embeddings (RoPE) have proven to be very effec- tive (Su et al., 2021; Shaw et al., 2018; Yang et al., 2025b) compared to positional embeddings (NoPE) (Kazemnejad et al., 2023). RoPE encodes positional information through point-wiee wise rota- tion of the query-key pairs. Other variants of RoPE have made attempts at improving RoPE in terms

---

Published as a conference paper at ICLR 2026
of its shortcomings in generalizing longer sequences by learning the position embedding (Li et al., 2024), framing it as a kernel design problem (Chi et al., 2022), or utilizing theoretical tools (Peng et al., 2024). Interestingly, our model generalizes RoPE by making angles input-dependent. In our experiments, we show the effectiveness of our proposed position embedding both in linear attention models and softmax Transformers. As shown in Section 2, applying RoPE linear Transformer is equivalent to operating in the complex domain and theoretically, this is essential for the universality guarantees of RNNs and SSMs (Orvieto al., 2024; Gu et al., 2020). Further investigation showed an improvement in the recall capabilities and expressivity of SSMs when operating in the complex domain (Ran-Milo et al.. 2024). However, later variants of these models removed the complex re- currence due to inconclusive evidence for their benefits in language modeling and implementation overhead (Gu & Dao, 2023; Dao Gu, 2024; De et al., 2024). In this paper, we focus on the kernel view of softmax attention, providing connection between it and linear attention models operat- ing in the complex domain. The resulting design principle provides a connection between softmax attention, complex linear attention, the gating mechanism, and position embeddings.
CONCLUSION
We introduced Selective RoPE, input-dependent rotary position embedding that generalizes RoPE from fixed to arbitrary, learnable rotations. Our theory shows that (i) softmax attention admits com- plex linear formulation that implicitly performs selective rotations, and (ii) this complex formulation introduces spectral leakage, which can be suppressed through the forget gate mechanism. Empiri- cally, equipping certain sequence models (namely, GLA, Gated DeltaNet, and softmax Transform- ers) with Selective RoPE improves recall-centric synthetic tasks and strengthens language modeling downstream performance for linear attention variants. Furthermore, we show that this improvement in performance comes at very little computational cost, with an easy implementation thanks to the RoPE trick.
Future work. There several aspects of Selective RoPE and the proposed design principle in- troduced in our paper that require further investigation. Firstly, we note that incorporating RoPE is notoriously detrimental to the santth-xxtrapolation capabilities of sequence models (Li et al., 2024). In this paper, we do not investigate this aspect since we consider to be out of the scope of our research. Secondly, in Selective RoPE, namely the bias term and the phase gate, can be fruitful direction for future research. Thirdly, we consider the impact of choosing diagonal as opposed to be an interesting question, since our theoretical justification for forget gates is only concerned with an exponentially decaying component in the sequence model, and not the dimensionality of it. Finally, given the existing variants of RoPE (Black et al., 2022; Su et al., 2021), 1), we believe it to be important to also incorporate the progress on the positional embedding front into future work.
believe that further investigation of the effect of the extra components used
scalar forget gate
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We would like to thank Philipp Nazari, Julie Naegelen, Felix Sarnthein, and Gwendolyn Neitzel for constructive discussions and feedback. We would also like to thank Songlin Yang for pointing out an error in a previous version of our manuscript which overstated the expressivity of Selective RoPE. This research was partially supported by the following sources: PNRR MUR Project PE000013 CUP JS3C22003010006 Future Artificial Intelligence Research (FAIR), funded by the European Union NextGenerationEU. and EU Project ELSA under grant agreement No. 101070617. TAI- LOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215; the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828; the European Research Council (ERC) Consolidator Grant 'Deep Learning 2.0' (grant no. 10). This research was partially funded by the Deutsche Forschungsgemeinschaff (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH 2698644) and the state of Baden- Württemberg. Frank Hutter, Antonio Orvieto, Sajad Movahedi, and Timur Carstensen acknowledge financial support by the Hector Foundation. The authors acknowledge sup- port from ELLIS and ELIZA, funded by the European Union. The authors gratefully acknowledge the computing time made available them the high-performance computers and the NHR Centers at TU Dresden and KIT. These centers are jointly supported by the Federal Ministry of Research, Technology and Space of Germany and the state govermments participating in the NHR. Arshia Afzal acknowledges support under project ID) #37 as part of the Swiss Al Initiative, through a grant from the ETH Domain, and computational resources provided by the Swiss National Su- percomputing Centre (CSCS) under the Alps infrastructure. This work was funded by the Swiss National Science Foundation (SNSF) under grant number 2000-1-240094, Research was sponsored by the Army Research Office and accomplished under Grant Number W91 19F1N-F24-1-000 1-048... Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Baden-Viurttembeg
by European Unior
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Niccol Ajroldi. plainlm: Language model pretraining in pytorch. https: com/ www.iiiioooaaaaiiaeeaaaaaaaaaaaa 2024.
S. Arora, Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, Rudra, and C. Ré. Zoology: Measuring and Improving Recall in Efficient Language Models. In The Twelfth International Conference on Learning Representations (1CL ICLR, 2024a.
Arora, Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, Zou, A. Rudra, and C. Re. Simple linear attention language models balance the recall-throughpe tradeoff. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (eds.). Proceedings of the 41st International Conference Machine Learning (ICMI.'24) volume 251 o Proceedings of Machine Learning Research. PMLR, 2024b.
M. Beck, K. Poppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. xLSTM: Extended Long Short-Term Memory. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.). Proceedings of the 37th In- ternational Conference on Advances Neural Information Processing Systems (NeurIPS 244), 2024.
S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, Phang, M. Pieler, U. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. arXhv:2204.06745 [cs.CL], 2022.
T.-. Chi, T.-H. Fan, P. Ramadge, and A. Rudnicky. KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation. In S. Koyejo, S. Mohamed. A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Proceedings of the 35th International Conference on Advances in Neural Information Processing Systems (NeurlPS'22). 2022.
K. Choromanski, V. Likhosherstov, D. Dohan, Song, A. Gane, Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D.. Belanger, L.. Colwell, and A. Weller. Rethinking attention with per- formers. In The Ninth International Conference on Learning Representations (ICLR'2)) 21). ICLR, 2021a.
K. Choromanski, H. Chen, H. Lin Y. Ma, A. Schanobish, D. Jain, M. Ryoo, J. Varley, A. Zeng, V. Likhosherstov, D. Kalashnikov, V. Sindhwani, and A. Weller. Hybrid Random Features. In The Tenth International Conference on Learning Representations (ICLR (12). ICLR, 2022.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane. Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with per- formers. In International Conference on Learning Representations, 2021b.
N. M. Cirone, A. Orvieto, B. Walker, C. Salvi, and T. Lyons. Theoretical Foundations of Deep Selective Spate-Space Models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Proceedings of the 37th International Conference Advances in Neural Information Processing Systems NeurlPS'24) 2024.
T. Dao. FlashAttention-22: Faster Attention with Better Parallelism and Work Partitioning. In The Twelfth International Conference on Learning Representations (ICLR'24). ICLR, 2024.
T. Dao and Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, Scarlett, and F. Berkenkamp (eds.), Proceedings of the 41st International Conference on Ma- chine Learning (ICML (2), volume 251 of Proceedings of Machine Learning Reseurch. PMLR, 2024.
S. De, S. L. Smith, A. Fernando, A. Botev, G. Iiistan-Muuaan A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. De Freitas, and C. Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models. 2.2.:2442..1921 [cs. 2024.
12

---

Published as a conference paper at ICLR 2026
Alexandre François, Antonio Orvieto, and Francis Bach. An uncertainty principle for linear current neural networks. In Nika Haghtalab and Ankur Moitra (eds.), Proceedings of the 38th Conference on Learning Theory (COLT'25), volume 291 of Proceedings of Machine Learning Research, pp. 2143-2187. PMLR, 2025.
L. Gao, J. Tow, B. Abbasi, Biderman, Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, Le Noac"h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang. L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang. K. Wang, and A. Zou. The language model evaluation harness, 2024. URL https: /te//eeooooo// org/ records (atpardd/11220080 126006026
R. Grazzi, J. Siems, A. Zela, J. Franke, F. Hutter, and M. Pontil. Unlocking State- Trae--rrakking in Linear RNNs Through Negative Eigenvalues. In The Thirteenth International Conference on Learning Representations (ICLR'23) 25). ICLR, 2025.
A. Gu and T. Dao. Mamba: Linear time sequence modeling with selective state spaces. arXiv:2312.00752 [cs.LG), 2023.
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Re. HiPPO: Recurrent Memory with Optimal Poly- nomial Projections. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin (eds.), Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurlPS 20)), 2020.
A. Gu, K. Goel, and C. Re. Efficiently Modeling Long Sequences with Structured State Spaces. In The Tenth International Conference on Learning Representations (ICLR 22). ICLR, 2022a.
A. Gu. A. Gupta, K. Goel, and C. R. On the Parameterization and Initialization of Diagonal State Space Models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Proceedings of the 35th International Conference Advances in Neural Information Processing Systems (NeurIPS'22), 2022b.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR 2022), 2022c.
Harris. On the use of windows for harmonic analysis with the discrete fourier transform. Pro- 2005.
ceedings of the IEEE,
A. Henry, P. Dachapally, S. Pawar, and Y. Chen. Query-Key Normalization for Transformers. In B. Webber, T. Cohn, Y. He, and Y. Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 1735 1780, 1997. Based on TR FKI-207-95, TUM (1995).
J. Hu, Y. Pan, J. Du, D. Lan, X. Tang, Q Wen, Y. Liang, and W. Sun. Comba: Improving Bilinear RNNs with Closed-loop Control. NKKY::2006.0477 [Cs. 2025.
Smith III. Spectral audio signal processing. (No Titl), 2011.
S. Jelassi, D. Brandfonbrener, S. Kakade, and E. Malach. Repeat After Me: Transformers Better than State Space Models at Copying. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (eds.), Proceedings of the s1 International Conference on Machine Learning (ICML'24) volume 251 of Proceedings of Machine Learning Research. PMLR, 2024.
A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, Lample, L. Saulnier, L. Lavaud, M.-A. Lachaux, Stock, T. Le Scao, Lavril, T. Wang, T. Lacroix, and W. El Sayed. Mistral 7B. arXiv:2310.0688 CL.) 2023.
Kacham, V. Mirrokni, and P. Zhong. PolySketchFormer: Fast Transformers via Sketching Poly- nomial Kemels. arXiv:2310001655. /1s0000.555//...GG. 2023.
13

---

Published as a conference paper at ICLR 2026
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers RNNs: Fast Autoregressive Transformers with Linear Attention. In H. Daume III and A. Singh (eds.), Proceedings of the 37th International Conference Machine Learning (ICML 20), volume 98. Proceedings of Machine Learning Research, 2020.
A. Kazemnejad, I. Padhi, K. Natesan, P. Das, and S. Reddy. The impact of positional encoding on length generalization transformers. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurlPS 233), 2023.
T. Salimans D. Kingma. Weight Normalization: A simple reparameterization to accelerate training of deep neural networks. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.). Proceedings of the 30th International Conference on Advances in Neural Information Pro- cessing Systems (NeurIPS' 16), volume 29, 2016.
Li, C. You, G. Guruganesh, J. Ainslie, S. Ontanon, M. Zaheer, S. Sanghai, Y. Yang. S. Kumar, and S. Bhojanapalli. Functional Interpolation for Relative Positions Improves Long Context Trans- formers. In The Twelfth International Conference on Learning Representations (ICLR 24). ICLR, 2024.
Z. Lin, E. Nikishin, X. He. and Courville. Forgetting Transformer: Softmax Attention with Forget Gate, In The Thirteenth International Conference on Learning Representations (ICLR'25) ICLR, 2025.
B. Liu, J. Ash, S. Goel,, A. Krishnamurthy, and C. Zhang. Transformers Learn Shortcuts to Au- tomata. In The Eleventh International Conference Learning Representations (ICLR 23). ICLR, 2023.
I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In The Fifth International Conference on Learning Representations (ICLR' 17). ICLR, 2017.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. The Seventh International Conference on Learning Representations (ICLR' (CLR' ICLR, 2019.
E. Martin and C. Cundy. Parallelizing linear recurrent neural nets over sequence length. In Interna- tional Conference on Learning Representations, 2018.
W. Merrill, J. Petty, and A. Sabharwal. The Illusion of State in State-Space Models. In R. Salakhutdi- nov, Z. Kolter, K. Heller, A. Weller, N. Oliver, Scarlett, and Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning (CCML'24) volume 251 of Proceed- ings of Machine Learning Research. PMLR, 2024.
D. Okpekpe and A. Orvieto. arXiv:2508. 19029 [cs.LG), 2025.
When recalling in-context, Transformers are not SSMs.
A. Oppenheim. Discrete- time signal processing. Pearson Education India, 1999.
A. Orvieto and R. Gower. In search of adam's is secret sauce. arXiv:2505.21829 [cs. (,,,L,,,200 2025.
A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning (CMML'33,, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023a.
A. Orvieto, S. De, C. Gulcehre, R. Pascanu, and S. Smith. Universality of Linear Recurrences Fol- lowed by Non-linear Projections: Finite-Wiae Width Guarantees and Benefits of Complex Eigenvalues. In R. Salakhutdinov. Z. Kolter, K. Heller, A. Weller. N. Oliver, J. Scarlett, and F. Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learning (ICML '24), vol- ume 251 of Proceedings of Machine Learning Research. PMLR, 2024.
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L Smith. Universality of linear recurrences followed by non-linear projections: finite-width guarantees and benefits of complex eigenvalues. arXiv preprint arXiv:2307. 1888, 2023b.
14

---

Published as a conference paper at ICLR 2026
G. Penedo, H. Kydlíček, L. Ben allal, A. Lozhkov, M. Mitchell, C. Raffel, L. Von Werra, and T. Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Proceed- ings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS 24), 2024.
B. Peng, J. Quesnelle, H. Fan, and E. Shippole. YaRN: Efficient Context Window Extension of Large Language Models. In The Twelfth International Conference Learning Representations (ICLR 24)). ICLR, 2024.
B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. Wind, T. Wu, D. Wutkke, and C. Zhou-Zheng. RWKV-7 "Goose" with Expressive Dynamic State Evolution. S.Kiv:2503.144555 [cs. [cs.CL], 2025.
H. Peng, N. Pappas, D. Yogatama, R. Schwartz. N. Smith, and L. Kong. Random Feature Attention. In The Ninth International Conference on Learning Representations (ICLR'21), ICLR, 2021.
M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Bjö"m Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Re, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning (ICML 24), volume 251 of Proceedings of Machine Learning Research. PMLR, 2024.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong, cosFormer: Rethink- ing Sofumax in Attention. In The Tenth International Conference on Learning Representations (ICLR 22). ICLR, 2022.
Z. Qin, S. Yang, and Y. Zhong. Hierarchically Gated Recurrent Neural Network for Sequence Modeling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Proceedings of the 36th International Conference Advances in Neural Information Processing Systems (NeurIPS 2023.
Z. Qin, S. Yang. W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated Linear RNNs with State Expansion. arXiv:2404.00 00904 [cs.CL], CLJ, 2024.
A. Rahimi and B. Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Proceedings of the 21 International Conference Advances in Neural Information Processing Systems (NeurIPS 077), 2007.
Y. Ran-Milo, E. Lumbroso, E. Cohen-Karlik, R. Giryes, A. Globerson, and N. Cohen. Provable Benefits of Complex Parameterizations for Structured State Space Models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS 24), 2024.
I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML'21), volume 139 of Proceedings of Machine Learning Research. PMLR, 2021.
Shaw, Uszkoreit, and A. Vaswani. Self-attention with relative position representations. [cs. CL), 2018.
J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi. DeltaProduct: Increasing the expressivity of deltanet through products of householders. arXiv:2502.1029) [cs.LG), 2025.
J. Su, Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.094 04400664444444 [cs.CL.) 2021.
Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive Network: A Successor Transformer for Large Language Models. arXiv:2307.0862) [cs.CI.). 2023a.
15

---

Published as a conference paper at ICLR 2026
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint ar XXXv:2337.08621,, 2023b.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and efficient foundation language models. arXiv:2302.13977 [es. (cs,CL), 2023.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS' 17). Curran Associates, Inc., 2017.
R. Waleffe, W. Byeon, D. Riach, B. Norick, V. Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayanan, G. Kulshreshtha, V. Singh, J. Casper, J. Kautz, M. Shoeybi, and B. Catan- zaro. An Empirical Study of Mamba-based Language Models. 2XXv:2440600788 [cs.LG), 2024.
B. Widrow, , and M. Hoff. Adaptive switching circuits, pp. 123134. MIT Press, Cambridge, MA, USA, 1988.
S. Yang and Y. Zhang. Fla: A riton-based library for hardware-efficient implementations of linear attention mechanism, January 2024. URL attps://githun wwt.a/uw..ocooooeeeeeeeeeeeeeeee oogg/f/a
S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated Linear Attention Transformers with Eafficet-Efffcieent Training. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, Scarlett, and F. Berkenkamp (eds.). Proceedings of the 41st International Conference on Ma- chine Learning (ICML 22), volume 251 of Proceedings of Machine Learning Research. PMLR, 2024a.
Yang, B. Wang. Y. Zhang, Y. Shen, and Y. Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS 2024b.
Yang, Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations (ICLR'25) ICLR, 2025a.
S. Yang, Y. Shen, K. Wen, S. Tan, M. Mishra, L.. Ren, R. Panda, and Y. Kim. PaTH Attention: Position encoding via accumulating householder transformations. arXiv:2505. 16381 [cs.CI.), 2025b.
M. Zhang, K. Bhatia, H. Kumbong, and C. R. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. In The Twelfth International Conference on Learning Repre- sentations (ICLR'S 24). ICLR, 2024.
16

---

Published as a conference paper at ICLR 2026
The supplementary is structured as follows:
Appendix A contains all derivations and proofs:
A.I shows that parameterizing a linear transformer with a unitary diagonal state transition can be implemented by applying RoPE to the queries and keys of the same models.
A.2 shows that one can use Random Fourier Features (RFFs) to approximate the expo- nential kernel and thereby softmax attention and, when limiting the approximation to the D-dimensions, can expressed as a recurrent model that can be implemented using an input-dependent variant of RoPE.
A.3 derives the optimal variance for the RFFs used Appendix A.2.
A.4 shows that complex diagonal SSMs can be understood spectral analyzers that suffer from spectral leakage. A well known remedy for spectral leakage is using real-valued decaying window functions, which can also seen as forget gates, a prevalent component in sequence models. This highlights the complementary roles of both imaginary and real parts of a gate in recurrent sequence models, with the former rotating and the latter decaying the past observation.
Appendix A.5 derives the connection between rotation using RoPE and Householder prod- ucts used in DeltaNet.
Appendix B lists the experimental details for language modeling and synthetic tasks and includes code listing of the implementation of Selective RoPE.
Notation. We use the following notation for mathematical objects: Lower-case letters denote scalars (a,, 8). Upper-case bold letters denote matrices (W, A). Lower-case bold letters denote vectors (v, ks ②) T denotes the transpose operator. H denotes the conjugate transpose operator. denotes the Hadamard-produet. Taking the real imaginary component of expression is denoted by either R or 3. Expressing vector as diagonal matrix is denoted by diag(-). Block- diagonalizing of square matrices denoted by blockdiag (.). Concatenating vectors denoted by æ, concat (I.-1"). By yo denote the argument of complex number.
A MATHEMATICAL DERIVATIONS AND PROOFS
A. RoPE AS IMAGINARY-VALUED LINEAR TRANSFORMER
We start by unrolling the linear Transformers recurrence:
S. S R + U.
O, R(Sigg}
0, R
Uyk H Ri-T
Σ R 2
q: 3
Therefore, the attention score applied to value e, is:
Atter = R
ki R''- q:
17

---

Published as a conference paper at ICLR 2026
Since R is diagonal, expand the expression
4/2
Att
R2
R + i +i elualt-r)) (k + i ki
4/2
@tnll ipp(@n)) clanlt-T) |ke,pu isp(eenn))
R
d/2
gg.nl k c((ua(t-)) ()- -0-((())) (ldan)-p((ee -((nn))p((x.n)))
R
n=1
4/2
1g.nl |kr,ul cos (wn 7) -pp(@r.))) (p(ker,))
(14)
where n))..)) and n) denote the complex phases (angles) of the n-th component of qi and ky, respectively. Equation (14) shows that an imaginary forget gate rotates the query-key pairs at each index n with distinct frequency wa- We now demonstrate that this is equivalent to applying RoPE. Replacing the cosine in eq. (14) with its matrix multiplication equivalent:
cos(ann T) Lunn
coos [ sin LQi,n)
cosfunn (t (1=)) T))
st (t-r) 73
(t
[ood (k.n)) siin (kr,,)
cos(ws (f-))
Plugging above in eq. (14) we achieve:
4/2
Toos ( sin Lqr.))
(t sin(wn (t-T))
- -sin(wn (t- cos(wnn (t T))
cood Zkr sin <k,
Att,,
Σ
At.n
k
T))
n=1
4/2
Σ
At.x
Cood LQr.n) )
sin <Qt,m)
Coodwin - -))
sin(wn -])
- -siu(un (t cos(win( (1 +))
|k.al
[oool (1(r,) sin (ko,,
4/2
Coo Wn (t T))
-sin(wn (t - -}) cos(wnntt T))
KR k
(15)
q
sin(wn (t - T))
Using the definition of:
d/2
d/2
q.
θ
k
I
n=1
T.
we can write Equation (15) as:
d/2 Σ
Att.r
qt.n R
kr.n.
which is theoretically equivalent to applying RoPE to query-key pairs q:, kr, RoPE interleaves the real and imaginary parts of complex queries and keys across the hidden dimension, then applies rotations to each pair.
A.2 RANDOM FOURIER FEATURE APPROXIMATION SOFTMAX ATTENTION
We start with the definition of softmax attention:
va q: kt)) Uy 2 exp Va q. k.r
0 21
St
exp
where qt, kr Rd. For simplicity, we omit the normalization factor 1/vd and first focus the numerator of the output, specifically the exponential kernel. As Equation (2), the denominator scaling be handled separately through external state z,-
18

---

Published as a conference paper at ICLR 2026
To approximate the exponential kernel exp(-) -), we use Random Fourier Features (RFF) (Rahimi & Recht, 2007) with frequencies w E F (o, o I). The feature map is defined as
(w(x) exp
i TwTxx x
so that
exp(q, T k.) [ ((0 [oh (g:) To (k=)))
for a - 1. By applying this feature map, the linear attention formulation in Equation (2), we can approximate the exponential kernel in softmax attention. Continuing the approximation:
11 + | 12
exp(q k-)) exx
By ((E=-N((.)) exp(iw T q:) exp(-id in k+]]}
2
Let wy ~ N 22) for j E (1,2,...
DJ. Then due to the law of large numbers we have:
11 1 exp(q,1:.) = exp
|| 11
1 D
R
lim D-
exp qi) exp(-iww 1s,)
2
D
Therefore, we can approximate exp q, k;) as the dot product of the random exponential projection of the query and the key using D random wjs:
1 8 D = D
q
1 +
|
exp(iw] q.) (ep(--iw] k.)) 2r-
exp
1=1j=1
This allows us to compute the softmax attention as the linear attention parameterized by:
119.12
| 2
( exp
exp(ist q(), o(k,) exp
exp( -is2 -s))),))))
with limp-ss R {P))
Ei= exp(g, k.)) er and Ω [.,,... wp). Omitting the superscript
D for simplifying the notation, let us focus on one random feature wj and its contribution to the output:
22
k
BCJ
exp
exp
exp(iw] q:) exp( (-iww kr) vr.
In this case, we have 8P have:
S 1, where S.D
$t E CdDD Now note that 1 81.2
8t
[1 exp
11 -111--11q--11
exp( a:-1) exp (q. q(-1)) (xw(-iw]] k.) V+ (16)
+ exp
exp
[| 2
exp زمن (q: kt)) Ur.
(17)
=exp
||
|
112
exp (iw) (q: q:-)) 8/_ ow,, (q:) ow, (k:) Uz
(18)
2
Note that the real exponential component in Equation (18) can introduce instability to the recurrence. Therefore, following the standard in both linear Transformers (Yang et al., 2024b;a; 2025a; Lin al., 2025) and deep softmax Transformers (Henry et al., 2020), we assume L2 normalization over the query and the key, i.e.. ||q. 112 = ||q.-1lla Thus, recurrence presented in Equation (18) simplifies to:
$t exp [w)] (q: q)-1)) $1-1,5 + ,w, (9t dws, (ke) U2.
with B.J being the jth column of SP scaled by the values exp( (g: - q)-1)). Therefore, we write the recurrence over $ as:
sPP = S + (Ø(g:) o(k))) 6 = 1 5P1
- 5P
where (xx) is vector with its jth element equal dw,, (x), and R, is:
R. diag(exp(1s) (g: qt- q(-1)))
(19)
19

---

Published as a conference paper at ICLR 2026
Focusing Equation (19), we observe that exponential kernel in softmax attention implicitly ap- plies a form of input-dependent (Selective) RoPE (see Sec. 2). However, instead of learning the frequencies 52, they randomly sampled from normal distribution.
Similarly, we can also approximate the normalizing factor 24 as:
9 12
exp(iwe q:) exp(-iis kz))
exp
2
Separating the contribution of each random feature, we have:
119:11 2
| sap(iw)] qi) exp( - k,)
exp
exp
2
Finally, defining 2,- [
212
D.
we arrive similar result. The full recurrence
of softmax attention, therefore, can be written as:
SP ō, 211
S,P = SP__ R. or (o(g)) okk))) 2P= 2P R. + ((q:) o
which again highlights the importance of the gate R as selective rotation.
A.3 OPTIMAL VARIANCE FOR RANDOM FOURIER FEATURES
Theorem Let the expected error of the RFF kernel over wi ERR 19c, k,] = Eww, (bE)= ow, )w,((w)) (qi) Ow, (kr) - -exp(1. k.,) )
N (0,0021) be as follows:
Then, for given a pair of
arecos q:
L2 normalized query and key, the optimal value of is equal to tan
Proof We start by writing down the error:
22
ERR qt: kr]
exp (we + wy) (q: ki-)
D
f
2e
R (ixp( (q: kr))]] explq. kr) + const.
D ==1
2 E cos" (iw T (g: ((:-))))] + ²² (D² - D) [cos(iw (g: - kr)))
1
D22
2e E [cos liw (qi kr))] exp q. k-) + const.,
where the const. term corresponds to the terms constant w.r.t the variance of the distribution o². Plugging in the expectation of the cos(-) and cos?(, functions (Choromanski et al., 2021a), we get the following optimization problem:
2-40 40²
exp( (xp((-
D- D
2- 20" exp(-202E) 26 20--0" exp((1 - -0 -0²)
min
2D
D
where for simplicity, set q. kr E E [0, 1). Since in most cases, D is sizable number, we try to solve this optimization problem in the limit D - oo, which is equivalent to:
min 2-20 21+8) (0(1-02)
with the optimal value equal to:
1
1+
20

---

Published as a conference paper at ICLR 2026
Considering normalized queries and keys ||ke|| =lleel 1 replace the & qi k, with cos(8) therefore above also simplifies to:
cos(O) + cOs(O)
tan(0/2).
This completes our proof.
A.3.1 PARAMETERIZATION OF THE TEMPERATURES
We can generalize the parameterization of proposed temperatures that of RoPE introduced by Su et al. (2021) as follows. Let e be small enough number. Then, we have:
RoPE:
& arange (0, D//2)
(1-C)T
Selective RoPE: arange (0, D//2)
- = tan(
(D//2-1
Here, E can be seen as the inverse of the base frequency in RoPE (Su et al., 2021), and the upper- bound on the angle between the queries and keys in our temperature scheme. A visualization of the temperature distribution in Selective RoPE compared to standard RoPE shown in Figure 2. Our proposed variation of the temperature has extremely similar distribution, but with slightly faster decay to 0.
A.4
ROLE OF REAL AND IMAGINARY PARTS IN DIAGONAL SSMS
We start our analysis with non-selective diagonal SSMs and show the distinct roles of the real and imaginary components. SSMs can be derived from continuous-time representations, expressed as
ds(t) As(t) + kv()), (t)) s(), K(t) q Atk, o( - K(t) * v(), di
(20)
where we assume the continuous value signal v(() and the continuous output signal o({) to both be scalars. Inspired by S4D (Gu al., 2022b), which is an SSM with diagonal A, we initialize the imaginary part of the state matrix as A. iwn, (n E [0, N). roots of unity), from which the output derived as:
(1) Σ en So c-mw ( (T)),, me(T)
1, OSTSL 0, O.W.
(21)
where (T) is step-window function. The integral in Equation (21) equivalent to computing the Fourier Transform of the windowed signal v(T) (T) at frequency wa. Duality between convolution in the time domain and multiplication in the frequency domain simplifies eq. (21) to:
(()) Σ kn Urarn)) U2 sin(wt)
e
(22)
w
with Vu,, and Ue denoting the Fourier transforms of u(T) and u, (T),, respectively. The input spectrum V., is convolved with the window spectrum Ur.., causing distortion, a phenomenon known as spectral leakage. In the discrete domain, the integral in eq. (21) becomes a summation:
N
OF
km
exp(
2minr N
Ur.
(23)
0
where W,. 2mmi N and A N. Thus, S4D with purely imaginary state matrix A acts as spectral analyzer: it accurately computes the -oooint DFT of the value v, for t s N. But for > N,
For consistency within our notation, we replace the common SSM notation for the B and C matrix and the input with our self-attention based notation, i.c., B denoted as the key k. C denoted the query q, and the input signal denoted the value For detailed comparison, refer Table 2 from Yang al. (2024b).
21

---

Published as a conference paper at ICLR 2026
this spectral analysis suffers from spectral leakage since the state size at most represent N frequencies. Therefore, the higher frequencies are being aliased or overwritten.
In Signal Processing, spectral leakage is addressed by windowing (Harris, 2005). In S4D, this is achieved implicitly by using a complex state matrix A with the real part acting as a window function, classical solution spectral leakage (Oppenheim, 1999). Concretely, with A exp(-a,, + 2winA), S4D performs windowed DFT using a Poisson window (III, 201 1), thereby avoiding spectral leakage. Its output can be written as:
02
On k Σ exp((
2xint Uy exp(-a,,
(24)
where W, is the Poisson window and A N is chosen for clarity in the DFT formulation. Thus, the real part A S4D acts as a window, suppressing spectral leakage and enabling undistorted spectral representations. Therefore, to summarize: the real and imaginary parts of state transi- tion matrix A serve distinct but complementary roles; Imaginary parts extract spectral information, while Real parts suppress leakage and ensure clean representation the spectrum.
A.5 COMPLEX ROTATIONS AND HOUSEHOLDER MATRICES
Another approach towards introducing rotations to the queries and keys is using Householder reflec- tion matrices (Yang al., 2024b; 2025b). this approach, the rotation the query and key pair is limited to single reflection along the direction of an input-dependent vector. Specifically, let w, be an input-dependent unit vector. Then, the positional information encoded through the product of Houscholder reflection matrices as:
q. R2 qq II 23 -w,w... k,.
Therefore, the positional information between the t and thh token is encoded through rotation consisting of - reflections.
Conveniently, we can also write the complex diagonal rotation matrix in Selective RoPE in terms of the product of Householder matrices. Specifically, we can write the realification of the rotation matrix R, as the product of d Householder reflections, each of which performs the reflection over single pair of adjacent elements:
0
0,
0j
Oj cos (wir /2) sin (we. 22)
R2 II
d
1
1-2
cos (wr.j /22 sin (we.s/2) /2)
0
0
Od---22
Oa
08-s-2
O 08-5-2
where define O, E Rm vector with all zeros. Assuming split adjacent elements in the query-key into the real and imaginary components, then Selective RoPE is performing two reflec- tions over each adjacent element pair of the input, with one of them parametric reflection, and the other negating the first element.
This interpretation also explains why gain more expressivity when using Selective RoPE: due to the block-diagonal structure, there is a channel mixing happening between the adjacent query-key elements. Channel mixing is key component in improving the expressivity of sequence mod- els (Cirone et al., 2024), thus improving the state-tracking abilities of the network (Siems et al., 2025).
A.6 KERNELIZED LINEAR ATTENTION WITH FIXED PARAMETERS
In order to further contextualize our results for the RFF kernel and its relationship to other variants of kemeliiedd linear attention, we try to replicate our mathematical derivations for two more variants of these models: namely Performer and CosFormer Choromanski et al. (2021b); Qin et al. (2022). We then present the results side-by-side to provide further insights into the importance of complex parameterization in Linear attention. We provide summary of the models in Table 4.
22

---

Published as a conference paper at ICLR 2026
Model
Transition Matrix
Input-dependent
Performer (PRF-based)
R. diag (exp (S2 (ge q(-1)))
RFA (RFF-based)
R2 = diag exp (is) (4: 41 41-1)))
Input-indepent independent
RoFormer (RoPE) CosFormer (diagonal RoPE)
R = diag(exp(iw))
R exp(iw))
Kernelized linear attention models under the shared recurrence S, S, + v The top block lists input-dependent transitions, while the bottom block lists input-independent ones. Among these, the Performer transition is not norm-preserving in general.
Table 4:
A.6.1 PERFORMER
We use the notation provided in Appendix A.2. To approximate the exponential kernel exp()), we use Positive Random Features (PRF) (Rahimi & Recht, 2007: Choromanski et al., 2021b) with random
features E Id (N, o2 I). The feature map is defined as
|
(wr))) = exp
w x
so that
exp(g, k7) = (0, hww (q:) &w (k:r)}
for a 1. By applying this feature map, the linear attention formulation in Equation (2), we can approximate the exponential kernel in softmax attention. Continuing the approximation:
| 2 + |
exp(q.k:: exp
(Ew.N( [exp(w> q:) exp(w kr)]
2
Let w N 00) for j (1,2,...
D}. Then due to the law of large numbers have:
|1qel12
||,
11
D
exp(q. k.)) exp
lim D-xe
exp(w] q:) exp(w) kr).
2
j=1
Therefore, can approximate exp(g, (g. k.r) as the dot product of the random exponential projection of the query and the key using random wjs:
E
q
1 |
1
exp(w) q:) exp(w] k,) Vy.
exp
2
r=1j=1
This allows us to compute the softmax attention as the linear attention parameterized by: |
|
(qq:) exp
exp(S2 qr), o(k,) exp
exp(St kr),
with limp- R { = Ei=r exp(q. k,) v and 52 = [W]. Wp). Omitting the superscript
D for simplifying the notation, let us focus random feature wy and its contribution to the output:
|
k l2
8
exp
exp
explw q:) exp(wx kr) v.-
2
2
In this case, we have s. we have:
1, S 1,
where S.P
[8r,1
Sr Rd Now note that S1,2
exp
19: 11:-111--111 1|::-111
exp(w) qi-1) exp(ws] (q: qr-1)) exp(w krr) vr (25)
1=1
exp
||
exp
| exp [ (q: ki)) vt.
(26)
exp
1a. || 111
exp(w> [ (qi q(-1)) ay dw, (q:) (ki) 11
(ki)
(27)
23

---

Published as a conference paper at ICLR 2026
Note that the real exponential component in Equation (27) can introduce instability the recurrence. Therefore, following the standard in both linear Transformers (Yang et al., 2024b;a; 2025a; Lin et al., 2025) and deep softmax Transformers (Henry et al., 2020), we assume L2 normalization over the query and the key, i.e., | = 11q--11l 11/2- Thus, recurrence presented in Equation (27) simplifies to:
$Lj exp(w) (q: q:-1)) $t-1, Φw, (q:) Pws (ki) Vt:
with 8t, being the jth column of SP is scaled by the values exp(w] (q: 9--1)). Therefore, can write the recurrence over S; as:
1 SP1.
= + os (Ø(q:) (kk))
s/
where (xx) is vector with its jth element equal ow,, (I). and R2 is:
R² diag(exp(S (q: qt-1)))
(28)
Focusing on Equation (28), we observe that the PRF approximation no-longer results in an input- depedent rotation matrix, but rather a more general form of real-valued input-dependent state transition matrix. However, the features l2 are randomly sampled from a normal distribution.
Similarly, we can also approximate the normalizing factor 2, as:
2P = D
1
D
q/ + ||k 12
exp(w] q:) exp(w]]::) kr).
exp
2
r=1j=1
Separating the contribution of each random feature, we have:
1:
[k-l2
21. = Σ exp
exp(w] qi) exp(w] kz).
exp
2
we arrive similar result. The full recurrence of softmax attention, therefore, be written as:
Finally, defining 2,- = 2
212
D.
SP1 2P'"
S.P = SP_ or (á():) o(k:))" 2 2P olq.) (kk)) o,
Interestingly, unlike the results provided in Appendix A.3, this case we won't be able to rely on spectrum of choices for a in order to limit the approximation error for finite number of random feature samples. More concretely, we provide the following theorem to highlight this issue:
Theorem Let the expected error of the PRF kernel over wi
N (0, be as follows:
Eww, EbE ow, (qi) Ow, (kr) -exppx.pppppp k.,) k
ERR (qt, k. k= =
Then, for given a pair of
L2 normalized query and key, the optimal value of is equal to =
Proof 2 We start by writing down the error:
-
ERR qt, kr]
E
cxp (we + w (qi - --kr))
D
3='=1
2c
E (exp(w) [w] (q: kt))] exp(ai (T)),))))))));))) + const.
D
D exxp(2w' T (q: + ko k,))]] e (D² (D2- -D) DD) [exp(w] (q: + k,)))
D2
2c E Te (q: + k-))] exp(q, kr) const.,
where the const. term corresponds to the terms constant w.E.t. the variance of the distribution o². Plugging in the expectation of the exp(-) function (Choromanski et al., 2021a), get the following optimization problem:
min
e
(1+
+ e 2 2D exp(202 + 22 1exep((22(1 6)
D
D
24

---

Published as a conference paper at ICLR 2026
where for simplicity, set q. ke = [0, 1]. Since in most cases, D is sizable number, we try to solve this optimization problem in the limit D - oo, which is equivalent to:
20 (1+E)-2 2e" 2 (145)+8-1
min
with the optimal value equal to a
This completes our proof.
Following Theorem 2, we observe that for a fixed pair of queries and keys, the optimal variance is independent of the angle between the two vectors. This is in stark contrast of the RFF-related results in Theorem 1, where the variance of the random feature is a function of the angle between the query and key vectors.
At first glance, this may seem good incentive to instead adopt the transition matrix introduced recurrence. However, one also need to consider that there is no stable generalization of recurrence based on this transition matrix, there won't be any guarantees about the matrix being contractive or orthogonal. On the other hand, the rotation matrix Equation (19) orthogonal matrix. This makes RFFs much more
Equation (28)
will always be stable, it corresponds suitable choice for a random deatures-basee approximation of the softmax attention function.
A.6.2 COSFORMER
We start by defining the output in the CosFormer model (Qin et al., 2022):
S.q.
(1--7)
क (1 (1-TT)
0.
S. = Σ COs
v,k 2, =
Σ cos
kr,
(29)
2, 9:
2M
2M
where M corresponds to pre-defined, large scalar value. First, we focus on the state S, in Equa- tion (29), which we can write as:
π
(t T)T
S. = c0S
008
2M
2M
7=1
(L
-
1 7)
[-1
(1 -T)
sin
sin
.okk? + viki
2M
2MM
-=1
cos
π
S
sin
11-1 + v,k]
2M
2M
where we define:
(t T)T
S
sin
-.KKT
2M
sin
(t
T)T
k + cos
π
Σ sin
(t (t-1- T)
v
2M
2M
2M
2M
sin
π
S + cos
π
S
2M
2M
Now, let us define:
$, = [S. Spin]
for which we can write the following recurrence:
S, S
cos
2M sin
sin
2M
I
[uik] 0]
n 2M
cos
2M
This recurrence is equivalent to complex-domain domain recurrence where S, corresponds to the real component of the state and Sain to the imaginary component:
25

---

Published as a conference paper at ICLR 2026
S. S vykk
π exp
(30)
2M
Similarly, we define the complex-domain normalization factor 2n as:
Σ; = [(2 2
where we define z!th as:
sin sin
π
cos
sin 21-1-
2M
2M
This recurrence also equivalent complex-domain recurrence of the following form:
2; = - ke.
R = R=expl I.
2M
The model introduced in Equation (29) is complex-domain linear attention model, in which the rotation matrix is defined as a diagonal matrix with the same values on the diagonal (see Equa- tion (30)). Note that demonstrated by Qin al. (2022), the rope trick be applied further simplify the computation.
A.7 RELATIONSHIP BETWEEN Selective RoPE, ALIBI, AND FOX
The of exponential decay in softmax Transformer has been common practice for a while (Press et al., 2021; Jelassi et al., 2024; Lin et al., 2025), proving advantageous for length generalization and position encoding. In this section, investigate ALiBi and FoX examples of such models and provide context for their success based on the findings of this paper.
A.7.1 ATTENTION WITH LINEAR BIASES (ALIB1)
ALiBi (Press al., 2021) originally introduced position encoding technique aiming improving the length generalization of softmax Transformers. Borrowing the notation from Equa- tion (I), we write the attention mechanism in ALiBi as:
8,
exp
q
kr +
Σ log A k=T+1
Vr: 21
exp
va
9
kr
+
Σ
log
N=++1
where log A 0 be scalar diagonal matrix. Note that re-write St and by moving the input-independent summation outside the exponent:
exp
Ja aT k.
A A
21
exp Va9. k.) -
which can be interpreted as Transformer layer with exponential decay.
A.7.2 FORGETTING TRANSFORMERS (FoX)
FoX (Lin et al., 2025) is a softmax Transformer that aims to augment the attention mechanism with a forget gate, inspired by the success of the exponentially decaying gating mechanisms linear RNNs and linear Transformers (Yang et al., 2024a; Gu & Dao, 2023). Borrowing the notation from Equation (I), we write the attention mechanism in FoX as:
cxp
1 V29
kr +
Σ x=
log Ar Ur, 2: = Σ cxp r=1
v2 kr + Σ
log Ar
T=1
k=T+1
26

---

Published as a conference paper at ICLR 2026
where log Ax 0 can be scalar or diagonal matrix. In this case. A, is an input dependent factor and a function of the nh token. We can re-write 8 and 2, by moving the input-dependent summation outside of the exponent:
81
Σ exp k.y 7=1
II #=T+1
A,
exp
1 Va
q k.)
II A N=7+1
r=1
which can be interpreted as a Transformer layer with selective exponential decay.
A.7.3 THE COMPLEMENTARY ROLE OF ROTATION AND DECAY
The success of models belonging to the family of softmax Transformers with forget gates, namely ALiBi and FoX, demonstrate that the exponential decaying factor is missing component in these models. This observation aligns with our proposed unifying framework in Section 3. Interestingly, in the softmax setting, Selective RoPE closely parallels FoX: it can be seen as endowing the decay term a4 with a rotation component R..
B EXPERIMENTAL DETAILS
In this section we provide additional details experimental setup for the tasks considered in the paper.
B.I LANGUAGE MODELING
We use PlainLM (Ajroldi, 2024) together with an adapted version of Mlash-lineae aeiiinnaratteetee tentioo for all of our language model trainings. We train 80GB VRAM GPUs including NVIDIA A100, H100 and B200. One model training (370M parameters, of such GPUs and takes anywhere from 48
1OB tokens) is performed hours (on A100) to 9 hours on 8 B200. We use Distributed Data Parallel (DDP) for multi-GPU training.
single node with
Table 5: Optimizer and learning-rate schedule hyperparameters for language modeling.
Optimizer
Symbol
Base learning rate (candidates)
[Se-4, Le-3, 20-3, 40-3, 80-3, 1. 6e-2)
Adam &1
0.9
Adam 2
0.95
Weight decay
X
0.1
Numerical epsilon
Grudient clipping (global norm)
clips,
LO
I.R Schedule/ Training Horizon
I.R start (schedule)
Ratur:
le-5
LR cnd (schedule)
Yond
1o-4
Warmup (fraction steps)
Total optimizer stops
T
66,758
B.2 SYNTHETIC TASKS
B.2.1 MAD
For MAD, we take the implementation from mad_lab and implement Selective RoPE in GLA. We follow the exact experimental setup outlined in the paper (Poli al., 2024) and run all variations of task difficulty and optimizer hyperparameters which results in 66 task settings x 6 optimizer settings 396 trained models per considered setting (i.e., GLA with Selective RoPE, RoPE NoPE). We provide the logs from the experiments in our supplementary.
B.2.2 STATE TRACKING
For state tracking we adopt the exact experimental setup as described DeltaProduct (Siems et al., 2025) and Grazzi et al. (2025).
27

---

Published as a conference paper at ICLR 2026
Table 6: Training state tracking configuration.
Training Loop
Parameter
Value
Epochs Batch
100
4096
Optimization
Learning rall
81
0.9
82
0.999
Optimizer
Ic-8
Weight decuy
LR scheduler
cosine
Precision Compile
Mixed precision
DType
bfloat16
Data
Train size
2,000,000 sequences
Train sequence length
128 tokens
set size
500,000 sequences
Eval sequence length
512 tokens
Seeds & Eval
Seeds
[555, 666, 777, 888, 999] 128
Eval batch size
B.2.3 MQAR
We have carefully followed the training recipe of Arora et al. (2024a) for all models including: GLA (Yang et al., 2024a), DeltaNet (Yang et al., 2024b), Mamba2 (Dao & Gu, 2024) and Transformert+ (Touvron et al., 2023). The learning rate for all models was swept within the range [0.0001, 0.01] for 8 different values per each model ranging uniformly from 0.01 to 0.001. All other configuration and the model dimensions were remained the same as original reference Arora al. (2024a).
B.2.4 COPYING
THE USE OF LARGE LANGUAGE MODELS (LLMs)
While preparing this manuscript, we used Large I. Language Models (LLMs) to limited extent. Their role was restricted to assisting with editing and polishing the writing, such as improving clarity, grammar, and flow. All conceptual idcas, methods, experiments, and analyses presented in this paper are entirely the work of the authors. No ideas, algorithms, or research contributions were generated by LLM. The LLM served only as a tool to refine the presentation of the text without influencing the substance of the research.
28

---

Published as a conference paper at ICLR 2026
Table 7: Optimizer and Data parameters for Copying
Optimizer
Leurning rate
5. 5.0c-5
Weight decay
0.1
0,9
8g Optimizer e Gradient clipping (global norm)
0,999
1.0e-8
1.0
Scheduler
Scheduler
linear
Warmup (fraction of steps)
0.1
Seeds & Eval
Seed
Eval batch size
256
Data
Vocub size
26
n n-gram
0
Answer length
0
Train task
copy
Eval task
copy
Sequence length
420
Min length (train)
2
Max length (train)
Min length (eval)
2
Max length (eval)
$12
Sampler type
sequential null
Sampler seed
29


Reviews:
[{"review_id": "h6PRb8jVaZ", "reviewer": "Reviewer_NYgW", "summary": "The paper introduces Selective Rotary Position Embedding (Selective RoPE), an input-dependent mechanism designed to generalize standard Rotary Position Embeddings (RoPE) by performing rotations at arbitrary, selective frequencies.", "questions": "Do you expect GLA  and Softmax Transformers to benefit equally from Selective RoPE?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Interesting theoretical insights, such as:\n- Softmax attention implicitly applies a selective rotation, to encode relative positional information between tokens\n- Linear Transformers can be enhanced by using both forgetting via real decay and rotation via imaginary gate.", "weaknesses": "Very limited evaluation for language modeling.  Would be good to have at least RoPE as baseline (Table 3) and evaluate Selective RoPE in different settings (i.e context length)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QLlAawBCMK", "reviewer": "Reviewer_zi2k", "summary": "The paper proposes a selective Rotary Position Embedding (RoPE) that uses an input-dependent rotation to enhance the performance of models with Gated Linear Attention. The paper provides a theoretical analysis of how softmax attention performs a hidden form of rotation and further proposes to determine the rotation angle via a linear projection of the query. Experiments are conducted on GLA showing that selective RoPE achieves better performance than NoPE and RoPE.", "questions": "* It would be appreciated if the authors could provide additional experimental results of applying selective RoPE to softmax attention with more details on the experiments.\n\n* Since rotations are composable. Can the proposed method be equivalently viewed as applying a rotation to the queries and keys before the RoPE operation? What is the significance of doing so?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is clear with summarized insights and clear figures.\n\n* The analyses on the implicit rotation of softmax attention are interesting.\n\n* The paper provides an in-depth analysis from the RFF perspective.", "weaknesses": "* While the paper takes a lot of effort in the derivation of implicit selective rotation in softmax attention, the proposed method is applied to gated linear attention. Given that the derivation heavily relies on the Random Fourier Features (RFF) approximation, the practical impact of the proposed method has not been validated.\n\n* Limited experiments. The experiments are conducted on small-scale models, which raises my concern about the stability and scalability of the proposed method. The leaned rotation angle may lead to unstable training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6H5GN7zbKm", "reviewer": "Reviewer_uVEP", "summary": "This paper presents a version of RoPE with learned, input-dependent arbitrary rotations. Theoretical analysis provided indicates that softmax attention implicitly performs selective rotations, motiving the proposed architecture. Selective RoPE uses a learned linear projection and cumulative sum to produce input-dependent rotations. An analysis of diagonal SSMs is provided which shows distinct roles for the real and imaginary parts of the state matrix, motiving the incorporation of Selective RoPE with GLA to provide better memory. Experiments on language modeling and show improvements with Selective RoPE compared to RoPE and softmax attention in GLA models.", "questions": "Did you compute RoPE setting for Table 3?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper gives a detailed theoretical justification for the architecture design. The analyses of softmax attention as implicit rotation and spectral leakage in SSMs may be useful to future work. Experimental evidence is provided to support claims.", "weaknesses": "The conclusion that softmax attention applies implicit selective rotation is based on the RFF approximation and additional normalization assumptions. The paper does not prove that the resultant normalized approximation converges in the limit to softmax attention, and so this analysis may be overstating the connection. \n\n\nThe real-data language modeling results in table 3 omit the RoPE condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZDa85mPL54", "reviewer": "Reviewer_Rgdu", "summary": "The authors generalise RoPE to a mechanism allowing it to choose angles in a way that is input dependent. The authors perform analysis mainly on linear-attention models and show that selective rope seems to improve performance over baselines such as RoPE or NoPE.", "questions": "Please see questions in the weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The connection between SSMs, linear attention, and RoPE is interesting. I particularly liked the presentation in Table 1.", "weaknesses": "My main area of research is in Transformers and not linear attention although I have some experience with linear attention and RFF.s \n\nI do not think I quite understand the point of \"Softmax attention implicitly applies a selective rotation, to encode relative positional information between tokens.\" Are you arguing that the rotations come from the relationship between the softmax kernel and RFF? So you can view the softmax kernel as applying RoPE but where the angles are sampled IID from a Gaussian. This however would really only be true if your angle samples tend to infinity of course. Is this how you are connecting RoPE with a \"NoPE\" softmax?\n\nI found the notation slightly hard to follow especially as someone not coming from SMMs. I mainly found confusing that RoPE is really a method used in Transformers, but the paper seems to only implemented the selective mechanism for linear attention and was not implemented for normal quadratic Transformers? Is there something stopping you from implementing for a normal Transformer? \n\nMinor\nTypo in abstract: rotation in all angels -> rotation in all angles", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)