A Circular Argument: Does RoPE need to be Equivariant for Vision?
Authors: Chase van de Geijn
Timo Lüddecke
Polina Turishcheva
Alexander S. Ecker
Venue: NeurIPS 2025 poster ()
Keywords: Positional Encoding
RoPE
Lie algebras
LieRE
Vision
Primary area: deep_learning

Abstract: Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such as images and videos. The success of RoPE has been thought to be due to its positional equivariance, i.e. its status as a \textit{relative} positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for $M$-dimensional data, if we require commutative generators -- a property necessary for RoPE's equivariance. However, we question the necessity of equivariance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but with the assumption of anti-commutative generators -- relaxing the equivariant condition. Empirically, we find Spherical RoPE to have the equivalent learning behavior as its equivariant analogues. This strongly suggests that relative positional embeddings are not as important as is commonly believed. We expect this discovery to facilitate future work in positional encodings for vision that are faster and generalize better by removing the preconception that they must be relative.

A Circular Argument: Does RoPE need to be Equivariant for Vision?
Chase van de Geijn Timo Lüddecke', Polina Turishcheva' Alexander S. Ecker
Institute of Computer Science and Campus Institute Data Science, University of Göttingen
2 Max Planck Institute for Dynamics and Self-Organization,, Göttingen, Germany
chase. wcwse..eieeee.......... geijnuniiinninnieeeee wwwnjjjnnn-gggttinnn -oattggggetnnnn.... eckeroca uni wwggggiinggnn.... de
Abstract
Rotary Positional Encodings (RoPE) have emerged highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such images and videos. The success RoPE has been thought to be due to its positional equivariance, i.e. its status relative positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for M-dimensional data, if we require commutative generators property necessary for RoPE's equivariance. However, we question whether strict equivariance plays a large role in RoPE's performance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but assumes con-commutative generators. Empirically, we find Spherical RoPE to have the equivalent or better learning behavior compared its equivariant analogues. This suggests that relative positional embeddings are not as important as is commonly believed, at least within computer vision. We expect this discovery to facilitate future work in positional encodings for vision that can be faster and generalize better by removing the preconception that they must relative.
Introduction
Deep learning is in the age of transformers 77 At their core, transformers are built on attention 4
61], which is permutation-invarian operation 74]. making them agnostic to word or token position within corpus. To break this symmetry, tokens must be modified with position embeddings [25] 83
Recently, Rotational Positional Encodings (RoPE) [69] have gained popularity, touting an emphasis
on the relative position between two tokens rather than their absolute positions 22 26 46
However, some of the original claims of RoPE have been called into question leading to confusion as to why it works: Su et al. [69] claimed the attention scores to decay with distance between tokens. This was found to be true only for attention with the same query and key [5]. Moreover, transformers with causal masking have been shown to require positional encodings to be capable of recovering absolute position [24), making RoPE's relative (shift-equivarian) claim questionable. However, many new methods continue be motivated by RoPE's benefit from shift-equivariance (27 60 35 To
guide future research in positional encodings. it is important to discover whether shift-equivariance truly makes RoPE successful and needs be preserved when extending it.
Both transformer and RoPE were originally designed for one-dimensional sequences such as language. RoPE encodes position by pairing dimensions within the query and key vectors within transformer and rotating the paired dimensions. Transformers have become the current staple across all AI Naturally, RoPE's recent popularity in NLP has also spread Vision
fields [17
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Transformers (ViT), where the data is two- or stree-dimensionall corresponding to images and videos. How to extend RoPE to other modalities nontrivial and assumptions must be made to maintain equívariance 44
60 The most commonly used approach for extending RoPE to ViTs is through Axial RoPE 15
26 77), partitioning the embedding dimensions into dimensions rotated independently either by the horizontal or the vertical position of the tokens. However, this approach does not allow for diagonal attention patterns where horizontal and vertical information "mix", which have been hypothesized to enhance generalization; consequently, learned Mixed RoPE was proposed [26]. Even more recently, LieRE [52] generalized Mixed RoPE from pair-wise rotations to higher dimensional rotations using learned skew-symmetric Lie algebras. If one defines rotations to be special orthogonal transformation, LieRE is the most general form of rotation encoding. However, while general, LicRE does not guarantee equivariance.
In this paper, we investigate the relationships across these different forms of positional encoding. In Section [ we mathematically show RoPE with parameterized rotation speeds to be equivalent to LieRE for one-dimensional data. When the number of positional dimensions is higher dimensional, LieRE is not guaranteed to be equivariant unless constraints are placed the Lie algebras. Using this insight, we derive Axial RoPE by imposing a "mutual exclusivity" constraint on the eigenvalues of LieRE's generators. Further, we will show that if one loosens this constraint requiring the Lie algebra to be commutative between the generator - then one arrives at Mixed RoPE. To be a relative encoding. this commutativity property is necessary 44 60]. thus making Mixed RoPE the most general form of LicRE which maintains equivariance. However, it has been noted that requiring the positional embedding to be relative is inductive bias whose necessity RoPE's success is unclear 15 24
The perceived necessity of equivariance has led to a circular argument where positional embeddings are assumed to perform well because they are relative, and all new embeddings must be relative because relative embeddings perform well. To break this cycle, we believe that it is imperative to establish the importance of equivariance embeddings for multi-dimensional RoPE. In Section + we propose alternative methods to establish a cause-effect experiment to evaluate whether equivariances a predominant contributor to RoPE's s faster training dynamics and generalization. To this end, we propose Spherical RoPE which takes non-ommmutaiive assumption, thus breaking equivariance, and Uniform RoPE, which maintains equivariance, but has only a single shared rotation speed.
In Section 5 we find that Spherical RoPE has the same training behaviors as its equivariant analogues and we find that Uniform RoPE outperforms the standard learned encodings, while performing worse than other RoPE methods. We conclude that our evidence suggests that the performance of RoPE over traditional embeddings is not explained by equivariance.
Background
In this section, we review concepts and notation from previous work on rotary positional embeddings. We introduce the methods in both historical and progressively general order which we will use to prove in Section that For a broader literature review on positional embeddings see Appendix C symbols, see Appendix D
that Mixed RoPE is the most general M-D rotary embedding with equivariance. For compact overview of
2.1 Attention
We use the standard attention mechanism from Vaswani et al. [73) given by
QKT
Z Attention Q, V) softmax
V.
(I)
Vole
We consider only single-headed attention to simplify notation, so here Q, K, and V are elements of I XN where T' the number of tokens and N is the network's latent dimension. We will primarily use index notation, where the above equation is expressed as, 2; = Σj= a(q,, ky) We define the attention mechanism, a(q;, k,), as
ks)):
(2)
Σ/=1 k)))

---

where what we refer to as the attention score is given by
a a(q. = q
(3)
This formulation of attention is equivariant to permutations of the token order. To break this symmetry, the position of the tokens must be "encoded" into the attention scores. Thus, we re-express the attention score as a function of the content of the query token x; E R and key token Xj E RR, and their positions ps. Pj E IR
Ozjy == (q kj) ==
= (x,, Xy. Pt.
(4)
Throughout this paper, we will abuse the notation of a and use these expressions interchangeably for ease of notation. the position affects the query and key directly, as in RoPE, we will introduce the notation (((o(I;;)))) (P)) for positional encoding function 4p.
2.2 Absolute and Relative Positional Encoding
Absolute Positional Encoding (APE) is common way of embedding token positions in transformers
by adding position-dependenn vectors, i.e. (xx p) == x + PE(p), where x is a token embedding, p is its position, and PE Z RR-V Previous work has suggested learning per-position token as PE 17 20). However, this restricts the network to fixed context length, removing the ability to extrapolate to different sequence lengths. The alternative is to add deterministic function to the embedding. Vaswani et al. [73 proposed to add Fourier modes,
sin (pwy): cos (DW[))). if mod 2
n mod 2 = 0
(E,
(5)
where n is a dimension within the positional embedding vector and wn. is a frequency term which increases with dimension. Note that this pairs elements in the embedding vector with each pair being transformed by the same frequency.
For ease in future notation, will use D == N/2 the number of pairs and interpret the embedded token as a x 2 tensor. One can also interpret this tensor as representing the coefficients a complex number, the first representing the real and second representing the complex part. Then can succinctly write this form of positional encoding as
(xx, = x + eup
where we use to indicate complex-valued vectors, & E CP, For this notation, we should also adjust the attention score for complex numbers,
aqq.k) Re [q k
where q W (x,
p). with Wg. W. E CDD and T is assumed to be the Hermitian transpose. With Eq. implied, we will continue with the notation Eq.
p) and k = W (&
Relative Positional Encodings Positional embeddings rely on being able to assign position values to each token. However, how assigns positions can often be arbitrary. One could just as correctly assign the first token the value zero and consider natural numbers, or assign the middle token of corpus zero and consider integers. We can relax the assumption of canonical way of labeling positions in APE by relying on relative distances between tokens, resulting in Oij- axx;, Xj. (y-pp) This is called relative positional encoding. We refer this property as embeddings having relative positional bias, or equivalently, having shift- equivariance (see Appendix E. for discussion of the cquivalence). In this manuscript, we will simply use the term equivariance with the implication that the attention score is invariant to shifts in the query and key.
2.3 Rotary Positional Encodings (RoPE)
There four common properties that are often preferred for positional embeddings: equivariance, key-query separability, linearity, and locality. For further details and why one may want these properties see Appendix B
From the properties, Rotary Positional Embeddings (RoPE) were derived by Su al. (69. Rather than adding positional embedding to the patch embedding, RoPE proposed to modify the queries

---

and keys by rotating them in pairs. By interpreting queries and keys as complex vectors, we can express this rotation as
(0 eawapa qd
(kks, p) swaP ld-
(8)
Since we assume the same operation is applied the queries and keys, from now on, will use to refer to operations which act on both. In matrix form, this is given by eleap can be represented as
(os(wwa)) sin(wap)
sin(wap) cos(wap)
dwap
Rwap:
(9)
where Ruap, is rotation matrix. While the rotation matrix is more intuitive, the complex exponential form will be useful for the mathematics in Section 3 so we will alternate between the two. Recall, we use the convention that real valued queries and keys will have dimension N and the complex interpretation will have dimension D.
One can represent the effect of RoPE as the application of block diagonal of rotation matrices,
pw1
21
RoPE(z.p) R
R
:
ipwz
2,
(10)
DD
where Zg is a query pair. We introduce this block-diagonal form as it was the notation used in Su et al. [69]. However, we will primarily stick to the index notation in Eq.
2.4 2D RoPE Embeddings
RoPE is constrained to operate on (ID) sequences. Motivated by the success of RoPE in language modeling, there have been growing efforts to extend it to multi molti-dimmnsional positions [10 [5], which outline below. We will use M to refer to the dimensionality of the position, but will primarily focus on images, where M = 2 and Pt. Pj E IR?.
Trivial 2D RoPE. One could trivially encode p (pz, Py) using rotation matrices Respet Reapy:
p(za. p) RRap, Rwap, qu (p.
(11)
However, in this case all positions with ps + Py e would get the same positional encoding Ruger
Axial RoPE. More practically, RoPE is extended to multiple dimensions by letting x and y act on different dimensions,
R
(Zzd p) =
0
R
Rpyea
Zdb
(12)
where queries and keys now split into four-dimensional vectors, 2
(x) 21 22
The block-diagonal matrix can once again be viewed as tensor of shape N/2M x M 2, where M once again the dimensionality position in this case M for horizontal and vertical position. This gives the index notation
kp (Z Pm) = R Pm.. 7d
(13)
for E {z, y}. From a programming perspective, one interpret this form of batched matrix multiplication.
While this method eliminates the symmetry, it treats x and as independent. The result is a separable
attention score of the form
Oij - 0 o (v)
(14)
(v) where a and 2y are components of the attention score which depend only on Pa and py- respectively. The frequencies are restricted to the axes, hence it is called Axial RoPE. This over- emphasizes horizontal and vertical relationships at the expense of oblique directions creating gridded patterns shown in Figure [ To represent oblique patterns, the rotations would have to be performed along directions that contain both an x and y component, e. frequencies that are not aligned on the axis in F These frequencies have been referred to "mixed frequencies" (26).

---

A
B
Axial RoPE
W
x Axcil RoPE
Unifform Aotil Rope
Miseed RuPE
Learned Avist RoPE.
Mixed RoPE
Z
W
Figure 1: The attention patterns of Axial and Mixed RoPE. A. Each dimension pair in the query and key vectors is rotated based the position creating attention pattern. The pixel value of the attention pattern is oxxy, X3, P, 0), where attention pattern of individual component-pairs in the embedding vector is shown and, the right, the components are combined into the overall attention pattern for a randomly sampled query and key vector. B. Location of the rotations frequencies in 2D frequency space. Axial RoPE can only represent frequencies that lie on an axis resulting in the grid-like attention patterns. Unlike Axial ROPE, Mixed RoPE can assign different directions to each component-pair (A Bottom). When Axial RoPE uses fixed frequencies. the frequencies are spread exponentially. However, they can be implemented as learnable parameters. For Uniform RoPE, all frequencies are fixed to single value for each axis.
(i. - the pixel location. On the left, the
Mixed RoPE: Learned mixed frequencies. The inclusion of mixed frequencies has empirically been shown to positively impact learning and generalization (26). The naive approach in Eq. [ is only a problem when x and y rotate by the same frequency in every dimension. One could instead parameterize the frequencies with two separate frequencies in each dimension,
(zzz)) = R Rwayy 9d Rwa. qdl
(15)
By making the wi and Wy parameters learnable, the attention pattern learn mixed frequennryy patterns by constructing a superposition of different diagonal patterns, as shown in Figure
LieRE. Recently, RoPE has been interpreted through the lens of Lie algebras 44
60
For an
intuitive introduction how Lie algebras appear, see Appendix FE.3] Lie Rotary Position Encodings (LieRE) [52) extend Mixed RoPE by applying N-dimensional rotation matrices, rather than 2 x 2 matrices applied to pairs, using a linear combination of learned skew-symmetric Lie algebras,
(16) where exp is the matrix exponential, and the terms are x skew-symmetric matrices - which are Lie group generutors of subgroup of SO(N). Mathematically, LieRE is the most general rotary-based embedding method as skew-symmetric matrices are the generators of any N-D rotation. However, unlike the other two methods, LieRE is not guaranteed to be equivariant.
(z,, = exp( AgPy)a,
3 The Generality of Learned RoPE and Mixed RoPE
While LieRE is motivated as generalizing RoPE M-D rotations, in this section will show that LieRE in one dimension can be learned by implementing RoPE with parameterized frequencies. For M-D positions, LieRE
equivariant unless the generators commute. If the generators required to commute, we show that LicRE can be re-expressed as Mixed RoPE. Thus, we conclude Mixed RoPE be general solution for M-D equivariant rotary embeddings. In this section, will give informal proofs focused on high-level insights.
3.1 ID-LieRE equivalent to ID RoPE with learned frequencies
In this section, we prove that any one-dimensional LieRE can be expressed as RoPE with param- eterized rotation frequencies. Thus, we conclude RoPE to be computationally efficient way of expressing D-dimensional rotation, i.e., ID-LieRE.
5

---

Proposition 1. Any ID-LieRE can be parameterized by RoPE with learned frequencies.
To why Proposition holds, suppose have ID-LieRE embedding with learned generator A. By formulation, A is skew-symmetric, A -A. The positionally encoded attention between query q W and k Wix
a( Ty, = (exp(Ap,)))
exp(Apx)k.
(17)
Any kkew-symmetric matrix has an eigenvalue decomposition A = UA where Ay a diagonal matrix of purely imaginary (or zero) eigenvalues and U is unitary matrix, U U I. Moreover, the matrix-expee expriiiexponential of an eigenvalue decomposition simplifies to cxp(UA,7U) = cxp(Ar)U*.
This allows to express attention
a(z,, (eii))=--UuXx-xpppe U
(18)
=q" exp(-P,AA) exp(pAAy))
(19)
where q' = W with W = UW 4. and k' = Wix with W. U We. Because the eigenvalue oal matrix diagonal, the exponential is given by
Aop
edp
exp(pAz)=
(20)
O e e
Notice that this is the same as the complex formulation of RoPE defined in Eq. 10 where the eigenvalues of the generator correspond to the rotation frequencies of the rotation matrices. Thus, any ID-LieRE can be expressed as RoPE with learnable frequencies by absorbing the matrix of eigenvectors of A into the weight matrices Wa and Wk. Since ID-LieRE learns rotation in SO(D), RoPE be seen an efficient way to represent rotation in R.
3.2 Extending RoPE to more than dimension
While this proof works for ID positions, it does not generalize to M-D without introducing extra inductive biases giving up equivariance. By imposing constraints on Ar and Ay. categorize the other RoPE methods based on the assumptions made.
Generators rotate independent subspaces. For example, one impose the assumption that Pz and py rotate independent subspaces R.N. Mathematically. this assumption would imply that
Ad E [1, D) (()) = 0 or (1) = O,
(21)
where ( and x are the eigenvalues of Ar and Ay. respectively. This equivalent to rotating independent components of the query/key as done by Axial-RoPE.
Commutative generators. For LieRE be equivariant, we only need to ensure that the generators commute. If we make this assumption, then we arrive at Mixed RoPE.
Proposition 2. Any M-dimensioonll LieRE with commutative generators be parameterized by Mixed RoPE
To why Proposition 2 holds, suppose diagonalize A.r U.A.UU and A, U.A.U.T we take the assumption that Az and Ay commute,
A, A,
A, AAy Anl =.
(22)
Ag is the Lie bracket. This implies that Ar and Ay simultaneously diagonalizable (Lemma) Thus, commutativity implies that U = U, = U. We can write
A exp(App= A,pb)) U Uexp(A.pr UT
(23)

---

one-dimensional
multi-dimension
RoPE
Axial RoPE
Mixed RoPE
Spherical RoPE
P
P
quick texin
Figure Diagram of each rotary embedding's effect on the subvector, z While Mixed RoPE affects 2D vector pairs, Spherical RoPE affects 3D vector triplets. Axial RoPE rotates independent dimensions for Pas thus containing pairs of pairs, or effectively quadruples. Each z contains D sub-vectors rotating at different frequencies. While the order in which the rotations are applied does matter for Axial or Mixed RoPE, order matters for Spherical RoPE. Explicitly, the triplet first rotated around the axis associated with pa and then rotated around the axis associated with Py-
Thus, Mixed RoPE forms general solution for assumptions of commutativity. which is necessary for LieRE to be relative. This also mathematically shows Mixed RoPE to be the strict generalization of Axial RoPE.
which leads to Eq. 15
In summary, learned frequency RoPE embeddings represent an efficient way of learning much more general set of SO(D) than is commonly believed. However, in order to generalize to higher dimensions while retaining their status as relative positional encodings, assumptions must be made. Mixed RoPE generalizes this to M-D positions, spanning the entire solution class for relative LieRE. Thus, LieRE-like methods with commutative Ar and A,, such as Schenck et al. [60] are not more expressive than Mixed RoPE and any empirical differences in performance must be attributed to the learning dynamics due to different parameterizations. However, remains unclear whether equivariance the real reason for RoPE's success.
Experiments
When extending RoPE to more than one dimension, we must either constrain ourselves to commuting Lie algebras or give up relativity. We therefore ask the question: Why does RoPE work? Which properties should preserved for generalizing RoPE to vision? To explore this question, we propose two RoPE variants: Spherical RoPE, which takes non-commutative assumption, and Uniform- Frequency RoPE, which uses a single fixed rotation frequency across all dimensions. Below we provide high level outline the different embeddings. We compare the existing positional embedding methods APE [77,, Axial RoPE [3]. Mixed RoPE [10], and LieRE [52] to these two new variants to understand whether equivariance, oblique directions variety of spatial frequencies are important features of PEs for vision.
Spherical RoPE.
We propose Spherical RoPE as a method between Mixed RoPE and LieRE that minimally changes 2D RoPE to break equivariance. Spherical RoPE embeds position
p(za. p)
Vway Rusayy
(24)
where qu E 3 is now triplet instead of pair, and y is block diagonal of x 3 yaw matrices and R is block diagonal of roll matrices.
(os(waat)) sin(warx))
sin(was))) cos(w)
I cos cos(wayy)) sin sin(way))
Vwar
0
Rway
coo
sin sin(wayy) cos(way))
(25)
0
0
Intuitively, rather than RoPE rotating around a circle, Spherical RoPE rotates around a sphere using Euler angles.
7

---

Table I: Table listing the properties of each of the rotary-based methods.
Importantly, spherical rotations like LieRE are concommmutative mak- ing them not equivariant. In fact, their generators are strictly non- commutative, A, * Ay- While non-commutativinn does not mean Spherical RoPE is incapable of learning or approximating equiv- ariance throughout the network, is the component of LieRE re- moved by Mixed RoPE and works which enforce commutativity such Yu al. 85 and Schenck et al. 60
Positional Encoding Vision
Strietly Equivariant Directions Learning
Oblique
Requires
Rotary (RoPE) 69
X
N/A
X
Axial RoPE 68
/
X
x
Mixed RoPE 26 LicRF. 52
/
V
/
Spherical RoPE Uniform RoPE
X
X
X
We hypothesized Spherical RoPE to have a number of advantages. While Axial RoPE unable to express oblique directions, Spherical RoPE can. Like Axial RoPE, Spherical RoPE can use fixed frequencies making it computationally cheaper than LieRE and Mixed RoPE since sines and cosines of the frequencies can be precomputed. However, our main interest is that Spherical RoPE is comparable in terms of expressivity to Mixed and Axial RoPE while being non-equivariant.
For an initial evaluation on the impact of relative position, we propose Suiform-FFequeent RoPE. For this method, we perform Axial RoPE with single frequency shared across all rotation matrices. While still being relative, this serves as more restricted version of RoPE. If this method performs significantly worse than other methods, it indicates the importance of having a range frequencies. We implement uniform frequencies for Axial RoPE to gauge against relative importance of equivariance.
Uniform-FFrquencc RoPE.
In one extreme, the rotation frequency could be zero resulting no changes to the queries and keys. In the other extreme, the frequency could be set very high resulting in large changes to the queries and keys. As a note, is the frequency relative to the resolution of the image that is important. Frequencies higher than the sampling rate are equivalent to low frequencies. To ensure every position has unique encoding, we fix the frequency to perform one rotation cycle across the entire image.
Datasets and architecture. We test the different PEs on CIFARI00 [39] and ImageNet [58] using a standard Vision Transformer - the VIT-S implementation from the timm [79] library. For Learned APE, we use the baseline VIT-S which uses learned positional encodings rather than sinusoidal. We follow much of the DeiT-III training procedure proposed in Touvron et al. [72). However, for ImageNet, we do not use dropout, MixUp, or CutMix as we observed that they significantly increase the number of epochs necessary for convergence. For ImageNet, we evaluate models trained after 200 epochs and 400 epochs for CIFAR 100. We evaluate without any hyperparameter tuning directly on the validation sets. For further details on hyperparameters and experimental setup, see Appendix H Error bars were created using three models with different random seeds.
Generalization to larger image sizes. We also perform an experiment to test how well different PEs generalize across image sizes. Our approach to this experiment follows prior research [2 52 The learned embeddings in Learned APE cannot be extrapolated, so we interpolate new embeddings when changing the number of patches. For RoPE embeddings, parameterize position such that the top-left corner of the image corresponds to Pa. = Py = -r and the bottom-right corner correspond Pa: Py with all other positions are evenly spread between the two for training. When increasing the image size, we extrapolate by scaling the range by the ratio of the new image size the training image size while keeping the patch size constant.
take square dimensions and
Additional Evaluations Additional evaluations can be found in Appendix including method speeds, experiments with smaller data splits, a segmentation task, and evaluation of the learned weights.
8

---

Table 2: Performance comparison (top-1 accuracy) across datasets and methods.
Top-1 Accuracy (%)
Fixed Encuding
CIFARIOU
ImageNet
Learned APE
64.2an
72.7 201
42.
an2
Unitoem RoPE (Our Ablation)
70.5
74,9 ana
50
Sphorical RoPE (Our Ablation)
73.2 ap3
Leamed Sphor cal RoFE
Learned Encuding
Leamed Astel Roft
Mixed RIPE
Learned Axial RoPE
72.9
75.7 2014
Unilorm
LieRE
73.
Mixed RoPE
74,7ana
77.4 201
Learndd Spherical RoPF (Our Ablation)
77.4
150 200
test Imagge resoxition [oxl
250
300
350
400
450
500
Figure 3: Dependence of accuracy on image resolution for ViT-S with various positional embedding methods on ImageNet-Ik, Error bars reflect the standard deviation across three models trained with different seeds.
Results
To evaluate the importance of different properties of positional embeddings in vision transformers, we trained the same VIT with different positional embeddings on CIFAR100 and ImageNet- IK. We start by evaluating the models images of the same resolution during training. If equivariance is important, we would see Axial and Mixed RoPE to perform better than Spherical RoPE, which lacks strict equivariance. On the other hand, if oblique frequencies important, then would observe Mixed and Spherical RoPE to do better than Axial RoPF, which does not capture oblique directions. We find that the lack of equivariance does not hinder Spherical RoPE. It outperforms Axial RoPE and performs comparably to Mixed RoPE. Moreover, we would expect equivariant methods be especially effective in the low data regime. However, in Appendix [ we observe Learned Spherical RoPE performs the best despite its lack of inductive bias. This suggests that the benefits to performance and generalization on ImageNet for Mixed RoPE may be due its extra parameters. However, Axial RoPE and Uniform RoPE perform significantly worse suggesting oblique directions be more important than equivariance.
When comparing with absolute positional encodings, we observe that all forms of RoPE perform better than learned APE (Table This includes Uniform RoPE, the variant that uses only single frequency. Moreover, all forms of RoPE using diverse frequencies outperform Uniform RoPE and have similar performance (whether they are learned or not), suggesting that diversity of frequencies is important. Spherical RoPE adheres much closer to the vectorized implementation of other RoPE methods than LieRE. As our goal was primarily identify the most impactful properties of M-D RoPE and not maximize accuracy, none of our conclusions depend on precise performance numbers for LicRE.
Last, we asked how well different PEs generalize across image sizes. Equivariance is often thought to aid model generalization. However, when evaluating each model using higher resolutions images, i.e. increasing the number of patches, we found Spherical RoPE to be the most effective method (Figure B), suggesting equivariance may not be the reason for RoPE's generalization.
Discussion
Because see very little variation between Spherical RoPE and Mixed RoPE, we conclude that equivariance is only a minor contributor to the increased performance seen by RoPE for vision. In fact, Spherical RoPE appeared to extrapolate to higher resolutions better than Axial RoPE. This could suggest that oblique frequencies are important for extrapolation. However, extrapolation is only done on short length scales, this may not hold in language.
There are two important differences between vision and language transformers: context length and patch variation. Where LLMs have on the order of 128K context windows [22]. vision transformers

---

only have 16 x 16 patches. Moreover, patches have more variation as tokens than language tokens, thus allowing the content embeddings to store information about the relevance of oblique directions. Because the context size is small, we hypothesize that there could be methods that perform better than Mixed RoPE and are more general than LieRE for vision. While LieRE was proposed with skew-symmetric generators to generalize RoPE to N-D rotations, Lie algebras do not have to be skew-symmetric. The skew-symmetry is important for maintaining numerical stability over long contexts [69]. However, skmm-symmelry also results in Proposition 3.1 of Barbero et al. [5), which proves RoPE to be non-local. Since the context size is small for images, numerical stability is likely not issue, thus freeing the space of Lie algebras available to including Lie algebras that encourage locality.
We observe decrease in generalization when using uniform frequencies. This finding qualifies Barbero et [5]ss hypothesis that the various semantic lengths contribute to RoPE's performance. However, Uniform RoPE outperformed learned APE, suggesting the reason why RoPE performs well is not among the properties we tested. We speculate this could be a flaw in additive positional embeddings themselves; additive methods create a trade-off between the magnitude of position and content forcing tokens that vary significantly with position to have lower magnitude to be closer to the origin.
7 Conclusion
We conclude that Mixed RoPE a very general solution for M-D data if equivariance a necessity, However, we see little evidence that strict relative positional bias is impactful for vision transformers. However, RoPE methods have still been found greatly improve performance in ViTs. Thus, we conclude that evidence suggests that RoPE does not need strict equivariance constraints to boost performance over APE methods.
8 Acknowledgments
This project has received funding from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (Grant agreement No. (0101166)) We gratefully acknowledge the computing time granted by the Resource Allocation Board and provided on the supercomputer Emmy/Grete NHR-Nord Göttingen part of the NHR infrastructure (project nim00012).
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):493 500, 2024.
[2] Meta AI. Llama 4: Multimodal language models, 2025. BtLL:ttss:://// http://aaameaaaa.aeaam meta. co..tt/g/cm//b//////ooooo www.tttioommiiieee liinncc///// Accessed: 2025-05-15.
[3] Giorgio Angelotti. Hype: Attention with hyperbolic biases for relative positional encoding. arXiv preprint arxiv:23 10. 2023.
[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 0473, 2014.
[5] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06225, 2024.
[6] Erik Bekkers, Sharvaree Vadgama, Rob D Hesselink, Putri A Van der Linden, and David W Romero. Fast, expressive (n) equivariant networks through weight-sharing in position- orientation space. arXiv preprint arXiv:231.009970. 2023.
[7] Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang. and Phil Wang. Rotary embeddings: relative revolution. blog. blog.eleuther .ai/ 2021. [Online;
eleuther
accessed
10

---

[8] Johannes Brandstetter, Rob Hesselink, Elise van der Pol. Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:21 10.02905, 2021.
[9] Johann Brehmer, Sönke Behrends, Pim de Haan, and Taco Cohen. Does equivariance matter at scale? arXiv preprint arXiv:2410.23/79, 2024.
[10] Yiting Chen and Junchi Yan. What rotary position embedding tell us: Identifying query and key weights corresponding to basic syntactic or high-level semantic information. In The Thirty- Annual Conference Neural Information Processing Systems, 2024.
[11] Ta- Ta-Chung Chi, Ling-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kemel-
ized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386-8835 -3996 2022.
[12) Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. arXiv preprini arXiv:2212.10333 10356, 2022.
[13] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[14] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv: 102. 10882, 2021.
[15] Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. Visionllama: A unified llama interface for vision tasks. arXiv e-prints, pages arXiv-2403, 2024.
[16] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16 35:16344416359 2022.
[17) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.1199 19299, 2020.
[18] Gamaleldin Elsayed, Prajit Ramachandran, Jonathon Shlens, and Simon Kornblith. Revisiting spatial invariance with low-rank local connectivity. In International Conference on Machine Learning, pages 2868-2879. PMLR, 2020.
[19] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-133) 2015. doi: 12.1007//11223-01444
[20] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Con- volutional sequence to sequence learning. CoRR, abs/1705.03122, 2017. URL http
//axxiv.rr//ab//11 0ttp//155500022
[21] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint 24X. 2024.
[23] D.J. Griffiths. Introduction to Quantum Mechanics. CUP, 2018
[24] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203.16634 2022.
[25] Yu He, Cristian Bodnar, and Pietro Lio. Sheaf-based positional encodings for graph neural networks. In NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, volume 9, 2023.
[26] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289-305. Springer, 2024.
11

---

[27] Mohammad Mohaiminul Islam. Rishabh Anand, David R Wessels, Friso de Kruiff, Thijs P Kuipers, Rex Ying. Clara Sánchez, Sharvaree Vadgama. Georg Bökman, and Erik J Bekkers. Platonic transformers: A solid choice for equivariance. arXiv preprint wrXiv:2510.035//1 2025.
[28] Albert QJiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 1,4088, 2024.
[29] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):58335555 2021.
[30] Pentti Kanerva. Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors. Cognitive computation, 1:139-159, 2009,
[31] Pentti Kanerva. Hyperdimensional computing: An algebra for computing with vectors, Advances in Semiconductor Technologies: Selected Topies Beyond Conventional CMOS, pages 25 42. 2022.
[32] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learming. pages 5156-5166 5165. PMLR, 2020.
[33] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy. Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 01-24492-244229 2023.
[34] T Anderson Keller and Max Welling. Topographic vaes learn equivariant capsules. Advances in Neural Information Processing Systems, 34:28585-28559 2021.
[35] Anderson Keller and Max Welling. Neural wave machines: learning spatiotemporally struc- tured representations with locally coupled oscillatory recurrent neural networks. In International Conference on Machine Learning, pages 16168-16189. PMLR, 2023.
[36] David Knigge, David Wessels, Riccardo Valperga, Samuele Papa, Jan-Jakob Sonke, Erik Bekkers, and Efstratios Gavves. Space-time continuous pde forecasting using equivariant neural fields. Advances in Neural Information Processing Systems, 37:76553-7655 -65777, 2024.
[37] Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. aat--translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems, 34:6417-6429, 2021.
[38] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Pro- cessing Systems, 34:21618-21622 2021.
[39] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[40] Christopher J Kymn, Denis Kleyko, E Paxon Frady, Connor Bybee, Pentri Kanerva, Friedrich T Sommer, and Bruno A Olshausen. Computing with residue numbers in high-dimensional representation. Neural Computation, 37(1):1337, 2024.
[41] James Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.0388 2021.
[42] Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 33:4465-4478. 2020.
[43] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[44] Haiping Liu and Hongpeng Zhou. Rethinking rope: A mathematical blueprint for n-dimensional positional encoding. arXiv preprint arXiv:2504.063008 2025.
[45] Xuanqing Liu, Hsiang-Fu Yu, Inderjit S. Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. CORR, abs/22003.0922, 2020. URL.
https: / aariiiiv. org/ abs////3....//////////////aa/ 09229
12

---

[46] Yunwu Liu, Ruisheng Zhang, Tongfeng Li. Jing Jiang. Jun Ma, and Ping Wang. Molrope- bert: An enhanced molecular representation with rotary position embedding for molecular property prediction. Journal of Molecular Graphics and Modelling, 118: 108344, 2023.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[48] Zikang Liu, Longteng Guo, Yepeng Tang, Junxian Cai, Kai Ma, Xi Chen, and Jing Liu. Vrope: Rotary position embedding for video large language models. arXiv preprint arXiv:2502.1 1664, 2025.
[49] Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex valued autoencoders for object discovery. arXiv preprint Sttiv:2204000775 2022.
[50] Sindy Löwe, Phillip Lippe. Francesco Locatello, and Max Welling, Rotating features for object discovery. Advances in Neural Information Processing Systems, 36:59666-59335, 2023.
[51] Takeru Miyato, Sindy Löwe, Andreas Geiger, and Max Welling. Artificial kuramoto oscillatory neurons. arXiv preprint arXiv:2410. 18221, 2024.
[52] Sophie Ostmeier, Brian Axelrod, Michael E. Moseley, Akshay Chaudhari, and Curtis Langlotz. Generalizing rotary position encodings. arXiv preprint arXiv:2406. /0322, 2024.
Liere:
[53] Wonpyo Park. Woonggi Chang, Donggeon Lee, Juntae Kim, and Seung-won Hwang. Grpe: Relative positional encoding for graph transformer. arXiv preprint arXiv:2201.12787, 2022.
[54] Ngoc-Quas Pham, Thanh-Le Ha, Tuan-Nam Nguyen, Thai-Son Nguyen, Elizabeth Salesky, Sebastian Stüker, Jan Niehues, and Alexander Waibel. Relative positional encoding for speech recognition and direct translation. arXiv preprint 009900 2020.
[55] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
[56) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14)): 2020.
[57] Ali Rahimi and Benjamin Recht. Random features for scage-scale kernel machines. Advances in neural information processing systems, 20, 2007.
(58) Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::211-252, 2015.
[59] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural PMLR, 2021.
networks. In International conference on machine learning, pages
[60] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, al. Learning the ropes: Better 2d and 3d position encodings with string. arXiv preprint arXiv:2502.02555 2025.
[61]
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):331-139 1992.
[62] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pages 9377-9388. PMLR, 2021.
[63] Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey. Medical image analysis, 88: 102802, 2023.
[64] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint VrXiv:1803.02155. 2018.
[65] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2020. URL.
http:// //wpiv../ivv.v// org httg/abs/20000 2ttps///000000666v1
13

---

[66] Ben Sorscher, Gabriel C Mel, Samuel A Ocko, Lisa M Giocomo, and Surya Ganguli. A unified theory for the computational and mechanistic origins of grid cells. Neuron, 11((1))21-137, 2023.
[67] Steven H Strogatz and Ian Stewart. Coupled oscillators and biological synchronization. Scientific american, (69(6):002-10)) 1993.
[68] Jianlin Su. Transformer upgrade road: 4. rotating position coding of two-dimensional position,
May 2021. URL bttps:/ts:///e fm/ httprrrrchiieeeeeeeeeeeeeeeeeeee 8897 Accessed: 2025-04-28.
[69] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 127663, 2024.
[70] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances neural information processing systems, 33:7537-7547, 2020.
[71] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.085 088002 2024.
[72] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European conference on computer vision, pages 516-533. Springer, 2022.
[73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[74] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 17...//////...0.. 10903, 2017.
[75] Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404. 12224, 2024.
[76] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing Dision-languall model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024.
[77] Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang. Yuhang Cao, Jian Tong. Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding? arXiv preprint arXiv:2502.05// 205.50002.77.. 2025.
[78] David R Wessels, David M Knigge, Samuele Papa, Riccardo Valperga, Sharvaree Vadgama, Efstratios Gavves, and Erik Bekkers. Grounding continuous representations in geometry: Equivariant neural fields. arXiv preprint arXiv:2406.0555. 2024.
[79] Ross Wightman. Pytorch image models.
pytorch-image-an models 2019.
[80] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022. ISSN 1467-8659. doi: 10. 10.1111/cgf. 14505.
[81] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:I810. 00826, 2018
[82] Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy. arXiv preprint arXiv:2501. 18795, 2025.
[83] Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In Interna- tional conference on machine learning, pages 7134-7143. PMLR, 2019.
[84] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint
2019.
14

---

[85] Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, and Chun Yuan. Comrope: Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices, 2025. URL https
//arxiv. orgooaggagoo
/tt//org/////b//eeeeeeeeee abs/ 03737
[86] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032. 2019.
[87] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
15

---

A Broader Impact
This work is fundamental research. While this work could lead to the discovery of better positional encodings and higher performing visual foundation models, the positivity or negativity of this impact determined by the downstream task and not this work.
B
Limitations
While our results do not show relative embeddings to be detrimental, we believe them to be evidence that equivariance is not the reason for RoPE's success. However, our experiments were performed in Vision where the number of tokens is limited compared to the long context lengths of NLP. Moreover, the datasets are not what many believe to be "at scale". While Spherical RoPE and LieRE would intuitively be favored at scale over Axial RoPE, as they have less inductive bias, it is unclear whether inductive bias and equivariance is favored at scale [9].
It has also been shown that vision is not purely equivariant task and benefits from relaxed equiv- ariance [18. Our results do not show that equivariance is not useful in tasks that are grounded in physics and obey strict symmetries.
C Literature Review
C.1 Natural Language Processing
In natural language, positional encoding has been used to break the permutation, "bag of words", symmetry [73). Although this could be done by learning a vector per position, this is both memory- expensive for large context sizes making it practical apply only the first layer. Moreover, does not allow for extrapolation at test time to context sizes beyond training. Thus, is favorable to perform positional embeddings with predictable deterministic function. One way of doing this is to make the attention relative with local receptive fields, as is done implicitly in convolutional neural networks 12]. Sinusoidal positional embeddings were proposed due approximate local and shift- ihitt-invariant properties of Random Fourier Features [57). Since sinusoidal, other methods have been proposed get guaranteed shift invariance by explicitly parameterizing based on distance 164, [54] 55]. However, these methods require positional embedding for every pair of positions which is not supported by many of the efficient attention optimizations such as Flash attention [16 3
Rotary Positional Embeddings (RoPE) have become the staple in NLP having recently been adopted
by many of the large language models 776 22 71 43 28). However, these methods also use causal masking, which has been shown allow models with no positional embedding to recover absolute
position 24 82 75 33
This has led to questions on the importance of relative position [
In language, there has also been extensions to RoPE proposed through NTKs and kernel methods [I]. However, these methods have not, to our knowledge, seen use in vision.
C.2 Vision and Video
Vision transformers were introduced in Dosovitskiy et al. 17 and, though they tried sinusoidal position encodings, found learnable position encodings to perform best. For convolution -eeque models such as SWin transformers, relative positional encodings have been popular [47 14] More recently, RoPE has been shown to an efficient and simple way to have relative embeddings and has been extended to 2D using Axial and Mixed RoPE. Going beyond 2D to Video data, Axial RoPE has become increasingly popular. The extension was first attributed to Wang et al. (76) as RD-RoPE or M-RoPE, leading to two separate Video-RoPE papers from Wei et al. [77] and Liu et al. 48 Both of these focus on the order of the position enumeration and interleaving positions. However, this should not be problem if frequencies
not deterministic, or if frequencies indexed by both d and modality m as done in Eq [3 We highly recommend using either Mixed RoPE or LieRE which
extend naturally for videos.
LieRE embeddings have thus been the most general form of RoPE to N-D. However, Schenck et al. [60] has claimed the method to have large memory footprint and proposed STRING. This
16

---

paper, preprint released concurrently with the writing of this manuseript, follows much of the same math as this paper. However, they did not recognize that an orthogonal matrix implicitly learned by the query and key matrix. Moreover, their method relies on commuting Lie algebras. From our insights in Section 3 their method can likely viewed as a slower implementation of N-D Mixed-RoPE.
is also worth noting that positional encodings have also been explored within vision through the area of Neural Fields [80]. Traditional coordinate MLPs have been found to be biased toward
low-frequeney functions [7] leading to more advanced positional encodings such as Random Fourier Features [57] or sinusoidal activation functions [[6]] These implicit functions have been used to encode attention and message passing in graph neural networks with recent work being put in to
make these functions equivariant to symmetry transformations [59 g 36].
C.3 Graphs and Al in Science
Graphs are limited in their expressivity up the Seisfeiler-Lehmare (WL) graph isomorphism test [81], so positional encodings can break the isomorphism symmetry [25] 83]. Within this community, they propose spectral attention and graph Laplacians for positional encoding B8]. These methods seem extremely close to our analysis of RoPE, but from a very different perspective. We show that the frequencies of RoPE can be interpreted the eigenvalues of orthogonal transformation by taking the spectral decomposition.
Positional encodings are well studied within graph neural networks 42 53
In overlapping vein, relative position encodings have been studied in terms of equivariant graph neural networks, often for scientific disciplines such as molecular physics 88 62) or drug discovery [29] One method to achieve equivariance is through defining relative coordinate frames [37]. This corresponds to the learned relative positional method described in Shaw et al. [64], but can be generalized to higher dimensions and different transformation using bi-invariant distance functions The pesssage-passingg functions of these works correspond to a generalization of attention
36 78 21
scores
However, even in these tasks with physics-grounder grounded symmetries, the need for equivariance is hotly debated. While AlphaFold (29) was originally touted as the example of the success of equivariant inductive biases in science, AlphaFold [ explicitly stated that they benefited from removing this inductive bias at scale. However, while the harm of inductive bias at scale is the prevalent zeitgeist, it not an established fact 9..
C.4
Computational Neuroscience
Coupled oscillators have become growing area of interest within computational neuroscience
34 35 67]. By observing the projection of the RoPE circles onto the real axis, one can interpret RoPE as time progression in D uncoupled, undamped harmonic oscillators. This perspective naturally connects RoPE to Löwe et al. [49 's series of papers on complex autoencoders and their extensions
50 51
In another, vein of research, there has been some work in dyper-dimensional computing (30 31] in Phasor and Residue VSAs [40] which represent concepts rotations around unit circles in high- dimensional spaces. These representations have strong connections with RoPE. Additionally, progress has been made in hypothesizing how biological neural networks encode positional knowledge with hexagonal grid cells, which can be represented as a discrete sum of three periodic functions oriented the cubic roots of unity|66)
C.5 Generality of RoPE
The generality of RoPE has been found by others. Schenck et al. 60 Su
68
and Liu and
Zhou 44 all propose proofs similar Proposition [ However, Schenck et al. 60 miss that the orthogonal transformation can be incorporated into key matrix. Liu and Zhou 44 the assumption of reversibilily, which leads to the independent eigenvalue assumptions of Axial RoPE. All three works take the assumption of an abelian subgroup i.e. commutative generators, but miss the generality of Mixed RoPE. While Su 68 propose quaternions i.e. spherical rotations as a direction, they immediately dismiss it as no-go because they lack equivariance. This exemplifies the
and Su 668 take
17

---

"circular argument," " where equivariance is assumed to be necessary because work will not investigate non-equivariant positional encodings because equivariance is necessary.
Because our derivation was found independently of these works and the previous works are, to our knowledge, not published, we have left in Proposition [T] We would like to acknowledge their work, but retain the flow of this paper.
D
Notation
Symbol / Term
Dimension
Meaning
Notes
R
Patch/token/cootent vector token
Raw input embedding
Abstract content of token
Raw imput embedding
Pi
Rd
or
Position of token i, can be M/D or ubstract P
Scalar (ID) or vector (2D)
711
Modality index
e.g., 2, 3, time
M
2.
Number, space, Modalities
D
2
Hidden dimension
Number of puinstriplessqqudnnles
T'
2.
Number Tokens
W W.. W.
RTXD
Query, Key, Value Matrices
9
g
q:
Query vector
RN
k,
W
Key
R
W.
Value vector
K. V
Rxxx
Query, Key, Values
T tokens, D latent dimensions
((z.p)
X
Positional Engoding function
2
RTRN
Output of Attention
Z Attention Q. K. V) Softmax of attention scores
a(t,))
X
Attention Weight
alq, k)
R
Attention score
Inner product
Wa/ha
R
Rotation frequency for dimension
Equivalent to eigenvalue generutor
qu
2//9/4 D2
Query pairtripleoquarrulle at dimension d
After RoPE or LicRE applied
Rup
rotation matrix
Rotation based froquency and position
Table 3: Summary of Notations and Key Concepts
18

---

Positional Eacoding
Vision
Learned
Extrupolation
Separable
Relative
Linear Flow
Used
Absolaze (Sinusodda))
dix
Transdomet
23
(L.cumed)
GPT,
Relative (Learned)
Trasdoonier
Rocary (RoPE)
Contemporary LLMs k5
Axdil RoPE
JIX
VisceLLaMA
15
Qwes2 76
VideoRoPP
Mixall RoPF
LicRE
Sphemial KoFP.
718
Clos
Uniform RoPE
VIX
Ous
Table Comparison of positional encoding methods in transformer models. "NoPE makes some properties trivially true.
E Positional Encoding Properties
Rotary positional embeddings were derived in Su et al. [69 by drawing equations from assumed properties. While these appear as arithmetic assumptions and equations in their work, we formalize what properties these assumptions imply and why we may choose these assumptions this section. In their paper, derive their equations. they equivariance (relativity), query-key separability of the positional encoding. lincarity and incompressability, locality, and query-key symmetry.
1. Equivariance/Relatiiiyy Attention score should be affected only by the relative position of two tokens, e. have the form
alz,, Fz Ps. Ps) á(z;, zz Pp))
(26)
2. Key-query seperability: The positional encoding, i2, of the query should depend on the position of the key
ofz,, Pt. a((((z), =d)((E)))))( P2))
(27)
3. Linearity: The positional encoding should linear flow, see Appendix E.3
Namely,
((p(2, p,) = p(x, p4 + Pj).
(28)
4. Locality: The attention score between two tokens should decay with distance
lim 1e-pp/t+00
axx,, 2y Py. Pt.
(29)
E.1 Relativity and Equivariant
We use the term equivariant interchangably with relative. Strictly speaking, one should specify the transformation or group you would like to be relative to, e.g. shift/rotation or SO As previous literature always refers to relative positional bias in terms of shifts/translations. in the main text, this what we mean. because we would like to refrain from using the term "relativity" to describe the property of being relative PE too often due to connotation within theoretical physics. First, we define relative case of positional encodings in language
use the term equivariance to be the generalization of relativity beyond language
a jr Pi. Pÿ) 6( PP))-
(30)
In the rest of this section, we mathematically explore where this equation comes from.
The behavior we are trying to capture that if we renumber the words in the sentence, it should not affect the attentions score. Intuitively, if text is padded with spaces the beginning, that will not have a significant effect on the meaning of the sentences. We can ensure this by colloquially saying that the attention between two words should depend strictly speaking this is not a proper distance, since it can be negative; it is, instead, signed distance function. Though this may seem pedantic one dimension, two dimensions defining a distance function is less unique. For example, one may choose L1 or Le distance metrics. Because distance functions are more nebulous, it makes more sense to define relative terms of the transformations that we would like our attention score to be independent of.
the distance between them. Notice, that
TI. P,) 2(I,,,,,,(p)) T(;;))
(31)
19

---

These transformations can be combined to generate a set of transformations which leave the attention score unchanged, or symmetric. This set has the mathematical properties of group and known as a symmetry group. We can index transformations by elements in the symmetry group, 9 E G. and let the elements act on
a(T; 25. Pi. a T3. 9-Pe. 9-P...ppp)
(32)
As an example, g could represent an angle, 8, and it may act on a vector p as a rotation g.p Rop.
Connecting everything back to Eq. 30 Noether's 's theorem states that any continuous symmetry can be expressed as a conservation law. This allows us to introduce bi-invariant function 36 or "Noether charge", [p(x, Pj), that is invariant under the group action,
B(ps. Pj) = 3(g.pt.
3(),,))) ((g-pi.9.P)) = 0.
(33)
Thus, can express our symmetry group through isodistances of B,
2zt,, Pj) == Tz,
(34)
For example, we can pick the function
B(pt, PT) P: Pj (pr (pt-po) (p) -Po) :3(;: O. -Pa)
(35)
If we were define ((r;, Ip: -P:-ppll then we we would additionally be equivariant reflection of the order of tokens in a sentence. If we trivially define ((p. C, then we arrive at bag of words, or no positional encoding (NoPE). For a list of common transformations and their corresponding bi-invariants see Theorem of Bekkers et al. (6.
E.2 Query-Key Separability
Query and key separability is important for efficiency reasons. we can decompose our positional encoded attention score as,
(fi,, Pc, PT) - a(sp(s;, pi), (()),
(36)
then we can pre-compute the positional encoding for the queries and keys on time making the computation O(T). If the positional encoding is not separable, then it will need to be computed for
every pair, (i, 47 56 64 Although there are many symmetries that can be exploited make this not a quadratic computation, it removes the symmetries exploited by efficient attention mechanisms
32
E.3
Linear Flow Property
The property of being "flow" was first proposed in Liu et al. (45) however it is not often discussed. is a property inherently present in RoPE[69]. 52) and ALiBi (53) embeddings, specifically linear flow.
We use the term linear flow for this property because the embedding can found by repeated application of linear function. However, the term "linear" this is small misnomer because it is only locally linear. We define a flow as function
6 R N x R PR+RO
(37)
such that for all æ E X and Pl, P2 E I. the following conditions hold:
1. Initial condition (identity at time zero):
(0, =
(38)
2. Group property (flow property):
((p((,,)), P2)) = (p(x P1 + P2)
(39)
3. Continuity (or differentiability): yp is continuous with respect to its variables, depending on the context
20

---

Strictly speaking, continuity is not necessary for positional encodings as positions tend to be integer values. What we really wish to capture with this property is for the positional encoding to be recursively defined. It may be strange to wish to apply the positional encoding multiple times; however, by having the positional encoding as an endomorphism it can allow for more predictable behavior when extrapolating to larger contexts, which we suspect helps the model train.
We define a position embedding to be linear flow if the flow has the form:
(((x Ap) = Ax,
(40)
for A E IR N and x E RR,, where Ap is the increment rate for position. By Eq. 39 any position p: = Pu Ap can then be attained by,
(xx
x
(41)
This can be seen as a geometric series if A is a scalar as seen in Press et al. [55] If we let At become infinitesimal, then we can express the recurrence relationship as the ODE,
App
(42)
Ōt
which we can integrate to get,
(x, p) = exp(Ap)>
(43)
This A is our generator of the flow, which is also a generator for a matrix Lie algebra, which we focus on in the main text. The matrix exponential, exp: RN R NNN, can be unstable for long contexts; similar to the scalar exponential function e the function can quickly become large for high values of z. However, this can be stable value O. since it always results in one. Similarly, the matrix exponential can be stable if the divergence of the flow trace of the generator - is zero. We call flow "incompressible" or "divergence- of A unit. If fluid dynamics, this is called incompressibilityy For fluids, this implies that the flow conserves mass.
if the trace of A is zero, making the determinant
If there are more than one generator of the Lie group, Ai and A2, then Eq. 39 must be modified to, (l(((x, P1), = ş(x, P1 P2),
(44)
where o is the group product. By the Baker Camkbell HaapbellHHausdoffff formula, exp AIp + A2p2 iff the commutator of AlpI and A2p2 is zero, i.e. the matrices commute. If they do commute, then
exp A1P1 exp A2ps
(45) thus making commutative and having the same properties as addition, o = =+* and Eq. 39 will hold. In this case, the group/flow is known an abelian Lie group, or abelian flow, However, if they do not commute, then will not commute and they are known as non-abelian. This also makes the flow non -on-integr
s((((x,))) p2)) = p(x(xx22). = px =(xp)
E.4 Locality
Locality is often conflated with relativity. The general idea is that tokens far from each other should be independent of one another i.e. attention should decay as distance grows. This often motivates the definition
lim a(x,, Ps: 0
(46)
P. P-----
for Pi E IR and Ty E R.D. However, this definition is both relative and local. We instead define local as,
lim alTi, Lz, Po) = O. [pe-pol-o -000
(47)
The difference being that Po is the origin position. If an embedding is relative, then the origin is arbitrary and can be defined as p, or Pj. In Press et al. (55). they define the origin vector as the next word. However, they can only do this because of the causal mask.
In general, the most natural way to measure locality is through the concept of the quantum mechanical concept of the variance of an operator. We will simply use exponential decay, but we point interested readers to Chapter of Griffiths (3). This formalism works for RoPE as it a linear transformation and the attention mechanism defines a Hilbert space.
To be clear, RoPE and LieRE are not local embeddings. This was shown for RoPE in Barbero et al. [5]. Because they are orthogonal matrices, they have unit determinant, which naturally precludes locality.
21

---

E,5
Other properties
For completeness, there are two additional assumptions that are common.
Adjoint symmetry of the Positional Encoding We implicitly assume that the positional encoding symmetric for the query and key. That is, we assume that the query and key are from the same domain, so the positional encoding has the same representation. More generally, the positional encoding can act differently on the query and key.
( Pt). (0(z) pý)) a(p(z;, p(z),
(48)
where is the positional encoding function for queries. More generally, we can have a relative embedding by letting & act on queries differently from the keys. For example, if we let
( p) exp(Ap)
((r, exp( (xp)
(49)
where A is diagonal matrix. We end up with,
al Pi), (p)), q exp(A(p) pz))k;
(50)
where RoPE can be interpreted as a simple harmonic oscillator, by weakening the symmetry require- ment, one could incorporate damping. This can also be used to incorporate graph Laplacian positional encodings into the framework.
Reversibility Reversibility means that the positional encoding an injective map that is, every coordinate is mapped to unique rotation, thus position can be recovered. This property is important Liu and Zhou [44] and Su [68] to derive Axial RoPE. While prevents Eq. [1] it is necessary only for the D case. More generally, Mixed RoPE can learn an injective map for large D. Moreover, while having a "lossless" positional encoding is nice mathematically, its practical utility has yet to be soundly justified, especially if the positional encoding is learnable.
22

---

Fast Implementation
We follow a vectorized implementation for Spherical RoPE similar to the "fast implementation" proposed in Su et al. 669
First, apply the rotation directly after the other:
zd | (ws(W)) zx[1] sin (wy()) =3
(51)
2n (in()) zal 2a[1] + cost (w,y 2x[3].
(52)
then
[2] (ws(p))))) za sin(w>>P>) [3
(53)
3 sin(w,Pe) 2x[2] + cos(w,Pp) 2, [3]
(54)
where steps 51 and 52 happen simultaneously, and steps 53 and occur at the same time.
G Experimental Setup
Models We use the ViT-S backbone from the timm library [79]. The network always has depth of 12. We keep N as close constant across models as we can. For CIFARI00, the embedding dimensions are changed from 64 Nheads to 60 x Nheads to be compatible with pairs, triplets and quadruples. For ImageNet, we make the embedding dimension 63 Nhead for Spherical RoPE and 64 x Meads for other methods. For classification, we use class token to pool the tokens and predict. Unlike the patch tokens, the class token is not affected by any positional encoding.
CIFARI00 All experiments CIFARIOS 100 were performed on one A100 GPUs with a batch size 256. We use patch size of 4 x on the original image size 32 x 32. The training uses heavy regularization and augmentations including dropout, MixUp [87] and CutMix [86]. The models are trained for 400 epochs, taking ~ 40 seconds per training loop.
ImageNet All experiments ImageNet-Ik 1k were performed on four A100 GPUs with batch size 256. We used cosine learning rate with learning rate of 3e for 200 epochs with 5 epochs of linear warm-up. We used patch size of 16 x 16 on the cropped and resized 224 x 224 image after applying 3-Augment [72]. We use the LAMB 844 optimizer. All experiments took ~0 hrs with to minutes to complete training loop depending method.
Positional Encodings For testing with different resolutions, the images from ImageNet's validation assigned position [- -,,
set were normalized, resized and cropped. On training, the patches and for evaluation, the patch positions were extrapolated to the range [- 1% T). For Learned APE, the positional embeddings are instead interpolated. The fixed frequencies were given by Wa = 1/1002d/D, where d is the index of the pair/upllequadruple.. One frequency is shared between both x and y in our implementation of Axial RoPE
23

---

H Hyperparameters
Table Hyperparameters for ImageNet- IK Training
Category
Setting
Model Architecture
Patch Size
16x16
Heads
6
Latent Dimension
64 (63 for Spherical) x Heads
Depth
12
Pooling
[CLS]
Stochastic Depth
No
Dropout
No
LayerScale
1
Optimization
Optimizer
LAMB 84
Base Learning Rate
4e-3
Weight Decay
0.05
Learning Rate Schedule
Cosine Decay
Warmup Schedule
Linear
Warmup Epochs
5
Epochs
200
Batch Size
512
Gradient Clipping
Precision and Backend
Precision
Mixed (bfloat16)
Backend
torch.autocast
Data Augmentation
Train
Crop
RandomResizeddCrpp (192~2244
Flip
3-Augment
Color Jitter
(0.3, 0.3, 0.3,
Mixup 88
X
Cutmix [86]
x
Normalization
ImageNet-IK Statistics
Data Augmentation Test
Resize
Resize + Resolution
Crop
CenterCrop
Normalize
ImageNet-IK IK Statistics
24

---

Table 6: Hyperparameters for CIFARI00 Training
Category
Setting
Model Architecture
Patch Size
16x16
Heads
12
Latent Dimension
60 Heads
Depth
12
Pooling
[CLS]
Stochastic Depth
0.1
Dropout
0.1
LayerScale
/
Optimization
Optimizer
LAMB 84
Base Learning Rate
4e-3
Weight Decay
0.05
Learning Rate Schedule
Cosine Decay
Warmup Schedule
Linear
Warmup Epochs
5
Epochs
400
Batch Size
1024
Gradient Clipping
Precision and Backend
Precision
Mixed (bfloat16)
Backend
torch.autocast
Data Augmentation
Train
Crop
RandomResizedCrop (32)
Flip
3-Augment
Color Jitter
(0.3, 0.3, 0.3, 0.0)
Mixup [87
0.8
Cutmix [86]
1.0
Normalization
CIFAR Statistics
Data Augmentation Test
Normalize
CIFAR Statistics
25

---

I
Additional Evaluations
In this section, we include extra evaluations including, basic data scaling, segmentation and speed. We also include additional experiments on the effect of rotation frequencies on Uniform RoPE.
L11
Data Scaling
Below evaluate the data scaling of each method. We partition the CIFARI00 dataset into smaller subsets. The number of epochs is scaled, so that the number of training steps is matched on the data subsets. Each model is trained only once cach data split. This experiment tests whether commutative constraint beneficial in smaller data regimes as an inductive bias.
Table 7:
Performance on different portions of CIFARI00.
Dataset Size
Spherical (Learned)
Axial (Learned)
Mixed
Uniform
APE
0.2
56.04 (57.2) 22)
55.3 (56.6)
56.9
52.82
45.9
0.4
63.6 34)
63.3 (62.5)
64.4
59.7
53.4
67.6 (69.8)
66.0 (66.78)
70.0
64.1
57.7
0.8
69.8 (72.6)
69.9 (69.1)
71.6
65.8
59.0
Equivariance, in theory, should provide better performance at small scales due to its inductive bias. However, we observe that learned Spherical RoPE performs on-par or better than Mixed RoPE with less parameters. The small gap
1,2 Segmentation
Below we include rudimentary experiments on segmentation to show that the equivalent performance
of Spherical RoPE is not caused by the simplicity of classification as a task. For these experiments, we use the models trained on ImageNet- Ik as pretrained backbones and fine-tune for Pascal VOC Segmentation [19] The heads of the models are replaced with single MLP which is used to get patch logits for each of. Bilinear interpolation is used to create individual pixel logits.
Table 8: Segmentation results (IoU) on VOC with and without augmentation.
Spherical
Axial (Learned)
Mixed
Uniform
VOC (No Aug.) VOC (Simple Aug.)
0.45(0.46)
0.42 (0.43)
0.44
0.41
0.498 (0.5022.03)
0.474+.011 (0.468+.o10)
0.502+.00s
0.461
1.3 Wall Clock Time
Below we include the wall clock time for each method. Beyond vectorization described in Appendix no optimizations were made for speed. LieRE was implemented following the pseudo-code in
Ostmeier et al. [52]]
Table 9: Time comparison across different positional encodings
Time comparison
Spherical (Learned)
Axial (Learned)
Mixed
LieRE
APE
Uniform
Without torch, autocast With torch, autocast
16.6s (16.6))
16.5s (16.7s)
15.7s
27.4s
13.1s
16.5s
6.7s (5.8s)
6.5s (5.7s)
5.2s
13.6s
3.9s
6.6s
The experiment was performed by rumning dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backhone on one A100 gpu. This simulated training time, so the rotation matrices were recalculated with every pass for learnalle methods.
Note, Mixed RoPE is faster due to naive the use of naive vector partitioning operations and broad- casting. The main conclusion is that learning parameters and Spherical RoPE cause negligible computational overhead.
L.4 Learned Frequencies
When the frequencies of Spherical RoPE are learned, it possible for the model learn equivariance in particular layer. Like Mixed RoPE, if the rotation frequencies in a layer is set to zero, then the
26

---

Learned Spherical RoPE
Learned Axial RoPE
Mixed RoPE
Proquency Frequency
Figure 4: The distribution of learned frequencies in each layer of the ViT. Every method tends to learn low frequency positional encodings in the later layers of the network, meaning representations in the later layers are more invariant to position.
Mixed Learned Learned Spherical
0
Layer Number
Figure 5: The scatterplot of learned and wy. Note, though Axial RoPE is plotting we and wy together, the rotations will always be axial, so there no importance to the pairing.
attention score is position invariant. If one of the rotation frequencies is set zero, then Spherical RoPE will become trivially equivariant in the remaining direction. This makes it interesting to observe what weights the model learns. Below show the learned frequencies in each layer of the network after being trained on ImageNet-1k.
Because frequencies progressively trend toward the axis in deeper layers of the network which makes the positional encodings equivariant in that direction, one could argue that Spherical RoPE learns an equivariant representation in its later layers. However, this same trend be in Mixed RoPE and more notably in Axial RoPE. Because Axial RoPE assumes mutual exclusivity, the frequency pairing arbitrary. Since we still see the trend toward the axes, the observation that later layers use lower frequencies could be an artifact of backpropagation rather than a necessity for the model to learn an equivariant representation.
Interestingly, every method has notable clusters at zero frequencies. This suggests that much of the information the images may be position agnostic. This further explains why setting low frequencies
to zero in traditional RoPE improves performance as observed in Barbero et al. [5]. An additional
cluster can be observed most notably in the later layers of Spherical and Axial RoPE. We hypothesize
this frequency corresponds some information about the resolution of the image, i the spacing of
grid. Some insight on how to generalize to higher resolutions may come from how this frequency corresponds to training data.
27

---

Proofs and Lemmas
Axial RoPE Separability
Proposition 3. Axial RoPE is sepurable and y, that is, the attention score can decom- posed into,
(z) 0zj + oj
(v)
a (x,, Xj. Ps. P5) =
Proof. Suppose we define the dot-product attention score
a(q. k) q
We incorporate Axial Rotary Positional Embeddings by rotating each 2-dimensional subvector of the query (and likewise the key). Concretely, if the hidden dimension 2n, we partition
[9r,1: 9y.1, .... 92,a 9y.x
k == k= ky.1.... k kg.n]
(55)
where each qr.d, qv,d, kg,d, ky.d E RR?. At spatial location p = (Pz: Py). we apply rotations
Ledd R[wa Pa)) Qa,ds
L3.d R (wia (w) qy.d.
and similarly for k. Here R(0) Rx22 is the planar rotation by angle For tokens at positions p: = (Pi,a, Pi.y) and Pj (Pj.xs Pj.y), their rotated queries and keys yield
aij Σ (ana)) R(wa (pj. pi kz,d + (qydd) R R(wa (Pi.r Pi.wi)) ky.d
Define the horizontal and vertical components by
((q.a)) R(wa (Pi. pi,,)) k
(1) aij Σ
(q
R(wa (Pis Ph,)) ky.d-
1=1
Hence the total attention decomposes additively:
(1) 0
demonstrating that axial rotary embeddings factorize the positional dependence along each axis.
L
Matrix Exponentiation Computing the matrix exponential by exponentiating the eigenvalues is common result linear algebra and numerics, however we provide it here for those unfamiliar.
Lemma 1. Let A be diagonalizable matrix A UAU-1, then the matrix exponential of A is given by
exp(A) U exx U-1
Proof.
Recall the power-series definition of the matrix exponential:
exp(A) = Σ k! A
(56)
Since A is diagonalizable,
A = (UAU 12 = UA
(57)
Substituting into the series gives
DC
exp(A)
(UA-U-))- k!
A
(58)
k=0
28

---

Because A is diagonal, the series Ek=0 1 A is itself the diagonal matrix of scalar exponentials, kl
exp(A) diag(e e...,
(59)
Hence is well defined, and
exp(A) (A) U exp(AAUU
(60)
Simultaneous- Sinnulaaneuss-Diiggnaaaiiibii Diaionalizzabbiity The proof that two (diagonalizable) matrixes are simultaneous diagonalizability and only if they are commutative is also a standard result. However, we once again provide it here:
Lemma 2. Let Ag and Ag be skew-symmelric. Then Ag and Ay are simultaneously diagonal- izable if and only if Ag = AyAs
Proof.
Suppose A, and A, are simultaneously diagonalizable. Then, because they are skew-symmetric, there exists unitary matrix U such that
UA = A. and UA Ay
(61)
where A, and Ay are diagonal matrices. Then,
AsAy = UUAJUU,,UU = UA = UA = AAAy
(62)
Hence, Ar and A, commute.
Now suppose Az and Ay commute, AnAy = Ay Since Ar and Ay are skew- Akew-symmetric, they are diagonalizable in CD2D, thus there exists a basis of eigenvectors of Az. Because Ag commutes with A., the eigenspaces of Ar are invariant under Ay. That is, for any eigenvalue À of Ag, the corresponding eigenspace
= (vE CP, A =
(63)
Ag -nvvariaa: if E Ex, then
Ayu)) An
(64)
Now, restrict Ag to each eigenspace Ex. Since is algebraically closed and A, is a linear operator finiteddimeniionl space, Ay is diagonalizable on Ex. Thus, can choose a basis of eigenvectors for Ay in each Ex.
Putting these together, we get a basis for CN consisting of vectors that are eigenvectors for both Ar and Ay Therefore, Ar and Ay are simultancously diagonalizable.
1-D LieRE is equivalent to RoPE this section, we will more formally prove that the traditional
RoPE with learned rotation frequencies is equivalent to 1-D RoPE as proposed in Section 3
Proposition 1. Any D-dimensional rotation can be parameterized by RoPE with learned
frequencies.
Proof.
We define a rotation to be orthogonal matrix with positive determinant: that is, it is an element of R. SO(N). We can write any element of SO(N) via the exponential map R e where A E soNN), i.e. AA is skew-symmetric matrix. It is well-known that the eigenvalues of real, skew- symmetric matrix are purely imaginary (or zero), and such matrix is unitarily (i.e. orthogonally) diagonalizable over C, resulting in a spectral decomposition with a purely imaginary eigenvalue matrix. Thus,
A UAiU
(65)
and, by Lemma I
exp (A) U exp (Ai) U
(66)
where, because A is diagonal, exp(A) is simply the scalar-exponential of each element. The positional encoding of token to a query can be written as,
p(x,p) exp(Ap) W x U ex p) W)
(67)
29

---

where W = W. We assume the same encoding for the key with different matrix, W and
the same generator, A. This equation can be rewritten as (p p) = URoP (x,, p) by Eq I0 If the attention score is given by a(q, k) = q"k, where i denotes the Hermitian transpose, then the attention score be expanded into,
axx x, P5)
P UUURPPPE
(68)
= RxPE((; P.) RoPE(x,,P)).
(69)
Hence, uny LieRE of one generator can be expressed as RoPE with learned rotation frequencies.
Any commutative LieRE is equivalent to Mixed RoPE We now prove that multi-dimensional
LieRE with commutative generators generalizes directly to Mixed RoPE.
Proposition 2. Any M-dimensional LieRE with commutative generators can be parameterized by Mixed RoPE.
Proof.
Let A1...., Aus (NN be skew-symmetric generators such that [Am An. 0 for all m, By Lemma 2 commuting normal matrices are simultaneously unitarily diagonalizable. Thus, there exists unitary U and diagonal matrices A,,..., AM such that
Am Ut for all m ....., M.
(70)
For position vector p (P1, .Ps. RM, the LieRE positional encoding is
LicRE(x, p) cxp Σ
Ampm
Wqx,
(71)
which, using Lemmas 1 and 2
can be written
M
LieRE(x, p)
U Uexp
Σ
Ami Pmo
U'Wax.
(72)
Let W = U.W, Then
LieRE(x, p) UMixedRoPE(x, p),
(73)
where MixedRoPE applies elementwise complex rotations
(A1 (4) p1 P pM)
(74)
to each channel k, with frequencies (k) learned from Am-
If the attention score is given by alq, k) qlk, then
a(x;, Xj, Pi P)) MixedRoPE(xx,
UMixedRoPE(xx Pj)
(75)
MixedRoPE(x, p.) MixedRoPPE(x,p))
(76)
Hence, any M dimiennsional LieRE with commutative generators is equivalent to a Mixed RoPE parameterization with learned rotation frequencies.
30

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide proofs and theoretical evidence on benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We emphasize that our conclusions are limited to vision and have limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

While we do not formally list the assumptions, we implicitly make assumptions on positional encoding through assuming N-D LieRE.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We do our best to provide ayper-parameters for reproducing our results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
We intend to make the code public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuii wwwiiicgguie//eeeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide them to the best of our ability.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We trained models from several random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
33

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide basic information about the GPUs used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We do not believe there any violations.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include section in the appendix, however, is mostly not applicable for our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper is more theoretical.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: We cite libraries used and datasets, however they are standard libraries and benchmarks. There no other specialized assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
36

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
37


Reviews:
[{"review_id": "wDQWW1GA8B", "reviewer": "Reviewer_HWL4", "summary": "The core subject of this paper is positional encoding in two-dimensional data (vision), in particular, whether relative positional encodings are beneficial in this setting. The paper proved the Lie RoPE can be expressed with learned Mixed RoPE, solving an ambiguity in research. They also provide sufficient conditions for Lie RoPE to be equivariant. They then introduce Spherical RoPE, a generalization of Mixed RoPE to frquencies in 3D space, which is non-equivariant. Experimentally, they observe that equivariant approaches such as Axial RoPE do not hold an advantage.", "questions": "Please address the weakness section. At large, I still don't fully understand the following:\n1. Can't we look at experiments without the Spherical RoPE, as it is just an example of a RoPE that isn't equivariant, while Lie RoPE is already a known one that isn't equivariant?\n2. The experiments don't show IMHO that equivariance is not beneficial, it seems Axial RoPE performs better with fewer parameters (or rotational degrees of freedom).\n3. Empirical evaluation is quite limited. Are there any more experiments performed? (Suggestion: Lie RoPE with commuting generators vs arbitrary generators. This might show the inverse result, that equivariance might be beneficial.)", "limitations": "None", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Spherical RoPE is a strong generalization essentially from 2D space to 3D allowing for more refined rotational schemes\n2. Mathematically coherent argument of the expressivity of Lie vs. Learned Mixed RoPE.\n3. Experiments justifying claims\nWeaknesses:\n1. Lie RoPE isn't equivariant if the generators don't commute, making the spherical RoPE redundant in terms of figuring out whether equivariance is important (we can just compare Lie RoPE to itself with/without commutative generators)\n2. Experiments are shown without standard deviation; thus results aren't clear, the results themselves are quite close to each other (other than in comparison to APE).\n3. Axial RoPE (which is equivariant) performs consistently better or at the same level as Spherical with fewer degrees of freedom and parameters, so it may indicate that equivariance is important...", "was_revised": true, "final_justification": "The additional experiments and running time ablations show me that even if there is no positional equivariance, positional encoding learn just as well with similar running time. Due to the paper's empirical results and theoretical grounding in that Lie generators can act as a particular case of equivariant methods when their generators commute, I update my score. I do acknowledge that I understand less than what I thought about this field.", "rebuttal": "# Organization\n\nBecause there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews, we will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).\n\n------\n\n**Weaknesses:**\n>Lie RoPE isn't equivariant if the generators don't commute, making the spherical RoPE redundant in terms of figuring out whether equivariance is important (we can just compare Lie RoPE to itself with/without commutative generators)\n\nBy Proposition 2, all LieRE with commutative generators are theoretically equivalent to some form of Mixed RoPE, as we show it is the spectral decomposition of the generators. Thus, to compare \"unconstrained\" LieRE to LieRE with commutative generators, we can theoretically compare the numbers to Mixed RoPE which is done in their [4] or add positional embeddings as done in [5].\n\nHowever, these do not isolate equivariance. Spherical RoPE has three main advantages as an ablation:\n\n\n- **LieRE is capable of learning commutative generators where (fixed) Spherical RoPE is strictly unable to (in one attention layer).** Similarly, modifying Mixed RoPE with APE as done in the Mixed RoPE paper (Heo et al.(2024)) makes the model not equivariant, however it is not a precise ablation. In a similar way to how Mixed RoPE constrains to LieRE to be commutative generators, Spherical is constrained to be non-commutative generators -- outside of the degenerate case where the frequencies are learned to be zero. This is why we choose it to evaluate whether equivariance is why RoPE performs well. In summary, the important property is **Spherical RoPE is strictly the component of LieRE which breaks the \"relative positional encoding\".**\n- **LieRE does not have as comparable of an implementation**. LieRE requires exponentiating skew symmetric matrices which: 1. Requires more parameters 2. Introduces numerical caveats. **Spherical RoPE on the other hand can be implemented in an almost identical way to Axial and Mixed RoPE making it more appropriate as an ablation.**\n- **The learned behavior of Spherical/Mixed/Axial RoPE is more concise.** We will add a section visualizing the weights learned to the appendix in each layer by Spherical, Axial and Mixed RoPE. We observe that that they learn to set many frequencies to zero. This also means weight decay on the frequencies will bias the model to be position invariant. This type of intuition is more difficult when learning Lie matrices with LieRE or other commutative variants.\n\n[4] Ostmeier, S., Axelrod, B., Moseley, M. E., Chaudhari, A., & Langlotz, C. (2024). Liere: Generalizing rotary position encodings. arXiv preprint arXiv:2406.10322.\n\n[5] Heo, B., Park, S., Han, D., & Yun, S. (2024, September). Rotary position embedding for vision transformer. In European Conference on Computer Vision (pp. 289-305). Cham: Springer Nature Switzerland.\n\n\n\n>Experiments are shown without standard deviation; thus results aren't clear, the results themselves are quite close to each other (other than in comparison to APE).\n\nThis is one of the main points. Even completely removing equivariance by constraining the generators to be non-commutative the results remain close outside of Uniform RoPE and APE. We have improved results for learned PEs and included error bars from multiple runs (see above evaluations).\n\n>Axial RoPE (which is equivariant) performs consistently better or at the same level as Spherical with fewer degrees of freedom and parameters, so it may indicate that equivariance is important...\n\nSpherical RoPE outperforms Axial RoPE, neither of which require any parameters. Here is the table for training on subsamples. Spherical RoPE performs consistently better.\n\n| Portion of CIFAR100 | Spherical | Axial |\n| -------- | -------- | -------- |\n| .2     | **56.04**     | 55.3     |\n| .4     | **63.6**     | \t63.3     |\n| .6     | **67.6**     |\t66.0     |\n| .8     | 69.8     | **69.9**     |\n| Full     | **73.2(+-.4)**     | 72.1+-.6     |\n| Added Parameters     | 0     | 0    |\n\n\n**Questions:**\n>Can't we look at experiments without the Spherical RoPE, as it is just an example of a RoPE that isn't equivariant, while Lie RoPE is already a known one that isn't equivariant?\n\n**See response to weaknesses**\n\n>The experiments don't show IMHO that equivariance is not beneficial, it seems Axial RoPE performs better with fewer parameters (or rotational degrees of freedom).\n\n**See response to weaknesses**\n\n>Empirical evaluation is quite limited. Are there any more experiments performed? (Suggestion: Lie RoPE with commuting generators vs arbitrary generators. This might show the inverse result, that equivariance might be beneficial.)\n\nIn our paper, we show that Lie RoPE with commuting generators is the same (mathematically) as Mixed RoPE, so your suggestion boils down to comparing their performances. See first response for more detail.\n\n\n# Blanket Statement\n\n----\n## Significance\nAs we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is *not* that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.\n\nBecause RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform *slightly* better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem *necessary* for the main benefits of RoPE. \n\nWe believe that this is significant in context of *very* recent work. Many contemporary works seek to impose constraints, such as commutativity, to \"extend\" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:\n\n[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. \n**Learning the ropes: Better 2d and 3d position encodings with string.** (ICML July 2025 Spotlight Poster)\n\n[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). **ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices.** In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).\n\n[3] Liu, H., & Zhou, H. (2025). **Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding.** (Preprint)\n\n---\n## Additional Experiments\n\nWe will add results when they are available.\n\nWe improved the frequency initialization for learnable RoPEs and have run these additional evaluations:\n\n|  Accuracy with STDev | Spherical <br>*Learned*   | Axial <br> *Learned*   | Mixed | Uniform | APE| LieRE|\n| --------           | --------     | -------- | -------- |-------- | -------- | -------- | \n| CIFAR100 (5 Runs)  | 73.2 +- .4 (*74.1 +- .4*)| 72.1+-0.6 (*72.9 +- .6*)|**74.7+-.3**| 70.5+-.2 |64.2 +- 0.9|74.2\n|ImageNet (In Progress) |||\n\n| Segmentation (IoU) | Spherical | Axial (*Learned*)    | Mixed | Uniform | APE |\n| --------           | --------     | -------- | -------- |-------- |-------- |\n| VOC  (No Augmentation)              |     .45     | .42     | .41|\n|VOC (Simple Augmentation)|.50|.46 (*.47*)|.50|.45||\n\n| Portion of CIFAR100 | Spherical (*Learned*) | Axial (*Learned*) | Mixed | Uniform | APE |\n| -------- | -------- | --------   |------ | ------ |------ |\n| .2     | 56.04 (**57.2**)    | 55.3 (*56.6*)       | 56.9 | 52.82 |45.9\n| .4     | 63.6  **(65.34)**   |   63.3 (*62.5*) | 64.4 | 59.7 | 53.4\n| .6     | 67.6  (*69.8*)   |   66.0 (*66.78*)    | **70.0** | 64.1| 57.7\n| .8     | 69.8     | 69.9     |**71.6**| |59.0|\n\n----\n\nBelow we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.\n\n| Time comparison | Spherical (Learned) | Axial (Learned)| Mixed | LieRE| APE | Uniform |\n| -------- | -------- | -------- | -------- |-------- |-------- |-------- |\n| Without torch.autocast     | 16.6s (16.6s)| 16.5s (16.7s)   | 15.7s  | 27.4s| 13.1s | 16.5s\n|With torch.autocast| 6.7s (5.8s)|6.5s (5.7s)| 5.2s | 13.6s| 3.9s |6.6s"}, {"review_id": "xgEYIjeyCt", "reviewer": "Reviewer_ZRjm", "summary": "The paper explores strategies for extending RoPE (rotary positional embeddings) for application in images. Rather than using standard positional embeddings that embed token position using addition, RoPE does it by rotating the token. However, in the context of image encoders, we need to embed a 2D location rather than a 1D position which is standard. There have been various RoPE variations that have been proposed here including adding the rotation matrices, concatenating them, and using different frequencies to embed different dimensions. This paper explores two new RoPE variants: spherical rope and uniform frequency RoPE. These new variations are compared to the previously proposed methods in the context of some standard image classification tasks. The results are nuanced.", "questions": "none", "limitations": "See my comments in strengths and weaknesses. I am most concerned that the paper does not make the right comparisons and does not zero in on the aspects of the rotary embedding that is most important.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n-- I thought the paper was relatively easy to understand. I liked the development of RoPE and the various rotary position embeddings considered. \n\n-- I think researchers who are interested in rotary positional embeddings will find this work useful, although the conclusions are nuanced and not clear cut.\n\n\nWeaknesses:\n\n-- The experiments (image classification) seem appropriate, but there is not clear outperformance of the proposed methods.\n\n-- It is not clear to me how important these results are for classification performance. The results seem relevant and interesting, but in many cases the differences in performance seem small, especially on ImageNet. It might be nice to find evaluation domains/benchmarks that illustrate these differences more precisely. In particular, it would be interesting to know if there are particular types of (image) classification problems for which these methods are particularly relevant. \n\n-- Related to the point above, I wonder if it might make sense to create some ablation studies here that could help pinpoint which aspects of the rotary embedding are most helpful.\n\n-- It struck me odd that Prop 1 and 2 did not have formal proofs. I would convert the \"explanation\" text into a proof or at least include the proof in the appendix (if I missed it in the appendix, please ignore this comment).", "was_revised": false, "final_justification": "", "rebuttal": "# Organization\n\nBecause there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).\n\n-----\n**Weaknesses:**\n\n>The experiments (image classification) seem appropriate, but there is not clear outperformance of the proposed methods.\n\nOur point is not to outperform, but rather to ablate whether relativity is necessary.\n\n>It is not clear to me how important these results are for classification performance. The results seem relevant and interesting, but in many cases the differences in performance seem small, especially on ImageNet. It might be nice to find evaluation domains/benchmarks that illustrate these differences more precisely. In particular, it would be interesting to know if there are particular types of (image) classification problems for which these methods are particularly relevant.\n\nWe have included a segmentation benchmark in the appendix. Once again, our point is to ablate whether relative position is important and not to show dominance of Spherical RoPE. In our view, ImageNet and image classification are biased *toward* relative PEs as the translations equivariance of convolutions have proved powerful inductive biases in the past. Thus, the comparable or better performance illustrates our point more than an anti-equivariance tailored datasets. These would not provide evidence that equivariance is not a powerful inductive bias for vision. In addition, image classification is the most common benchmark for comparing 2D RoPE methods in the closely related works of Mixed-RoPE[5], LieRE[4], ComRoPE [2] and STRING[1]. \n\n\n\n> Related to the point above, I wonder if it might make sense to create some ablation studies here that could help pinpoint which aspects of the rotary embedding are most helpful\n\nThere seems to be a misunderstanding. Spherical RoPE and Uniform RoPE are meant to be ablations to \"pinpoint which aspects of the rotary embedding are most helpful\". We have modified the tables from \"(Ours)->(Our Ablation)\" to make that more clear and to emphasize Spherical RoPE not necessarily as a practical method, but as an ablation.\n\n\n> It struck me odd that Prop 1 and 2 did not have formal proofs. I would convert the \"explanation\" text into a proof or at least include the proof in the appendix (if I missed it in the appendix, please ignore this comment).\n\nThanks for calling it out. We will add more formal proofs to the appendix.\n\n\nLimitations:\n> See my comments in strengths and weaknesses. I am most concerned that the paper does not make the right comparisons and does not zero in on the aspects of the rotary embedding that is most important.\n\nIf this is not resolved please clarify and I will try to respond.\n\n\n----\n\n# Blanket Statement\n\n----\n## Significance\nAs we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is *not* that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.\n\nBecause RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform *slightly* better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem *necessary* for the main benefits of RoPE. \n\nWe believe that this is significant in context of *very* recent work. Many contemporary works seek to impose constraints, such as commutativity, to \"extend\" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:\n\n[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. \n**Learning the ropes: Better 2d and 3d position encodings with string.** (ICML July 2025 Spotlight Poster)\n\n[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). **ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices.** In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).\n\n[3] Liu, H., & Zhou, H. (2025). **Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding.** (Preprint)\n\n-----\n\n## Additional Experiments\n\nWe will add results when they are available.\n\nWe improved the frequency initialization for learnable RoPEs and have run these additional evaluations:\n\n|  Accuracy with STDev | Spherical (*Learned*)   | Axial  (*Learned*)   | Mixed | Uniform | APE| LieRE|\n| --------           | --------     | -------- | -------- |-------- | -------- | -------- | \n| CIFAR100 (5 Runs)  | 73.2 +- .4 (*74.1 +- .4*)| 72.1+-0.6 (*72.9 +- .6*)|**74.7+-.3**| 70.5+-.2 |64.2 +- 0.9|74.2\n|ImageNet (In Progress) |||\n\n| Segmentation (IoU) | Spherical | Axial (Learned)    | Mixed | Uniform | APE |\n| --------           | --------     | -------- | -------- |-------- |-------- |\n| VOC  (No Augmentation)              |     .45     | .42     | .41|\n|VOC (Simple Augmentation)|.50|.46 (.47)|.50|.45||\n\n| Portion of CIFAR100 | Spherical (Learned) | Axial (Learned) | Mixed | Uniform | APE |\n| -------- | -------- | --------   |------ | ------ |------ |\n| .2     | 56.04 (**57.2**)    | 55.3 (56.6)       | 56.9 | 52.82 |45.9\n| .4     | 63.6  **(65.34)**   |   63.3 (62.5) | 64.4 | 59.7 | 53.4\n| .6     | 67.6  (69.8)   |   66.0 (66.78)    | **70.0** | 64.1| 57.7\n| .8     | 69.8     | 69.9     |**71.6**| |59.0|\n\n----\n\nBelow we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.\n\n| Time comparison | Spherical (Learned) | Axial (Learned)| Mixed | LieRE| APE | Uniform |\n| -------- | -------- | -------- | -------- |-------- |-------- |-------- |\n| Without torch.autocast     | 16.6s  (16.6s)| 16.5s (16.7s)   | 15.7s  | 27.4s| 13.1s | 16.5s\n|With torch.autocast| 6.7s (5.8s)|6.5s (5.7s)| 5.2s | 13.6s| 3.9s |6.6s"}, {"review_id": "4U9kbKMWw5", "reviewer": "Reviewer_Kc7V", "summary": "This paper investigates whether shift equivariance is really the key factor of success for RoPE in transformers. The authors analyze recent works such as Mixed RoPE, LieRE, Axial RoPE and discuss how commutative generators may not be necessary. They introduce Spherical RoPE (non-commutativity) and Uniform-frequency RoPE to test their hypotheses and empirically show that Spherical RoPE performs similarly to equivariant positional embeddings.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- This paper tackles the interesting problem of why RoPE is successful and provides theoretical analyses and empirical results to support their claims.\n- Propositions 1 and 2 gives good insights into the relationship between RoPE, Mixed RoPE and LieRE. In particular the authors show that Mixed RoPE is the most general equivariant PE.\n- The paper is very well written and the presentation is straightforward.\n\n## Weaknesses\n- I think the main weakness of this paper is its limited evaluation and experiments. While the theoretical contributions are solid, only the smallest ViT-S model and only two datasets CIFAR100 and ImageNet was used. As the main claim of this paper hinges on the fact that Spherical RoPE performs similarly to Mixed RoPE, the conclusion (\"equivariance is not necessary\") requires more thorough evaluation at multiple model sizes and with various other datasets.\n- A smaller weakness of this paper is that although the authors show that equivariance is not necessary for performance, there is no explanation or hypothesis as to why this may be true. This would greatly strengthen the paper in my opinion.", "was_revised": false, "final_justification": "", "rebuttal": "# Organization\n\nBecause there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).\n\n---\n\nReviewer 3 : \n----\n\n## Weaknesses\n\n> I think the main weakness of this paper is its limited evaluation and experiments. While the theoretical contributions are solid, only the smallest ViT-S model and only two datasets CIFAR100 and ImageNet was used. As the main claim of this paper hinges on the fact that Spherical RoPE performs similarly to Mixed RoPE, the conclusion (\"equivariance is not necessary\") requires more thorough evaluation at multiple model sizes and with various other datasets.\n\nWe have provide more evaluations on various training splits, segmentation tasks, and have included error bars from multiple runs. These can be found below.\n\n\n>A smaller weakness of this paper is that although the authors show that equivariance is not necessary for performance, there is no explanation or hypothesis as to why this may be true. This would greatly strengthen the paper in my opinion.\n\nWe would like to refrain from being too speculative. In the Discussion, we do hypothesize this may be because the small number of tokens, so it may not hold for language. However, there has been evidence in the past that equivariance as a hard constraint is too strict for vision, specifically relaxed weight sharing has beneficial for vision in [6]. \n\n\n[6] Elsayed, G., Ramachandran, P., Shlens, J., & Kornblith, S. (2020, November). **Revisiting spatial invariance with low-rank local connectivity.** In International Conference on Machine Learning (pp. 2868-2879). PMLR.\n\n---\n\n# Blanket Statement\n\n----\n## Significance\nAs we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is *not* that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.\n\nBecause RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform *slightly* better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem *necessary* for the main benefits of RoPE. \n\nWe believe that this is significant in context of *very* recent work. Many contemporary works seek to impose constraints, such as commutativity, to \"extend\" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:\n\n[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. \n**Learning the ropes: Better 2d and 3d position encodings with string.** (ICML July 2025 Spotlight Poster)\n\n[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). **ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices.** In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).\n\n[3] Liu, H., & Zhou, H. (2025). **Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding.** (Preprint)\n\n---\n## Additional Experiments\n\nWe will add results when they are available.\n\nWe improved the frequency initialization for learnable RoPEs and have run these additional evaluations:\n\n|  Accuracy with STDev | Spherical (*Learned*)   | Axial  (*Learned*)   | Mixed | Uniform | APE| LieRE|\n| --------           | --------     | -------- | -------- |-------- | -------- | -------- | \n| CIFAR100 (5 Runs)  | 73.2 +- .4 (*74.1 +- .4*)| 72.1+-0.6 (*72.9 +- .6*)|**74.7+-.3**| 70.5+-.2 |64.2 +- 0.9|74.2\n|ImageNet (In Progress) |||\n\n| Segmentation (IoU) | Spherical | Axial (Learned)    | Mixed | Uniform | APE |\n| --------           | --------     | -------- | -------- |-------- |-------- |\n| VOC  (No Augmentation)              |     .45     | .42     | .41|\n|VOC (Simple Augmentation)|.50|.46 (.47)|.50|.45||\n\n| Portion of CIFAR100 | Spherical (Learned) | Axial (Learned) | Mixed | Uniform | APE |\n| -------- | -------- | --------   |------ | ------ |------ |\n| .2     | 56.04 (**57.2**)    | 55.3 (56.6)       | 56.9 | 52.82 |45.9\n| .4     | 63.6  **(65.34)**   |   63.3 (62.5) | 64.4 | 59.7 | 53.4\n| .6     | 67.6  (69.8)   |   66.0 (66.78)    | **70.0** | 64.1| 57.7\n| .8     | 69.8     | 69.9     |**71.6**| |59.0|\n\n----\n\nBelow we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.\n\n| Time comparison | Spherical (Learned) | Axial (Learned)| Mixed | LieRE| APE | Uniform |\n| -------- | -------- | -------- | -------- |-------- |-------- |-------- |\n| Without torch.autocast     | 16.6s  (16.6s)| 16.5s (16.7s)   | 15.7s  | 27.4s| 13.1s | 16.5s\n|With torch.autocast| 6.7s (5.8s)|6.5s (5.7s)| 5.2s | 13.6s| 3.9s |6.6s"}, {"review_id": "dkpTtGtiLb", "reviewer": "Reviewer_5p81", "summary": "This paper investigates the role of equivariance in Rotary Positional Encodings (RoPE) for vision transformers. The authors formulate a unifying theoretical framework for various multi-dimensional RoPE extensions (such as Axial RoPE, Mixed RoPE, and LieRE) and formally show that Mixed RoPE is the most general form of equivariant rotary encoding under the assumption of commuting generators (a necessary condition for equivariance). The paper questions the belief that this equivariance (i.e., being a relative positional encoding) is the key to RoPE's success. To test this hypothesis, the authors propose \"Spherical RoPE\", a non-equivariant RoPE variant based on anti-commuting generators (SO(3) rotations), designed as a minimal modification to equivariant counterparts. Through experiments on CIFAR-100 and ImageNet-1K, they demonstrate that Spherical RoPE performs on par with equivariant RoPE variants. The authors conclude that properties other than equivariance, such as the diversity of rotation frequencies, might be more critical for the performance of rotary embeddings in vision tasks.", "questions": "1. Have the authors tried to interpret what kind of positional relations architectures with \"unrestricted\" RoPE variants (i.e. not enforcing translational equivariance) learn? For example, this could be analysed by looking at the learned attention maps for different (shifted) versions of the same image: If the attention maps transform (shift) in a similar manner, it would indicate that models have learned approximate translational equivariance. If not, it would reveal that such models can learn attention patterns that are (by construction) inaccessible to architectures that use translationally equivariant RoPE variants. Both outcomes would be interesting.\n2. The authors should report error bars on the results, so that their statistical significance can be assessed. Ideally, they should extend the experimental evaluation to include other vision tasks (e.g. segmentation) and architectures to see whether the findings hold in general.\n3. The authors should re-run experiments for LieRE using their training framework, so that the different results in Table 2 can be compared more directly.\n4. In Figure 3, it seems that the highest ImageNet top1-accuracy is obtained for image resolutions *above* the training resolution for all tested RoPE variants. This feels counterintuitive to me (I would have expected that performance peaks for the training resolution), can the authors provide any insights on potential explanations for this trend?\n5. The paper uses the term \"equivariant\" synonymous with \"translation equivariant\". However, 2D positional encodings can also be *rotation* equivariant. This means that for a relative displacement $p_{ij}$, the attention pattern is the same no matter in which direction $p_{ij}$ points, only the length of $p_{ij}$ would be important (In Figure 1A, this would correspond to a circular attention pattern). I believe neither of the discussed RoPE variants has this property, but it is conceivable that this could be advantageous for some vision tasks (e.g. classifying objects on a table from a bird's eye perspective, see https://arxiv.org/abs/1612.08498 and similar works for context). I believe it would be valuable to discuss the distinction between translation and rotation equivariance in the manuscript.\n\n\n\nTypos and other minor formatting issues (these do not influence the rating):\n\n- \"effects\" on line 87 should probably be \"affects\" (see also the caption of Figure 2, where there are similar mixups)\n- inline equation on line 91 uses $p$, but the text on line 92 refers to $p_t$\n- $_1^{(y)}$ on line 140 should probably be $z_1^{(y)}$\n- \"guage\" on line 253 should be \"gauge\"\n- \"that\" on line 286 should be \"than\"\n- \"method\" on line 302 should be \"methods\"", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper challenges the widely-held assumption that the success of RoPE stems from being relative (translational equivariance) and provides empirical evidence that built-in equivariance may not be a requirement for successful positional encodings in vision applications.\n2. The paper offers a valuable theoretical clarification by unifying various RoPE extensions within a common framework.\n3. The proposed \"Spherical RoPE\" is an elegant way to test the central hypothesis of the paper.\n4. The paper is well-written and the presented arguments are easy to follow.\n\n**Weaknesses**\n1. The authors find that built-in (\"hardcoded\") translational equivariance may not be necessary for positional encodings, but they do not analyse what the models with \"unrestricted\" RoPE variants actually learn. It is conceivable that those models still learn approximate (\"good enough\") translational equivariance.\n2. The presented results seem to be from a single run and do not report any error bars. For the main performance comparison in Table 2, an indication of statistical significance seems necessary to arrive at the conclusion that performance truly is \"equivalent\". Similarly, for the extrapolation experiment in Figure 3, the statistical significance of the results is unclear and care should be taken when drawing conclusions such as \"Spherical RoPE generalizes better to larger image sizes compared to other RoPE variants\".\n3. The experiments are limited in scope; they were performed only for a single architecture (ViT-S), a single task (classification) and on two datasets (CIFAR100 and ImageNet-1K). It is conceivable to me that the benefits of translational equivariance only become apparent in different (perhaps larger?) architectures, or on different tasks like object detection or segmentation, which might rely more heavily on spatial relationships. The strong conclusion that equivariance is not needed for vision (in general) is not fully supported by the limited experimental evidence.\n4. The authors compare their methods to LieRE using results from the original LieRE paper, acknowledging that the training protocol is different. This makes a direct comparison of performance numbers difficult. While the authors' main conclusions do not hinge on these results, a re-implementation of LieRE within their own experimental framework would allow a more direct comparison and seems to be straightforward.\n5. While the authors' findings are certainly interesting, their impact on the field seems limited to me. While relative positional embeddings may not be required and similar performance can be reached without translational equivariance, as long as there is no advantage to be gained from dropping this constraint (either increased performance or efficiency), it is not clear why this insight is useful.", "was_revised": true, "final_justification": "The authors ran some additional experiments, answered my questions, and improved the discussion/interpretation of the results.", "rebuttal": "# Organization\n\nBecause there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).\n\n---\n\n\n**Weaknesses**\n\n>The authors find that built-in ...\n    \nIt is a good question, though to be pedantic, Spherical RoPE is not unconstrained; it is contrained to *not* be commutative. It is true that the models learn to be almost completely translation equivariant (see the Questions section for elaboration).\n\nWhile it is an interesting question, we don't think it is relevant enough to cram into the main body of this paper. The point still stands that the hard constraint does not appear to be why RoPE performs well.\n\n>The presented results seem to be from ...\n\nWe have included additional experiments below.\n\n>Similarly, for the extrapolation experiment in Figure 3, the statistical significance of the results is unclear and care should be taken when drawing conclusions such as \"Spherical RoPE generalizes better to larger image sizes compared to other RoPE variants\".\n\nWe will include results from three runs when those results are available (expect Saturday/Sunday).\n\n>The experiments are limited in scope; ...\n\nOur claim is that ***RoPE*** does not need to be strictly equivariant, that is the benefits from RoPE are not caused by equivariance. We would like to make it clear that we are not claiming that \"equivariance is not needed for vision (in general).\"\n\nWe have included segmentation experiments on VOC Segmentation, however this tends to require a pretrained backbone for transformer based architectures to have good results. Even in this very low data regime, we see that Spherical RoPE performs on par which is contrary to the main motivation of shift invariance as an inductive bias.\n\n> The authors compare their methods ...\n\nThis has now been included. \n    \n\n>While the authors' findings ...\n\nRoPE is currently the most popular way of doing positional encodings and the question of how to extend it to N-D is a very relevant and active topic right now. It is very relevant for the field to understand why RoPE works to move forward. It is currently assumed that it is because it is a relative positional encoding (see blanket statement).\n\n\nQuestions:\n\n>Have the authors tried to interpret ...\n\nYes we have. The Learned Spherical RoPE learns to push all the $y$ frequencies to zero in the last layers which collapes the rotation to the identity. This is a special case where Spherical RoPE becomes strictly equivariant to $x$. We will try to provide a small section in the appendix because it *is* interesting. \n    \nHowever, this requires more testing such as whether this is determined by the order of the roll/yaw matrix. Moreover, whether it learns to be approximately equivariant or not isn't relevant for this paper's focus question of whether it is *necessary*. Indeed, it almost needs to learn approximate invariance for these datasets since we do random cropping augmentations. Because of the history and perceived importance of equivariance in these datasets, our results are even more evidence that equivariance is not the core reason RoPE works.\n\n\n>The authors should report error bars on the results...\n\nWe have provided these results below, though some results are in progress.\n\n>The authors should re-run experiments for LieRE using their training framework, so that the different results in Table 2 can be compared more directly.\n\nCIFAR100 provided. ImageNet in progress.\n\n>In Figure 3, it seems that the highest ImageNet ...\n    \nThis is a standard result and one of the draws of RoPE-like embeddings as it allows for training on lower resolution images which can be done much faster [1-5]. We are unfamiliar if there is any proper explanation of this phenomena, but the layman intuition is that the larger resolution provides more information and more patches as context for the classification. The change in zoom in the content of the patches is not an issue because of random cropping while training.\n\n>The paper uses the term \"equivariant\" ...\n\nThanks for pointing this out. We discuss this concept in detail in our discussion of equivariance in the Appendix, but we have now added a sentence stating that we assume equivariance -> shift equivariant explicitly in the main body.\n\nWe have thought about shift equivariance. To make a rotationally equivariant RoPE, one simply needs to parameterized position in log-polar coordinates. However, SE(2) is much more complicated. One needs a position and orientation [6] of the patch and the orientation is not a known quantity. More generally, one could construct a bi-invariant, which has been shown to be equivalent to the steerability property in [7]. \n\n[6] Kofinas, M., Nagaraja, N., & Gavves, E. (2021). **Roto-translated local coordinate frames for interacting dynamical systems.** Advances in Neural Information Processing Systems, 34, 6417-6429.\n\n[7] Knigge, D., Wessels, D., Valperga, R., Papa, S., Sonke, J. J., Bekkers, E., & Gavves, E. (2024). **Space-time continuous pde forecasting using equivariant neural fields.** Advances in Neural Information Processing Systems, 37, 76553-76577.\n\n>Typos and other minor formatting issues (these do not influence the rating):\n>\n\nThank you for the diligence!\n\n---\n\n\n# Blanket Statement\n\n----\n## Significance\nAs we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is *not* that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.\n\nBecause RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform *slightly* better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem *necessary* for the main benefits of RoPE. \n\nWe believe that this is significant in context of *very* recent work. Many contemporary works seek to impose constraints, such as commutativity, to \"extend\" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:\n\n[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. \n**Learning the ropes: Better 2d and 3d position encodings with string.** (ICML July 2025 Spotlight Poster)\n\n[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). **ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices.** In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).\n\n[3] Liu, H., & Zhou, H. (2025). **Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding.** (Preprint)\n\n---\n## Additional Experiments\n\nWe will add results when they are available.\n\nWe improved the frequency initialization for learnable RoPEs and have run these additional evaluations:\n\n|  Accuracy with STDev | Spherical (*Learned*)   | Axial  (*Learned*)   | Mixed | Uniform | APE| LieRE|\n| --------           | --------     | -------- | -------- |-------- | -------- | -------- | \n| CIFAR100 (5 Runs)  | 73.2 +- .4 (*74.1 +- .4*)| 72.1+-0.6 (*72.9 +- .6*)|**74.7+-.3**| 70.5+-.2 |64.2 +- 0.9|74.2\n|ImageNet (In Progress) |||\n\n| Segmentation (IoU) | Spherical | Axial (Learned)    | Mixed | Uniform | APE |\n| --------           | --------     | -------- | -------- |-------- |-------- |\n| VOC  (No Augmentation)              |     .45     | .42     | .41|\n|VOC (Simple Augmentation)|.50|.46 (.47)|.50|.45||\n\n| Portion of CIFAR100 | Spherical (Learned) | Axial (Learned) | Mixed | Uniform | APE |\n| -------- | -------- | --------   |------ | ------ |------ |\n| .2     | 56.04 (**57.2**)    | 55.3 (56.6)       | 56.9 | 52.82 |45.9\n| .4     | 63.6  **(65.34)**   |   63.3 (62.5) | 64.4 | 59.7 | 53.4\n| .6     | 67.6  (69.8)   |   66.0 (66.78)    | **70.0** | 64.1| 57.7\n| .8     | 69.8     | 69.9     |**71.6**| |59.0|\n\n----\n\nBelow we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.\n\n| Time comparison | Spherical (Learned) | Axial (Learned)| Mixed | LieRE| APE | Uniform |\n| -------- | -------- | -------- | -------- |-------- |-------- |-------- |\n| Without torch.autocast     | 16.6s  (16.6s)| 16.5s (16.7s)   | 15.7s  | 27.4s| 13.1s | 16.5s\n|With torch.autocast| 6.7s (5.8s)|6.5s (5.7s)| 5.2s | 13.6s| 3.9s |6.6s"}]

Author rebuttal:

Meta-review: The main concern initially raised by a subset of reviewers was on the main finding of the paper; after an engaging discussion round, it seems there is now understanding that the main point of the paper was not to promote the advantages of Spherical RoPE or Uniform RoPE, but to perform ablation studies relative to these two methods to better understand why RoPE works. All reviewers find value in the findings reported in the paper. Although one reviewer believes additional and more comprehensive study is needed to make the argument that equivariance is not the primary reason for RoPE's success, the study itself and its results are still considered meaningful and of value. The final version of the paper will, I believe, be considerably revised to reflect the many constructive suggestions of the reviewers, adding more nuance to the claims and better explaining the main findings about the relationship between equivariance and RoPE's success.
Decision: Accept (poster)