LieRE: Lie Rotational Positional Encodings
Authors: Sophie Ostmeier
Brian Axelrod
Maya Varma
Michael Moseley
Akshay S Chaudhari
Curtis Langlotz
Venue: ICML 2025 poster (2025)
Keywords: Position encoding
Vision Transformer
Primary area: deep_learning->attention_mechanisms
TL;DR: Lie group relative position encodings for 2D and 3D data.

Abstract: Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure.
Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information.
We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.

LieRE: Lie Rotational Positional Encodings
Sophie Ostmeier Brian Axelrod
Maya Varma
Michael Moseley? Akshay Chaudhari? 21
Curtis Langlotz"
Abstract
dependencies erucial for tasks ranging from natural lan- guage to image understanding (Vaswani et al., 2017).
Transformer architectures rely on position encod- ings model the spatial structure of input data. Rotary Position Encoding (RoPE) is widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transform- ers. Instead of fixed 2D rotations, LieRE learns dense symmetriit matrices (Lie algebra cl- ements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learn- able, and continuous, encodings of both relative and absolute positional information. We demon- strate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining compu- tational efficiency. The code and checkpoints are publicly available https: / / giittn thub. com/ StanfordMIMI MOMMM/LLiee /LieRE.
This challenge has sparked extensive research into posi- tional encodings, which inject order information into the otherwise order-agnostic attention mechanism. The field has evolved from early approaches using fixed sinusoidal positional embeddings (Vaswani et al., 2017) to more so- phisticated learned embeddings (Shaw et al., 2018a; Devlin et al., 2019; Dosovitskiy et al., 2020). Recent advances have introduced increasingly dynamic methods, including relative position representations (Shaw et al., 2018b; Doso- vitskiy et al., 2020) and rotary positional embeddings (Su et al., 2024; Heo et al., 2024), demonstrating the critical role of position encoding in enabling transformers to effectively process ordered sequences.
In particular, Rotary Position Encoding (RoPE) has emerged as an elegant solution for encoding relative positional infor- mation in transformers (Su et al., 2024). RoPE works by applying a rotation matrix to each token's 's keys and queries, where the rotation angle depends on the token's 's absolute position in the sequence. The key insight is that when two tokens interact through the attention inner product, their rotated representations naturally encode their relative dis- tance. For example, when token at position five attends to a token at position two, RoPE enables the model to under- stand that these tokens are three positions apart, regardless of their absolute positions in the sequence. This translation- invariant property makes RoPE particularly efficient cap- turing position-dependeen patterns in text, which has led to its adoption in popular open-source language models such as LLaMA and Mixtral.
1. Introduction
The attention mechanism, particularly within transformer architectures, has revolutionized machine learning across diverse domains. However, attention is inherently permuta- tion invariant-it cannot leverage the sequential order of its inputs information directly. This fundamental limitation necessitates additional mechanisms to encode positional in- formation, enabling models to capture sequential and spatial
Despite RoPE's success in sequential tasks (Touvron et al., 2023; Chowdhery et al., 2023), it faces several fundamental limitations. First, RoPE is designed for one-dimensional sequence data with an exponential decay in frequencies, making it suboptimal for spatial relationships in images or spatiotemporal patterns in videos where distant pixels may have strong correlations. Second, when handling higher dimensional data, the challenge compounds significantly- learned relative position encodings must capture exponen- tially many relative positions for n-dimensional data, as demonstrated in 2D image data where positions must be en- coded both horizontally and vertically (Shaw al., 2018a).
'Equal contribution 'Computer Science Department, Stanford University, USA 'Radiology Department, Stan- ford University, USA. Correspondence to: Sophie Ost- meier <3ossm stanford.edu>. dresearch @ gmail. com'> Equal senior authorship. Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Brian Axelrod <baxclro-

---

LieRE; Lie Rotational Positional Encodings
N-dimensional Input i.e.N=3
LieRE Position
Encoding Matrices
LicRE Scaled Dot Product Attention L (d, 1), k (d,) 1)
A = (N, d, d), R = (N, d, d)
4 R
R k
Agx
+
Rotate
Rotate
exp
Ay +
A
Attention
Figure 1. Overview of how LicRE encodes spatial information, where N denotes number of input dimensions, i.c. N 3 3D image. A is learnable skew symmetric matrix and H, exp(A(æ; 34 zz) E gedd the rotation matrix for the ith patch in the flattened input, where j corresponds to a different patch. is the head dimension.
Third, RoPE's reliance on sparse, block-diagonal rotation matrices with handcrafted basis functions constrains its abil- ity to learn complex spatial dependencies (Chu et al., 2024).
highlighting their crucial role in spatial understanding.
2. Related Work
Our key insight is that Lie group theory provides a natu- ral framework for generalizing relative position encodings to higher dimensions. We introduce Lie Relative Encod- ings (LieRE), which replaces RoPE's rotation matrices with learned, dense rotations derived from Lie group genera- tors. LieRE learns a basis of skew-symmetric matrices A;, computes rotation matrices R(p) = exp(Ei=o PtA.) for n-dimensional positions, and applies these rotations keys and queries the attention mechanism. The relative posi- tions then naturally captured through the inner product of the rotated keys and queries.
2.1. Position Encodings
We split the review of positional encodings into: a) absolute, b) relative, and c) contextual.
Absolute position in this context refers position with respect to consistent reference, usually the start of a text or top left corner of an image. Absolute encodings generally operate on a per token-level, modifying the embedding of token to encode the location the token in the input. Methods such as sinusoidal and learned absolute encodings directly add vectors to the input token embedding (Vaswani et al., 2017; Devlin al., 2019; Dosovitskiy et al., 2020).
This approach addresses RoPE's limitations in two key ways: (1) it handles higher-dimensional spatial relationships through Lie groups and (2) inereases representational capacity through variably dense, learned rotation ma- trices while requiring only 0.68% of parameters in ViT-B model. Importantly, LieRE can be implemented with single modification to existing architectures.
Relative position encodings, in contrast, encode the relative positions of two tokens. One strategy is to learn an set of embeddings for position deltas which can be incorporated into the attention mechanism (Shaw al., 2018b; Liu et al., 2021: 2022). However, this incurs quadratic computational cost in terms of the number of tokens ding is required for every pair of tokens. Rotary Position Encodings (RoPE) avoid this cost by rotating the key and query vectors before the attention inner product. The al- gebraic properties of the block-diagonal rotation matrices used in RoPE ensures only relative positional information captured in the attention mechanism (Su al., 2024). RoPE is quite widely used in open source LLMs including the PaLM, Llama and Mixtral models (Touvron et al., 2023; Chowdhery et al., 2023; Jiang et al., 2024). However, RoPE
separate embed-
To assess the impact of LieRE and other positional encod- ings on VIT performance, we evaluate several encoding schemes across diverse tasks, including 2D and 3D image classification. Additionally, we investigate a fundamental spatial reasoning task where models must identify where an points. Despite their sophistication, contempo- rary multimodal LLMs struggle with this seemingly simple lask. Our experiments reveal that successful completion of this task specifically requires relative position encodings,
2

---

LieRE: Lie Rotational Positional Encodings
LicREa4
RoPE-Mixed
VisionLIaMA
LieREg
Basis matrix A
LiieREg
13
MILA F
13
MILA &
13
MIA
Rotation matrix exp(Az)
Rxll
Rxll
Rxll
L2
MHHA F
L2
MIAA F
L22 MHAA &
in Model
Rxll
Rxll
L.
FF
L.1
RxH
RaH
RaH
Imput
Impst
Impst
Figure 2. Comparisons of basis matrix A of work, Rope-Mixed (Heo al., 2024), VisionLlama (Chu et al., 2024). First row: Iearnable parameter (yellow), not icaraalle parameter (bluc) in basis matrix. Sccond row: rotation matrices exp(Az) shared across the stem (gray) different for each layer and head (colorful). MIIA Rotation Matrix per Head, LieRE, == the number of LieRE parameters.
FF == Feed Forward, L[index] == Layer index, RxHI
can perform poorly on inference for larger context sizes than the model was trained on. This has spurred an active line of work extending RoPE to longer contexts, work which we review later.
cally focused on adapting RoPE to image tasks. Both Vision- Llama and RoPE-Mixed present relative position encodings inspired by RoPE capable of encoding 2D positional data (Chu et al., 2024; Heo et al., 2024). The primary difference is that RoPE-Mixed has learnable component, whereas VisionLlama does not.
We refer the last category of positional encodings as contextual position embeddings. This category is defined by encodings that aim to capture semantic positional in- formation lost in traditional absolute and relative position encodings, often motivated by reasoning or mathematical tasks. Contextual Position encodings achieve (CoPE) this by allowing the model to learn how the position is computed (Golovneva et al., 2024). Abacus embeddings enable trans- formers to learn how handle arithmetic by better exposing the digit structure of numbers (McLeish et al., 2024).
2.3. Efficient Scaling beyond Sequence Data
There is also an extensive line of work improving the perfor- mance of transformers for modalities with dimensionality greater than one. Axial Attention (Ho al., 2019) reduces computational complexity by applying attention along spe- cific axes (e.g., rows and columns in images), enabling trans- formers to scale efficiently with high- nigh-dimensional data. Per- ceiver utilizes latent tokens to compress nigh-dimensional inputs into a smaller set, improving scalability as input size and dimensionality grow. These methods address the inefficiencies of traditional transformers when applied to high-dimensions data (Jaegle et al., 2021). Additionally, techniques like Swin and Vmamba optimize compute for vi- sual data through structuring how information flows through the network (Liu ct al., 2021; 2024). Swin Transformer in- troduces a hierarchical approach with shifted windows, lim- iting attention to local regions to reduce complexity while capturing global context. Vmamba, on the other hand, pro- poses visual state space model that represents images as collection of spatial states, allowing attention to be applied
2.2. Extensions of RoPE
The efficiency and popularity of RoPE have led to several lines of work building off of it.
One notable one is context extension, which aims to address the fact that RoPE NLP models trained on short documents tend to perform poorly on long documents. Methods like NTK-aware context extension, YaRN and LongRoPE focus on enabling already trained models to handle long context, both with and without finetuning (Ding al., 2024; Peng et al., 2023; Tworkowski et al., 2024; Chen et al., 2023).
The final, and most relevant, line of work has been specifi-
3

---

Let X E Rexd be the set of imput embeddings and Wo, WK, W. be learnable matrices. Let Q XW K = XWK. V XWv be the keys, queries and values re- spectively. The outputs are computed as scores QK" va," W softmax(scores) and final outputs WV. We let LieRE: Lie Rotational Positional Encodings
efficiently across large-scale visual inputs by exploiting spa- tial locality and reducing redundant computation.
allowing us to strengthen the statement in (1) to
exp(U V) exp(U) exp(V)-+ = exp(VV)) exp(U).
(2)
2.4. Equivariant Networks
A related branch of work encoding problem structure fo- cuses equivariance. We say that model T is equivariant with respect to if T(f(x)) Where with relative position encoding we often want to be able to encode translation invariance, equivariance pro- vides a more general framework. Equivariance has been applied to improve performance on problems with a wide ar- ray of structures, ranging from inttriiniiv (Esteves et al., 2017; 2018; Worrall al., 2017), 3D reference frame- invariance (Liao & Smidt, 2022; Fuchs et al., 2020) and many others. The subset of these works that focus on gen- erating equivariant token embeddings for transformers can be combined directly with LieRE or another rotation-base based position encoding.
Our work examines the tradeoff between using the stronger property in (2) or increased capacity and the weaker property (I).
(x(())) (Tai et al., 2019).
3.2. Attention Mechanism
LicRE is modification of the standard attention mechanism to introduce positional information, which we review below. The modification we propose is independent of whether we use multiple heads, so we focus on single-header attention for simplicity.
2.5. Lie Groups in Machine Learning
Qi and K; denote the ith rows of Q and K respectively.
Lie groups have also had extensive use in machine learning. The range of works is diverse, ranging from algebraic sig- nal processing (Kumar et al., 2024), automated discovery of symmetries (Forestano al., 2023) to state estimation (Falorsi et al., 2019). Furthermore (Gallier & Quaintance, 2020) provides friendly introduction to differential geom- etry and lie groups that may be of interest to the reader.
4. Method
LieRE is a simple modification to the attention mechanism that is presented in Algorithm 3a. Recall that we assume that positions are n-dimensional vectors, matrix A is skew- symmetric if AA -A, and that the matrix exponential of akew-symmetric matrix, call it A, is always high dimensional rotation matrix.
3. Backkround
3.1. Lie Groups in the Context of Attention
When encoding positions E R", LieRE learns a skew- symmetric basis of matrices {A,} for i E [n]. It en-
In this section, we aim to provide a minimal introduction to Lie groups so that the reader is able understand the mathematical motivations behind LieRE. Lie groups are well studied, especially in the context of representation theory, and standard texts including (Fulton & Harris, 2013) are able to provide a more extensive introduction to the subject.
codes a position by writing it in this basis. Σ PrA,, We ==0
then map the resulting skew-symmetric matrix to a high- dimensional rotation via the matrix exponential. R(p)
exp EE: (Figure 1). Learning in the space of skew-
symmetric matrices allows us to sidestep some of the dif- ficulty that would come from learning on the manifold of rotation matrices (Figure 2).
In this context, Lie groups are smooth sets of matrices that closed under matrix multiplication and inversion. For every Lie group, the matrix exponential provides a smooth bijective map from subset of Rax*, also known as the Lie Algebra, to the Lie group. The exponential map is a diffeomorphism and has the following key property for U, V E Baxa close together:
LieRE uses the rotation matrix computed above to modify the keys and queries of the standard attention mechanism. LieRE's final step is to modify token i's query and keys as Q R(p, QQ and K'- = R(p: K,. This modifies the score between tokens i, j to be X?T (TP:) R(p.)" (T(p)))((p)))))))))))))))))))))) WK XX, Re- calling that R. = R for any orthogonal matrix R helps illustrate the encoding of relative positions in (1). Note that the only difference hetween LieRE and RoPF-Mixed is that the latter constrains the rotations to be block-diagonal with block size two.
exp(U V)- exp(-V + +U) exp exp(U) (xp(()))exp((U) (1)
Both RoPE (in the context of text) and RoPE-Mixed use block-diagonal rotation matrices with 2D rotations as blocks
3b. These form a special Lie group that commutative,
We include the pseudocode for the LieRE attention Algo-

---

LieRE; Lie Rotational Positional Encodings
Algorithm 2 RoPE Attention
Algorithm 1 LieRE Attention
1: procedure RoPE(X, p, d)
1: procedure LIERE ROTATIONS(P, A)
2: O + 1000024/4 Vi [0, d)
2: d dimensssion(p)
3: for it O to d step 2 do
3: return matrix_exp
Σ A:Pi
4:
X rot f X' cos p0 X Xi+1 sin po
5: X + X sin pO + x cos pO
4: end procedure
rot
6: end for
5: procedure LIERE_ AATRRAAATTETTOOOO,AA K, V.
7: return X
6:
P f
8: end procedure
7: R LIERE ROTATIONS(P,A)
9: procedure RoPE_ ROPE_ATTENTIONC, K, V)
8: // Multipll each key and query vector by
the rotation for that token.
10:
p f- tokenPositions
9:
K 4- BATCHMATMUL (R, (R, K)
11:
d <- embeldingDimennsoon
10: Qro < BATCHMATMUL Q)
12:
K <- RoPE p,
(
13:
Qrot <- ROPE(Q.P..)
Smul (Vim((s)
11: Attention softmax
V
14:
Attention < softmax
V
Jd
12:
return Attention
15:
return Attention
13:
end procedure
16:
end procedure
(a) Lie Rotary Embedding (LieRE) attention mechanism.
(b) Rotary Position Embedding (RoPE) attention mechanism.
Figure 3. Comparison of the LieRE and RoPE-Mixed attention mechanisms.
Table 1. 2D image and 3D video classification Top-1 Accuracy (95% confidence intervals) results. All models use 85.2M parame- ters for 2D tasks and 88.7N parameters for 3D task (Krizhevsky al. 2009; Deng et al.. 2009; Soomro al.. 2012; Flanders et al., 2020) equivalent to DeiT architecture, equivalent to Vivit (spatio-temporal) architecture.
Table 2. P-values comparing RoPE-Mixed and LieREs across
datasets.
Dataset
P-value (RoPE-Mixed vs. LicREs)
CIFAR-10
1.3 10-5
ImageNet-
10-3
UCFI01
10
Method
CIFAR-100
ImageNet-1k
UCF101
384 384 (Resolution Invariance)
4.0 x 10-4
Abs. Pos, E =
63.9 (62.9-65.8)
66.1 (65.7-66.5)
40.9 (40.5
VisionLlama RoPE
65.5 (64.6-66.5)
65.4 (5..0
45.0 (44.6-45.4)
RoPE-Mixce
68.8 (67.9-69.7)
(68.4-69.2)
46.3 (45.9-46.7)
LieRE's
70.3 (69.4-71.2)
69.6 (69.2 (69.2-70.0)
47.0 (46.6-47.4)
LicREsa
70.0 (69.1-70.9)
69.3 69.7)
44.7 (44.3-45.1)
5. Experiments
We evaluate LieRE across a range of tasks to assess its impact on transformer performance in 2D and 3D vision, spatial reasoning, and resolution generalization. Our goal is to isolate the contribution of positional encodings by using consistent architectures, training procedures, and hy- perparameters across all methods. To this end, we utilize a standard recipe and build on top of the vision transformers repository (Yoshioka, 2024) and verify that our baselines perform similarly to prior work (Lee et al., 2022). We avoid using pre-trained weights in order to help reprodueibility and comparability of the results between methods. We pro- vide confidence interval estimates using bootstrap (B=1000, a = 0.05). We evaluate two versions of LieRE, distin- guished by the basis matrix block-diagonal sizes of 64 and 8, referred to as LieREs1 and LieREs, respectively. Notably, tile size of corresponds RoPE-Mixed.
rithm 3a in addition to the standard RoPE attention (Algo- rithm 3b). In practice, we compute the rotation matrices at the start of the forward pass. By default, the skew symmet- ric bases are learned separately for every layer and attention head except the experimental section focused on shar- ing parameters across heads and layers. Adjusting the skew-symmetric basis matrices' block width allows us to incrementally adjust the capacity allocated towards position encoding. We specify the basis block width subscript, eg. LicREs. When not specified, the block size is equal to the head dimension. If we set the block size to 2, we recover RiPE-Mixed (Heo et al., 2024). For a ViT-B model the head dimension 64.
5

---

LieRE; Lie Rotational Positional Encodings
5.1. 2D Image Classification
Table 3. Comparing different positional encodings synthetic
We begin with CIFAR-100 and ImageNet-Ik benchmarks to evaluate LieRE in 2D vision tasks. All models use ViT- based architectures trained from scratch with standard data augmentations (Rand/uggeet). We compare LieRE to ab- solute positional encodings, RoPE-Mixed (Heo et al., 2024), and VisionLLaMA (Chu et al., 2024). Table shows that LieRE outperforms all baselines. On CIFAR-I00, LieREs achieves a statistically significant relative improvement in top-1 accuracy relatively of 10.0% over absolute encodings, 7.3% over VisionLLaMA, and 2.2% over RoPE-Mixed. Sim- ilar trends hold on ImageNet (Table 2).
task on base model (85M)* equivalent to DeiT architecture across
different resolutions, 108 x 108, 168 x 168, 276 x 276
Mathod
108
168
276
Abs. Pos. E."
45.1 (43.1-47.7)
(390043)))))))
40.1 (382-42.0)
Abs. Pos. E." (2M examples)
47.2 (45.2-4..)
Iasi RoPE
46.4 (44.0-47.9)
48.6 (46004999)
48.6 (46.2-50.1)
RoPe-Mixed
(99.5-100)
98.5 (97.6-977)
(87.-99))
LieRE;
99.5 (99.2-99.8)
99.7 (99.4-99.9)
99.7 (99.5-99.9)
LickEs4
(99.5-100)
100 (99.6-00))
100 (99.7-100)
5.2. Synthetic Spatial Reasoning Task
We also investigate performance across model sizes (ViT- Tiny, Base, Large). As shown in Table and Figure 8, LieREs consistently outperforms other methods.
A
D
100
70
Figure 5. The model is asked to identify the direction of the arrow pointed to by base of the Y. Here said arrow points down. This task requires understanding spatial relationships.
65-
Max Accuracy
60
Recent observations indicate that even advanced models like ChatGPT-4 and Claude Somnet 3.5 struggle with basic spa- tial reasoning tasks. To investigate whether these limitations stem from positional encoding mechanisms, we designed a synthetic image classification task (Shah al., 2024). The task presents 108 x 108 pixel image containing a 9 grid (81 cells). Within this grid, we randomly place eight arros and six letters (A, B, C, D. E, and Y), ensuring that one arrow is placed in the direction of the base of the "Y". The objective is to identify the direction of this specific arrow. To introduce visual complexity, we include four spurious letters and seven additional arrows as distractors. Figure illustrates an example of this task setup. We train the mod- els on 800,000 examples and observe that they generally converge after the first 400,000 examples.
55
50
LieRE_ LieRE_60 64
45
RoPE-Mixed
VisionLIaMA Absolute PE
40
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Relative Dataset Size
Figure 4. Performance comparison of different positional embed- ding methods across varying dataset sizes. The plot shows the peak accuracy achieved by LieREs, LieREss. RoPE-Mixed, Vi- sionLlaMA, and Absolute PE when trained on different fractions of the CIFAR-100 datasct. Both LicRE variants and RoPE-Mixed consistently outperform other methods, with their advantage he- coming particularly pronounced in data-scarce scenarios.
We verified that none of ChatGPT 4o, Claude Sonnet 3.5 and Gemeni Pro 1.5 are able to solve this task.
Table outlines the performance different positional en- coding methods on synthetic task using base model (85M parameters) equivalent the DeiT architecture. We evaluate the models across three different input resolutions (108 x 108, 168 168, and 276 x 276 pixels), revealing differences in scalability and effectiveness. While absolute positional encoding shows degraded performance as reso- lution increases (from 45.1% 1% to 40. 1%) and RoPE-Mixed demonstrates strong but degrading performance at higher
To evaluate robustness in low-data regimes, we perform data ablation study. Figure 4 shows that LieRE variants and RoPE-Mixed maintain significantly higher accuracy than baselines when training on only 20-90% the CIFAR-100 dataset. This highlights LieRE's data efficiency.
6

---

LieRE: Lie Rotational Positional Encodings
47.5
Table Accuracy with parameter sharing over heads and layers
for VIT-B sized models on CIFAR-I0U 1000
11.0
Shared
Shared
73.3-
FLOP
All Shared Across Layers Across Heads
RoPE-Mixed LicRE LicRE,
23.0
5.684G
68.8
70.0
70.3
5.684G
68.7
69.5
69.8
03.3
5.613G
69.5
69.7
69.7
455
5.613G
V
68.3
69.4
69.5
00.0
45.0
61.5
05.0
resolutions (from 100% to 88.6%), both LieRE variants maintain near-perfect accuracy across all resolution scales, with LieREsa achieving 100% accuracy consistently. Quali- tative analysis shows that absolute and visionllama position encoding attend less clearly to the "Y" token than RoPE- Mixed and LieRE. Please refer the appendix for attention map examples, Figure 10 and Figure 12.
9.236
22,518
E4.532
130.240
285,596
Figure 6. Performance varies with skew-symmetric basis learnable dimensions, referred to as LieRE parameters. This is equivalent to increasing the tile size in the kyemeeymmetric basis (2 x 2, x 4, 8 x 8, 16 16, 32 x 32, 48 x 48). For both 2D (CIFAR- 100) and 3D (UCF101) LieRE with tile size 8 x 8 performs superior.
5.3. 3D Classification
Table 5. Relative accuracy drop for 2D image classification (CIFAR-100) and Video recognition (UCFI01) after patch shuf- fling
To assess LieRE's performance on 3D data, we use the UCFI0I video classífication benchmark (Soomro et al., 2012). All models use ViT-style backbone with 3D patch tokenization, trained from scratch with no hyperparameter tuning and the dataloader from (Tong et al., 2022). The full set of hyperparameters may be found in appendix B. We observe a relative accuracy improvement of the LieRE- based transformer of up to 15.1% compared to absolute position embeddings and at least 1.5% compared to RoPE- inspired position encodings (table 1).
CIFAR- 1-AAR-10::20 (20))))002))))))))000
UCFI0l (SD)
Method
Hetore Shuttingt
After
Deep(())
Before Shuffling
Atter Shuthing
Drope
Shutting
40.9
Visielhuu RoPU-Mised
45.0
17.7
LiRRE, LiRRE
70.3
47.0
70,0
Mixed.
5.4. LieRE Capacity and Parameter Efficiency
5.5. Patch shuffling: Measuring Positional Dependency
LieRE introduces minimal overhead only 580k addi- tional parameters (0.68% for ViT-B)-yet offers flex- ible mechanism for increasing representational capacity. To explore the impact of this marginal capacity, we vary the density of the keew-symmetric basis and examine parameter sharing across heads and layers.
Shuffling patches and frames allows us to see how much the model is able to use the positional information in its inputs. A model whose architecture does not allow/encourage the use of positional information will converge a representa- tion similar in spirit to a bag-of-words, where the relative locations of pixels/voxels do not matter. A greater drop-off in accuracy during shuffling is indicative that the model more heavily utilizes positional information.
We control capacity via imposing a block-diagonal structure on the basis matrices. Smaller blocks (e.g.. 2) repli- cate RoPE-Mixed, while larger blocks increase expressivity, with LieREat using fully dense basis. We adopt RoPE- Mixed's initialization for fair comparison. We observe that RoPE-Mixed is more sensitive to initialization than LieRE (Appendix Table 7).
We evaluate models using the decline in accuracy when evaluating on shuffled patches. We observe the most sig- nificant decline LieRE-based transformers, leading to the conclusion that LieRE models rely more on positional infor- mation as expected. The complete results are displayed for CIFAR- 100 and table for UCF101 (table 5).
Figure 6 shows performance as function of block size. Both in 2D (CIFAR-100) and 3D (UCF101), accuracy im- proves with block size, peaking around x 8 S-suggesting this is sweet spot between capacity and regularization.
5.6. Multi-resolution Classification
In this section we compare the ability of methods to gen- eralize to image resolutions not seen during training. We evaluate two training recipes inspired by (Heo et al., 2024).
We also assess the effect of parameter sharing on CIFAR 100 (Table 4). Learning LieRE parameters independently per head and per layer yields the best results. Sharing across lay- ers or heads reduces accuracy, but still outperforms RoPE-
The first recipe matches the rest of the paper and consists of training the models on images of size 224 224 for 200
7

---

LieRE: Lie Rotational Positional Encodings
VIT-B: Training at 224x224
VIT-B: Pre-training at 224x244, Fine-kune at 256x256
74
72
Validation Accuracy (%)
70
68
Training Resolution
training Rezolution
66
LieRE 64 RoPE-Moaee
LieRE 64
RoPE--Mood
192
224
256
320
384
448
192
224
256
384
448
Image Resolution
Image Resolution
Figure 7. Validation accuracy comparison between LieRE 64 and RoPE-Mixed positional embeddings across different image resolutions ImageNet. Left: Models trained at 224 224 resolution. Right: Models pre-trained at 224 224 and fine-tuned at 256 x 256 resolution. Both approaches show similar performance trends to 320 x 320, after which LicRE 64 demonstrates significantly better accuracy retention higher resolutions, particularly in the fine-tuned scenario.
epochs. The second adds an additional fine-tuning step at size 256 x 256 for 30 epochs. The full details can be found in appendix B.1.
coding methods. Our analysis indicates that LieRE-based ViTs effectively leverage spatial reasoning capabilities un- available transformers using only absolute positional codings. In addition to accuracy gains, LieRE offers notable data and compute efficiency. Its simplicity. flexibility, and strong capacity to learn spatial structure make it broadly applicable. With no tokenizer modifications beyond posi- tion output and no additional architectural changes, LieRE provides simple approach for controlling the amount of positional information into transformers.
We evaluate the accuracy the ImageNet validation set with varying inference resolutions. Specifically, we scale the input images to resolutions of 196 x 196, 256 x 256, 320x x 320, 384 x 384, and 448 x 448 pixels per dimension, and present the resulting accuracies in figure 7.
For position assignment, adopt a sequential approach where token positions are scaled proportionally to the image dimensions. For example, doubling the length of an image in each dimension doubles the range of positional indices. This method outperforms rescaling positions to fixed range, as demonstrated by superior results for both RoPE-Mixed and LieRE across the evaluated training recipes.
7. Limitations
While LieRE shows promising results for 2D and 3D inputs, several limitations are worth noting. For ID input, LicRE re- duces to RoPE with learnable phases (proof appendix A). Our method designed modify the inner product, making it compatible with most attention mechanisms, including standard softmax attention and linear attention. However, this may limit its applicability to other architectures -ucceessssuhhhsshusscc such as convolutional neural networks- do not rely on the attention mechanism. Future work could adapt the method to a broader range of architectures. The current formula- tion encodes vector positions in Rdd While sufficient for many applications, it may directly apply tasks that require pose encoding in SE((3 (c.g., robotics). Lastly, in its current implementation, LieRE relies on the accuracy and numerical stability of the matrix exponential in PyTorch. Future work may explore more efficient and robust imple- mentations or approximations of this operation. Despite
6. Conclusion
We proposed Lie group Relative position Encodings (LieRE), a positional encoding method that modifies the attention method via dense, learned, high-dimensional rota- tion matrices. As compared to the more widely used block- 2D rotation matrices typically used to encode positions, dense rotation matrices can encode both relative and abso- lute positional information and allow greater portion of the model's 's learnable capacity be allocated to spatial reason- ing, Experiments on 2D image classification (CIFAR-10,, ImageNet- k) and 3D video classification (UCF101) show that LieRE consistently outperforms existing positional en-
8

---

LieRE; Lie Rotational Positional Encodings
these limitations, believe our approach provides valuable insight into improving model performance and reducing training costs by encoding relative positional information across various input dimensionalities.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence ofe North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Impact statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending Ilm context window beyond 2 million tokens. 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom,
Acknowledgements
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
We would like to thank Maksim Maydanskiy for helping us understand Lie groups and Lie group representations. We would like to thank Aradhana Sinha for suggesting the shuf- fling experiment and providing early feedback on the paper. Furthermore, lucidrains suggested adding the proof of that in the ID setting, LieRE has identical representational ca- pacity to RoPE up to the Iearnable Sosis Ostmeier was supported by the German Research Foundation (DFG), Walter-Benjamin fellowship (ID: 51 517316550), This project was supported by Google Cloud and Azure credits. This work was also supported in part by the Medical Imaging and Data Resource Center (MIDRC), which is funded by the Na- tional Institute of Biomedical Imaging and Bioengineering (NIBIB) under contract 75N92020C00021 and through the Advanced Research Projects Agency for Health (ARPA-H).
Esteves, C., Allen-Blanchette, C., Zhou, X., and Dani- ilidis, K. Polar transformer networks. arXiv preprint ABKiv//709.08800 2017.
Esteves, C., Allen-Bili Blanchette, C., Makadia, A., and Dani- ilidis, K. Learning so (3) equívariant representations with spherical cnns. In Proceedings of the european confer- ence on computer vision (ECCV), pp. 52-68, 2018.
Falcon, W. A. Pytorch lightning. GitHub, 2019.
Falorsi, L.,, de Haan, P., Davidson, T. R., and Forré, P. Repa- rameterizing distributions on lie groups In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3244 3253. PMLR, 2019.
References
Chen, S., Wong, S., Chen, L., and Tian, Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15552 2023.
Flanders, A. F., Prevedello, L. M., Shih, G., Halabi, S. S., Kalpathy-Crann Cranmer, J., Ball, R., Mongan, J. T., Stein, A.. Kitamura, C., Lungren, M. P., et al. Construction of machine learning dataset through collaboration: the rsna 2019 brain hemorrhage challenge. Radiology: Artificial Intelligence, 2(3)=e190211 2020.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
Forestano, T., Matchev, K. T., Matcheva, K., Roman, A., Unlu, E. B., and Verner, Deep learning symmetries and their lie groups, algebras, and subalgebras from first principles. Machine Learning: Science and Technology, 4(2):025027, 2023.
Chu, X., Su, J., Zhang, B., and Shen, C. Visionllama: A unified Ilama interface for vision tasks. arXiv preprint 2rXiv:2403.00522. 2024.
Fuchs, F., Worrall, D., Fischer, V., and Welling, M. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970-1988 1981, 2020.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Ran- daugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
pattern
Fulton, W. and Harris, Representation theory: first course, volume 129. Springer Science & Business Media, 2013.
Deng, J., Dong, W., Socher, L.-J, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Gallier, J. Q. and Quaintance, J. Differential geometry and lie groups, volume 12. Springer, 2020.
9

---

LieRE; Lie Rotational Positional Encodings
Golovneva, O., Wang. T., Weston, J., and Sukhbaatar, S. Contextual position encoding: Learning count what's important. arXiv preprint S.3iv:24005187115 2024.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007), 20000 2023.
Heo, B., Park, S., Han, D., and Yun, S. Rotary posi- tion embedding for vision transformer. arXiv preprint arXiv:2403. 13298, 2024.
Shah, K., Dikkala, N., Wang, X., and Panigrahy, R. Causal language modeling can elicit search and rea- soning capabilities on logic puzzles. arXiv preprint arXiv:2409. 10502, 2024.
Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional transformers. arXiv preprint arXiv: /9/2. 2. 12 180, 2019.
Shaw, P.. Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations, 2018a. URL https:
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with it- erative attention. In International conference on machine PMLR, 2021.
/ //tpa//v..rg/bb///8000....
Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv: htl::8803.2115 2018b.
learning, pp. 4651
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 04088,, 2024.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.04/1..440. 2012.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding.
Neurocomputing.
Kumar, H., Parada-Mayorga, A., and Ribeiro, A. Lie group algebra convolutional filters. IEEE Transactions on Sig- nal Processing, 2024.
127063, 2024. ISSN 0925-2312. doi:
URL https:/www.scceenddiecctt sci science/article// sateee//artaceee/p////2222222222
Lee, S., Lee, S., and Song, B. C. Improving vision trans- formers to learn small-size dataset from scratch. IEEE Access, 10: 10:123212-123224, 2022.
Tai, K. S., Bailis, P., and Valiant, G. Equivariant transformer networks. In International Conference on Machine Learn- ing, pp. 6086-6095, PMLR, 2019.
Liao, Y.-L. and Smidt, T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11 11t.22206.19990 1/990, 2022.
Tong, Z., Song, Y., Wang, J., and Wang, L. Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training. Advances neural infor- mation processing systems, 35:10078-1000 2022.
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., Jiao, J., and Liu, Y. Vmamba: Visual state space model. Advances in neural information processing systems, 37: 103031-103063, 2024.
Touvron, H., Cord, M., and Jégou, H. Deit iii: Revenge of the vit. In European conference on computer vision, pp. 516-533. Springer, 2022.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEF/CVF international conference computer vision, pp. 10012-10022, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini 2xXiv:2302.139777 2023.
Liu. Z.. Hu, H., Lin, Yao, Z., Xie, Z., Wei, Y.. Ning, J., Cao, Y., Zhang, Z., Dong, I.,, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12009-12019, 2022.
Tworkowski, S., Staniszewski. K., Pacek, M., Wu, Y., Michalewski, H., and Mitoš, P. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
McLeish,, S., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B. R., Kailkhura, B., Bhatele, A., Geip- ing, J., Schwarzschild, A., et al. Transformers can do arithmetic with the right embeddings. arXiv preprint arXiv:2405. 17399, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
10

---

LieRE; Lie Rotational Positional Encodings
Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1p28--205031 2017.
Yoshioka, K. isiion-ransformmes-ccff1100:: Train- ing vision transformers (vit) and related models cifar-10. https://github. /enr//entaroy7/ oissioontrraassooomrrrrc--fllll 0, 2024.

---

LieRE; Lie Rotational Positional Encodings
A. Equivalence of RoPE and LieRE in one Dimension
Though focused on higher dimensional inputs LieRE remains compatible with ID tasks. It turns out that in the ID setting, LicRE has identical representational capacity to RoPE. This the case for higher dimensional inputs
Recall that in the ID setting positions scalars. The LieRE rotation is R exp(t.A) for some learnable skew-sy seew-symmetric matrix A. Recall that skew-symmetric matrices can be written in the form 8 A.S where and is orthogonal A has the structure denoted below.
0
0
A
0
A1
We can then use an identity of the matrix exponential to break down the LieRE rotation matrix.
R = exp(ts"AA) 5 exp(tA)S.
For two tokens in positions t, we denote the embeddings for a specific attention head as It, xV. If K, Q denote the corresponding key and query linear transformation matrices we can write the attention inner product with LieRE explicitly.
2. KR Qe = =s exp(IA)5 Sexp((
I,KS exp(tA)² exp(t' SSQee KT exp(tA)? exp(t' A)SQre
We let K' KS and Q' SQ, since these matrices are all learnable fold the matrix into parameters of the key and query linear layers for the given head, allowing us simplify the above expression.
2 K' exp(tA)" (p)'''
Now we use the fact that each block is skew symmetric. In the case of two dimensions,
exp ((xx 1)) (-s
cos(A) sin(A) sin(A) cos(A))
If we let Rx denote block diagonal rotation matrices with 2D rotations of angles Ao.... Ans we can rewrite the above expression in a more familiar form.
K R. Re Q'æe
This is exactly the formulation of the original RoPE position encoding. This also makes more clear how LieRE is different from RoPE-Mixed in the high dimensional setting. The above proof depends on the fact that we can decompose every rotation into a matrix of the form 5 AS with not dependent on the position, allowing us to fold the orthogonal matrices into the key and query matrices. This decomposition with constant S is guaranteed because the inputs the matrix exponential differ by only a scalar factor. This is longer true once switch to more than -inedineniional basis of skew symmetric matrices.
B. Experimental Details
B.I. Experimental Hyperparameters
The backbone for all experiments is configured as ViT-B, with 12 layers, hidden dimension of 768, and an intermediate dimension of 3096. We use dropout of O.1. We used CL.S pooling our implementation to facilitate comparability with
12

---

LieRE; Lie Rotational Positional Encodings
existing literature in the field. Further experiments revealed substantial performance improvement with mean pooling and LieRE. We use the pytorch lightning framework for all experiments (Falcon, 2019).
B.2. 2D Image Classification
The CIFAR experiments where trained 8xL4 GPUs with 24GB of VRAM each and all took under 30 minutes complete. The basis capacity scaling experiment was conducted using RTX6000 GPUs. The ImageNet experiments trained on 8xL40 GPUs and all took less than 2 days and 5 hours of runtime including time lost due to preemption and resource sharing, We use a cosine learning rate schedule with an initial learning rate of 1E 4 and train for 200 epochs. We use an effective batch size of 512. We use patch size of x 4 on the original 32 x 32 image for CIFAR-100 and a patch size of 16 x 16 on the randomly cropped and resized 224 x 224 image. All vision experiments used RandAugment (Cubuk et al., 2020). We use the ADAM optimizer with betas of 0.9 and 0.999 and le 8. The hyperparameters were tuned with RoPE-Mixed and selected before conducting the LieRE trainers as ensure fair comparison.
B.3. 3D Video Classifications
The 3D classification experiments were conducted on either A100 40GB GPUs or x A100 80GB GPUs with the effective batch size held constant either by using gradient accumulation or increasing the batch size. Similar to 2D classification, we use initial learning rate of E 4 with cosine decay, trained for 200 epochs, and had total batch size of 64 and patch size of 2 x 16 x 16 the randomly cropped and resized 8 x 224 224 video/image. We use the ADAM optimizer with betas of 0.9 and 0.999 and c = le 8.
B.4. Multi-resolution Classification
The second training recipe consists of 30 epochs with initial learning rate of 1E-5 with cosine decay. This mirrors the DEIT III training recipe that first pretrains at lower resolution and finetunes at higher resolution.
B.5. CIFAR-100 Performance Across Model Scales
We further evaluate the impact of incorporating LieRE across different model sizes CIFAR- 100, shown in Table 6. LieRE consistently outperforms the baseline with statistically significant gains across all three model scales. However, these results may be sensitive to dataset size, as all models are trained from scratch in this study. This is reflected in the performance drop observed with the ViT-Huge model.
75
70
Acturnry 190
65
60
55
VIT-Base
VITLarg
Figure 8. Performance behavior on CIFAR-100 100 (2D Image Classification) over ViT-Tiny (22M), VIT-Base (85M), ViT-Large (302M) for LieRE RoPE-Mixed and Absolute Encoding (Appendix, table 6).
B.6. Initialization Sensitivity
In order to understand the sensitivity of the methods to the initialization, we explore several scaling factor for the weights initialization. The results are presented table 7.
13

---

LieRE; Lie Rotational Positional Encodings
Table Comparison of Position Encoding Methods for Different ViT Models Sizes CIFAR-100, Accuracy (hootstrapped C5%CTT
Position Enc.
ViT-Tiny (22M)
ViT-Base (86M)
ViT-Large (302M)
Abs. Pos.
57.2 (56.2-58.1)
63.9 (62.9-65.8)
60.5 (59.5-61.4)
LaMA RoPE
58.2 (57.2-59.2)
65.5 (64.6 -66.5)
62.3 (60.4-64.2)
RoPE-Mixed
65.4 (64.5-66.4)
68.8 (67.9-69.7)
68.8 (67.9-69.7)
LieREs
65.6 (64.7-66.6)
70.3 (69.4-71.2)
69.9 (68.9-70.8)
LieRE84
65.3 (64.4-66.3)
70.0 (69.1-69.7)
68.9 (68.0-65 0-69.8)
Table 7. Paired Z-test between 2x and 1 initialization for LieREs and RoPE-Mixed. LieREg (2mr vs init) RoPF-Mixed (2x vs init)
Metric
Z-statistic
-1.81
-3.85
P-value
0.070
0.00012
Difference between means
-0.0118
-0.0255
95% Confidence Interval
[-0.0246, 0.0010]
[-0.0385, -0.0125]
C. Python implementation of LieRE rotation matrix computation
Baais_rrw_params nn. Parametet (
torch, rand
input_di www.nsimmaaaaiii.m
head_dim,
head dim,
math.pi & optional, inspired from RoPE-Mixed paper
upper_ triangle =
torch. (basis_raw_param)) diagonal=1)
skew_bases upper_triangle torch. transpcseeseeeeeee (upper_triangle -1, -2)
in_basis_positions in in_basis positions
positions. reshape (list (positions.shapa [1] * 2) skew_bases
rotation_ torch. wtw.summiinaaassoossssoeieooeeee dim=-3)
rotation torch her.etaaatrrkeeeerttttttooooooo http:t/ddttteetttch....ooo
rotation tttatioc.tooddyyppppstttt....... )ttp))
D. Compute efficiency
We demonstrate that LieRE-based transformer achieves comparable performance to the Absolute Position Embedding baseline (DeiT III (Touvron et al., 2022)) on CIFAR-100 with fewer training epochs. This represents notable advancement over recent methods such as VisionLlama and RoPE-Mixed (Chu et al., 2024; Heo et al., 2024). Figure 9 illustrates that LieRE enables x reduction in training compute while maintaining the accuracy achieved by absolute position encodings after 200 epochs.
14

---

LieRE; Lie Rotational Positional Encodings
Delf Baseline
3.5
Muttpiicaatve Reduction in Taining Compute
3.0
2.5
1.0
0.5
LieR 64
LieRE
Rope-Mlad
VistenLiMM
Method
Figure The LieRE spatial encoding allows the madel match the performance of absolute position encodings with substantially less training time.
D.1. FLOPS Comparison of methods
We find that since all methods we examine introduce a computational cost that at most linear the number of tokens, and runtime dominated by the quadratic attention component, there no substantial difference in computational efficiency between the methods. We list inference FLOP of the various methods in table 8.
Table 8. FLOP analysis with percentage increase compared to absolute position encodings
Position Enc.
ViT-Tiny (22M)
ViT-Base (85M)
ViT-Large (302M)
Abs. Pos. E.*
0.963G
5.607G
19.856G
VisionI RoPE
0.963G (+0.001 %)
5.607G (+0.002%)
19.856G (+0.000%)
RoPE-Mixed
0.964G 104%)
5.609G (+0.036%)
19.863G (+0.035%)
LieREs
0.968G (+0.519%)
5.617G (+0.178%)
19.882G (+0.065%)
LieREg4
0.970G (+0.727%)
5.684G (+1.375%)
20.061G (+1.033%)
E. Validation Losses
Table 2D image and 3D video classification T Validation loss (95% confidence intervals) results. All models 85.1M parameters for 2D tasks and 88. 7M parameters for 3D task (Krizhevsky et al., 2009; Deng et al., 2009; Soomro et al., 2012; Flanders et al., 2020) equivalent DeiT. equivalent Vivit (spatio-temporal).
Method
CIFAR-100 100
ImageNet- 1k
UCF101
Abs. Pos. E.
1.56 (1.47-1.56)
1.84 (1.81-1.86)
2.94 (2.92-2.96)
VisionLlama RoPE
1.56 (1.51-1.61)
1.98 (1.94-2.01)
2.66 (2.63-2.69)
RoPE-Mixed
1.38 (1.33-1.43)
1.72 (1.68-1.74)
2.52 (2.49-2.54)
LieRE
1.36 (1.31-1.41)
1.73 (1.70-1.76)
2.47 (2.44-2.49)
Lie
1.37 (1.33-1.42)
1.73 (1.70-1.76)
2.64 (2.62-2.67)
15

---

LieRE; Lie Rotational Positional Encodings
F. Basis parameters scaling
Table 10. Accuracy Results for Different LicREg Parameters, relative the model size of 85.2M and 88.7M
Dataset
LieREg Parameter Absolute
LieREş Parameter Relative
Tile Size
Accuracy (%)
CI (95%)
CIFARION 100
9216
0.01 %
2
68.84
(67.93-69.75)
CIFAR 100
27648
0.03 %
4
69.28
(68.38-70.18)
CIFAR 100
64512
0.08 %
8
70.32
(69.42-71.22)
CIFAR TOO
138240
0.16 %
16
69.85
(68.95-70.75)
CIFARIO0
285696
0.34 %
32
69.65
(68.75-70.55)
CIFARION 100
580608
0.68 %
64
69.99
(69.09-70.89)
UCF101
9216
0.01 %
2
46,30
(45.89-46.71)
UCF101
27648
0.03 %
4
45.67
(45.26-46.08)
UCFI01
64512
0.08 %
8
47.03
(46.62-47.44)
UCF101
138240
0.16 %
16
46.86
(46.45-47..7))
UCF101
285696
0.32 %
32
46.42
(46.01-46.83)
UCFI01
580608
0.66 %
64
44.68
(44.27-45.09)
G. Attention Maps
We show the attention scores to the CLS token, averaged across heads for every layer. Each grid is rescaled that the minimum element has value zero and maximum value of one. Red indicates the maximal score, and blue indicates the minimal value.
16

---

LieRE; Lie Rotational Positional Encodings
Figure 10. Attention Maps (normalized), 108 x 108: RoPE-Mixed, LicRE, Absolute and Visionllama posi- tional encoding (x-axis). Layers 1-12 (y-axis). While RoPE-Mixed and LieRE learn to look the "Y", Ab- solute and Visionllama less and concentrate the arrows.
Figure 11. Attention Maps (normalized), 276 x 276: RoPE-Mixed, LicRE, Absolute and Visionllama posi- tional encoding (x-axis), Layers 1-12 (y-axis). While LieRE still learns look the "Y". Absolute and Visionllama do less so.
Figure 12. Comparison of attention maps across different resolutions and position encoding methods. 17


Reviews:
[{"review_id": "9WYobYmMue", "reviewer": "Reviewer_SwFq", "summary": "The paper introduces a positional embedding encoding based on Lie Groups. The idea of the paper is to parameterize the positional embeddings using skew symmetric matrices. The authors show the benefit of the proposed method in terms of generalization, data efficiency and compute needed.\n\nOverall, the idea is novel and interesting. The authors have empirical evidence that validates the quality of their work. However, the paper quality needs to be improved, both in terms of presentation as well as writing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and thorough review and writing feedback which has helped strengthen the paper. We have addressed the typos and writing style in the revision based on your comments.\n\n**Equivariant work comparison**: We are excited about the equivariant line of work! We believe it is key to learning sample efficient representations for domains with extensive symmetry. We will include a new section of the related work that relates LieRE to the suggested works on equivariance and other work on lie groups in ML.\n\nAs a complimentary architectural change, equivariance transformers do not directly compete with position encodings such as LieRE. Position encodings such as LieRE, RoPE-Mixed and absolute position encodings are minimally invasive modifications to the base transformer architecture, making them compatible with many different architectures. In the case of LieRE, this enables things like finetuning existing LLM weights to become multimodal models capable of handling high dimensional data. \n\nCombining them would be an exciting area of research. One of the observations present in both this work and the most closely related work [1] was that, sometimes, translation-invariance is helpful, but in other cases having access to the reference coordinate system can help performance. This is supported by the fact that adding absolute position encodings to RoPE-Mixed, and LieRE performs best in the regime where the attention patterns are not necessarily constrained to be translation invariant (recall that the attention patterns are translation invariant only when the block size is constrained to two). Fortunately, this is compatible with many of the designs in the equivariant line of work.\n\nThe natural question is how do we let models benefit from both the sample efficiency of equivariance and the fact that sometimes the absolute coordinate system does contain useful information. Combining these methods in the right way is an exciting future direction but has enough technical complexity that it is hard to incorporate into the current paper without losing focus and diluting individual learnings around positional encodings. The contributions in this paper are easiest to understand when comparing to earlier works that modify the same aspect of the transformer architecture. We hope you agree that the revisions to the paper provide a stronger connection to equivariant line of work and an extended related work accelerates future research in the area.\n\n**Equivariance Related work**:A related branch of work encoding problem structure focuses on equivariance. We say that a model T is equivariant with respect to $f$ if T(f(x)) = g(T(x))$ for some $g$ [8]. Where with relative position encoding we often want to be able to encode translation invariance, equivariance provides a more general framework. Equivariance has been applied to improve performance on problems with a wide array of structures, ranging from rotation-invariance [10,13,14], 3D reference frame-invariance [9,12] and many others. The subset of these works that focus on generating equivariant token embeddings for transformers can be combined directly with LieRE or another rotation-based position encoding.\n\n**Lie Groups in Machine Learning**:Lie groups have also had extensive use in machine learning. The range of works is diverse, ranging from algebraic signal processing [15], automated discovery of symmetries [16] to state estimation [18]. Furthermore [17] provides a friendly introduction to differential geometry and lie groups that may be of interest to the reader.\n\nWe thank the reviewer for highlighting this connection and helping us improve the paper.\n\nReferences are replaced with links to respect character limits.\n[1] https://arxiv.org/abs/2403.13298\n\n[2] https://arxiv.org/abs/2403.00522\n\n[3] https://arxiv.org/abs/2404.02905\n\n[4] https://arxiv.org/abs/2212.09748\n\n[5] https://arxiv.org/abs/2112.10752\n\n[6] https://arxiv.org/abs/2411.04097\n\n[7] https://arxiv.org/abs/2406.15955\n\n[8] https://arxiv.org/abs/1901.11399\n\n[9] https://arxiv.org/abs/2206.11990\n\n[10] https://arxiv.org/abs/1709.01889\n\n[11]https://proceedings.neurips.cc/paper_files/paper/2021/file/2a79ea27c279e471f4d180b08d62b00a-Paper.pdf\n\n[12] https://arxiv.org/abs/2006.10503 \n\n[13] https://arxiv.org/abs/1711.06721\n\n[14] https://arxiv.org/abs/1612.04642\n\n[15] https://arxiv.org/abs/2305.04431 \n\n[16] https://arxiv.org/abs/2301.05638 \n\n[17] https://link.springer.com/book/10.1007/978-3-030-46040-2 \n\n[18] https://arxiv.org/abs/1903.02958 \n\n[19] https://arxiv.org/abs/1912.12180\n\n[20] https://arxiv.org/abs/2103.03206\n\n[21] https://arxiv.org/abs/2103.14030 \n\n[22] https://arxiv.org/abs/2401.10166", "claims_and_evidence": "The paper claims are well supported by the experiments.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense.", "theoretical_claims": "There are not theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper expands the equivariant work in terms of positional embeddings for Lie groups.", "essential_references_not_discussed": "The paper overlooks several works in equivariant transformers and neural networks:\n\n- Equivariant transformer networks, Tai et al\n- Equivariant Neural Functional Networks for Transformers, Tran et al\n- Equiformer: Equivariant graph attention transformer for 3d atomistic graphs, Liao et al\n- Polar transformer networks, Esteves et al\n- Efficient equivariant network, He et al, \n- Se (3)-transformers: 3d roto-translation equivariant attention networks, Fuchs et al\n- Learning so (3) equivariant representations with spherical cnns, Esteves et al\n\nAlso a lot of work in equivariance:\n- Harmonic networks: Deep translation and rotation equivariance, Worral et al \n\nEven in lie groups ML, there are plenty of references missing:\n- Lie group algebra convolutional filters, Kumar et al.\n- Deep learning symmetries and their Lie groups, algebras, and subalgebras from first principles, Forestano, et al\n- Differential geometry and lie groups, Gallier et al\n- Reparameterizing distributions on lie groups, Falorsi et al", "other_strengths_and_weaknesses": "Comments:\n\nC1  - Use a definition for the property of equations (1) and (2). For example, for eq 2 use\nComm(U,V).\n\nC2 - Page 4, line 216 - Algorithm should be capitalized. All the subsequent calls to the word Algorithm should also be capitalized. \n\nC3 - Page 5, a lot of blank space. I believe it would improve the quality of presentation to fix this. \n\nC4 - The text under Figure 1 is confusing and needs a better structure. Some quantities are define but never used. \n\nC5 - In the experiments, a piece of the text was removed, making the sentence incomplete:\n\n“We train the models on 800,000 examples and observe that they generally converge after the first 400,000. The only exception to this is absolute position encodings, where we have variants trained on 800,000 and”\n\nThere is also a missing reference:\n“Please refer to the appendix for attention map examples, Figure 13 and Figure ??.”\n\nC6 - In page 15, the text covers the page number.", "other_comments_or_suggestions": "See before.", "questions_for_authors": "The authors do not include any equivariant transformer architecture and only compare with 2 other embedding works. Why are the authors not comparing themselves with any architecture of the ones mentioned before. \n\nI believe that the authors need to compare their results with some equivariant / quasi equivariant work of the ones mentioned before.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None."}}, {"review_id": "J3MlWaQAUb", "reviewer": "Reviewer_udrb", "summary": "The authors mainly proposed a new positional encoding method called Lie, to replace the previous wildly used RoPE. It is used to improve the spatial relationship representation, especially in 2D and 3D images. Extensive experiments are conducted on classification tasks, and with the proposed PE, the accuracy values are all improved by a significant margin.\n\n## update after rebuttal\nI carefully read the authors' rebuttal, and thanks so much for the responses.\nThe authors also agree that it currently lacks evidences on image generation tasks, and some of the experiments are limited in design and scope. There issues are not fully resolved actually, and the authors did not clarify how to address them in the final version. The authors mainly used these experiment to \"inspire\" other future works, which I feel not very informative. \nHowever, the generalization design of RoPE to Lie group itself is interesting. \nSo better AC can make the final decision to balance these factors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer udrb,\n\nWe appreciate your recognition of our work's theoretical merits and experimental contributions. We have carefully considered your feedback and would like to address each point.\n\n**Long Context for 1D**: LieRE is primarily focused on inputs with dimensionality greater than one. In fact, there is the [non-obvious] property that, in the 1D setting, LieRE has equivalent representational ability to RoPE with learnable frequencies. This is not the case in higher dimensions. We include a proof of this fact at the end of the response, and will include it in the paper as it has been requested by other readers. \n\n**Long Context for 2D and 3D**: This implies that the natural equivalent of long-context modeling is evaluating the model at resolutions higher than it was trained at. This is the focus of section 5.6. (Multi-Resolution Classification). We evaluate with up to four times as many inference tokens than during pretraining and finetuning (Figure 9).\nHigher Resolution image generation: We agree that high resolution image generation is an exciting application to benchmark position encodings. The long-context image classification examples were motivated in part to create apples to apples experiments when compared to prior works [1, 2], as these are more focused on image classification. We are very supportive of future work in that application. This application is especially exciting in light of recent autoregressive image generation techniques such as VAR [3] that scale more predictably, enabling smaller scale experiments. It would be particularly interesting to see how position encodings could influence global consistency.\n\n**Patch shuffling**: We hope to clarify that the intent of these experiments is to provide insight to the mechanics of why the methods perform differently rather than to show one method is better than the other. Patch shuffling allows us to see whether the model is actually using positional information. We agree with you that it does not identify which method is best at the ultimate applications. For that we depend on the other experiments. The focus of the experiment is to help the reader build an intuition of what is going on with various position encodings. \nEach method compared is distinct in both how and what kind of positional information it is capable of encoding. LieRE has the ability to use both relative and absolute positional information, and, ideally, we would like to rule out there being a simpler method that could perform just as well. The drop in performance when shuffling patches is one limited datapoint consistent with that hypothesis.\n\n**Synthetic task**: We wholeheartedly agree with the assessment that the synthetic task has limitations. Still, we can see clear failures of basic spatial reasoning even in frontier models trained with resources well beyond the reach of any academic lab. Reproducing similar patterns in simple experiments greatly improves the accessibility of studying these issuesIn fact, understanding these failures is an increasingly growing area of research [6,7]. We believe better position encodings are one of the ingredients that will be necessary to resolve these limitations and hope the data point in our paper is suggestive of that. It is important to note that many VLMs are still trained with absolute position encodings that perform substantially worse than the relative position encoding we benchmarked (Table 1). While paper coherence and practical considerations prevent us from fusing this work with a new research project focused on resolving these limitations on spatial reasoning, we hope this experiment serves as a datapoint used to inform future work on spatial reasoning.\nWriting and figures: Thank you for the feedback on the writing and figures. We have used it to improve the readability of the paper and make changes such as switching to more readable high contrast color schemes for the figures. (such as Figure 5: https://postimg.cc/D8w2tcKp)\n\nThank you for the thoughtful review, and we hope we addressed your remaining concerns.", "claims_and_evidence": "The evidences are mostly well supporting the claims. The proposed LiePE greatly improves the transformer-based classification model by a large margin. Figure 9 also shows great generalization capability of the positional encoding to higher resolution. Experiments also show that the compute increment is not significant.", "methods_and_evaluation_criteria": "The idea and the theory behind the proposal is elegant and interesting. Using the exponential of skew-symmetric matrix to generalize RoPE  is intuitive and smart, making the position encoding fully learnable and more expressive. Experiments on classification is a simple yet effective say of validating the idea, and the baseline comparison is clear and fair. \n\nGiven now RoPE is more verified in image generation task, it will be better to show the potential of LiePE on image generation task.", "theoretical_claims": "Not applied to this paper.", "experimental_designs_or_analyses": "The main concerns for the experimental analysis is, the newly proposed PE is only used for image classification task and lower-res images. It is not sufficient to prove the effectiveness and expressiveness of the new PE for very long-context modeling. The image understanding tasks on the synthetic data is also very limited to prove its effectiveness. The patch shuffle experiments are interesting, but with random patch shuffle, it's not that meaningful to compare the dropping rate when the accuracy from different approaches are similarly low.", "relation_to_broader_scientific_literature": "The proposed PE is supposed to be a very general approach and a plug-and-play components for all transformer-based models. The idea has its merits and it has great potential to be generalized to high-resolution image generation task. However, the experiments in this paper cannot well support the claims and might not bring significant impacts in the literature.", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "The overall writing is not that easy to follow. \nMissing figure numbers in line 307, and the formatting needs improvement. \nFigure5. is too hard to parse.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynusNde6cA", "reviewer": "Reviewer_gAmX", "summary": "LieRE extends the popular RoPE by replacing its block-diagonal 2D rotation matrices with learned, dense, high-dimensional rotation matrices derived from Lie group theory.\nThe authors show that LieRE addresses key limitations of RoPE, particularly for multi-dimensional data like images and videos. Specifically, while RoPE was originally designed for one-dimensional sequence processing (like text), LieRE generalizes position encoding to higher dimensions through the use of Lie groups. The method involves learning skew-symmetric basis matrices and computing rotation matrices for n-dimensional positions, which are then applied to keys and queries in the attention mechanism.\nThe paper evaluates LieRE against other positional encoding methods on several tasks, the results show that LieRE outperforms competing methods across these tasks, with particular advantages in data efficiency, resolution invariance, and when processing limited training data.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gAmX,\n\nThank you for your thorough and supportive review. We particularly appreciate your recognition of our mathematical foundations and empirical results. We have built upon your feedback to further improve the paper. In addition to the changes below, we have also expanded the paper to better relate our work to other architectural work such as perceivers and axial attention.\n\n**On sensitivity to initialization**: We originally discovered the sensitivity as we were iterating to reproduce the results of RoPE-Mixed. RoPE-Mixed seems to be particularly sensitive to the scale of the initial weights. LieRE is about half as sensitive. We have added a section to the appendix that explores this in greater detail, but in short, RoPE-Mixed shows up to a 2% drop in CIFAR100 performance while  LieRE shows a 1% drop. Outside of the new initialization sensitivity experiment, all experiments for both LieRE and RoPE-Mixed are performed with the setting where RoPE mixed performs best. The key table is presented below.\n| Metric | LieRE_8 (2π vs 1 init) | RoPE-Mixed (2π vs 1 init) |\n|---------|------------------------|---------------------------|\n| Z-statistic | -1.81 | -3.85 |\n| P-value | 0.070 | 0.00012 |\n| Difference between means | -0.0118 | -0.0255 |\n| 95% Confidence Interval | [-0.0246, 0.0010] | [-0.0385, -0.0125] |\n\n**1D evaluation tasks**: We have added a section characterizing the use of LieRE for 1D tasks. In short, LieRE is equivalent in representational capacity to RoPE with learnable frequencies for one-dimensional tasks. Please see the response to reviewer “udrb” for the proof of this fact that will be included in the paper. Our empirical experiment confirms this equivalence.\nHigher-dimensional evaluation tasks: Thank you for the suggestion! To our knowledge, LieRE is the first approach to extend rotational position encodings to 3D data, and we are eager to explore scaling to 4D. Given the dataset size required to train transformers on high-dimensional data, such extensions may currently rely more on synthetic data or alternative ways of structuring dimensional information. We see this as an exciting next step and appreciate your insights on pushing this further.\n\n**Related Work**: Thank you for pointing out the connection to Axial attention and Perceivers. We have added a section of related work focused on related works towards compute-efficient scaling beyond sequence data. \nAxial Attention [19] reduces computational complexity by applying attention along specific axes (e.g., rows and columns in images), enabling transformers to scale efficiently with high-dimensional data. Perceiver [20] utilizes latent tokens to compress high-dimensional inputs into a smaller set, improving scalability as input size and dimensionality grow. These methods address the inefficiencies of traditional transformers when applied to high-dimensional data. Additionally, techniques like Swin [21] and Vmamba [22] optimize compute for visual data. Swin Transformer introduces a hierarchical approach with shifted windows, limiting attention to local regions to reduce complexity while capturing global context.  Vmamba, on the other hand, proposes a visual state space model that represents images as a collection of spatial states, allowing attention to be applied efficiently across large-scale visual inputs by exploiting spatial locality and reducing redundant computation. \nIt would be great to compound these methods with LieRE, as in this work we use a plain encoder transformer. \n\nThank you for constructive feedback which we were able to use to further improve the paper. We are excited to share this work with the community. \n\n\n**=== LieRE vs. RoPE 1D proof  ===**\n\nThough focused on higher dimensional inputs LieRE remains compatible with 1D tasks. It turns out that in the 1D setting, LieRE has identical representational capacity to RoPE. This is not the case for higher dimensional inputs for reasons that will be clearer later in the exposition. We include a cut-down version of the proof we propose to add to the paper below.\n\nRecall that in the 1D setting positions are scalars. The LieRE rotation is $R=\\\\exp(tA)$ for some learnable skew-symmetric matrix A. Recall that skew-symmetric matrices can be written in the form $S^T \\\\Lambda S$ where $S$ and is orthogonal and $$\n\\\\Lambda = \\\\begin{pmatrix}\n0 & \\\\lambda_0 & & & \\\\\\\\\n-\\\\lambda_0 & 0 & & & \\\\\\\\\n& & 0 & \\\\lambda_1 & \\\\\\\\\n& & -\\\\lambda_1 & 0 & \\\\\\\\\n& & & & \\\\ddots\n\\\\end{pmatrix}\n$$\n\nWe can then use an identity of the matrix exponential to break down the LieRE rotation matrix. \n$R = exp(tS^T \\\\Lambda S) = S^T \\\\exp(t \\\\Lambda ) S$. For two tokens in positions $t, t’$ we denote the embeddings for a specific attention head as $x_t,x_{t’}$. If $K, Q$ denote the corresponding key and query linear transformation matrices we can write the attention inner product  with LieRE explicitly. \n\n**... continued at the end of next rebuttal ...**", "claims_and_evidence": "The claims made in the paper are well-supported by empirical evidence, but i just want to point out that the assertion that LieRE provides a \"unified approach\" for handling different dimensionalities is supported by experiments on 2d and 3d data, but testing on additional dimensionalities (like 1d sequences, higher-dimensional data) would make this claim more robust.", "methods_and_evaluation_criteria": "The methodology and evaluation criteria seem well designed, even though the paper focuses on classification tasks, which may not fully showcase the advantages of better positional encodings. Tasks requiring fine-grained spatial understanding (like segmentation or object detection) would provide a more comprehensive evaluation.", "theoretical_claims": "There's no formal proof that LieRE can handle \"exponentially many relative positions for n-dimensional data\" better than alternatives, though the empirical results are supportive.", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "The paper builds directly upon RoPE and its variants (RoPE-Mixed, VisionLlama), clearly identifying limitations and proposing extensions. The application of Lie group theory to positional encodings is a new connection between abstract algebra and deep learning architectures.\n\nThe paper doesn't extensively discuss connections to other approaches for handling multi-dimensional data in transformers, such as axial attention or perceiver architectures. \n\nOverall, while the paper makes a significant contribution to positional encoding research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other weaknesses:\n- The authors note that \"RoPE-Mixed is sensitive to the initialization of weights,\" suggesting that LieRE might share this sensitivity, but they don't thoroughly explore how different initialization strategies might affect performance.\n\nOther strenghts:\n- The paper provides a solid mathematical foundation based on Lie group theory, extending positional encodings in beyond the current state-of-the-art methods.\n- LieRE shows very good ability to generalize to image resolutions not seen during training, outperforming other methods especially at higher resolutions.\n- The patch shuffling experiments offer valuable insights into how much the model utilizes positional information, with LieRE showing the most significant performance drop when positional information is disrupted.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rxw4EVd8Pl", "reviewer": "Reviewer_yH2U", "summary": "The authors introduce a type of positional embedding which extends the RoPE embeddings by introducing learnable rotation matrices.\n\n## update after rebuttal\nI thank the authors for their thorough response. In light of this, I will increase my score to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer yH2U,\n\nThank you for your thoughtful review and detailed feedback. We understand your concerns about the incremental nature and effectiveness of our work, and would like to address these directly:\n\n**On novelty and the primary contribution of the work**: We split up our contributions broadly into (1) analysis and (2) method. \n\n**Analysis**: The closest work to ours is RoPE-Mixed. We build on their work with both a substantially larger performance delta and more extensive analysis. We aim to include coverage of technical details not present in the most similar prior work such as the sensitivity to weights initialization (see the table in the response to reviewer gAmX) and different methods of defining extrapolated token positions. In addition to an extended quantitative comparison, we aim to provide qualitative insights to how the position encodings affect inference and training dynamics with the attention maps and patch shuffling experiments. Finally, we extend experimental coverage to study the effect of the dimensionality of the input, a first even for existing position encodings. Concretely, to the extent of our knowledge, this is also the first work that benchmarks RoPE-Mixed for 3D data.\n\n**Method**: We believe strongly that the presentation of a technical work should be as easy to understand. This motivates us to keep the connection to existing methods simple, including being direct about the settings where the methods are identical and the additional complexity of LieRE is worth the additional complexity.\nNovel technical machinery is required in order to utilize high dimensional rotation matrices that are dense or have block size more than two. In particular, we introduce the method of encoding the positions in a basis of skew symmetric matrices which is then passed through the matrix exponential to obtain dense high-dimensional rotations. This use of Lie group theory is novel, and the key ingredient in enabling the use of dense high-dimensional rotations with allows LieRE to go beyond RoPE-Mixed with statistically significant performance improvements.\n| Dataset | p-value (Rope-Mixed vs. LieRE_8) |\n|---------|----------------------------------|\n| CIFAR-100 | 1.3e-05 |\n| ImageNet-1k | 6.3e-03 |\n| UCF101 | 7.1e-04 |\n| 384 x 384 (Resolution Invariance) | 4.0e-04 |\n\nYour observation about block size unpredictability helped us recognize the need to better articulate our findings: LieRE_8 consistently demonstrates optimal performance across both 2D and 3D experiments (Appendix B.9. Basis parameters, Table 8). We will incorporate clear guidelines for practitioners based on our systematic analysis.\n\n**Writing Improvements**: We will address the writing issues you identified: complete the unfinished sentence in section 5.2, fix the undefined reference and clarify Figures 1-2 with improved visual explanations (Figure 5: https://postimg.cc/D8w2tcKp).\nWe believe these revisions will better communicate both the theoretical contributions and practical benefits of our work.\n\nWe again thank Reviewer yH2U for their review of our paper. We hope that the above responses adequately address all concerns.\n\n\n**=== LieRE vs. RoPE 1D proof continued ===**\n\n$$\n\\\\begin{align*}\nx_t K R_t^T R_{t'} Q x_{t'} &= x_t K S^T \\\\exp(t \\\\Lambda) S S^T \\\\exp(t' \\Lambda) S Q x_{t'} \\\\\\\\\n&= x_t K S^T \\\\exp(t \\\\Lambda)^T \\\\exp(t' \\\\Lambda) S Q x_{t'} \\\\\\\\\n&= x_t K S^T \\\\exp(t \\\\Lambda)^T \\\\exp(t' \\\\Lambda) S Q x_{t'}\n\\\\end{align*}\n$$\n\nWe let $K’=K S^T$ and $Q’= S Q $, since these matrices are all learnable we can fold the S matrix into parameters of the key and query linear layers for the given head, allowing us to simplify the above expression. \n$$ x_t K’ \\\\exp(t \\\\Lambda)^T \\\\exp(t’ \\\\Lambda) Q’ x_{t’} $$\n\nNow we use the fact that each block is skew symmetric. In the case of two dimensions, \n\n$$\\\\exp\\\\left(\\\\begin{pmatrix} 0 & \\\\lambda \\\\\\\\ -\\\\lambda & 0 \\\\end{pmatrix}\\\\right) = \\\\begin{pmatrix} \\\\cos(\\\\lambda) & \\\\sin(\\\\lambda) \\\\\\\\ -\\\\sin(\\\\lambda) & \\\\cos(\\\\lambda) \\\\end{pmatrix} $$\n\nIf we let $R_\\\\lambda$ denote a block diagonal rotation matrices with 2D rotations of angles $\\\\lambda_0, \\\\ldots, \\\\lambda_n$, we can rewrite the above expression in a more familiar form. \n\n$$ x_t K’ R_{t \\\\Lambda} ^T R_{t’ \\\\Lambda} Q’ x_{t’} $$\n\nThis is exactly the formulation of the original RoPE position encoding. This also makes more clear how LieRE is different from RoPE-Mixed in the high dimensional setting. The above proof depends on the fact that we can decompose every rotation into a matrix of the form $S^T \\\\Lambda S$ with S not dependent on the position, allowing us to fold the orthogonal S matrices into the key and query matrices. This decomposition with constant S is guaranteed because the inputs to the matrix exponential differ by only a scalar factor. This is no longer true once we switch to more than a one dimensional basis of skew symmetric matrices.", "claims_and_evidence": "The authors present reasonable although somewhat limited experimental validation, by training ViT models on CIFAR-10, Imagenet-1k and UCF101.\nThe gap between their model and RoPE mixed (from which it is an incremental modification) is very small.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The litterature review is thorough, and I appreciate that the authors are honest in acknowledging similitudes with existing methods (\"Note that the only difference between LieRE and RoPE-Mixed is that the latter constrains the rotations to be block-diagonal with block size two\").", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Paper is well-written and easy to follow\n- Method seems to marginally outperform existing methods at little compute increase\n\nWeaknesses:\n- Novelty: as acknowledged by the authors, this method is an incremental modification of the existing RoPE-Mixed embeddings where instead of having block matrices of block size 2, the block size becomes a hyperparameter. \n- Effectiveness: I am not convinced of the benefits of this method. First, as shown in figure 5 and table 2, the increase in performance is rather marginal. Second, as shown in Figure 8, the effect of this hyperparameter on performance is rather unpredictable, which does not make this method particularly practical. Although the sections 5.2 and 5.6 of the paper are a bit more convincing, I remain lukewarm about the effectiveness of the method.\n- Unpolished: the paper seems to have been rushed nearing the deadline and feels unpolished. Consider section 5.2: the second paragraph ends with an unfinished sentence (\"The only exception to this is absolute position encodings, where we have variants trained on 800,000 and\") and the third paragraph contains an undefined reference. Additionally, figures 1 and 2 are not very clear in my opinion.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: The paper had initial mixed reviews, where the major concerns were about limited experimental validation, unconvincing results, and missing discussion of related work. On the other hand, reviewers agreed that the idea is elegant and interesting and the method is built on a strong mathematical foundation.

The negative reviewers increased their score after the rebuttal so the paper ended up with unanimous acceptance recommendation. I agree with the reviewers. Although submission would be stronger with more convincing experimental results, the idea of using skew-symmetric matrices to generalize RoPE is quite interesting and I'm glad to see it being implemented.
Decision: Accept (poster)