Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Authors: Connor Schenck
Isaac Reid
Mithun George Jacob
Alex Bewley
Joshua Ainslie
David Rendleman
Deepali Jain
Mohit Sharma
Kumar Avinava Dubey
Ayzaan Wahid
Sumeet Singh
René Wagner
Tianli Ding
Chuyuan Fu
Arunkumar Byravan
Jake Varley
Alexey A. Gritsenko
Matthias Minderer
Dmitry Kalashnikov
Jonathan Tompson
Vikas Sindhwani
Krzysztof Marcin Choromanski
Venue: ICML 2025 spotlightposter (2025)
Keywords: position encodings for Transformers
RoPE
Robotics
translation-invariance
Lie algebra
Primary area: deep_learning->algorithms
TL;DR: This paper introduces STRING: a general class of separable translation-invariant position encodings for Transformers (extending Rotary Position Encodings), and applies them in a wide range of 2D and 3D scenarios, in particular in Robotics.

Abstract: We introduce $\textbf{STRING}$: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides $\textbf{exact}$ translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. 
We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers.
We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods. Videos of STRING-based robotics controllers can be found here: https://sites.google.com/view/string-robotics.

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Connor Schenck Isaac Reid A Mithun George Jacob
Alex Bewley + If Joshua Ainslie If
David Rendleman *i
Deepali Jain" +1
Mohit Sharma *1 Avinava Dubey +3
Ayzaan Wahid
Sumeet Singh
René Wagner' Tianli Ding',
Chuyuan Fu'
Arunkumar Byravan' Jake Varley' Alexey Gritsenko
Matthias Minderer'
Dmitry Kalashnikov Jonathan Tompson'
Vikas Sindhwani
Krzysztof Choromanski
Successful diffusion policy conditioned on a STRING-eehanced Transformer vision encoder, attempting the double- insertion task Aloha-sim. Bottom: Same experiment, but with regular vision encoder for which the policy fails. STRING provides strong improvements for training dexterous robotics policies, outperforming previous position encoding algorithms such as RoPE.
Figure Top:
Abstract
1. Introduction and Related Work
We introduce STRING: Separable Translation- ally Invariant Position Encodings. STRING ex- tends Rotary Position Encodings (RoPE; Su et al., 2024), a recently proposed and widely used al- gorithm in large language models, via unify- ing theoretical framework. Importantly, STRING still provides exact translation invariance, includ- ing token coordinates of arbitrary dimensional- ity, whilst maintaining low computational foot- print. These properties are especially important in robotics, where efficient 3D token representa- tion is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the uni- versality of our methods. Vidcos of STRING- based robotics controllers can be found here.
Position encodings (PEs) (Zhao et al., 2023; Kazemnejad et al., 2023; Chen al., 2021; Kiyono al., 2021) inject information about the respective locations of tokens into transformers (Vaswani al., 2017). They are essential for good performance because vanilla attention is set function, equivariant under permutation. In contrast, the meaning of sequence of tokens in general depends on its ordering.
APEs and RPEs. Practitioners initially relied on absolute PEs (APEs; Vaswani et al., 2017; Kiyono et al., 2021; Wang et al., 2021; Liu al., 2020) which add or concatenate fixed, precomputed position embeddings tokens. These have since been replaced by relative PEs (RPEs; Shaw al., 2018; Raffel et al., 2020; Li et al., 2024; Chi et al., 2022; Press et al., 2022: Chi et al., 2023), which add a learnable bias term that depends on the distance between tokens to the pre-softmax attention logits. RPEs tend to generalise better than APEs over varying sequence lengths. However, they often require explicit computation for every query-key pair.
RoPE. To address the limitations of RPEs and APEs, re- searchers recently introduced rotary position encodings (RoPF; Su et al., 2024; Heo et al., 2025). These have been widely adopted large language models (LLMs; Dubey et al., 2024; Gemma Team al., 2024). RoPE acts on queries and keys by partitioning them into 2-dimensional blocks, each of which is rotated by angle proportional
'Equal contribution 'Random order "Work done Google Re- search *Senior lead. 'Google DeepMind Universityy of Cambridge ! Google Research. Correspondence to: Krzysztof Choromanski <kchoro www.goo.ooo..............
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
to the token's position in the sequence. Whilst queries and keys are rotated separately, the angle of relative rotation is proportional to their separation, combining the best proper- ties of APEs and RPEs. Mathematically, for query and key of dimensionality d, RoPE involves [ Givens rotations (Bindel et al., 2002) acting on disjoint 2D subspaces.
2. Preliminaries
Let {x:}i== E IR denote a set of N d-dimensional tokens. Assume that d is even. The ith query, key and value vectors are given by q, = W W k; W&x; and v W respectively, where W, q. Wk: W. E Raxd are learned pro- jection matrices (to keep the notation simple, we assume here the one-head setting). The attention mechanism, basic computational unit of the Transformer, can be written as:
Besides providing strong empirical gains, two attractive properties have driven the enthusiastic uptake of RoPE.
1. Separability. RoPE transforms each query and key inde- pendently, based on its position. This happens once per token; the PE'd tokens are not recalculated during sub- sequent processing like autoregressive generation. This makes K KV-ccching convenient. Separability also makes RoPE compatible with linear attention, e.g. Performers (Choromanski et al., 2020; Katharopoulos al., 2020). Here, the attention matrix is not instantiated in memory so explicit RPE mechanisms are not possible.'
Expx(a k2)v, X; exp(a, ki)
This updates the set of tokens, mixing them dynamically depending on the query-key softmax similarity scores.
Rotary position encodings. As described in Section 1, RoPE rotates tokens depending on their locations. In the ID data setting (e.g. text), for token z; E {fg, at position i E N, we take z; RoPE(iiz; with
2. Translational invariance. For a query-key pair at posi- tions (i.j) E N2, the relative rotation angle depends only on i j. This improves sequence-length generalization.
4/2
RoPE(i)z, = θ p(10 [2::22n1122m 1:20
n=1
However, RoPE is not the only position encoding algo- rithm with these desirable traits. In this paper, we pro- pose a more general algorithm called STRING: Separable Translationally Invariant Position Encodings. STRING is based on Lie groups. It generalises RoPE via a unifying the- oretical framework, incorporating the latter as special case. In fact, we later prove that STRING is the most general PE algorithm with the properties above, amongst a broad class.
p(0)
cos
-sin 8
sin 0
cos
Here, ① denotes the direct product, so that each 2 x matrix (p(i0x)) independently rotates a 2-element section of the query/key, and [2: (n:)2n-1:2n denotes the 2n - and 2n elements of z, Note that the matrix (i is d x d, but it is only nonzero on the 2 x 2 blocks on the diagonal. Since p(O) T = p(-0) and 2D rotations commute, we have that
STRING for robotics. The above features are especially important in robotics, where efficient 2D/3D token represen- tation and sensible physical priors are key. To demonstrate it, we integrate STRING into Vision Transformers (ViTs), showing strong improvements for vocabulaay object detection models and various robotics controllers. This showcases the real-world impact of our STRING.
RoPE(1) RoPE(j) = RoPB(j - i),
(4)
whereupon we are transforming q k, - q RoPE(j-ikk; The dependence on j i makes this translationally invari- ant. RoPE takes the of angles (0n) determining the rotation frequency of each 2 x 2 block, as hyperparameters. We suppress this dependence for notational compactness. The authors originally proposed the decaying sequence 0n = 2-2 with hase wavelength À = 10, 000, though variants have since been explored (see below).
Videos of STRING-based robotics controllers can be found here:
Key contributions.
1. We introduce STRING, new family of position encod- ings for multidimensional token coordinates that respect both separability and translational invariance.
RoPE in higher dimensions. Whilst RoPE was originally proposed for sequence data, recent work has extended it to encode higher-dimenssooaal position information (Heo et al., 2025). Now each token is equipped with vector r, E Rd and we require: RoPE(r,) RoPE(r,) = RoPE(r, - rz). Since 2D rotations commute, one approach is to define de
2. We rigorously analyse STRING's theoretical properties (Sec. 3). proving that it is more general than RoPE. We provide computationally efficient implementations.
3. We show strong accuracy gains across varied models using Transformers with STRING, on range of robotics and general vision tasks (see Fig. 1 and Sec. 4).
RoPE(r,) = II RoPE([r]]a),
where [r;]e is the kith coordinate of (with k E (1,..., d. This independently applies regular 1-dimensional RoPE (Eq. 2) for each dimension of the position vector. The
Implicit relative position encoding schemes, which avoid in- stantiating the attention matrix memory, have also been proposed (Reid et al., 2024; Choromanski et al., 2022; Luo et al., 2021).
2

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
rotation frequencies {0.) 1/2 can optionally differ between each coordinate axis.
There are many ways parameterize such a set; we give examples in Section 3.2. Remarkably, the following is true.
Generalisations of RoPE. Prompted by its success, num- ber of papers have since sought to understand the effective- ness of RoPE and propose better- perter-performing alternatives. One well-known method argues to increase the base wave- length À to 500, 000, slowing the rate of token rotation and improving learning with longer contexts (Xiong et al., 2023; Roziere al., 2023). Another suggests to completely trun- cate the lowest frequencies, setting them to zero, which helps preserve long-range 'semantic channels' (Barbero et al., 2024). Practitioners can also make the parameters {0, ] fully learnable, improving flexibility. Lastly, re- cent work has proposed to replace the block-diagonal RoPE matrices RoPE(i) by more general dense matrices in SO(d), parameterized by learned antisymmetric generators (Ost- meier et al., 2024). Whilst more expressive, this algorithm breaks translational invariance for position vectors with de > 1, and has large memory footprint. This makes it unsuitable for robotics applications. In Section 3, we will propose better alternative, STRING.
Theorem 3.2 (STRING is general), Consider the set of mappings R(-) R. - R that satisfy the group- like translational invariance property R(r,) [((r))((r))))))))))))( R(r; r.) ,,, RRo, are continuously differentiable with respect to F, and satisfy R(0)- La (with La the d- dimensional identity). All such mappings can be expressed as STRING with some set of generators {Lg] k= C CTRadd
dd.
In this sense, STRING is the most general of all translation- ally invariant position encoding mechanisms using matrix multiplication. Meanwhile, RoPE is a simple special case of STRING, taking particular choice of generators. This can be seen as follows.
Theorem 3.3 (RoPE is type of STRING #1). Consider the generators L = >n (82p..02.1111111 82p-1, 1,2p where 1d/2 sponds to RoPE with rotational frequencies {0p)p=r
{ C R and 81,5 is the delta function. This corre- d/2
Proofs of Theorem 3.2 and Theorem 3.3 are in Appendix A.
3. STRING: Separable Translationally Invariant Position Encodings
3.2. Computationally efficient STRING
Despite being general and notationally compact, the param- eterization of the STRING matrices R(r,) shown in Eq. 6 may not be convenient for practical applications. Given N tokens at positions {I, } one must in general exponenti- ate and store N dense d d matrices. This incurs O (N time complexity and O (Nd)) space complexity cost. The problem is exacerbated if the ( differ across training examples and batches, which occurs e.g for point cloud data or color plus depth channel (RGB-D) images. In this case, (R(r:)) cannot be cached and reused. This motivates the goal of this section: to find efficient STRING instantia- tions, nonetheless more general and expressive than RoPE. We begin with the following (proof in Appendix A):
Recall that goal is modify queries and keys depending on their respective positions, so that changes to dot products q. ky depend on r. ry. RoPE achieves this using matrix multiplication (Su et al., 2024). Here, we present STRING: a more general, better- setter-performin algorithm.
3.1. Introducing STRING
STRING is defined as follows.
Definition 3.1. STRING is the mapping R(-)
Rdd. + Rddd from dy-dimensional position vec-
tors to d x d matrices, given by de
Theorem 3.4 (RoPE is a type of STRING #2). For any STRING position encoding with generators {Lx ld. there exists an orthogonal matrix P such that
R(I,)= = exp
(6)
where (L= C gdxd is a set of Iearnable and commuting akew-symmetric generators. Given set of queries or keys {2: C & at positions (r,) Rede their positions are encoded
R(r:) PRoPE(r:)PP.
Note that the orthogonal matrix P is independent of the coor- dinates r,, so it can be learned and stored once per attention head and shared across all training examples. Meanwhile, RoPE(r,) is sparse it is only nonzero on the super- and subdiagonals so multiplying tokens only requires O(Nd) memory and O(Nd)) time, saving a factor of d. This is crucial in the contrastive learning setting where batch sizes can become large. Once again, one can see that RoPE is a special case of STRING, this time taking P = La- We emphasize that the parameterization of STRING in Eq. 9 remains just as general as Eq. 6. We also note that, since
2, > R(r:)z Vi e N1.
Here, exp(-) refers to the matrix exponential, defined by its series expansion exp(A) = Di= A' A'/i! and [ri]e is the kth coordinate of vector r;- By "commuting skew- symmetric generators', we mean that [I+] satisfy
L. L,] = and L -L, V i,j-
(8)
3

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
in regular attention one takes dot products between position- encoded queries and keys, the first orthogonal matrix P will always cancel with its counterpart. Therefore, in Trans- formers it is sufficient to take R(r,)= = RoPE(r,) with P E (()) learnable, without loss of generality?
Theorem 3.5 (Cireulant-STRIING is fast). Given generalors L Ck- Cj with Cs circulant, the position encoding exp(2 LA (r.]k) yzz for token zi at position I; can be com- puted in Od d) time and (dd) memory using the fast Fourier Transform (FFT).
Equipped with Theo- rem 3.4, our goal becomes to choose suitable parame- terization for the orthogonal matrix P. One option is to take the Cayley Transform,
Example 1: Cayley-STRING.
We provide a proof in Appendix A. Circulant-STRING pro- vides another efficient position encoding algorithm that scales gracefully to large, high-dimensional datasets and performs well in spatial applications (see Section 4).
Paayley == - (I, +
(10)
Learnable frequencies with STRING. Note that the STRING generators from Definition 3.1 are (in general) learnable. For Cayley-STRING, the angle-definig frequen- cies for RoPE and S, the skew-symmetric matrix from Equa- tion (10) are learned whereas in Cireulant- the scalars {co. Cd-1) i in Equation (11) are learned.
where S is a learnable (potentially sparse) antisymmetric matrix (Diele et al., 1998). Pcayley is convenient since, for token 2₁, we can compute (Iu + +S) z; efficiently using linear solver, avoiding expensive methods such matrix inversions and matrix exponentials. Note that wherever we use P Cayley, we refer to our algorithm as Cayley-STRING.
Train
Inference
The unreasonable effectiveness of STRING. In some sense, Theorem 3.4 makes it surprising that STRING out- performs RoPE so comprehensively in all our experiments (see Section 4), given that they are related by a change of basis. It appears that the ability to explicitly leam this ba- sis change via P (shared between queries and keys), rather than implicitly via existing network weights, substantially boosts performance. Conversely, when using linear atten- tion variants, the projected tokens Wyq. and W are pushed through nonlinearities such as (eI.U(-)) before tak- ing the dot product. Hence, in this case, including learnable P does increase the capacity of the network, rather than simply learning a basis change.
STRING
Space
Time
Space
Time
Cayley
O (d))
O (d
O (d)
Circulant
(ld
O (dlog)
Od) O (d
Table 1. Space and time complexity of the presented STRING methods in terms of the token dimensionality per head d.
Computational complexity. Table I lists the computa- tional complexity of the presented STRING methods. With token dimensionality per head d, Cayley-STRINN duces d² d'/2 parameters per head (from the d/2 angle- defining RoPE frequencies and d (d - 1) /2 parameters which fully determine the learnable skew-symmetrie matrix used to generate Pcayley). Circulant- -SRIIGGG--GIIGGGGGGGG introduces d parameters per head (from the learnable circulant ma- trix fully determined by d parameters). In our experiments, d = 64 resulted in negligible increase of trainable parame- ters.
intro-
Example 2: Circulant-STRING. We now present sec- ond efficient STRING algorithm within our framework. A square matrix is referred to as circulant if it takes the form
co
Cd-1
2
c1
During training, Cayley-STRING takes O (da) time due to the linear solver. However, during inference, the learned orthogonal matrix P Cayley can be absorbed into existing g/ projections at no extra cost (for vanilla attention). Circulant- STRING only takes O (d d) time via the Fast Fourier Transform.
Ci
co
Cd
C2
C
=
Co
(11)
2
ca-1
Cca-1
Cd-2
C1
All rows are composed of the same elements, and each row is rotated one element relative to the preceding row. The transpose of a circulant matrix CT is also circulant, and the sum of two circulant matrices is also circulant. It follows that C-C T is circulant and antisymmetric. Lastly, circulant matrices commute. With these properties in mind, we can simply define L, = C₁ C₁ for & E {l,.... de). with C₁ a learnable circulant matrix parameterized by d scalars (co, - Ca-11. We call this Circulant-ST S/RNG.. This special parameterization is convenient for the following reason.
Both approaches improve upon ROPE, yet Cayley-STRING in general leads to larger improvements (see Section 4), Thus, we have here classic trade-off between qual- ity and computational expense. For applications with strict training performance constraints. we recommend Gireulant-STRING due to its compact computational foot- print, whereas for other applications Cayley-STRING is recommended.
C
3.3. Loose ends
We dropped the transpose sign the second P. redefining as our learnable orthogonal matrix.
Here, we discuss further generalisations of STRING.
4

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Extension 1: &-STRING. So far, we have followed RoPE in assuming that our position encodings are applied via matrix multiplication. However, this can be relaxed whilst preserving separability and translational invariance. For example, one can transform tokens 2, via the outer product with position feature vectors f(r;) E R
coordinates (0, + (0 A0, o AΦ) - transformation looking much more complicated in the Cartesian basis.
STRING for Abelian Lie groups. It follows that, simply by replacing Cartesian coordinates {r,} with their canon- ical counterparts, we can repurpose STRING to construct position encodings that respect more general invariances.
2; + ver( @ z.).
(12)
4. Experiments
Here, @ denotes the outer product and vec denotes the 'vec- torizing' operation that flattens matrix to vector, so that (e: & q: dbb f(r;)) where a E {1,..., 2m} and b E {I,..., d}. Since the dot product of (flattened) outer products gives the product of dot products, we have
In this section, we provide an exhaustive empirical compari- son of STRING with RoPE and vision encoders leveraging regular APEs. To set the stage, we start with general non- robotics experiments in Sec. 4.1. On our way to robotics applications, we then test STRING for 2D and 3D object detection in Sec. 4.2. Finally, we present robotics manipula- tion experiments in Sec. 4.3 and Sec. 4.4.
vec(f(r;; @ q:)" veec(((r)) & kj) = q. ky f(r:) T((r))
(13)
Now suppose that we take the Fourier features
4.1. General Experiments: Classification and Retrieval
1 vm
f(r:)
[cos(W] r:), sin(w] r:) r:))
(14)
We tested STRING for image classification tasks on the 2ma1 (Deng et al., 2009) and Places365 datasets, with Vision Transformer (ViT) (Dosovitskiy et al., 2021) as our base model. We compare against RoPE and RoPE- Mixed (Heo et al., 2025), abbreviated to RoPE-M, to Circulant-STRING and Cayley-STRING (respectively ab- breviated to Circulant-S and Cayley-S). The results are shown in Table 2. For both datasets, STRING offers best models. For ImageNee2202, the top two models are STRINGs. Furthermore, ImageNet2012 STRINGs provide absolute gains larger than 1%, compared to regular ViTs, with only negligible set of extra trainable parameters.
where {wi]kEt C Ed are learnable d-dimensional fre- quency vectors. Then we have that (((,)) = m Ex= cos(wa(r, r2)) which is clearly a function of r. - r.. We refer to this novel position encoding variant, or- thogonal to previous RoPE-like approaches, as &-STRING.
1
Extension 2: General transformation groups. Having focused on Iranslational invariance, another natural question is whether STRING could be repurposed for other contin- uous transformation groups. These may be more suitable for data with different properties; for example, one might sometimes prefer a rotationally invariant position encoding.
VIT
RoPF
RoPF-M
Circulant-S
Cayley-S
ImageNNet
80.04
80.18
80.86
81.22
81.09
More formally, recall that Lie group with parameters 1b E RR is group of transformations of the form To Rd - Rd that are differentiable with respect to vh. Let the parameter xb = 0 correspond to the identity element, so that Tox- = x. A canonical coordinate system for G is an injective map p from Cartesian coordinates to a new coordinate system, sat- isfying p(Txx)) = p(x)= + Σi=
Places365
56.79
56.97
56.69
56.77
57.16
Mean
68.42
68.58
68.78
69.00
69.12
Table 2. Image classification % test accuracy. Best numbers are highlighted in bold and the second-best numbers are underlined.
1 thee; TT E G, where e; is the ith standard basis vector. Observe that the right hand side of this equation represents translation in the new basis. Canonical coordinate systems exist for all eeeaparaeeter Lie groups (A: = 1), and more generally for Abelian groups of dimension A < d (Segman et al., 1992; Rubinstein et al., 1991; Tai et al., 2019). They can be derived analytically by solving a set of first-order PDEs, though for many common transformation groups the canonical coordinate system is obvious. For instance, for azimuthal and polar rotations of points (Tz: Ty. r=) in 3D space (k- = 2), a canonical coordi- nate system is )), where sin 0 == /r2 + r// //lrlll and tan Φ : / Rotating' simply 'translates' the canonical
i2t@ 1 i2t@5
i2t@
10 1
12i@5
12i@ 10
Mean
VIT
53.88
73.17
78.49
53.98
73.83
79.29
68.77
RoPE RoPE-M
55.27
74.27
79.52
55.22
74.61
80.25
69,86
55.30
74.08
79.47
55.36
74.73
80.18
69.85
Circulant-S 55.52
74.69
79.91
55.68
75.03
80.45
70.21
Cayley-S 55.70
75.08
80.24
55.82
75.40
80.65
70.48
Table 3. Image-to-text (i2t) and text-to-image (t2i) WebLI recall @ rank (best numbers: in bold, second-best: underlined.)
4.2. Improving Vocebloccyy Object Detection
1.2.1. VEEN-VOCABBLLARY OBJECT DETECTION IN 2D
Next, we lift WebLI (Chen et al., 2023), a dataset of 10 billion image-text pairs across a variety of languages, into 3D by pre-processing a 60-million image subset with Depth-
Note that this differs from full 3D object pose invariance, for which the transformations do not form an Abelian group
5

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Baseline
RoPE
RoPE-M
Circulant-S Cayley-S of 4 million images of indoor and tabletop scenes with groundtruth 3D bounding box labels (Lin et al., 2025) (see App.F.1 for details), From this dataset, we hold out 80 images for evaluation. We evaluate both ViT and ViTD variants. The 3D intersetiinn-oee--nnioo (IOU) values for various RoPE and STRING variants on the evaluation data are shown in Table 5. For each configuration, we train models with different random seeds and take the best per- forming model (see Appendix F.2.4 for details). Fig. 2 shows example 3D detections for 6 different variants (see App.F for details). Note that STRINGs provide much more accurate prediction of the 3D bounding boxes for more chal- lenging to localize smaller objects than baseline and RoPE. For ViT, Circulant-STT Frrculant-STRING STRING is the best, providing 1.5% rela- tive improvement as compared the best RoPE variant. For ViTD, Cayley-STRING is the best, providing 2% relative improvement compared to the best RoPE variant. For both ViT and ViTD, two best models are STRINGs.
Ape000
32.44
33.66
32.74
33.24
33.47
ApIMs
21.98
22.71
22.43
22.60
23.01
Mcan
27.21
28.18
27.59
27.92
28.24
Table Average Precision (AP) % of the OWL.-VIT model on COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019). Best in bold, second-bes underlined.
Anything- V2 (Yang et al., 2024) for metric mono-depth es- timation. The dataset is filtered using methods from (Chen et al., 2024) to remove images on which the indoor finetundd model performs poorly such those with overlays, visi- ble groundplane, large outdoor scenes, or optical illusions.
We perform contrastive learning on the text to visual rep- resentation pairs in the WebLI-3D lifted dataset, where the visual representation may be in the form of an RGB image or an RGB-D depth image. Similar to CLIP (Radford et al., 2021), this done by maximizing the similarity between the embeddings of matching visual-text pairs, while minimiz- ing the similarity between embeddings of the non-matching pairs. This enables open-vocabulary detection and classi- fication by comparing the text embeddings of all possible classes against those of the visual representation, and select- ing the minimum distance pair. We compare against baseline in Table 3. For all six evaluations, Gayley-STRING is the best and Firculant-STRING
Baseline
RoPE
RoPE-M
Circulant-S
Cayley-S
ViT ViTD
49.77
58.09
57.17
58.95
58.85
67.60
71.21
70.90
72.36
72.67
Table S. Average 3D IOU % over all objects for the 3D bounding box prediction task. For each setting, models were trained with different random seeds and the max is reported. Baseline indicates no RoPE or STRING. Best in bold, second- -bstt underlined. 4.3. Simulation-Based Robot Manipulation: ALOHA
second best.
We demonstrate the efficacy of STRING on open-vocabulary object detection for localizing a 2D bounding box on stan- dard RGB image benchmarks. For a baseline, we use the official implementation of OWI-ViT* (Matthias Minderer, 2022) which applies a standard Vision Transformer (Doso- vitskiy et al., 2021) with weight classification and lo- calization heads for detection. Table 4 compares the baseline OWL-VIT model with RoPE and STRING variants. For all experiments, we followed the standard training pipeline for the B/32 backbone with the CLIP loss (Radford et al., 2021).
We evaluate the performance of STRING on dexterous robotics tasks using ALOHA 2, a bimanual parallel-jaw gripper workcell with two 6-DoF arms, within the ALOHA Unleashed (Zhao et al., 2024) simulation environment (see: Fig. 1). ALOHA Unleashed utilizes scalable teleoperation framework used to collect human demonstration data.
Even in this axis-aligned 2D detection setting - which is favourable for the standard RoPE variant- Cayley-STRING provides the best overall performance.
4.2.2. OPEN- VOCABULARY OBJECT DETECTION IN 3D
We tested STRING the open-vocabulary 3D object bounding box prediction task, similar to those from Sec. 4.2.1. Here we modify the output to be the full SE(3) pose and 3D size of the bounding box. We initialize the weights of the vision and text towers of our model with the weights from the models trained on the WebLI-3D dataset from Sec. 1. We replace the IOU loss from OWL-ViT with an 8-comer vertex loss, but otherwise keep the same match- ing and loss algorithm. We train simulated dataset
Figure 3. Hannover task for the ALOHA 2 robot: real (top) and the corresponding simulated (bottom) evaluation.
We trained ALOHA Unleashed's Transformer- baaooommer---bedddddd neural network with diffusion policies (conditioned on vision en- coders) on human demonstrations of 12 dexterous tasks (see Appendix B for descriptions and renders). The vision sys- tem utilized images from RGB cameras positioned overhead, on each wrist, and the level of the table. We also deployed
sceni ttt/ http//maannn/cccaeeee wwt.mmaaaaaaaamaaaaaaaaaaaaaaeee http:///w//./meeeeeeeee///ot//ss
6

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Baseline
RoPE
Cayley-STRING
edfo.a273: Trvio
PURSUI
TREAK PURSUI
195]: Blosk and Wt
Blokk and white
Ma ed Black
Book and white Po
Art
Pngll Beige
Purce Anice
Hutita F881 Delg
Suato Art
Prod[o.7863 Belg
VIT
Prod Tit
SA PURSUI
AA PURSU
29 Block and
AAL PURSUI
rdfo.aas) Black
DLLA D
28884887 63 Beig
red(o.9223: Delp
red(0.9443: Belg
eup
Figure Example outputs for the 3D detection task for baseline, RoPF, and Cayley-S. Green boxes: groundtruth. Blue boxes: predictions.
our policies on real ALOHA 2 robots (see Fig. 3 and Fig. 11). Due to the large observed variance of the on-robot evaluation for ALOHA 2, we focused on the evaluation in simulation to accurately rank different methods.
VIT
RoPE
STRING
BowIOnRack
0.90
0.80
1.00
DoubleInsertion
0.20
0.50
0.60
FMB-1
0.20
0.20
0.20
FMB-2
0.10
0.10
0.10
Table 6 reports the best task success rate of using RoPE (Heo baseline SigLIP
FruitBowl
0.30
0.30
0.30
et al., 2025), and Cayley-STRING B/16 256 (Zhai et al., 2023) ViT model.
GlassOnRack
0.60
0.60
0.60
HandOverBBnana
1.00
1.00
1.00
HandOverPen
1.00
1.00
1.00
The success rate is averaged over 10 trials of each check- point, taken every 10K train steps and over 1M train steps. Corresponding curves are given in Fig. 4. Cayley- STRING achieves superior results across all tasks on average (i.e. MultiTask). Additionally, it achieves equivalent superior results, as compared to RoPE (e.g. for the Doublelnsertion task from Fig. I) and ViT for all 12 tasks except for Mu- gOnPlate and PlateOnRack (second-best), Finally, STRING converges much faster than other methods (see: Fig. 4). Note that we applied the strategy of learning all angle- defining frequencies for both RoPE and Cayley-STRING.
MugOnPlate
0.70
0.90
0.80
PlateOnRack
0.60
0.70
0.50
Singlelnsertion
0.40
0.60
0.60
StorageBin 0.00
0.00
0.00
MultiTask
0.37
0.42
0.46
Table 6. Mean success rate (best in bold, second-best underlined) over evaluations of each ALOHA simulation task. RoPE (Heo et al., 2025) and Cayley-STRING are added to baseline SigLIP B/16 ViT (Zhai al., 2023). See Appendix B for details.
4.4. Real-World 3D Robot Manipulation: KUKA
Establishing STRING as superior to other methods on previ- ous tasks, we let operate on data to obtain new SOTA robotic manipulation policies. This resulted in policies di- rectly using depth and deployed on real hardware. Note that STRING can be naturally applied in that context since can be used for data equipped with general coordinate vectors r, associated with tokens (e.g. 3D).
RoPE
0.4
STRING
Mean Success Rate
0.3
0.2
0.1
4.4.1. SETTING
0.0
We evaluated STRING in the vision encoder of a generative policy applying energy-based models (Singh et al., 2024) and deployed on a real industrial KUKA robot arm (Udayan
0.0
0.2
0.4
0.8
0.8
1.0
Training Step
1e6
Figure 4. Mean success rate across all tasks (i.c. MultiTask) evalu- ated times every 10K train steps over 1M train steps.
7

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
al., 2023). The closed-loop feedback policy operates on RGB-D images, and is learned generative model with imitation learning. Its architecture (see Appendix G for de- tails) consists of a diffusion Transformer and 3D encoders. The policy was trained on a mixture of scripted and teleop- erated data collected for 3 different skills (pick, place and handover) on various objects. It is evaluated exclusively on the pick skill with a diverse set of objects, Each evaluation was an A/B test for total of 50 trials.
80
74.8
65.3
15.1
Success Rate
42.2
40
20
4.4.2. REGULAR EVALUATIONS
2D
2D + mmap
nmpp + +0 STRING
3D STRING
We experimented with two ways of using depth in the policy.
Figure 5. Performance of STRING with 3D input vs. baselines on real-robot tasks (with 2 seeds). 2D baseline performance without depth input is Rs 65%. Incorporating depth through surface normal maps (nmap) reduces performance to 42%. Using 3D STRING for incorporating depth improves the performance in both scenarios with and without normal maps to 53% and 74% respectively. Mcan/stdev shown above were calculated from 35 evaluation runs.
Implicit depth via normal maps: In the first approach, following (Tziafas & Kasaei, 2023), depth input is used to construct surface normal map with unit R² values per pixel. Both RGB and depth inputs are then processed via identical (shared weights) embedding layers. The embeddings are concatenated and processed through Transformer layers. Finally, the embeddings are split and fused to yield the final vision embedding. Our results in Figure 5 show that this approach of incorporating depth has a detrimental effect on the on-robot deployed policy. We hypothesize that this is caused by the significant amount of noise coming from the depth sensors, leading to imperfect surface normal maps.
policies from Section 4.4.2. As seen in Figure 6, 3D STRING policies outperform 2D policies across all OOD settings. For instance, with large distractors (middle), the 2D model's performance decreases from 65% to 57%, while 3D STRING maintains performance similar to non-OOD) settings (Re 74%). In some OOD cases, such as lighting changes, both 2D (~ 10%) and 3D ( 25%) policies ex- perience a performance decrease vs. the non-OOD setup. This drop in performance during lighting changes is likely due the significant alteration in image observations, thus affecting both 2D and 3D policies. Finally, the largest per- formance difference is observed in the table height variation scenario. Here, the 3D policies exhibit significantly higher robustness (Rs 50%) compared to the 2D policies ( 10%). This suggests that the 3D STRING policy leverages the raw depth signal to better generalize to table height variations, a change imperceptible fixed monocular cameras.
Lifting patches to 3D for STRING: In the second ap- proach, we compute the height for each patch via mean- pooling across depth values for all the pixels in the patch, followed by the learnable linear layer. The resulting 3D patches are then fed to the Transformer, with positional encoding given by STRING to incorporate depth into the vision encoder. Our results Figure show that STRING im- proves the success rate over the 2D base policy. Also, when STRING is combined with the first method, it drastically reduces the negative impact of noisy normal maps.
We used Circulant-STRING to obtain a particularly compact computational footprint. Note that in this setting, more computationally intense mechanisms, such as (Ostmeier al., 2024), ran out of memory and could not be trained.
Overall, our results show that 3D STRING policies highly robust to many variations and significantly improve over 2D policies. Fig. 7 shows a sample episode from the on-robot evaluation of the STRING generative policy.
4.4.3. DUT-OF-DISTRIBUTTON EVALUATION: STRING VS BASELINE
Lighting
Large Distractors
Table Height
To further compare STRING with the baseline and show the advantages of using 3D over 2D policies, we also perform out-of-distributtin (OOD) evaluations on the real robot.
100
108
We vary three different environment settings. These include: (1) lighting changes, (2) adding large distractor objects and (3) changing the height of the table from which the robot has to grasp the block. For each setting, we test multiple different variations, e.g., three different light settings.
75
75
.2.9
30
30
52
22
12.5
STRBNGG
STR
30
Indicates performance on standard (non-ood) environments
-D
3D STRING
Figure 6. Comparison of 2D baseline with 3D STRING on out-of- distribution scenarios for real-world Kuka robot evaluations.
Figure compares STRING with the 2D baseline for each OOD setting. For these evaluations, we choose the best
8

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
References
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Veličković, P. Round and round we go! what makes rotary positional encodings useful? arXiv preprint 24X0:2211000605,, 2024.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang. X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschan- nen, M., Bugliarello, E., Unterthiner, T., Keysers, D., Koppula, S., Liu, F., Gryener, A., Gritsenko, A., Houlsby, N., Kumar, M., Rong, K., Fisenschlos, J., Kabra, R., Bauer, M., Bošnjak, M., Chen, X., Minderer, M., Voigt- laender, P., Bica, I., Balazevic, I., Puigcerver, J., Papalam- pidi, P., Henaff, O.. Xiong. X., Soricut, R., Harmsen, J., and Zhai, X. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.077766 2024.
Figure 7. Rollout frames from a successful policy evaluation in the real world (top: RGB, bottom: depth). Best viewed zoomed-in.
We have already demon- strated (see the normal map approach from Section 4.4.2) that just adding depth signal does not necessarily improve performance. STRING does so and exhibits another feature that other approaches (e.g. adding depth as extra channel) do not: it can be trained from a regular 2D pre-trained check- point. This is the case since STRING incorporates depth by using it to modulate regular attention matrix, effec- tively disentangling 3D specific parameters (defining the modulation) from the core 2D backbone. All training runs in Section 4.4 started from the pre-trained 2D backbones.
From 2D to 3D with STRING:
Bindel, D., Demmel, J., Kahan, W., and Marques, O. On computing givens rotations reliably and efficiently, ACM Trans. Math. Softw., 28(2):206-238, 2002. doi: 10.1 10.11455578806557880. URL httpp:://oi.... 10. 00.1145/567806.557790
5. Conclusion
Calli, B., Walsman, A., Singh, A., Srinivasa, S., Abbeel, P., and Dollar, A. M. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. arXiv preprint arXiv: 0333,,3,.3... 2015.
We introduced a new class of translationally invariant po- sition encodings (PEs) for Transformers, called STRING. STRING is the most general of all translation-invarian PE methods using matrix multiplications (under weak smooth- ness assumptions) and contains the prominent class of RoPE methods as its special instantiation. We proposed to apply STRING in robotics for 2D and 3D modeling and provided its extensive empirical verification over a range of tasks, from standard classification and retrieval, through object localization, to diffusion robotic policies conditioned on Vision Transformers. In all these experiments, we showed consistent gains over RoPE, as well as baselines applying regular absolute position encodings.
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Flo- rence, P., Sadigh, D., Guibas, L., and Xia, F. Spatialvim: Endowing vision-language models with spatial reason- ing capabilities, 2024. URL httts:://aaiiiiii /ttp///axvv./r///////////////lll abs /oss/2401.2 12168.
Chen, P., Tsai, H., Bhojanapalli, S., Chung, H. W., Chang, Y., and Ferng, C. A simple and effective positional encod- ing for transformers. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 2974-2988. Associa- tion for Computational Linguistics, 2021. doi: 10.18653/ V V1/2021.MMNPPMMIIN.... URL https:/ doi. http:/...655/111//// http://22211111111111111111
Impact Statement
The goal of this work is to contribute to the advancement of the machine learning field which may result in potential so- cietal consequences. We acknowledge these potential risks, especially in downstream use-cases of advanced machine learning techniques and advocate for careful consideration of ethical implications in the development and deployment of these techniques. Additionally, as it is the case for all papers discussing training Transformer architectures, the corresponding carbon footprint needs to be taken into ac- count. STRING plays a positive role here since it reduces computational costs by providing ways of fine-tuning al- ready pre-trained architectures with a negligible set of extra trainable parameters.
236.
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski. P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J.. Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B. K., Riquelme, C., Steiner, A., Angelova, A., Zhai, X., Houlsby, N., and Soricut, R. Pali: A jointly- scaled multilingual language- image model, 2023. URL https:// http///awiiirrrrrs///aaaaa 02799/2992...22....

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Chi, T., Fan, T., Ramadge, P. J., and Rudnicky, A. KER- PLE: kernelized relative positional embedding for length extrapolation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022.
Downs, L.,, Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T. B., and Vanhoucke, V. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Confer- ence on Robotics and Automation (ICRA), pp. 2553-2560. IEEE, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21 2024.
Chi, T., Fan, T., Rudnicky, A., and Ramadge, P. J. Dis- secting transformer length extrapolation via the lens of receptive field analysis. In Rogers, A., Goyd-Graber, I.,, and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023. pp. 13522-13537. Associa- tion for Computational Linguisties, 2023. doi: 10.18653/ V122023ACL-LLNG..T
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhu- patiraju, S., Pathak, S., Sifre, I.., Rívière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and technology. arXiv preprint arXiv:2403.08295, 2024.
https: /twi//iioo///// org/ NO.18653/v1/20223 act. 755. 756.
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. arXiv preprint
Hall, B. C. Lie groups, Lie algebras, and representations. Springer, 2013.
NDw:2009914794, 2020.
Choromanski. K., Lin, H., Chen, H., Zhang, T., Schanobish, A., Likhosherstov, V. Parker-Holder, J.. Sarlos, T., Weller, A., and Weingarten, T. From block block-toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers. In International Con- ference on Machine Learning, pp. 3962-3983. PMLR, 2022.
Hendrycks, D. and Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606. 08415, 2016. URL http: httt:://axiv. org/ abs 6066..66600 08415.
Heo, B., Park, S., Han, D., and Yun, S. Rotary position em- bedding for vision transformer. European Conference on Computer Vision, pp. 289-305. Springer, 2025.
Collins, J., Goel, S., Deng, K., Luthra, A., Xu, L., Gun- dogdu, E., Zhang, X., Yago Vicente, T. F., Dideriksen, T., Arora, H., Guillaumin, M., and Malik, J. Abo: Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
Hertzberg, C., Wagner, R., Frese, U., and Schröder, L. In- tegrating generic sensor fusion algorithms with sound state representations through encapsulation of manifolds. CoRR, abs/ 107.1 2011. URL http::/arxx: /1000 1119.
Deng, J., Dong, W., Socher, R., Li, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference ma- chine learning, pp. 5156-5165. PMLR, 2020.
Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans,
Diele, F., Lopez, L., and Peluso, R. The cayley trans- form in the numerical solution of unitary differential systems. Adv. Comput. Math., 8(4):317-334, 1998. doi: 10. 001103//4111189080003588 URL https:://o/ org/10. 1t1.11.1022222111190000000000000
Dosovitskiy, A., Beyer, I.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICIR, 2021.
LA, USA, December 10 16, 2023, 2023.
Kiyono, S., Kobayashi, S., Suzuki, J., and Inui, K. SHAPE: Shifted absolute position embedding for transformers. Moens, M.-F,, Huang, X., Specia, L., and Yih, S.
10

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
W.-t (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3309-3321, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguistics. doi: 12.18633/11/0221emm/pp 266. URL https: http:://accatn htt////caaattoooooooeo org/222... emn momll-mmain matpmmain... 266/.
Ostmeier, S., Axelrod, B., Moseley, M. E., Chaudhari, A., and Langlotz, C. Liere: Generalizing rotary position encodings. arXiv preprint arXiv:2406. 10322, 2024.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: //openreview net
Kuhn, H. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-99) 97, 1955.
Li, S., You, C., Guruganesh, G., Ainslie, J., Ontañón, S., Zaheer, M., Sanghai, S., Yang. Y., Kumar, S., and Bho- janapalli, S. Functional interpolation for relative positions improves long context transformers. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: htttp::///ppereeeeee net forum? id id=rRO3qFesqk.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger. G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748- 8763. PMLR, 2021.
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Explor- ing the limits of transfer learning with a unified text- to-text transformer. J. Mach. Learn. Res., 21:140:1- 140:67, 2020. URL https: http://mml.r.rr//////// v21/20- 000//200074-000 html.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P.. and Zitmick, C. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12.22 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Reid, I., Dubey, K. A., Jain, D... Whitney, W., Ahmed, A., Ainslie, J.. Bewley, A., Jacob, M., Mehta, A., Rendleman, D., et al. Linear transformer topological masking with graph random features. arXiv preprint arXiv:2410.03462, 2024.
Lin, Y., Humplik, J., Huang, S. H., Hasenclever, L., Ro- mano, F. Saliceti, S., Zheng. D., Chen, J. E., Bar- ros, C., Collister, A., Young, M., Dostmohamed, A., Moran, B., Caluwaerts, K., Giustina, M.. Moore, J., Connell, K., Nori, F., Heess, N., Bohez, S., and Byra- van, A. Proc4gem: Foundation models for physical agency through procedural generation, 2025. URL https: aaaxiiiii. org/ abs/ /200..553.. 08593.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 129500 2023.
Rubinstein, J., Segman, J., and Zeevi, Y. Recognition of dis- torted patterns by invariance kernels. Patlern Recognition, 24(10):959-967, 1991.
Liu, X., Yu, H., Dhillon, I. S., and Hsich, C. Learning to en- code position for transformer with continuous dynamical model. In Proceedings of the 37th International Con-
ference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 6327-6333 PMLR, 2020. URL hwtp:/tpp///e... /ettt::/prrcee http//pocedeggss////// ml 1.a20n.hmm.
Segman, J., Rubinstein, J., and Zeevi, Y. Y. The canonical coordinates method for pattern deformation: Theoretical and computational considerations. IEEE Transactions on Pattern Analysis Machine Intelligence, 14(12):1171- 1183, 1992.
wttpeeediniimmm....eee..eeeeeeee press/v119/
Luo. S., Li, S., Cai, T., He, D., Peng, D., Zheng, S., Ke, G., Wang, L.. and Liu, T.-Y. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems, 34: 22795-22807 2021.
Shaw, P., Uszkoreit, J., and Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv: arXiv:1803. 1a0::8833000 883.002155, 2018.
Singh, S., Tu, S., and Sindhwani, V. Revisiting energy based models as policies: Ranking noise contrastive estimation and interpolating energy models. Transac- tions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https: http:::////poottt /ttp://///e/eieeee/e net / forum? dd=JmKAYb7100.
Matthias Minderer, Alexey Gritsenko, A. S. N. D. A. M. A. A. M. D. Z. S. X. W. X. T. N. H. Simple vocabulucy object detection with vision transform- ers. ECCV, 2022.

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
2023. doi: 1048550/ARXXV:331217744 URL https: //doi ///bb..org/1..445550//aaaa/
Zhao, T. Z., Tompson, J.. Driess, D., Florence, P., Ghasemipour, K., Finn, C., and Wahid, A. Aloha un- leashed: A simple recipe for robot dexterity. arXiv preprint 2rXiv:2410.13/22 2024.
Tai, K. S., Bailis, P., and Valiant, G. Equivariant transformer networks. In International Conference on Machine Learn- ing, pp. 6086-6099 60956 PMLR, 2019.
Tziafas, G. and Kasaei, H. Early or late fusion matters: Efficient rgb-d fusion in vision transformers for 3d object recognition. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9558-9565, 2023. doi: 10.1 10L10//RROS55522200000000/ 10341422.
Udayan, J. D., Addanki, V., Durgapu, S., Yerramreddy, D. R., and Kolla, D. Forward kinematics simulation of KUKA KR5 are robot with robo analyzer. In Pro- ceedings of the 2023 Fifteenth International Confer- ence on Contemporary Computing, IC3-2023, Noida, India, August 3-5, 2023, pp. 994-299. ACM, 2023. doi: 10.1 011144//360794733600000 URL https: /doi. org/10. 1145 /1111111//3177777711
Unity Technologies. Unity, 2023, URL /tttt:::/unii com/. Game development platform.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, V. N., and Garnett, R. (eds.). Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008 2017.
Wang, B. Shang, L., Lioma, C., Jiang, X., Yang. H., Liu, Q. and Simonsen, J. G. On position embeddings in BERT. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openrevient
id www.oooooooooooooo xMw.
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-comtext scaling of founda- tion models. arXiv preprint arXiv:2309. 16039, 2023.
Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., and Zhao, Depth anything v2, 2024. URL https: //arxiv. wrg/oogggooooo.oo abs abs /246 09414.
Zhai, X.,
Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. arXiv preprint arXiv:2303.1 15343, 2023.
Zhao, L., Feng, X., Feng, X., Qin, B., and Liu, T. Length extrapolation transformers: survey from the per- spective of position encoding. CoRR, ab4/2312.170444,
12

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
A. Proofs
commuting antisymmetric matrices. We write
A.I. Proof of Theorem 3.2
L(r,)
La(ralk
(21)
To begin, we provide a proof of Theorem 3.2: that STRING the most general form of transformation that respects the group-like property
with (Lx R=1 C Rdxd set of commuting antisymmetric generators and [ri]a the k-th entry of coordinate vector Ty This completes the proof.
]
R((r))((r)) = RR(r, - ri),
(15)
supposing that R(0)- = Ia and (r)) is continuously differ- entiable with respect to r. Note that this is a sufficient, but not necessary, assumption for translational invariance.
A.2. Proof of Theorem 3.3
Now, we prove that generators of the form L d/2 p=1 2p= 82p-1,29), recover RoPE, as described
Σ
Proof. Recall STRING is applied
in Theorem 3.3.
q - R(r:)q:
(16)
Proof. Let us initially consider the case de = 1, so that the token coordinate r = r E IR and we learn a single generator. Recall that powers of a block diagonal matrix will remain block diagonal. Each block of the generator Lg of the form
with R(:): Rede - RRddd so that R(r;) E Rdxd Then we
require that
q k, + q R(r:) R((r)),
(17)
0
(22)
Recall that R(0)= La. the d-dimensional identity, so that the logit of query and key at the same position (r; = ry) will be unmodified by the positional encoding. Then
La.
-0 0 0
Then note that
0 0
[ 02
0
clearly R(r;) E O(d), the orthogonal group in dimension d. For compatability with gradient- optimisers (a chief concern the machine learning setting), is convenient to specialise the connected component (normal subgroup) of O(d) containing the identity matrix: that is, the special orthogonal group SO(d). This means that (et(R(r;)) = 1. These transformations are the d-dimensional rotations.
-
0
0
=
0
- B2
(23)
It follows that
1
0 (-10//2/2
if n is even,
0
0
1
0
(-1-))
11/2)/2
1)/2
if fn is odd.
Since R(r:) E (O(d), the rotation can be written using its Lie group,
0
(24)
R(r.) (x.()(r.))
(18)
Combining and inspecting the Taylor expansions,
where the matrix L(r:) is antisymmetric (Hall, 2013). L(rs) is called the 'generator', representing an infinitesimal rota- tion. Here, exp(-) denotes the matrix exponential (not to be confused with the element-wise exponential of a matrix, as appears e.g. in softmax). Setting ry in Equation (15), it is clear that L(-r.) = -rrs). We then require that
0 0
cos
sin
cxp
0 =
sin 0 cos
0
(25)
which is clearly a rotation matrix a well-known result. This holds for all the d/2 blocks, each of which exponentiates to give 2 x 2 rotation at a different frequency. Therefore,
cxp(Lr) = RoPE(r),
(26)
exp(L(r:)) exp (L(,)) = exp(L(r: + +r)))
(19)
showing that with this particular generator STRING is RoPE.
= =exp(L(r,)) ex:)).r.)).
Clearly L(r:) and L(ry) must commute for all choices of coordinate vector (ra, ry). Therefore, we need
Now suppose that de > 1, so L(r,) = E La(rile In our special case, each generator is of the form L
Ep=1 4/2 (ō2p.2p-1 52p-1.2p) where {0p] can differ for different & (notationally suppressed for compactness).
L(r. + ry) L(r:) L(rs),
(20)
so L(-) is linear in its arguments. That is, L(-) is a lin- ear map from the set of diddinenso vectors to a set of
Observing that
a
0
g
a
'This means that you can optimise the position encoding trans- formations on the same manifold. You could in priniciple also incorporate reflections so that det (rr(R(r))) -1, but this unlikely to significantly improve performance.
3
0
(27)
0
a
3
0
a
13

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
different L3 [r commute. Then we have that
can be aggregated the columns an orthogonal matrix P. taking
de
de II exp (Ln(rikk)
exp
L
[x
1
1
t
k=
P
==
U1
W1
U2
W2
BRadd
(29)
de
(28)
II RoPE((r:]a) k=1
J
4
4
whereupon
RoPE(r),
R(r) PRoPE(r)P
(30)
where we used the definition for multidimensional RoPE I1
Note especially that the change of basis matrix P is indepen- dent of r, because P determines the axes of rotation whereas the (complex) eigenvalues depend on the amount of rota- tion. This is obvious from the definition R(r) exp(Lr); the matrix P needs to (block) diagonalise L, then this is sufficient to (block) diagonalise R, and the matrix L is independent of r.
from Equation (5). This completes the proof.
A.3. Proof of Theorem 3.4
Next, we prove that STRING can always be rewritten as RoPE in different basis.
Proof. As usual, we begin with de 1. Then task is to show that matrix R(r) exp(Lr). with L E Rdxd an antisymmetric matrix and r E IR, can be rewritten as RoPE with change basis.
Now suppose that de > 1. Recall that we have
de
R(r) = exp
LEFE = II k=1
exp exp LEPE = II [(())
Begin by noting that R(r) SO(d); it is special or- thogonal matrix. It is orthogonal since R(r) R(r) exp(Lr) exp(Lr)
k=1
(31)
where we used the fact that the generators { com- mute (from Definition 3.1). (R(rx)) must also com- mute so are simultaneously diagonalisable, whereupon
ex)(-Lr) exp(Lr) Le, and its determinant is since it is continuously connected to the identity (which occurs at 0). Consider an cigenvector v E Cd, with eigenvalue E C. Since R R = La. A| so A = 100 Taking the complex conjugate of Rv = Av, we have RV \*V, where X = e e-t and v is the complex conjugate of v. We used that R is real. Therefore, the eigen- values appear in conjugate pairs for conjugate eigenvectors.
de R(r) II PRoPE([T]A) km
P RoPE((T]R) PT (32)
de
Let u + iw and V = u iw with w E Rd real vectors. Inserting into the eigenvector equation, R(u + iw) = (cos(f) + i ( iw). Equating the real and imaginary parts, Ru = cos( sin(O)w w) and Rv = cos( + sin(O)u. u. This corresponds exactly to 2-dimensional rotation in the (u, v) plane. By normalisa- tion of (complex) |u/2 ww|2 1. But since À and X" differ, their corresponding eigenvectors are orthogonal under the Hermitian inner product, so we also have that viv = |u|2 - |w/ 2iu w = 0. So w = O, where- upon u and w are orthogonal vectors ddd Of course, from basic linear algebra the complex eigenvectors correspond- ing to different 0 will also be orthogonal in Cd, or in the case of degenerate & one choose orthogonal basis for the corresponding subspace using e.g. the Gram-Schmice process. It is easy to show that the corresponding real vec- tors (u; w:) will therefore be orthogonal for different i, ie. u, uy = بG, = w w,. To summarise: the real and imaginary parts of the the d orthogonal (complex) eigenvec- tors of R in 2d correspond d orthogonal (real) vectors in Rd. organised in planes in each of which R acts 2-dimensional rotation, at frequency that depends the corresponding eigenvalue. These real vectors (u, w:) d/2
(r
This concludes the proof.
A.4. Proof of Theorem 3.5
Proof. Note that it suffices show that the computation of p exp(C - C CT) can be computed in Old log(d) time with O(d) memory for circulant matrix C, defined by its first row c. We will leverage the fact that every circulant matrix can be factorized as follows:
C DFF x diag(DFTe) x DFT
(33)
for the Discrete Fourier Transform matrix DFT, and where diag(v) (v) stands for the diagonal matrix with the main diago- nal given by v. Therefore we have the following:
p = exp erp/DDF((dag
(34)
where u = c - and t stands for the first column of C. Here we leverage the fact that the transpose the circulant matrix also circulant. Therefore, by leveraging Taylor
14

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
series formula for exp, get:
open-jar-Ie -id: open the glass jar lid, handover other hand and put the table
p= DFT exp(din(DFTT)DDFTT
(35)
bowl-in-rack: put the bowl into the drying rack
DFFddiggep(DFTT)DDT"""
cup-in-rack: put the cup into the drying rack
where exp in the last formula is computed element-wise. Note that matrix-vecter product with diagonal matrices can be trivially conducted in linear time. Thus the calculation of can be conducted in ((d) memory and (d log(d) time complexity via Fast Fourier Transform (FFT) (to multiply with DFT matrices) and inverse FFT (iFFT) (to multiply with matrices DFT This completes the proof. 0
4. banana-handover: put banana in bowl with han- dover
5. open-drawer: open the drawer
6. remove-gears: remove the gears from the nist- board
fold-dress: fold the dress
B. ALOHA Unleashed Simulation Tasks
stack-cups: stack the cups
ALOHA leader robots are teleoperated in ALOHA simula- tion for data collection using an ALOHA station, an Oculus VR headset and controllers (Zhao et al., 2024). The teleoper- ators are instructed perform the following tasks using this setup. See Figure and Figure 10 for simulation renders of the following 12 tasks.
9, han-handover: put pen in container with handover
10. take-phone-oute take phone out of purse
After fine-tuning, the model then evaluated on the fol- lowing 5 tasks: bowl-in-rack, banana- bowl-in- fold-dress, remove-gears. See: Fig. 11 for the visualizations of selected ALOHA real world tasks.
1-3. (BowL/Glass/Plateennnnn the rack.
Place the item on
4. Si Single Innertnnst::on: Use the left arm to grab the blue socket and the right arm to insert the block into the socket.
D. WebLI-3D Dataset
We lift the WebLI dataset (Chen al., 2023), a dataset of 10 billion image-text pairs across variety of languages, into 3D by pre-processing a 60-million image subset using Depth-Anything V2 V2 (Yang et al., 2024) for metric mon- odepth estimation. The dataset is filtered using the method described in (Chen et al., 2024) for images that the indoor- finetuned model performs poorly on (pictures with overlays, no visible groundplane, large outdoor scenes, optical illu- sions removed).
5. Doubl CoubleTnserte insertioo: After completing the SingleInsertion task, insert another block into the other end of the socket.
6-7. Functional Manipulation Benchmark (FMB) 1 and 2: Insert a yellow block into the recess of a red base.
8. FruitBowl: Place all the fruits in the bowl.
9. StorageBin: Place all the snack boxes in the storage bin.
E. Classification and Retrieval Experiment Details
10-11. Ha HandOver (Banana/Pen): FendOver(Baraaa//ea)) Hand over the item and place it in the container.
Our experimental base model is ViT-B/16 (Dosovitskiy al., 2021), which has 12 layers, 768 model dimension, 3072 MLP size, 12 attention heads, and 16x 16 patch size. All RoPE and STRING variants retain these same model hyper- parameters. For vanilla RoPE, we use the common 10,000 max wavelength and simply split query/key dimensions be- tween axes in the 2D or 3D case, respectively. For RoPE-Mixed, we initialize with 100 max wavelength as suggested in Heo et al. (2025), which we also adopt for STRING.
12. MugOnP Mate Place the mug on the plate.
MultiTask aggregates results of of the above tasks.
C. ALOHA Real Tasks
ALOHA-real models are first pre-trained with human-teleop data collected on 300 diverse tasks, which were crowd- sourced based on relevance with real-world scenarios as well as feasibility for the ALOHA robot. Further, the model fine-tuned on the following 10 tasks.
For other hyperparameters like learning rate, batch size, warm-up schedule, etc., we left these the same as the de- fault values used for the base ViT-B/16 model. We use
15

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Noise
%
Eg
Ewi
Eg
Transformer Encoder
Proj
Proj
Prop
Proj
Proj
Proj
CNN
X-Attn
Transformer Decoder
CNN
Prop
Proj
Proj
Proj
Proj
CNN
9
By
5
Sea Pay
Noisy Actions
Proprioception
MLP
Diffusion Timestep
Figure 8. ALOHA Unleashed Architecture (Zhao et al., 2024).
the Adam optimizer with b1=0. 9 and b2=0. 999 for all experiments. For STRING, we experimented with sharing parameters across attention heads rather than the default of learning them all separately, and we found this could yield slight gains in both efficiency and quality. Finally, for Circulant-STRING
F. 3D Detection Details
F.1. Dataset
We train our synthetic datasets of indoor living room and cluttered tabletop scenes using a procedural generation recipe. We utilize the dataset described in (Lin et al., 2025), which we briefly deseribe here. We use open-sourced 3D assets, specifically a subset of assets from the Amazon Berkeley Objects (Collins et al., 2022) (ABO) dataset for background and tabletop objects and the YCB (Calli et al., 2015) and Google Scanned Objects (Downs et al., 2022) for tabletop clutter placement. The procedural generation recipe works by hierarchically generating sub-areas such as lounge, dining, office and reading area within a randomly- sized rectangular room together with freestanding pieces of furniture, sampled from the different classes within the ABO assets. For scenes with tabletop clutter, addition- ally randomly sample assets top of existing placement surfaces (e.g. tables) in the scene, and procedurally vary the packing fraction on top of the placement surface to achieve randomised clutter arrangements. We additionally vary light- ing, background colors, camera extrinsics, intrinsies, aspect ratios etc generate diverse datasets. Each example within dataset has accompanied metadata such RGB, Depth, Segmentation masks, Object poses, Camera extrinsics, in- trinsics, and 3D & 2D bounding boxes for each object in view. We render images using Unity (Unity Technologies, 2023) achieving a high-degree of photo-realism.
we swept over block sizes in the set {4, 8, 16, 32, 64) to find the optimal setting (often around 16).
E.1. ImageNet2012 and Places365 Classification
All experiments were trained from scratch, separately for Im- ageNet2012 or Places365. In both cases, we used 224 224 image resolution, batch size 4096, and trained for 300 epochs. Training used the cosine decay learning rate sched- ule with 0.001 base learning rate and 10,000 warm-up steps. For ImageNet2012 there were total of about 94k train- ing steps, and for Places365 there were about 130k. For Ciroulant-STRING, block size 16 yielded the best results.
E.2. WebLI-3D Retrieval
All experiments were trained from scratch and used 256x256 image resolution. The VIT baseline used RGB data, but all RoPE and STRING variants used RGB-D, where we incorporated depth as a third coordinate for 3D position representations. We trained with batch size 8192 for 20 epochs, amounting about 155k training steps using the SigLIP (Zhai et al., 2023) pretraining setup. Training used the cosine decay learning rate schedule with 0.001 base learning rate and 5% warm-up steps (about 8k). For Cireulant-STRIING, block size 32 yielded the best results.
We generated four different 1-million image datasets using the procedure above: a) living room scenes without table- top clutter (Fig. 12a), b) tabletop scenes with procedurally varying clutter, placed within randomly created living room scenes (Fig. 12b). c) tabletop scenes with multiple cluttered tabletops (Fig. 12c), and d) tabletop scenes with more com-
16

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Figure 9. Views from the wrist, overhead and table-level cameras in ALOHA sim. Top to bottom, Left to right: PlateOnRack, DoubleInsertion (insert peg into sockets on either end), MugOnPlate,, Singlelnsertion, BowlOnRack, and Functional Manipulation Benchmark--1 (FMB-1).
plicated object arrangements, primarily objects placed on top of trays and other flat objects within the scene (Fig. 12d). We held out the first 20 images for each these datasets for evaluation, and used the rest for training. Example images from our datasets showing the images and corresponding 3D bounding box labels can be seen in Fig. 13.
of the box head is
[< translation rotation >, size >
R is the SE(3) translation of the center of the box, rotation E R is the first 2 columns of the SO(3) rotation matrix of the box, and < size > E R3 is the length of each side of the box. Thus the output of the model for each predicted box is a 12D vector. All predictions are relative to the camera frame.
where translation
F.2 Implementation
We base our implementation for 3D bounding box detection described in Section 4.2.2 on OWL-ViT (Matthias Minderer, 2022). OWL-ViT predicts 2D, axis-aligned bounding boxes for the queried object names in the image. We modify this to predict full 3D bounding boxes. We utilized the vision and text towers from the SigLIP task in Section 4.1, and add the box and class prediction heads on top for the 3D bounding box task. The main differences with the original OWL-ViT described in the sections below.
F.2.2. LOSS FUNCTION
We modify the OWL-ViT loss terms the following way. We keep the class loss as-is. We also keep the L. loss di- rectly the 12D bounding box prediction vector. However, this loss is not necessarily optimal for SO(3) rotations and in future work we would like to look into better rotation losses such as (Hertzberg et al., 2011) 1). Finally, we completely replace the 2D, axis-aligned intersection-over-niii (IOU) loss. The algorithm for computing the full 3D, non-axis- aligned IOU loss over the 8 comner vertices of the box. For both the pre- dicted and target boxes, we compute the 3D coordinates of the 8 corners of the boxes, and then we sum the LI distance between the predicted and target corners (we assume a fixed ordering of the corners). We found this loss significantly increases overall performance of the learned models.
F.2.1. Box FORMAT
The output of the box head in OWL-ViT is the relative offset from the corresponding image patch for the 4 edges (top, bottom, left, and right) of the axis-aligned, 2D bounding box. In 3D, image patches do not align with predictions quite as well as in 2D, so instead, the box head predicts the absolute pose of the 3D bounding box. The output format
non-differentiable, we instead compute
17

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Figure 10. More ALOHA Simulation Tasks, Top to bottom, left to right: FruitBowl, FMB-2, GlassOnRack, HandOverBanana, HandOverPen and StorageBin.
Figure 11. Sampled ALOHA Real Tasks. Top to bottom, left to right: aake-phone-out stack-cup, remove gear, open-drawes bowl-in rack and fold-dress.
18

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
EROVIAC
(a) Living
(b) Tabletop with clutter
(c) Multiple cluttered tabletop scenes
(d) Objects on flat objects scenes
Figure 13, Example images drawn at random from our dataset used for 3D detection. Each subfigure shows images from each of generated subsets of the dataset. The green boxes show the ground truth object bounding boxes.
19

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
F.2.3. 3D IOU
outputs a relative offset for the bounding box, our box head directly outputs the absolute bounding box representation itself, as described above in Appendix F.2.1.
We use Monte Carlo algorithm to compute the intersection- over-union (IOU) between 2 boxes in 3D space, which we report in our evaluation numbers in Section 4.2.2. First, we sample 100k 3D points uniformly at random inside the predicted box. Next, we transform those points to be in the coordinate space of the target box and we compute the ratio of points that are inside the target box. Finally, we compute IOU as the intersection of the volumes divided by the sum of the volumes minus the intersection:
We use the same training method and optimizer as described in appendix A1.3 in (Matthias Minderer, 2022). replacing the gloU weight with a weight for our 8 corner vertex loss described above in Appendix F.2.2 but keeping the same values. To improve training speed, we randomly subsample 12 objects from each image for each iteration. However, during evaluation we include all objects in the scene when computing the 3D IOU. We train with a batch size of 1,024 for 250k iterations with an initial learning rate of -.
r * col()))))
IOU (bozp, boz,) vol(box,)) + (1 r) vol(box)p)
where r is the fraction of sampled points inside the target box and vol is the volume of the given box.
F.4. Results From All Runs
Baseline
RoPE
RoPE-M
Circulant-S
Cayley-S
F.2.4, BIPARTITE MATCHER
49.77
55.77
2.53
56.69
58.85
We found empirically that the Hungarian algorithm (Kuhn, 1955) for bipartite matching between predictions and tar- gets during training had the best performance. Specifically, the Hungarian algorithm was the only bipartite matching algorithm tested that was able to correctly detect objects far from the center of the image. We suspect that, due to the absolute bounding box predictions described above in Appendix 1, the box head was biased towards objects in the center of the camera frame, and the Hungarian algo- rithm encouraged it to predict boxes for further away objects. However, we also found the Hungarian algorithm to produce significant volatility in the performance of the model, with models sometimes getting stuck in local minima with poor performance during training (see Table 7). To counteract this, for every entry in Table 5 in Section 4.2.2, we trained 3 models with different random seeds. The values in the table are the max of the models' performance after 250k training iterations. Note that the parameters for the vision and text towers are loaded from the pre-trained checkpoints, so only the parameters for the prediction heads are randomly initialized.
VIT
49.10
52.11
57.17
58.95
58,43
47.85
58.09
2.46
13.12
57.47
65.88
71.21
70.78
72.36
70.36
ViTD
66.49
69.50
69.94
69.86
72.67
67.60
70.31
70.90
68.51
71.91
Table 7. Average 3D IOU over all objects for the 3D bound- ing box prediction task. For each configuration, 3 models were trained with different random seeds. Baseline indicates no RoPE or STRING. The maximum for each configuration is highlighted in bold. Higher is better.
For each configuration of RoPE/STRING and ViT/ViTD, we trained models with different random seeds. In Table 5 in Section 4.2.2 we report the maximum IOU of the for each configuration. Here, Table 7 shows the IOU for every run, with the maximum highlighted in bold. Note that for 3 runs (2 for ViT+ROPE-M and I for ViT=Circulant-STRING the models fell into local minima which they never left and thus had inferior performance. See Appendix F.2.4 for details on possible causes.
G. Details of Generative Robotics Policies
E33. Model and Training Configuration
The model is composed of 4 parts: the vision tower, the text tower, the class head, and the box head. The vision tower encodes the image as a set of tokens, the text tower encodes each input text sequence as a token, the class head predicts the class probabilities for the predictions given the query texts, and the box head outputs the bounding box parameters for each prediction. For the vision and text towers, we use the same model layout (e.g., number of layers) as the models trained on WebL.I33D in Section 4.1. For the class head we use the same layout as described in OWL-ViT (Matthias Minderer, 2022). We modify the box head to be a 3 layer MLP, with GELU (Hendrycks & Gimpel, 2016) non- linearities after the first layers. Unlike OWL--VT, which
Figure 14 shows the network architecture for the generative robotics policies for manipulation tasks. PaliGemma (Beyer et al., 2024) VLM with embedding size 256 and patch size 16 is used for image encoding. The policy is trained with Adam optimizer with le learning rate and 1e - weight decay. We use a linear learning rate warm-up for first 10000 steps of training. The policy is trained for total of 500000 steps with batch size 256.
20

---

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
Noisy Actions
Image Encoder
Taken Learner
Image Encoder
Context Fuser
Trajectory Diffusion
Multimodal Instruction Encoder
Pickup coke with left hand.
Figure Generative policy architecture robotic manipulation using Kuka- Suka-IIWA arms.
21


Reviews:
[{"review_id": "IyLgzdoJIH", "reviewer": "Reviewer_57Ag", "summary": "This paper introduces STRING (Separable Translationally Invariant Position Encodings), a new family of position encodings. STRING extends RoPE using a theoretical framework based on Lie groups, maintaining key properties like separability and translational invariance, while supporting 2D/3D token representations.\n\nThe authors propose effective implementations, such as Cayley-STRING and Circulant-STRING, and demonstrate through experiments that STRING outperforms RoPE and absolute position encodings (APEs) across tasks. \n\nThe experiments include:\n\nImage classification and retrieval.\nOpen-vocabulary 2D/3D object detection.\nRobotic manipulation tasks in both simulation and real-world settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the Reviewer for the comments. We provide detailed answers below.\n\n$\\textbf{Increasing the number of trials in testing (currently 10 trials / task for Aloha-sim and 50 trials for bi-arm KUKA)}:$\n\nWe sincerely thank the Reviewer for the comment. We would like to note that 10 trials / task is considered normal for the evaluation of the robotics controllers, both in sim and on hardware (e.g. [1]: 5 repetitions as stated in Sec 6.1, $\\textit{\"This is repeated \\textbf{5} times\"}$  and [2]: 10 repetitions in $\\textit{“We ran each of the learned controller \\textbf{10} times.”}$ at the beginning of Sec. B (“Results and Analysis”)). \n\nIn fact it is also considered normal for testing several regular machine learning algorithms. For instance, most papers evaluating Random Fourier Features and related methods, such as [3] ($\\textit{“All experiments are repeated \\textbf{ten} times”}$, p.7), [4] (caption of Fig. 2 $\\textit{“\\textbf{10} independent trials are executed”}$) that usually can be quickly evaluated with very limited computational resources, apply 10 repetitions to calculate empirical MSEs.  \n\nHaving said that, following Reviewer’s comment for the rebuttal, we re-ran the evaluations for the $\\textit{BowlOnRack}$ task (we chose one task, given rebuttal period time constraints) from the Aloha simulation portfolio, by increasing the number of trials $\\textbf{from 10 to 100}$. The ranking of different methods in that setting $\\textbf{stays exactly the same}$ as reported before: 1. STRING (83% accuracy) 2. regular ViT baseline (80% accuracy) and 3. RoPE (75%). We also would like to emphasize that all reported KUKA experiments are $\\textbf{on-robot}$ (and thus much more time-consuming) and in the paper we use many more trials for the corresponding evals. \n\n$\\underline{References}$:\n\n[1] Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation; Kalashnikov et al.; CoRL 2018; $\\textbf{Best Systems Paper, ~1.8K citations}$.\n\n[2] Deep Spatial Autoencoders for Visuomotor Learning; Finn et al., ICRA 2016; $\\textbf{764 citations}$.\n\n[3] Nystrom Method vs Random Fourier Features: A Theoretical and Empirical Comparison; Yang et al.; NeurIPS 2012; 450+ citations).\n\n[4] Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels; Avron et al.; ICML 2014\n\n===========================================================================================================================================================\n\n$\\textbf{Scaling Experiments to show that STRING still performs well}:$\n\nThank you very much for an interesting question about scaling to larger models. Following the Reviewer’s suggestion, we have run such experiments using the SoViT-400m class of models which surpass ViT-H and are competitive in performance with ViT-G [1]. As compared to baseline, STRING provides $\\textbf{>4.7}$% relative improvement on COCO AP and $\\textbf{>5.1}$% relative improvement on LVIS AP and improves also upon regular RoPE in the OWL-ViT-2D setting. For the task of 3D object detection with SoViT-400m models, using STRING led to additional accuracy gains as compared to the baseline ($\\textbf{1.2}$ of the baseline improvement coming from using a larger model). \nThus, we tested that STRING does perform well also on larger models. \n\nWe have also conducted additional scaling experiments, where we increased the resolution of images from 256 x 256 to 512 x 512. In the higher-resolution setting, STRING maintained an advantage over the baseline for the object detection task from Sec. 4.2.2 (3D bounding boxes): $\\textbf{6}$%+ for the 2D variant and almost $\\textbf{10}$% for the 3D variant. \n\nWe will include all these results in the camera-ready version of the paper.\n\n$\\underline{References}$:\n\n[1] Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design; Alabdulmohsin et al.; NeurIPS 2023\n\n===========================================================================================================================================================\n\n$\\textbf{Improvements Over Baselines}$:\n\nThank you very much for your question. The STRING variant provides highest average accuracy over all Aloha-sim tasks. However, what is as important and clearly seen in Fig. 4, it also provides significant convergence improvements in training, achieving accuracy level of regular RoPE in $\\textbf{1.6x}$ shorter time. We explicitly commented on that in l.367-368. This clearly distinguished STRING from regular RoPE. Furthermore, our 3D object detection experiments clearly show improvements coming from STRING (from approximately $\\textbf{1}$% to $\\textbf{1.5}$%), in both: 2D and 3D settings, as compared to $\\textbf{the best}$ RoPE variants (Table 4). These improvements are actually visible to the naked eye. We show it in Fig. 2, where predicted small 3D boxes (in blue) are clearly misaligned with ground truth bounding boxes (in green) for baseline and RoPE, while pretty well aligned for STRING.", "claims_and_evidence": "The paper’s claims are generally well-supported by clear evidence:\n\n\nSTRING generalizes RoPE with separability and translational invariance.\nSupported by rigorous theoretical proofs (Theorem 3.2 and 3.3).\n\nSTRING outperforms RoPE and APEs in tasks like classification, detection, and robotics.\nBacked by experimental results showing consistent improvements, e.g., higher accuracy in ImageNet and better IOU in 3D object detection.\n\nSTRING is computationally efficient.\nSupported by proposed implementations (Cayley-STRING, Circulant-STRING) and complexity analysis.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate:\n\nMethods: STRING is well-designed for tasks requiring translationally invariant position encodings, with practical implementations like Cayley-STRING and Circulant-STRING ensuring efficiency.\nEvaluation: Benchmarks like ImageNet, WebLI-3D, and robotics tasks (ALOHA and KUKA) are relevant and suitable for testing STRING's performance.\n\nHowever, the ALOHA simulation tasks (10 trials per task) and KUKA real-world experiments (50 trials) have limited evaluations, which may not fully capture variability or ensure robustness. Larger-scale testing is needed to strengthen the conclusions.", "theoretical_claims": "The theoretical claims, including Theorem 3.2 and 3.3, were reviewed and appear to be correct, with clear proofs provided in the appendix.", "experimental_designs_or_analyses": "Please see the Methods And Evaluation Criteria.\n\nThe ALOHA simulation tasks (10 trials per task) and KUKA real-world experiments (50 trials) have limited evaluations, which may not fully capture variability or ensure robustness. Larger-scale testing is needed to strengthen the conclusions.", "relation_to_broader_scientific_literature": "The paper builds on prior work in position encoding and equivariant representations, extending these ideas with the STRING framework. It specifically addresses limitations of fixed encodings in transformers and draws inspiration from group theory and circulant structures, contributing a novel approach aligned with recent advances in efficient, invariant neural architectures.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper focuses on the important area of position encoding, which is highly relevant to the broader field of machine learning. It introduces novel theoretical derivations and technical contributions, which are innovative and valuable.\n\nWeaknesses:\n\n**Limited Scalability Experiments:** The paper lacks scaling experiments. It is unclear if Circulant-S and Cayley-S perform well on larger models, such as ViT-H or ViT-G.\n\n**Marginal Improvements Over Baselines:** I noticed that Circulant-S and Cayley-S do not show significant improvements over baselines like RoPE, especially in the ALOHA simulation task, where the results are nearly identical across many tasks.\n\n**Robotics Task Evaluation:** The evaluation on robotics tasks could be more robust. For example, testing on more trials and tasks would strengthen the claims.\n\nWhile the theoretical contributions are clear, I am not deeply familiar with this field, which limits my ability to assess the novelty of the contributions in the broader context of existing research.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I give a Weak Accept because of the solid theoretical proofs and novel technical contributions. However, I am not very familiar with this field, so I cannot guarantee my judgment with full confidence. I did not give a higher score because the experimental results show limited improvements over the baselines, especially due to the lack of scaling-related experiments, which raises concerns about the actual effectiveness of the proposed methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "yMsLDmJua7", "reviewer": "Reviewer_CQUV", "summary": "This paper proposes a class of positional encoding for high-dimensional tokens with both separability and translation invariance. The proposed positional encoding is a generalization to the rotary positional encodings (RoPE) based on Lie groups. The authors also provide a computationally efficient implementation of it. Experiments show its gains with transformer models in robotics and vision tasks.\n\n## update after rebuttal\nThank the authors for their responses! My questions are well-answered.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the Reviewer for the comments. We provide detailed answers below.\n\n$\\textbf{Theorem 3.5}$:\n\nThe proof relies on the fact that the computation of $exp(\\mathbf{M})$ for a given matrix $\\mathbf{M}$ can be re-written as: $\\exp(\\mathbf{M}) = \\Sigma \\exp(\\mathbf{D}) \\Sigma^{-1}$ for a given factorization: \n$\\mathbf{M}=\\Sigma \\mathbf{D} \\Sigma^{-1}$ (from the Taylor-series definition of matrix exponentiation). The next observation is that, for the matrices of interest in the proof, the factorization can be done efficiently with $\\Sigma$ and $\\mathbf{D}$ being: (1) a DFT matrix and (2) a diagonal matrix with the main diagonal given as the action of $\\Sigma$ on an easy-to-compute vector, respectively. The final observation is that the DFT matrix supports fast matrix-vector multiplication (via the Fast Fourier Transform) and the diagonal matrix exponentiation is obtained by element-wise exponentiation. We will clarify it in the final version of the paper by providing a sketch of the proof in the main body of the paper.\n\nThank you very much for an excellent question regarding the backpropagation pass. Even though the analysis in the paper was made for the forward pass, it can be extended to the backward pass. This comes from the fact that the computation of the gradient in the backpropagation over networks leveraging Circulant-STRING matrices also involves multiplications with DFT matrices. The analysis of the backpropagation is more technical, but basically follows steps from Sec. 4.1 of [1]. We will provide a paragraph to discuss computational gains in backpropagation in the camera-ready version of the paper.\n\n$\\underline{References}$:\n\n[1]  An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections; Cheng et al.; ICCV 2015.\n\n===========================================================================================================================================================\n\n$\\textbf{Statistics or discussions on the number of parameters, memory consumption, or training/inference time}$:\n\nThank you very much for the comment. STRING introduces a negligible amount of extra trainable parameters. With token dimensionality per head $d_{\\mathrm{head}}$, Cayley-STRING introduces $d_{\\mathrm{head}}(d_{\\mathrm{head}} - 1) / 2 + d_{\\mathrm{head}} / 2$ parameters per head while Circulant-STRING adds $d_{\\mathrm{head}}$ parameters per head. \nThus for instance for the: (1) Cayley-STRING variant with no parameter sharing across heads, (2) Cayley-STRING with parameters shared across heads and (3) Circulant-STRING with no parameter sharing across heads, parameter count increase is marginal: (1) 0.34%, (2) 0.028% and (3) 0.016% respectively. In terms of training time and memory footprint, the performance of Cayley-STRING and Circulant-STRING is similar to that of the RoPE-M model (~$\\pm 7 \\$%). We will include these numbers in the supplementary material of the camera-ready version of the paper.\n\n===========================================================================================================================================================\n\n$\\textbf{Theorem 3.3 and 3.4}$:\n\nWe thank the Reviewer for the comment, and will ensure that our notation is consistent and clear. Theorem 3.3 shows that RoPE is a special case of STRING for a particular choice of an antisymmetric generator. Meanwhile, Theorem 3.4 shows that RoPE is a special case of STRING with basis transformation matrix P = I. Both theoretical results show that our new method is more general than that previous SOTA, but from complementary perspectives. As shown in the paper, this increased generality unlocks better performance in experiments. We will make sure this is clear in the text and highlight the differences between these important, closely-related results.\n\n===========================================================================================================================================================\n\n$\\textbf{Clearer introduction to the notation at the beginning}$:\n\nThank you for the question. $d$ refers to the dimensionality of the latent representations in the Transformer, i.e. the length of queries, keys and values. Meanwhile, $d_c$ refers to the dimensionality of the position coordinates encoded using STRING, i.e. 1 for sequences (text), 2 for 2D data (images), and 3 for 3D data (images with depth). We will make sure this is clear in the text.", "claims_and_evidence": "The three key contributions claimed in the paper are:\n- \"We introduce STRING, a new family of position encodings for multidimensional token coordinates that respect both separability and translational invariance.\"\n\t- The proposed method is indeed a \"family\" of positional encodings with different variants.\n\t- Both separability and translation invariance are well discussed in the paper.\n- \"We rigorously analyse STRING’s theoretical properties ( Sec. 3), proving that it is more general than RoPE. We provide computationally efficient implementations.\"\n\t- The theoretical properties are well provided.\n\t- The relations between STRING and RoPE are well discussed.\n\t- For computational efficiency, there are some theoretical discussions about computational complexity, but it would be good to also have some statistics or analysis on its time or memory consumption in the experiments.\n- \"We show strong accuracy gains across varied models using Transformers with STRING, on a range of robotics and general vision tasks (see Fig. 1 and Sec. 4).\"\n\t- The experimental results support this argument.", "methods_and_evaluation_criteria": "The method and evaluations look reasonable to me. The evaluations are mostly done on public benchmarks with widely adopted metrics.", "theoretical_claims": "I briefly went through the proofs for Theorem 3.2-3.4. Theorem 3.2 is mostly based on the general conclusions from group theory and I think they are good. The intuitions for Theorem 3.3 and 3.4 look right, where I briefly followed the proofs but didn't check all computations in details.\n\nFor Theorem 3.5, I am not very familiar with the computational complexity of FFT and DFT algorithms, and thus I cannot say too much about the proof. But a small question I have is, it seems that this complexity analysis is mostly about the forward computation? How would the complexity be like for backpropogation?", "experimental_designs_or_analyses": "The experimental designs look reasonable to me, especially the focus on comparing STRING (with its variants) to previous positional encodings. The Training curve in Fig. 4 also looks good, as it shows that networks with STRING positional encodings are also easy to optimize for.\n\nIt may be good to also have some statistics or discussions on the number of parameters, memory consumption, or training/inference time, as efficient implementation is discussed in the key contributions.", "relation_to_broader_scientific_literature": "My personal feeling is that the proposed method is a generalization of RoPE (and thus RoPE is its closest related work). But this generalization is non-trivial -- it covers the most general cases and shows different variants. Computational efficiency is also studied in this generalization. So overall I think the proposed method has good novelty and well-developed details.", "essential_references_not_discussed": "The related works and preliminaries have provided a good background for the topic studied in this paper.", "other_strengths_and_weaknesses": "To summarize, the strenghts and weaknesses are:\n\nStrengths:\n- A general framework with theoretical proofs, well-designed details, and different variants.\n- Good experimental results.\n\nWeaknesses:\n- Computational efficiency: This point can be better justified if there could be some statistics or discussions on the number of parameters, memory consumption, or training/inference time.\n- More explanations on Theorem 3.3 and 3.4 and the notations could make the paper easier to read.", "other_comments_or_suggestions": "- It may be good to have clearer explanations of the differences between Theorem 3.3 and 3.4, instead of calling both of them \"RoPE is a type of STRING\". Based on my understanding, Theorem 3.3 is saying that RoPE is a special case of STRING (and thus STRING is a generalization of RoPE), and Theorem 3.4 is about, and Theorem 3.4 is saying that all STRING positional encodings can be expressed with RoPE.\n- It may be good to have a clearer introduction to the notations at the beginning. For example, it took me some time to find what are $d_c$ and $d$.", "questions_for_authors": "I don't have specific questions in addition to the comments I made above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymO9QcWov8", "reviewer": "Reviewer_1d62", "summary": "This paper proposes STRING, which generalizes the 2D rotation matrix in position encoding in RoPEs to a more general form of rotation, which is parameterized by the linear combinations of skew-symmetric generator matrices. This allows the framework to obtain exact translational invariant or rotational invariant, depending on the needs. The proposed method is tested on 2D and 3D object detection tasks, as well as a robot policy learning task, in which the proposed method is shown to obtain better performance than ViT and RoPEs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the Reviewer for the comments. We provide detailed answers below.\n\n$\\textbf{Discussion on trade-offs regarding using Cayley-STRING and Circulant-STRING}$:\n\nThank you very much for an excellent comment. With token dimensionality per head $d_{\\mathrm{head}}$, Cayley-STRING introduces $d_{\\mathrm{head}}(d_{\\mathrm{head}} - 1) / 2+ d_{\\mathrm{head}} / 2$ parameters per head while Circulant-STRING adds $d_{\\mathrm{head}}$ parameters per head. This is the case since Cayley-STRING trains a full skew-symmetric matrix and d / 2 frequencies, while Circulant-STRING uses a learnable circulant matrix which is fully determined by its first row. Furthermore, multiplication with Cayley-STRING matrices takes quadratic time in $d_{head}$ during training, whereas multiplication with Circulant-STRING matrices takes time $O(d_{\\mathrm{head}} \\log(d_{\\mathrm{head}}))$ via Fast Fourier Transform (FFT). During inference, quadratic cost can be in principle absorbed into existing q/k projections at no extra cost (for the vanilla attention mechanism). Both approaches improve upon regular RoPE, yet Cayley-STRING in general leads to larger improvements. Thus we have here a classic quality-speed tradeoff. For applications with strict memory constraints, we recommend to use Circulant-STRING due to its very compact computational footprint, whereas for other applications Cayley-STRING is recommended. We will add the above discussion to the camera-ready version of the paper.\n\n===========================================================================================================================================================\n\n$\\textbf{The impact of discretization in the sensory data on the performance of the system}:$\n\nThank you for your question. We conducted our experiments with images of various resolutions, seeing consistent gains across the board, as compared to the baselines. The statement that STRING is effective even if the sensory data is not accurate (due to discretization or noise) is also supported by our 3D experiments with the bi-arm KUKA robot. In all those experiments, depth sensors provide a relatively noisy signal, yet STRING is capable of leveraging depth sensors to provide $\\textbf{8}$%+ accuracy improvements in regular in-distribution tasks and from $\\textbf{15}$% to $\\textbf{42}$% improvements for out-of-distribution tasks. We will add this discussion to the camera-ready version of the paper.", "claims_and_evidence": "The main contribution of this paper is to generalize the position encoding. Another contribution is that this position encoding can be extended to 3D. The authors provide a detailed explanation of their position encoding design and proof of its invariance.", "methods_and_evaluation_criteria": "Evaluation metrics are standard. There are two ways to learn the skew-symmetric matrix: Cayley-STRING and Circulant-STRING. While they are each introduced clearly, a discussion about the trade-off and when to use which would be beneficial.", "theoretical_claims": "The paper provides theorems of the position encoding based on the generators, which proves how STRING is a generalization of RoPE and the translational invariant property.", "experimental_designs_or_analyses": "The authors provide extensive experiments on 2D object detection, 3D object detection, and policy learning for robot manipulation. All of them are compared with standard baselines ViT and RoPE. While the improvements are marginal, this is the nature of generalizing RoPE. The out-of-distribution evaluation shows the robustness if the framework is extended to 3D. \nAdditional discussion on the differences between Cayley-STRING and Circulant-STRING can strengthen the paper.", "relation_to_broader_scientific_literature": "Overall, the paper is a great addition to the literature. It generalizes RoPEs by a natural choice of position encodings. The paper is both theoretically sound and well supported by extensive experimental results.", "essential_references_not_discussed": "The references are addressed adequately.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does the discretization in the sensory data (for example, limited image resolution/token) affect the performance of the system?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsnvcO0xpB", "reviewer": "Reviewer_PFCP", "summary": "This submission proposes STRING, a novel method that generalizes the popular Rotary Position Encodings (RoPE) used in Transformers. Unlike RoPE, which is naturally suited for 1D inputs and then extended to higher dimensions, STRING offers a theoretical framework that accommodates multi-dimensional (2D/3D) inputs while preserving essential properties like translational invariance and separability. Through extensive experiments—covering image classification, open-vocabulary detection, simulation-based robotics tasks, and real-world robot manipulation—STRING consistently outperforms or matches RoPE and standard baselines. The authors also provide in-depth theoretical justification showing that STRING is, in fact, the most general family of rotation-based position encodings (under certain smoothness constraints).\n\n## After Rebuttal\nThe authors' reply not well addressed my concerns on 3D imitation learning part. I might keep my score as weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the Reviewer for the comments. We provide detailed answers below.\n\n$\\textbf{Additional tests, e.g. 3D-based imitation learning methods from [1,2]}$:\n\nThank you very much for the comment. The main goal of the 3D bi-arm KUKA experiments was to combine the best techniques to obtain new SOTA. We have already established STRING as providing gains on the top of the regular 3D algorithms, that in turn improved upon their 2D counterparts in Sec. 4.2.2., where we focused on 3D object localization (see: Table 4). We explicitly describe this goal, putting our plan for bi-arm KUKA experiments in the context of our previous findings,  at the beginning of Sec. 4.4 (l.372-375: $\\textit{“Establishing STRING as superior to other methods on previous tasks, ...”}$). \n\nHaving said that, we still included in the bi-arm KUKA section 3D-based imitation learning methods that do not leverage STRING. We trained via imitation learning a policy directly using depth to produce normal maps (paragraph: $\\textit{“Implicit depth via normal maps”}$ at the beginning of Sec. 4.4.2). This is based on well established methods for incorporating depth into robotics tasks from [1]. As we show in Fig. 5, STRING provides ~$\\textbf{11}$% improvement accuracy-wise, as compared to that baseline.   \n\nWe would like to note one more thing. The 2D policies we compared against in the bi-arm KUKA section were strong enough to outperform various 3D counterparts given similar computational budget (e.g. point-cloud-based methods that were used in the first paper pointed by the Reviewer; we will explicitly clarify it in the camera-ready version). This was the case since point cloud (PC) approaches suffer from the very high computational footprint. The average number of points in the bi-arm KUKA PCs was of order 5K+. Training/deploying policies directly using all the points was infeasible and various tokenization techniques to reduce the number of 3D tokens had a detrimental effect on the performance. Furthermore, base 2D policies applied in that section used vision encoders pre-trained on massive amount of vision data. Trained-from-scratch 3D encoders were not a match for them. Thus it was important for us to build on the 2D policies in that section. The approach we used there with the positional encoding mechanism in the form of STRING applied on RGB-D images via the 3D patch-lifting (see paragraph: $\\textit{“Lifting patches to 3D for STRING”}$, p.7) enabled us to successfully do it. First of all: it did not require working with PCs and retained the same number of patches as its 2D counterpart. That was critical for computational efficiency. Secondly, it could easily reuse 2D-pretrained checkpoints, leveraging vision input pre-processing from the 2D baseline as is, only to add depth signal to modulate attention computation. The STRING variant re-used all pre-trained parameters of its 2D counterpart, introducing a negligible amount of additional 3D-specific trainable parameters. Finally, STRING variant could be set up in such a way that at the very beginning of its training it was equivalent to the 2D variant. This was achieved by setting up a depth scaling parameter for the patch-depth (obtained by averaging over depth values of its constituting pixels) as zero at the beginning of training. We commented on some of these in the paper (e.g. a paragraph “From 2D to 3D with STRING”, p. 8), but will provide more detailed discussion in the camera-ready version.\n\nWe also would like to note that we added several additional experiments for the rebuttal, confirming the effectiveness of STRING: \n\n1. We re-ran the evaluations for the $\\textit{BowlOnRack}$ task (we chose one task, given rebuttal period time constraints) from the Aloha-sim portfolio, by increasing the number of trials $\\textbf{from 10 to 100}$. The ranking of methods in that setting stays exactly the same as reported before with STRING outperforming others.\n2. We run experiments for 2D and 3D object detection using the SoViT-400m models which surpass ViT-H and are competitive in performance with ViT-G. As compared to baseline, STRING provides >$\\textbf{4.7}$% relative improvement on COCO AP and >$\\textbf{5.1}$% relative improvement on LVIS AP and improves also upon regular RoPE in the OWL-ViT-2D setting. For 3D object detection with SoViT-400m models, using STRING led to additional accuracy gains as compared to the baseline ($\\textbf{1.2}$ of the baseline improvement coming from using a larger model). \n3. We also conducted additional scaling experiments, where we increased the image resolution from 256 x 256 to 512 x 512. STRING maintained an advantage over the baseline for the object detection task from Sec. 4.2.2 (3D bounding boxes): $\\textbf{6}$%+ for the 2D variant and almost $\\textbf{10}$% for the 3D variant. \n\n\n$\\underline{References}$:\n\n[1] Early or late fusion matters: Efficient rgb-d fusion in vision transformers for 3d object recognition; Tziafas et al.; IROS 2023.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The experiments cover several aspects, including vision classification and retrieval, object detection, and robotic manipulation.\nThe authors even conduct the real-world 3D robot manipulation to show the real world effectiveness of String.\nHowever, for the real-world 3D Robot Manipulation part, the authors only show that 3D String is better than 2D, which is actually widely shown [1, 2]. Could authors try some simple 3D-based imitation learning methods [1,2] and showcase whether String could directly improve them?\n\n[1] Ze et al. \"3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.\" arXiv preprint arXiv:2403.03954 (2024).\n[2] Ke e tal. \"3d diffuser actor: Policy diffusion with 3d scene representations.\" arXiv preprint arXiv:2402.10885 (2024).", "relation_to_broader_scientific_literature": "The proposed method generalizes RoPE with a unified rhetorical framework, which is related and useful to the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The introduction is well written, clearly introducing the context and the motivation.\n2. The experiments covering several domains are very extensive.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "As mentioned above, for the 3D robot manipulation part, it will be more convincing to showcase how STRING improve 3D-based imitation learning methods instead of 2D baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors introduce STRING (Separable Translationally Invariant Position Encodings), a novel family of position encodings for Transformer models. STRING generalizes the widely used Rotary Position Encoding (RoPE) using a Lie group theoretical framework. This allows it to natively handle multi-dimensional inputs (1D/2D/3D) while preserving key properties like separability and translational invariance, and makes it especially relevant for robotics/computer vision. The authors propose efficient implementations (Cayley-STRING, Circulant-STRING) and demonstrate performance improvements over RoPE and absolute position encodings across diverse tasks, including image classification/retrieval, 2D/3D object detection, and simulated/real-world robotic manipulation.

The reviewes were generally positive, with reviewers particularly highlighting the theoretical soundness of the proposed method, novelty, sufficiency of experimental evaluation, as well as the well written nature of the paper. Some concerns were also raised about the number of trials in robotics experiments and the lack of scaling experiments to larger models. These were largely addressed in the rebuttal with additional experiments (larger SoViT models, higher resolution, more trials for one sim task). The authors also addressed other reviewer asks such as runtime performance/memory numbers; tradeoffs between Cayley-STRING and Circulant-STRING, which will be added to the final version of the paper.

Though one additional concern from reviewer PFCP was that the authors only show that using STRING, methods that use RGBD outperform the method that used RGB - but this does not necessarily show how well STRING handles native 3D representations. The authors respond that the intent was to practically show that the best performance by adding 3D info efficiently to very strong pre-trained 2D models, and direct 3D models (such as those that operate on point clouds) are more expensive to train and experiment with. The AC agrees with this, and also feels that in light of the other evidence in support of the method, as well as the depth-normal map experiment the authors performed as a 3D baseline, the efficacy of the method is indeed being shown. 

Overall, given the strengths of the paper, the theoretically sound contribution, and positive consensus from all the reviewers, I recommend the paper be accepted,
Decision: Accept (spotlight poster)