Group Representational Position Encoding
Authors: Yifan Zhang
Zixiang Chen
Yifeng Liu
Zhen Qin
Huizhuo Yuan
Kangping Xu
Yang Yuan
Quanquan Gu
Andrew C Yao
Venue: ICLR 2026 Poster (2026)
Keywords: position encoding
group theory
Primary area: learning theory

Abstract: We present GRAPE (Group RepresentAtional Position Encoding), a unified framework for positional encoding based on group actions. GRAPE brings together two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $\operatorname{SO}(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $\mathrm{GL}$.
In Multiplicative GRAPE, a position $n\in\mathbb{Z}$ (or $t\in\mathbb{R}$) acts as $\mathbf{G}(n)=\exp(n\,\omega\,\mathbf{L})$ with a rank‑2 skew generator $\mathbf{L} \in \mathbb{R}^{d \times d}$, yielding a relative, compositional, norm‑preserving map with a closed‑form matrix exponential. RoPE is recovered exactly when the $d/2$ planes are the canonical coordinate pairs with log‑uniform spectrum. Learned commuting subspaces and compact non‑commuting mixtures strictly extend this geometry to capture cross-subspace feature coupling at $O(d)$ and $O(rd)$ cost per head, respectively.
In Additive GRAPE, additive logits arise as rank‑1 (or low‑rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) as exact special cases while preserving an exact relative law and streaming cacheability. Altogether, GRAPE supplies a principled design space for positional geometry in long‑context models, subsuming RoPE and ALiBi as special cases.

Published as a conference paper at ICLR 2026
GROUP REPRESENTATIONAL POSITION ENCODING
Yifan Zhang'
Zixiang Chen
Yifeng Liu
Zhen Qin Huizhuo Yuan?
Yang Yuan
Kangping Xu
Quanquan Gu
Andrew Chi-Chih Yao
Princeton University UUniversity California, Los Angeles III,, Tsinghua University
yi riffhang@princetton.. edu qguêcs wuwaalaaa.oo. edu undrewcyad@tsinghun edu.cn cn
ABSTRACT
We present GRAPE (Group RepresentAtional Position Encoding), unified framework for positional encoding based on group actions. GRAPE brings to- gether two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in (O(d) and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group GL. In Multiplicative GRAPE, position n E Z (or E IR) acts as G(n) = exp (n w L) with rank-2 skew gen- erator E Rdxd, yielding relative, compositional, norm-preserving map with closed-form matrix exponential. RoPE recovered exactly when the d/2 planes are the canonical coordinate pairs with log-uniform spectrum. Learned commut- ing subspaces and compact non-commuting mixtures strictly extend this geome- try to capture cross-subspace feature coupling at O(d) and O(rd) cost per head, respectively. In Additive GRAPE, additive logits arise as rank-1 (or low-rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) ex- act special cases while preserving an exact relative law and streaming cacheability. Altogether, GRAPE supplies a principled design space for positional geometry long-context models, subsuming RoPE and ALiBi as special cases. Project Page:
INTRODUCTION
Positional information is essential for sequence modeling with Transformers (Vaswani et al., 2017), whose self-attention is otherwise permutation cermutation-invariaat. Early work injected absolute positional codes (sinusoidal or learned) into token representations (Vaswani et al., 2017). Later, relative en- codings depending on offsets (Shaw et al., 2018) and linear logit biases such ALiBi (Press al., 2021) were introduced, the latter offering strong length extrapolation with negligible overhead.
Rotary Position Embedding (RoPE) (Su et al., 2021) realizes relative positions orthogonal planar rotations of queries and keys, preserving norms and yielding exact origin invariance of attention scores. Despite its appeal, RoPE fixes coordinate planes and typically a log-uniform spectrum, limiting cross-subspace coupling and contextual warping of phase. More broadly, absolute codes break translation equivariance; aable-based relatives add window deeenddd overhead. A new for- mulation is needed because current methods isolate the essential properties of stability, monotonic distance penalty, and expressivity. These observations motivate unified formulation that (i) pre- serves RoPE's orthogonality and exact relativity when desired, (ii) also covers additive/forgetting mechanisms such as ALiBi (Press et al., 2021) and Forgetting Transformer (FoX) (Lin et al., 2025), and (iii) admits learned and contextual generalizations with clean streaming.
We therefore propose Group RepresentAtional Position Encoding (GRAPE), group-theoretic framework that unifies two complementary families of positional mechanisms (see Figure/ 1 for an overview). The multiplicative family (Multiplicative GRAPE) models positions as nres-presgrving rotations in SO(d) acting (q. k); the additive family (Additive CRRAPE/Pat-Integee Addi- tive GRAPE) models positions as unipotent actions in the general linear group GT. that yield
"Core contribution; 'Coresponniig authors.

---

Published as a conference paper at ICLR 2026
GRAPE
Group Representational Position Encoding
General Relative Law: G) =G
(n) = exp(n Generator)
Multiplicative GRAPE
Additive GRAPE
Operation: Rotation
Operation: Translation
Manifold: (O(d)
Manifold: GL((++) (Unipotent Lift) Generator: (Low-rank Nilpotent)
Generator: L (Rank-2 Skew)
0 exp = =++A
A
(Lp(L)
dis
L
birss
64
Tarslation
pas
Norm-preservie preserving Isometry Fast Matrix Exponentials
Homogerneos Space
Addilive Translations
Recovers: RoPE
Extensions: Learned Basis
Recovers: ALiBi, FoX
Extensions: Path Integral
Also extends to contextual forms via state-dependent generators
Figure 1: Overview of the GRAPE Framework. We unify positional encodings via group actions Gnn) exp(nwL). Left: Multiplicative GRAPE recovers RoPE via rank-2 skew generators in Additive GRAPE recovers ALiBi and FoX via low-rank nilpotent generators in the unipotent subgroup of GL((d + ke) (k = 1 or 2).
(O(d). Right:
linear-in-ofiie logit biases (including content- gated and path-integral forms). This perspective re- covers RoPE and ALiBi as exact special cases, proves that FoX an exact instance of Additive GRAPE, and supplies principled, streaming -friinnyy contextual extensions on both sides.
Concretely: (a) Multiplicative GRAPE (GRAPE-M) encodes n (or R) as an element of (O(d) via a rank-2 skew generator; and (b) Additive GRAPE (GRAPE-A) and Path-Integral Additive GRAPE (GRAPE-AP) lifts to the general linear group GL using homogeneous coordinates to produce linear-in-offset logit biases (recovering ALiBi and FoX).
For Multiplicative GRAPE, positions are mapped as
Gn) = exp (n L) E SO(d), L= ba" T so(d),
where b E Rd define rank-2 skew generator L and w > 0 is a frequency. The action is an isom- etry, and G + +m) = (m)) guarantees exact origin invariance attention logits. We derive closed-form Rodrigues-type formula (Rodrigues, 1840; Hall, 2013), enabling fast linear-time ap- plication with stable derivatives and no explicit matrix materialization. RoPE is recovered when d/2 commuting rank-2 generators act disjoint coordinate planes with prescribed frequencies.
For Additive GRAPE, positions are mapped via the matrix exponential Gaaa (m) = Ax)(nwA) = I+ n in lifted homogeneous space. Here, the generator A E gl(d + 1) is nilpotent matrix of rank one. While this additive transformation not an isometry, it preserves the exact relative law, ensuring attention scores depend only position offsets. This formulation provides rigorous group-theoretic foundation for additive biases, recovering ALiBi and FoX as exact instances.
Our contributions are highlighted follows:
2

---

Published as a conference paper at ICLR 2026
1. We propose GRAPE unified group -hou-ttheoeetic view that subsumes multiplicative orthogonal rotations in (O(d) and additive unipotent (all eigenvalues equal to 1) mechanisms in general linear group GL, recovering RoPE and ALiBi exact special cases and proving FoX is exact instance (Appendix C).
2. Multiplicative GRAPE. We derive a closed-fone rank-2 matrix exponential with fast application and stable differentiation; we show RoPE is special multiplicative GRAPE in possibly learned orthogonal basis.
3. Additive GRAPE. We show that linear-in-offset logit biases arise from rank-1 (or low-rank) unipotent actions in the general linear group GL. with an exact relative law and streaming cacheability. This includes query- key-gated slopes, commuting dictionary of additive com- ponents, and exact recoveries of ALiBi and FoX in closed form (Sections 4, 4.2, Appendix C). We also formalize path-integral additive biases that remain causal and support efficient training. (Section 5).
MULTIPLICATIVE GROUP REPRESENTATIONAL POSITION ENCODING
We propose the Multiplicative GRAPE, as a Lie-group positional map with a closed-form rank-2 matrix exponential, exact relative law, and streaming/cache methodology. The intuition the special orthogonal group SO(d) "(Hall,
to encode position as a norm-preserving rotation 2013). A single skew-symmetric generator L so(d) produces the entire family of rotations via the matrix exponential. We begin with notation and the rank-2 generator.
2.1 PRELIMINARIES AND RANK-2 GENERATOR
The generator L is formally defined as an element of the corresponding Lie algebra, so(d). Let so(d) [L E Rddd L } denote the Lie algebra of SO(d). The simplest non-trivial generator defines rotation within a single 2D plane. We construct such a rank-2 generator from two vectors, and b, that span this plane of action. For a E Rd, define the rank-2 generator L(a, b) as
L(a, b) ab ba a ||a|| : B |bl a A 0g
VA. (2.1)
Rank-2 structure. Let span b). The rank-2 generator L has useful geometric property: applying it twice projects onto the action plane U and scales. A direct calculation shows
L = - Pus
where Pu is the orthogonal projector to the space U. Hence spectrum of L (the set of its eigen- values), denoted (L), (=iis,...... 0} and the minimal polynomial is A(>² + s²). A detailed derivation is given in Appendix
Initialization. Write A - Ja b] Rdx2 and J = -i- so that L = AJA For any M E SL(() (the 2 x 2 real matrices with determinant 1, see Table 3), MJM J and thus A + AM leaves invariant; for general M E GL(2) the group of invertible 2 x 2 matrices), L scales by det(M). Therefore the oriented plane Ut = span a, b} and the scalar
VaB - 72 determine the action. We fix gauge at initialization by ||a|| = ||b]| = and a Tbb = (absorbing scale into w).
Canonical 90 rotation operator. Fix block-diagonal complex structure J E so(d) with JT -J and J2 = -I (for odd d. act on the top-left 2 [d//2] coordinates and leave the final coordinate unchanged). Concretely,
+ d/2] -1) ). For any Rd, write a Ja. which equals "a rotated by 90S" within the canonical 2D blocks and satisfies a and ||a_| ||||]
2.2 EXACT RELATIVE LAW
For fixed L E so()), define G(n) exp(nL) E SO(d), which forms one-parameter subgroup. The exact relative law property for positional encoding implies:
G(t x) Gs)) G()), G(n) [G)))=== II
Definitions (O(d) and other mathematical terms are postponed Table 3 in the Appendix.
3

---

Published as a conference paper at ICLR 2026
Here G(n) SO(d), so the transpose coincides with the group inverse, G(n) = G the identity above is exactly the relative-position law for a one-parameter subgroup. A concise summary of SO(d), GL(d) and SL(d) is collected in Table 3. This algebraic property enables relative positional encoding: interactions depend only on offsets.
(n) exp(nwL), G + m) ((m) G(0) =I, and G(-n) G()
Crucially, this exact relative property relies solely on the one-paramett subgroup structure (G(n + m) G(m)), holding regardless of whether the generator implies commuting or coupled non-commuting subspaces.
2.3 CLOSED-FORE FORM FAST MATRIX EXPONENTIAL
Based on the minimal polynomial mentioned Section 2.1, 1 the exponential map exp(L) for a rank-2 generator can be expressed as a quadratic in L. This yields convenient closed-form solution, often referred to as a Rodrigues-tyy ttype formula (Rodrigues, 1840; Hall, 2013):
eLp = I+ sin 8 L + 1 cos I
Geometrically, the formula is best understood via L as a projector onto U. Since L = - the exponential can be written
sin $
exp(L) I- (1 cos s) Pu
L,
which reveals its action explicitly: it is rotation by angle within the plane U span(a, b) and the identity on the orthogonal complement The vectors a and b thus define the plane of action for the positional rotation.
Cost of application. For a single rank plane, computing y = (n))x requires two inner products (a, x). v (b,x), followed by y x fi (n) -u) (n) hy(c bu) Bau abu),, where (ar, 3, 7) are plane scalars and f1, are trigonometric scalars (with series guards as + 0). This is (d) flops with small constant and no materialization of Gnn); derivative expressions
in Appendix I.
2.4 THE b = Ja CONSTRAINT
We now consider an important special case by setting b Ja. This constraint, which makes the plane vectors a and b orthogonal and equal in norm, significantly simplifies the generator's 's structure and reveals direct connection to canonical RoPE formulation. With this constraint, the scalars simplify: a b a Ja 0, Moreover, on the 2D subspace = span a, Ja} one has
|b|l2 ||al a, and hence
vaB -
L(a, Ja] (Ja)a,
I(a,Ja) Ja aa,
so L(a, Ja)lu = Jau and Jaa
0. Therefore
exp (mwL(a, Ja)) =I I (1 cos(nwa)) Pu sin(nwax) JPu:
This expression follows by substituting Llu = a The lu and L = PP into the Rodrigues formula exp(nwL) I sim(nws) L (-cos(mess) 1 with steps. It a pure planar rotation by angle nwa on 14 and the identity on 44_
a see Appendix I for the algebraic
Corollary 2.1 (Frequency-nont coupling). If ||a|| 1, the rotation angle reduces to nw. Without normalization, the effective frequency is weff = wi ||aal so the scale can be absorbed into w.
2.5 APPLICATION TO RELATIVE ENCODING AND EQUIVARIANCE
We demonstrate how the GRAPE-M operator G(n) is applied in practice. As established in Section 2.2, the operator's group structure guarantees the exact relative law. We first transform the query and key vectors, qi and ky, into position- aware representations, qi and ay:
q. G(i)qi
k G(jkk)
4

---

The simplest way to combine generators is ensure they act on mutually orthogonal subspaces, which guarantees they commute. Let d be even. For i = 1,...,d. d/2, we can define set of rank- generators }L,), each acting on distinct 2D plane. RoPE the canonical example of this Published as a conference paper at ICLR 2026
It follows from the exact relative law established in Section 2.2 that the attention score between these position-awwed aware vectors simplifies to:
q kj q. G()k) - q: G i)k,
Hence, the attention score depends solely on the relative offset j i, not the absolute positions.
Streaming and caching. At inference, cache kj = Gjjkk, once when token j arrives. At step form q. G())q, and compute logits q. kj- No cache rotation is needed when t increments; complexity matches RoPE. A full integration into multi-head attention (per-head formulation, logits, and streaming) is detailed in Section B.
MULTIISUBSPPCE MULTIPLICATIVE GRAPE
A single rank-2 generator touched. To encode position across the entire hidden dimension, we can combine multiple genera- tors. This leads to the Mubti-Subspace (MS) Multiplicative GRAPE (GRAPE-M) model, which forms the basis for both RoPE and more expressive types. Detailed rank-2 algebra appears in Ap- pendix I.
2D subspace, leaving the rest of the d-dimensional space un-
3.1 MULTI-SUBSPACE GRAPE-M AND RoPE AS A SPECIAL CASE
construction. We further discussed non-commuting multiplicative GRAPE in Appendix D.
- and the coordinate selector be U; [e2:-1 e2] E [ddx2, We set the rank-2 generators as L, = U,JU L(e2--1, e2i) and assign per-plane frequencies 0g O. The total generator is the commuting sum:
Let the 2 canonical skew matrix be
d/2
LROPE = Σ O,L:
with
L, = 0 for i #
1=1
Then
d/2
G(n) exp (nLRRO))
II
exp(n0:L.)) blockdiag (R2(n01), R2(n0a/22),
(3.1)
where (f) denotes the standard 2 x 2 rotation matrix introduced in Table and the last equality holds because each term exp(n0;L.) is identity except for a single 2x2 rotation block on its diag- onal. Eq. (3.1) is precisely the RoPE mapping: block-diagonal product of planar rotations with per-subspace angles n00;-
Equality holds when the planes (U.) the coordinate 2D blocks and (0,) follow the canonical log-uniform spectrum.
Proposition 3.1 (RoPE a multiplicative GRAPE), Choose d/2 mutually orthogonal vectors (a,) and b, Ja; with per-plane angles 0,- Then the commuting MS-GRAPE G(n)
I exp(no,, Ja,)) equals the standard RoPE map in a (possibly learned) orthogonal ba- sis. If the planes are the canonical coordinate pairs and (0:) follow the log-uniform spectrum, recover the canonical RoPE exactly.
Spectral parameterization. Classical RoPE chooses 0, on a log-uniform grid across In GRAPE, 8; can be learned shared/tied across heads or layers. The MS-GRAPE view also al- lows replacing the coordinate selectors U, by a learned orthogonal basis B E SO(d) so that preserving commutativity while learning subspaces.
L Σ, BU,JJU, .
Multimodal GRAPE. Please refer to Appendix G for 2D and 3D GRAPE for Vision and Multi- modal Position Encoding.
5

---

Published as a conference paper at ICLR 2026
4 ADDITIVE GROUP REPRESENTATIONAL POSITION ENCODING
This section shows that additive positional mechanisms (absolute shifts of features and additive logit biases, including ALiBi (Press et al., 2021)) also admit group-theoretic formulation. The key is homogencous lift to an augmented space and a one- one-parameter subgroup of the general linear group GL that acts by unipotent (all eigenvalues equal 1) transformations. This yields an exact relative law and streaming/cache rules analogous to Section 2.5.
4.1 HOMOGENEOUS LIFT AND A UNIPOTENT ACTION
To produce additive biases from multiplicative group action, employ the homogeneous lift. This a standard method linear algebra for representing affine transformations (such as translations) as linear transformations in nigher-dimensiona space. Let & E Rd4* denote homogeneous augmentation of x E IR We work within the general linear group GL(d+k) and its Lie algebra gl(d+k). Fix a nilpotent generator A such that A 0. Its exponential is unipotent:
Gada (n) := exp(n w A) = I + nw A E GL(d+k).
Crucially, this action forms a one-parameter subgroup: G (n++m) Gadd (n) (m).
Canonical rank-1 generator in (d+1))-lift. convenient concrete choice (used in Appendix J.3) the rank-1 index-2 nilpotent
A
Oaxe
Ushift
E (d++))x(++)
=
(4.1)
where Usin RR; acting on x = [x; 1] yields (n)a [x + u 1), i.e., a translation in the original d-dimensional space.
Application and exact relative law in GL. For queries/keys augmented as q and k,. define
q Gadd qi k,- Gj)a(j) kj.
(4.2)
We use the shorthand Gadd (j) = ()) to emphasize that we first take the group inverse in GL and then transpose it. This composition results in the final form:
q. kj- q. Gadd k,, depending only on j-i.
(4.3)
Closed form and content-gateds additive term. To incorporate content dependence while strietly maintaining the unipotent group structure, we decompose the action into separate commuting gen- erators for queries and keys. Let Ag be the canonical rank-1 nilpotent basis in ddmmensiino lift (augmenting q = [q: 1; and = [k; 0; 1)):
Ag = e E (8+2))x(4)+)
(4.4)
We modulate this basis via non negative scalar gates (q:) = softplus q:/Vd) (qtp/v/) and Au (s(k))
softplus(u) (aruus/uuu///iii k/)v)..
Because the scaled generators share the same nilpotent basis, they commute. Let Aij (q.))) + Ak, 2 0 and define the effective (endpoint-dependent) nilpotent generator Aij := -A An- so that A, AA; AJ- Their composition in the relative offset m = - yields a summed decay rate. The unipotent inverse transpose admits exact closed form that converts the offset m (negative in causal attention) into a penalty:
Gadd (m)- = mw A,, = I mw (A,(q:) + Ak(k;)) AT;
m m-j-i.
(4.5)
This formulation is rigorous: the softplus(-) function ensures the effective decay rate is strictly non- negative. When j < i, the offset m is negative, resulting in a monotonic penalty. Applying this operator yields the exact affine score:
q k, = q. k, + (i-i) [softpluss (v qq/va + softplus(u" (uThpllusuTk//V[]) kv/V/)).
(4.6)
|q: 1: k k; 0: 1], the base inner product satisfies q. k; = q.- k, (no extra constant), while the rank-1 update contributes exactly (j (j-i)w(AA, + A&), which is non-positive for causal offsets learnable, content-adaptive linear bias (slope) from first principles: it is the sum of query-dependent and key-dependent unipotent actions in the general linear group.
Note that under the asymmetric lift
which we denoted as GRAPE-A-QK. This derives

---

Published as a conference paper at ICLR 2026
4.2 EXACT ALIBI AS A RANK- I UNIPOTENT IN GL(d+2)
ALiBi adds a head-specific scalar slope (i-) to the logits that is independent of content. This is captured exactly by augmenting with constant coordinates:
q: = qq:
0] Rd+
kj = [ky; 0; E
and choosing the rank-1 nilpotent generator
A = 3h cd+1 0242
An = Sn. Cut2 02411
(A =0.
(4.7)
Then Gadd, (1))
= m J and
q Gudd q,j, + (j-i)
ie., the ALiBi term emerges as a unipotent GL((++) action with exact relative composition. Remark 4.1 (Why GL(d+))). While general additive biases, such as key-gated slopes, can be realized in GL(d+1), the content-independent nature of exact ALiBi necessitates the (d+2)-- dimensional lift. Generating non-zero scalar bias via a nilpotent generator requires the interaction of two distinct constant coordinates, one for query, one for key, to avoid self-loops that would violate the trace-zero property of the generator.
FoX as GRAPE-A. Let f. E 1] be per-token forget scalars and set wi == log Si. The resulting additive bias is j(t. Σi=f+1 we, which coincides with FoX's forgetting bias Dij- A full derivation and the unipotent path product are given in Appendix C.
5 PATH INTEGRAL ADDITIVE GRAPE
Additive GRAPE (GRAPE-A) realizes exactly relative additive logits via one-parameter unipotent action in the general linear group GT,; the bias depends only on offset m = j- (or contextual phase difference - when using cumulative phases). Here the "phase" 4, is a scalar path variable, typically defined as a cumulative sum 2, = Erer we of per-token frequencies we, so that 4 4, plays the role of an effective relative position. In practice, amount of additive nncouragemenn/uppressttt the endpoint (e.g., the current syntactic semantic needs of the query token), while preserving causality, boundedness, and clean composition with the orthogonal GRAPE acting on k). We formalize this by a rigorously defined path-integral sum, deriving conditions under which the exact relative law of Additive GRAPE is recovered.
sometimes want the between key at j and a query at to depend on
Definition (Path-integral bias). Fix a head h and per-head scale on > 0. For each time u, let Pu,A E Rd be a positional embedding obtained from token-local features (a linear projection fol- lowed by RMS normalization in our implementation). Let J be the canonical block coock-dagoonal 90- operator (Section 2.4), and define R, := exp(E T)) (a fixed commuting rotation). For link function g IR + (-o0, 0) that is monotone increasing and 1 1-Lipschitz', define the edge potential
(t, on (
R RePLa)) (PL..R/PEA)) S 0, t<t.
(5.1)
The vectors Pr, and Pt,h here are the positional embedding. The aath-integral additive bias from key position query position is the causal sum
(t.j) = Σ (n(t,) S 0.
(5.2)
E=j+1
The attention logit combines this additive term with either the raw or orthogonally-rotary bilinear part:
1
l.j.h
Va A.nken + bx(t,)) or 4.j. = vd qe, Gn + ba(t.)). (5.3)
Our experiments take g(2) log(Sigmoid(2)) then o' gz(2 1 Sigmoid(2) E (0, ensuring 1- Lipschitzness.

---

Published as a conference paper at ICLR 2026
Group-theoretic formalization and path composition. Let E ((d++)x(4+2) be a fixed rank-1 nilpotent with E² = (e.g., E = e2+28241 as in Section 4.2). For each fixed endpoint t, define endpoint-indexed unipotent factors
1 (t) = I - (t, E). H h
Since E² 0, the path product along (j, (] collapses additively:
t
II H (0= E-j+1
Σ (n((t bx(tt) E.
(5.4)
Scoring in homogeneous coordinates as in Section with the paired inverse-transpose removes multiplicative anisotropy and yields exactly the additive term bt,t,j), cf. Eq. (4.3), since (I + by [I, and G. E E Kj.A 1 under the asymmetric lift. The rowwise semigroup law is preserved (Eq. (5.4)), while the t-dependence of the factors intentionally relaxes the global one-pane mee-parmeterr group law.
()
Relation to GRAPE-A. GRAPE-AP strictly contains GRAPE-A as the special case in which edge potentials do not depend on the endpoint:
vh (t, E) = O2 de
On 0,20, 0, 0
bn (t.j) On Σ
- On (Ai - -1,) S 0, Au = Σ ae.
ø=++
E<aa
Two important instances follow directly:
Exact ALiBi. ag = gives by (t,j) -t On(j-t); setting On. Bn recovers the ALiBi
term from Section 4.2.
Phase-modulated Additive GRAPE. If ae we with =((x)) 2 0, then by, (t.5) = -00 (+=
22) with 4 = Eecu We.
In both cases, (t, depends only on a (possibly contextual) phase difference and thus obeys the exact relative law with the same streaming/cache policy Section 4. Outside these endpoint- independent regimes, GRAPE-AP provides strietly more expressive, path-integral biases while pre- serving row-wise path composition (Eq. (5.4)).
Computation and streaming, For each head / and decoding step 4, compute the row {vh (1, })ee by single similarity sweep É - (pr,a, RePE,.) (the rotated probes RÆPe, can be cached on arrival), apply the link g. and take a prefix sum to obtain j + by (t..). This yields O(t) per-step overhead with ((1) recomputation per cached key; memory O(L) per head for the cached probes (or (d) if the per-f rotations are recomputed on the fly).
Spectral and stability. Each factor H (2) (c)
e/n (t, E)E unipotent with all eigenvalues and at most two singular values deviating from 1; the full path product equals I - by (t.j)E (Eq. (5.4)). As in Appendix J.3, the paired transppe used for scoring cancels multiplicative distortions and delivers exactly the additive bias b, (t, operator norms remain controlled linearly in [b, (f,
more extensive spectral analysis, including eigenvalue structure and singular-vale behavior across GRAPE variants, is provided in Appendix J. There, we also give an explicit comparison to PaTH Attention (Yang et al., 2025b), which is shown to be contractive and near singular. These properties may impair PaTH's effectiveness in long-context modeling.
6 EXPERIMENTS
In this section, we evaluate the performance of GRAPE on the language modeling task comparison with baseline positional encoding mechanisms, including RoPE (Su et al., 2021), AliBi (Press al., 2021), as well as Forgetting Transformer (FoX) (Lin et al., 2025).
6.1 IMPLEMENTATION DETAILS
Based on the nanoGPT codebase (Karpathy. 2022), our experiments are implemented based on the Llama model (Touvron et al., 2023a). We only change the positional encoding mechanism
8

---

Published as a conference paper at ICLR 2026
and keep the rest of the model architecture the same Llama. We choose Fine Web-Eeu 100B dataset (Lozhkov et al., 2024), which contains 100 billion training tokens and 0.1 billion validation tokens, and randomly choose 50B tokens for training. Our medium models (350M) have 24 layers and heads, with hidden size of 1024; large models (770M) have 36 layers and 10 heads, with hidden size of 1280 and head dimension of 128. We applied QK RMSNorm for training stability (Yang et al., 2025a). The context length is set to 4,096, and the batch size is 480. All the large models (770M) are optimized by AdamW optimizer (Loshchilov and Hutter, 2019), with maximum learning rate of 2 x 10 for medium models and 1 10 for large models, (31, = 0.9, 0.95, and weight decay of 0.01. We use cosine learning rate scheduler with 2,000 warm-up iterations, and the minimum learning rate is 3 x 10 We also clip the gradient to 1.0 for stabler training. The frequency of RoPE set to 10,000. Moreover, for fair comparison, we do not use FoX-Pro and disabled the V-sshift module within it.
6.2 RESULT ANALYSIS
The curves for training and validation loss of models with variant positional encoding mechanism are displayed in Figures and 3. This analysis provides specific insight into the source of the frame- work's stability and performance. other mechanisms, including RoPE and FoX. Moreover, the model with RoPE suffers from training instability shown in Figure 3 (a), while the model with GRAPE embedding steadily improves during the training process.
can be observed that GRAPE can keep a persistent edge over
Table 1: The evaluation results of medium models with different positional encoding mechanisms pre-trained using the FineWeb-Eao 100B dataset (0-shot with Im-evaluation-hammes)) Avg. is re- computed over the remaining tasks. The best scores in each column are bolded, and the second best are underlined. Results are ranked separately for w/o K VV-shift and w/ KV-shift.
Method
ARC-E
ARC-C
HellaSwag
OBQA
PIQA
WinoGrande
SciQ
Avg.
RoPE
56.36
30.38
44.65
35.20
68.77
52.33
74.40
51.73
GRAPE-M-cIR
56.31
29.95
44.37
33.40
68.23
53.28
76.90
51.78
GRAPE-M-nonck
56.82
28.84
44.36
32.80
68.93
53.67
77.10
51.79
AliBi
58.21
29.78
45.38
34.60
70.08
53.51
78.50
52.87
FoX
58.38
30.89
45.80
35.00
69.37
52.88
78.40
52.96
GRAPE-A-Q
58.00
30.89
45.70
36.20
68.93
52.96
77.00
52.81
GRAPE-A-K
58.54
30.72
45.13
34.60
68.
53.35
76.30
52.43
GRAPE-A-QK
57.95
32.00
45.77
33.40
69.37
53.51
79.00
53.00
GRAPE-AP
59.26
31.31
45.42
35.60
68.17
53,28
79.70
53,25
RoPE (w/ KV-shift)
58.04
30.80
45.04
35.40
68.39
53,99
78.60
52.89
AliBi (w/ KV-shift)
59.51
31.31
45.93
35.40
69.15
52.88
78.
53.18
FoX (w/ KV-shift)
58.84
30.97
46.25
34.20
67.85
55.41
79.70
53.32
GRAPE-AP (w/ KV-shift)
57.32
46.18
35.80
55.64
79.60
53,46
Medium Model, FineWeb-edu1008
Medium Model, FineWeb-edu1008
RoPE
RoPE
3.0
Fox
3.0
Fox
GRAPE-AP
GRAPE-AP
2.9
Validation Loss
2.9
Taaninng
2.7
2.7
2.6
2.6
2.5
10
20
30
40
50
10
20
30
40
50
Training tokens
Training
(a) Training Loss
(b) Validation Loss
Figure 2: The training and validation loss of medium-size models (355M), with different positional encoding mechanisms on the FineWee-Edt 100B dataset.

---

Published as a conference paper at ICLR 2026
Large Model, FinWWebeedu1008
Large Model, FineWeb-edu1108
3.1
3.1
RoPE
RaPE
3.0
AliBi
3.0
AliBi
Fox
Fox
GRAPE-AP
GRAPE-AP
Velidation Loss
2.8
2.8
Waining
2.7-
2.7
2.5
2.5
2.4
2.4
20
30
40
Trairing tokens (B)
Training tokens (B)
(a) Training Loss
(b) Validation Loss
The training and validation loss of large-size models (770M), with different positional encoding mechanisms on the FineWeb-Edu 100B dataset.
Figure 3:
The evaluation results of large models with different positional encoding mechanisms pre- trained using the FineWeh-Fdu 100B dataset (O-shot with nn-evaluatioon-haneese The best scores
Table 2:
in each column are bolded, and the second best are underlined. Results are ranked separately for
w/o KV-shift and w/ KV-hiift
Method
ARC-E
ARC-C
HellaSwag
OBQA
PIQA
WinoGrande
SciQ
Avg.
RoPE
62.63
32.76
51.01
36.40
71.33
55.72
80.50
55.76
GRAPE-M-cI.
60.52
32.51
50.13
35.40
70.67
54.38
79.50
54.73
GRAPE-M-noneee
60.61
33.45
49.47
35.60
70.51
53.35
80.70
54.81
AliBi
62.67
34.39
51.33
36.60
71.11
56.27
82.70
56.44
FoX
61.07
33.11
51.85
37,80
71.27
55.33
83.70
56.30
GRAPE-AP
63.89
34.22
51.52
35.40
71.98
56.99
84.40
56.91
RoPE (w/ KV-shift)
62.12
33.70
50.13
35,80
71.49
53.51
80.50
55.32
AliBi (w/ KV-shift)
63.68
34.56
52.04
37.80
71.38
55.09
83.90
56.92
FoX (w/ KV-shift)
63.55
33,96
52.72
38.40
71.71
56.12
83.20
57.09
GRAPE-AP (w/ KV-shift)
63.72
33.11
52.29
39.00
71.65
54.78
83.50
56.86
7 CONCLUSION
GRAPE provides a general framework for positional encoding based on group actions, unifying mul- tiplicative and additive mechanisms. Multiplicative GRAPE offers closed-form form. rank-2 exponen- tial that is relative, compositional, and norm-presene corm-preserving: recovers RoPE and yields learned- basis and non-commuting extensions at controlled cost. Additive GRAPE realizes ALiBi and FoX ex- actly via unipotent general linear group GI. lifts with the same streaming/cache policy. The GRAPE framework integrates seamlessly with existing Transformer models and offers a principled, extensi- ble design space for future architectures.
USE OF LARGE LANGUAGE MODELS
We used Large Language Models (LLMs) assistive tools polish part of this paper, and the roles of LLMs in this work are restricted to improving presentation.
ACKNOWLEDGEMENTS
We thank the anonymous reviewers and area chairs for their helpful comments. We also thank Zhixuan Lin, Songlin Yang, and others for their constructive feedback on the experimental settings.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličkovic. Round and round go! what makes rotary positional encodings useful? In International Conference on Learning Representations (ICLR 2025), 2025. URL https:
/ saazxiv.org/as//2200 06205. Also arXiv:2410.06205.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.06. 150, 2020.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via position interpolation. arXiv preprint arXiv:2306. 2023. URL https: //apvv/aaavvv///aa//eaa org/ abs// 2306. 15595.
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. Kerple: Kernelized rel- ative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386-8399, 2022a..
Ta- Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. arXiv preprint arXiv:22 222/222.03 10356, 2022b.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, William Cohen, Ruslan Salakhutdinov, and Jaime Carbonell. Transformer-Li Attentive language models beyond fixed-length context. In Proceedings of ACL, pages 2978-29999 2788, 2019.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. BERT; bidirectional transformers for language understanding. In Proceedings of NAACL-IILT, pages 2019.
Pre-training of deep
Olga Golovneva, Jackie C Lou, Daniel Holtmann-Rice, Aditya Kusupati, Chengeng Cai, Zijian Hu, Prateek Vijay Kumar, Tim Dettmers, Pratyusha Sharma, Behnam Neyshabur, Jason D. Lee, and Mohammad Bavarian. Contextual position encoding: Learning to count what's important. arXiv preprint ar wxx.::2405.177111 2024. URL https: //t.aaiivv.//./////v//aaa//aaaia wwtpxi.w..aaas////ooot.ao.ootaat /tt.ab///4445 2405 18719.
Brian C Hall, Lie groups, lie algebras, and representations. In Quantum Theory for Mathematicians, pages 333-366. Springer, 2013.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203. 16634, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint Ttl:2006.0365342 2020.
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289-305. Springer, 2024.
Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend Il context window without tuning. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235. pages 22099-22114, PMLR, 2024.
Andrej Karpathy. NanOGPT. https: httptgttttttccce wtt.:/ooo//aaaaee piithubb.com/karppttttrrnnnoor 2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pages 5156-5165. PMLR, 2020.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances 2023.
in Neural Information Processing Systems,

---

Published as a conference paper at ICLR 2026
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint TrXiv:2006115993 2020.
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of EMNLP, pages 3309-3333 3321,, 2021.
Konstantinos Kogkalidis, Jean-Philippe Bernardy, and Vikas Garg. Algebraic positional encodings. Advances in Neural Information Processing Systems, 37:34824-34888 2024.
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang. Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418 2023.
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. CAPE: Encoding relative positions with continuous augmented positional embeddings. In Ad- vances in Neurul Information Processing Systems (NeurIPS), volume 34, pages 16079-16092, 2021.
Mengtian Lin, Ji Lin, Wei-Ming Chen, and Yonglong Tian. Forgetting transformer: Softmax atten- tion with forget gate, arXiv preprint arXiv:2503.02130, 2025. abs/2503 02130.
Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. In International conference machine learning, pages 6327-6335. PMLR, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https: http:///hggggggafeeeeeee co datasets/ HuggingF www.gggaaaa/aacooo f Feneeb-eedu.
Masato Neishi and Naoki Yoshinaga. On the relation between position information and sentence length in neural machine translation. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 328-338, 2019.
Sophie Ostmeier, Brian Axelrod, Maya Varma, Michael Moseley, Akshay S Chaudhari, and Curtis Langlotz. Liere: Lic rotational positional encodings. In Forty-second International Conference Machine Learning. 2025.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam:: Efficient context window extension of large language models. arXiv preprini (tf::230000007// 2023.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.11 12409, 2021.
Zhen Qin, Weixuan Sun, Kaiyue Lu. Hui Deng. Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Linearized relative positional encoding. arXiv preprint arXiv:2307.00927, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(1):5485-555
2020.
Olinde Rodrigues. Des lois géométriques qui régissent les déplacemens d'un système solide dans l'espace et de la variation des coordonnées provenant de ces déplacemens considérés indépendamment des causes qui peuvent les produire. Journal de Mathématiques Pures et Ap- pliquées, 5:380-440, 1840.
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Ben- nani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305. 16843, 2023.
12

---

Published as a conference paper at ICLR 2026
Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendle- man, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string, arXiv preprint arXiv:2502.02562, 2025.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprini arXiv:8803.00m 1800.02255. 155, 2018.
Koustuy Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. The curious case of absolute position embeddings. arXiv preprint 1.Xiv:22/0.12574, 2022.
Jianlin Su, Yuancheng Zhang, Shengfeng Pan, Shengyu Ge, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 2021.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A lengthhextrrapllatbbe transformer. arXiv preprint wrXiv:2212.10554 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2024,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. In Proceedings of ICLR, 2020.
Benyou Wang. Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang. Qun Liu, and Jakob Grue Simonsen. On position embeddings in BERT. In Proceedings of ICLR, 2021.
Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. In Findings of the Association for Computational Linguisties: ACI. 2024, pages 14024 14040, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 10.18653/v120244 http:///........s-eee.a.e 834. URL https: / aaaalcaannn aclanthology. org 2tgg/022 2224. findings acl. 834/.
Ulme Wennberg and Gustav Eje Henter. The case for translation-invarian self-attention in bransformer-based language models. arXiv preprint vrXiv:2106.01955 2021.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. Da-transformer: Distance-aware transformer.
arXiv preprint 2xXiv:2010.0692 2020.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025a.
Songlin Yang. Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim. Path attention: Position encoding via accumulating householder transfor- mations. arXiv preprint arXiv:2505.1638 2025b.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297 2020.
13

---

Published as a conference paper at ICLR 2026
Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng. Xin Jiang, Zhenguo Li, et al. Dape: Data-adaptive positional encoding for length extrapolation. Advances in Neural Information Processing Systems, 17726659-26700 2024.
Dawei Zhu, Nan Yang, Liang Wang. Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of Ilms via positional skip-wise training. arXiv preprint 17X1:22309.1400, 2023.
14

---

Published as a conference paper at ICLR 2026
Appendix
A Related Work
16
Application Multi-Head Attention
17
Forgetting Transformer Special Additive GRAPE
17
D Non-Commuting Multiplicative GRAPE
18
Composition of Additive GRAPE and Multiplicative GRAPE
19
F Comparison with LieRE
19
G 2D and 3D GRAPE for Vision and Multimodal Position Encoding
20
Algorithmic Details and Pseudo Code
20
I Differentiation and Fast Application of Rank-2 Matrix Exponential
20
J Spectral Analysis of GRAPE and Other Methods
21
J.1 Rank-2 Plane: Exact Spectrum and Geometric Interpretation
J.2 Multi-subspace GRAPE-M and RoPE
21
J.3 Additive GRAPE
22
J.4 Comparison to PaTH Attention
23
15

---

Published as a conference paper at ICLR 2026
Table 3: Summary of Notation and Definitions.
Symbol
Definition
GL(d)
General Linear Group:
The group of all x d invertible matrices.
(O(d)
Special Orthogonal Group: The group of d x d ootthgoonal matrices
with determinant (R. R I, det = 1).
SL(d)
Special Linear Group: The group x matrices with determinant 1.
gl(d)
general linear algebra: The Lie algebra of GI((d), consisting all d x matrices.
so(d)
special orthogonal algebra: The Lie algebra (O(d), consisting of d Sxmm-yymeetric matrices (L =)).
exp(.)
Exponential Map: A map from Lie algebra (generator) Lie group (operator).
coss 0 sin 0 sin 0
(2(
2D Rotation Matrix: The matrix
cos
G(n)
Transpose (in SO(d)) For G (Odd), the transpose is the group inverse (G T G
G(n)-T
Inverse Transpose (in GL(d)): The transpose of the matrix inverse, (G (G-I)T.
Unipotent
Unipotent Transform: A linear transformation whose eigenvalues are all 1.
Pu.A
Positional Beabbedding/Reppesenttttiin A vector derived from token-local features,
obtained via linear projection followed by RMS normalization.
A RELATED WORK
Positional information in Transformers mainly can categorized into these classes: (a) absolute encodings (sinusoidal or learned) (Vaswani et al., 2017; Devlin et al., 2019; Neishi and Yoshinaga,
2019; Kiyono et al., 2021; Likhomanenko et al., 2021; Wang et al., 2020; Liu et al., 2020; Wang al., 2021; Sinha et al., 2022; Wennberg and Henter, 2021; Ke al., 2020); (b) relative encodings that depend on offsets (Shaw et al., 2018; Dai et al., 2019; Raffel et al., 2020; He et al., 2020); and (c) linear logit biases with strong length extrapolation (Press al.. 2021: Chi et al., 2022arb; Li et al., 2023; Ruoss et al., 2023), all shaping recencylextrapolation behavior (Haviv et al., 2022; Kazemnejad al., 2023).
Multiplicative position encoding. RoPE realizes offsets as block-diagonal planar rotations of queries/keys, preserving norms and exact origin invariance; and modalities (Su et al., 2021; Touvron et al., 2023a;b; Heo et al., 2024). Angle/spectrum designs improve long-context fidelity (e.g., xPos) (Sun et al., 2022); LRPE formalizes separable relative transforms for linear attention models (Qin et al., 2023); mechanistic work analyzes frequency us- age (Barbero et al., 2025). These methods are also compatible with sparse/lincar attentions (Beltagy et al., 2020; Zaheer al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2020) and with context-scaling procedures (Xiong et al., 2023; Chen et al., 2023; Peng et al., 2023; Zhu et al., 2023; Jin et al., 2024). Beyond ID language modeling, 2D RoPE and variants adapt rotary encodings to 2D grids by applying rotations along spatial axes, and have been shown to improve high-resolution extrapolation in Vision Transformers and related vision models (Heo et al., 2024). Recently, LieRE (Ostmeier et al., 2025) learns dense skew-symmetric generators whose exponentials pro- duce high-dimensional rotations for multi-modal,, n-dimensional inputs, while STRING (Schenck et al., 2025) designs separable, translation invariant RoPE-style encodings that scale to 2D and 3D coordinates in vision and robotics settings (Ostmeier et al., 2025; Schenck et al., 2025). GRAPE-M identifies RoPE commuting rank-2 exponentials in SO(d) and extends it to learned subspaces and compact non-commuting mixtures in closed form and much faster way. Compared with LieRE, which parameterizes a dense skew-symmetric generator and applies numerical matrix exponen- tial (c.g.. torch. mentrix_eex x_exp) with (dd)) time and O(d2) parameters per head, Multiplicative GRAPE decomposes the action into rank-2 subspaces and uses the closed-form Rodrigues-tye -ypp formulas from Section 2.3, so we only need vector-vectu operations with ((d) cost per head (a detailed comparison between LieRE and GRAPE is presented in Appendix F.)
widely deployed across LLMs
16

---

Published as a conference paper at ICLR 2026
Additive position encoding and forgetting mechanisms. Additive schemes such ALiBi (Press et al., 2021) and related kemnelized/randomizee forms (Chi et al., 2022a;b; Li et al., 2023; Ruoss et al., 2023) captured exactly by GRAPE-A unipotent actions in the general linear group GT. that preserve the same relative law and streaming cacheability. Importantly, forgetting mech- anisms are additive: the Forgetting Transformer (FoX) implements learnable per-head expo- nential decay in the attention logits and is a specific GRAPE-A / GRAPE-AP instance impos- ing distance-dependent attenuation (Lin et al., 2025). FoX's data-dependent forget gates yield path-additive bias D that we show is exactly the pendix C for constructive equivalence and its streaming implementation (Lin al., 2025).
GRAPE-AP case; see Ap-
Contextual position encoding. Content-adaptive position modulates effective phase or distance via token features through gating/scaling and algebraic parameterizations (Wu et al., 2020; Zheng al., 2024; Kogkalidis et al., 2024), and contextual counting (CoPE) (Golovneva et al., 2024). GRAPE introduces phase-modulated and dictionary-basse -based contextual variants that replace linear phase with cumulative ooken-adaptive phases (single multi-subsppace) while retaining exact headwise relativity and streaming caches. Finally, models can length-generalize without explicit encodings ("NoPE") under suitable training (Wang et al., 2024), which corresponds to the trivial generator = 0 in our view.
B APPLICATION IN MULTI-HEAD ATTENTION
Building upon the algebraic foundation for relative encoding established in Section 2.5, this section details the concrete integration of the rotational map G(n) into the full Multi- Attention (MHA) architecture, covering the per-head formulation, streaming policy, and implementation complexity.
Per-head formulation. Let H be the number of heads and d the per-head width. For head h E [H], let (qt,,, kg,,, Ve.h) E Rdd denote the query/key/value at position A GRAPE-M position map realized as orthogonal operator GR, (O(d) applied to (gr.A, k.,)):
4 G..t It,s,
kg,i = Gb,t ke,h,
V.A Vt.h-
The headwise attention logits and outputs are then
Kin
qT,x
YeA
Σ
softmax
(ettmax((e..)).,..) (Ce..h) Ji
(B.1)
Vd
Vd
with the usual output projection applied after concatenation across heads.
Exact relative law. G arises from one-parameter subgroup Gn(n)) = exp(n Ln) (commuting MS-GRAPE-M. including RoPE and learned commuting bases), then
q1 G. (i-)) K3
G Gh. = Gt-)tt)
b.j.A
Vd
so logits depend only on the offset j-t (exact origin invariance).
Streaming cache. Applying the rotational map G(I) independently each query and key vector is the core property that enables an efficient streaming cache policy. For any type where G, is known at token arrival (non-contextual and modse-mod cache Kj,AA GGi once and never rewrite it; at step f, compute qr.A GR, and use logits L,,j,h K. k,,k////
C FORGETTING TRANSFORMER AS A SPECIAL ADDITIVE GRAPE
The Forgetting Transformer (FoX) introduces a scalar forget gate f. (0, 1] per head and timestep and adds the cumulative log-gate as an additive bias in the attention logits. Concretely, for head h,
f. = o [W xe bynds
FIJ.
II Sen E=j+1
Dig, = log Fij.
log Se.
t=j+1
and the attention is
O = softmax
1 LKQT + +D
(FoX)
17

---

Published as a conference paper at ICLR 2026
We now show that Eq. (FoX) is exactly realized by GRAPE-A framework using the path-additive specialization of Section
FoX GRAPE-AP with andpoint-independer edges. In GRAPE-AP (Section 5), head-wise additive logit bn j) arises as a causal path sum
bn (t,j))
Σ (n(t,)). E-j+1
the edge potentials do not depend on the endpoint, i.e. (1, {) = 0gg then by (1, reduces to difference of per-time potentials:
(t,j)
Σ
GEP = Uth - UgAr
Unh == Σ agg
t=j+1
tcu
FoX corresponds to the choice athh = log fex I 0, yielding
(n, = Dis.h
Σ
log St,
(=
Thus, the FoX forgetting bias Dj is precisely the GRAPE-AP path-integra additive bias with endpoint-independerr edges.
Unipotent GL lift (GRAPE-A view). Let == e4+20 the rank-l nilpotent used in Sec- tion 4.2. For a fixed head h and endpoint I, define per-link unipotent factors
(C) I+ (t,
(t, log fê.
Since E² = 0, the path product collapses:
II
H0 = I+
Σ
log fe,h E = I+
f=++1
Ł=j+1
Scoring homogeneous coordinates as Section 4 with the paired trverse-transpose,
q. Kj,b
-TTAA (I Dijh Dij.n E) Kyn = qr,nkj, + Dij.h-
recovers Eq. (FoX) exactly (up the standard 1/va factor include throughout). Hence, FoX is an exact GRAPE-A GRAPE-AP instance realized by a rank-1 unipotent path with endpoint-independent edges.
Streaming and complexity. Compute prefix sums ULA = -Eer log Sen once per step; then DiJ.1 = UzA - Ug is obtained by subtraction, preserving the O(L) rowwise cost and the streaming cache policy from Section Secctio 5. The headwise gates fr, add O(1) parameters and negligible computation.
Special cases and composition. If fi.h - (constant per head), then Dij.A -1x(1-) and FoX reduces to exact ALiBi (Section 4.2). More generally, FoX composes additively with the multiplicative (orthogonal) GRAPE acting on (q, k) as in Eq. (5.3), preserving porm-preeervation of the rotational part while adding bounded, non-positive, content-adaptive path biases.
D NON-COMMUTING MULTIPLICATIVE GRAPE
Consider the thin compression = EL, with E d orthonormal and L, E so(r). Then o (L) = (L, (0)--r, af (cxp(nL)) a exp(nn,)) U (1)d-r.
If If = T T. is the real-Schur form, then the nontrivial eigenvalues are (// and tin for the exponential. Thus, the expressive power of non-contextual non-commuting MS- GRAPE is fully captured by the r/2 mode angles {0,); the ambient lifting via E preserves the spectrum.
18

---

Published as a conference paper at ICLR 2026
E COMPOSITION OF ADDITIVE GRAPE AND MULTIPLICATIVE GRAPE
For the unipotent forms of Additive GRAPE, applying the inverse transpose Gadd (m) -T requires only simple scalar-vector operations per active component. Thus, the per-head overhead is ( and negligible relative to attention matrix multiplications.
Multiplicative GRAPE (Section 3) and Additive GRAPE (Section 4) compose naturally, This can be viewed either additively at the logit level:
lij.h va 9r,6 Gn ()(j-t)k5, + (i-t) (Zg.t Ak.s).
Additive GRAPE bias
rigorously as single group action in the general linear group GL(d++).. Concretely, we construct the joint lift by direct sum. Let q = [q: 1; 0] and k = [k; 0; 1] be the augmented vectors. Define the joint generator L E gl(d+2) as the direct sum of the rotational generator L E so(d) and the gated nilpotent generator AA (where A >g + As captures the content modulation):
(x
0
0
Gjoin(() exp(mLL)
0
mwa 1
E (L((d+)).
0
This Block-diagonal structure unifies the mechanisms: the top-left block applies the norm- cem--reeeervng RoPE rotation to the features, while the bottom-right unipotent block generates the additive scalar bias via the auxiliary coordinates. Scoring with the paired inverse-transpose strictly preserves the exact relative law for the combined system:
q Gjoin(j-i)] kj = q. exp (U-i))) + (j-1) + const,
exactly reproducing the sum of multiplicative (rotary) and additive (slope) components. In this unified view, both mechanisms are simply projections of a single one-parameter subgroup action in high-dimensional space, retaining exact relativity and efficient streaming caches.
F COMPARISON WITH LIERE
Lie Rotational Position Encodings (LieRE) (Ostmeier al., 2025) encode positional information by learning skew-symmetric generator in SO(d). The method then applies the matrix exponential of this generator get rotational position map. For each attention head, the method learns one skew matrix. Its exponential gives dense orthogonal operator on queries and keys. Positions then match elements of parameerettrr subgroup on the rotation manifold. This picture a compact Lie theoretic version of RoPE style encodings. Different heads learn distinct rotational geometries, and the map keeps the norm and an exact relative position law.
Formally, for head h the generator is G₁, E so().. The positional map is x + exp(1w) Gr )z. A direct implementation has cost TIiRR((d = (dd²) per head for the matrix exponential and needs (dd)) parameters and the same order of memory.
Multiplicative GRAPE and LieRE both use rotations in SO(d) that come from skew-symmetric generators. LieRE gives each head dense block skew matrix. It forms the positional operator with the full matrix exponential exp(G). This creates very rich rotations but needs (d2) time for the exponential and ((d) parameters and memory per head. GRAPE-M restricts the generator to sum of rank 2 planes and uses a closed form Rodrigues-type formula for the exponential (Section 2). For one token, the positional mapping then reduces to few inner products and vector updates. So the cost is (dd) time and (dd) memory per head.
This choice of parametrization has two main effects in practice. First, the GRAPE-M scale cleanly translates to contextual versions where frequencies or phases depend on the token content. The closed-form expression can be computed quickly for each token, and there no large matrix expo- nential. In the LieRE setup, one needs a new dense matrix exponential for each content-dependent generator. This step is much more costly and makes such contextual use harder to deploy in real models. Second, GRAPE gives single group-theoretic picture for multiplicative and additive mech- anisms. The multiplicative part lives in SO(d) and additive or forgetting style terms (ALiBi, FoX, GRAPE-A, GRAPE-AP come from unipotent actions in GL with the same relative law and the same streaming cacheability (Sections 4-5). LieRE only targets rotational encodings and does not model additive logit biases or forgetting terms.
19

---

Published as a conference paper at ICLR 2026
G 2D AND 3D GRAPE FOR VISION AND MULTIMODAL POSITION ENCODING
Extending GRAPE beyond mne-dimensional token positions is easy. The construction only needs chosen group action on coordinates.
For images with integer pixel coordinates (u, u) E Z we pick two generators (x) and L(w) A token at (u, v) then gets the encoding
G2 v) exp( (w L()) exp exp(1 ww (w) E SO(d).
The two generators act on 2D planes that can be disjoint in the base design. In that case, the map reduces to RoPE-style separable encoding. A learned choice of planes inside Rd gives the GRAPE- M variant again.
For 3D coordinates (u, v, w) that mark video space time tokens or point clouds, we follow the same pattern. We introduce three commuting generators and define
G3 u w) = exp( u w,w.L. ( exp(uw, L ()) exp( w W.wL (*))
In the non-commuting case, we use the thin Schur mode compression from Appendix D. The closed- form rank 2 matrix exponential from the main text still applies. The per token cost stays O even for higher-dimensional coordinate spaces.
On the additive side, GRAPE-A and GRAPE-AP handle 2D or 3D structures through the scalar offset m. The value m can be any function of coordinate differences. For an image, we can take
a U4 Wj) + (xy( U5):
and this keeps the same algebraic template. For 3D settings,
set
m
ryyl
with r, and r, in R The update matrix then stays unipotent, and the exact relative composition law still holds. This gives clear way to impose axis-aligned radial recency bias in vision and multimodal models.
H ALGORITHMIC DETAILS AND PSEUDO CODE
This appendix contains the detailed pseudocode.
Algorithm Commuting Multi-Subspace GRAPE-M
Require: Q K E BBxxxxxxd orthogonal E E Rdxd, frequencies {wh.s d/2 positions n E g
1: for h = to H do
2: Q'(:,;,,, h,
Q h E; K(A,), K h;
3: for [ 0 to
do for j 1 to do
4:
5:
0 + ne Wh.j apply 2 x 2 rotation Gz( to coords (2j-1,2j) of Q'i:, h, and
K'[:, h
6:
end for end for
7:
8:
: + KEAAI-KE K'E
9: end for
10: return @Q.
I DIFFERENTIATION AND FAST APPLICATION OF RANK-2 MATRIX EXPONENTIAL
Differentiation and stability. Let f1(z) and fz(2) 1-cos with nws. Then
exp(nwL) = I [+ f2(=)L??
20

---

Published as a conference paper at ICLR 2026
For any scalar parameter 0 {w} U fentries of a,
dg eXp(nwLL) = (=) 0,L + f2(2) (L 0gL + + +g (f(2)+++++ 0g2 = no 03s + nLS Ogw.
a=s= = 2 0 7²).
2/2 0(*1). These formulas Use series for |a| <: (i(=) 1 -े + ((=1) and f2(=)
enable siixed-reccsioon backprop with small-s guards.
Fast application. For any Rd
Lx ab, b(a, x):
L >x(, x) b(a, x)) aa x) b(b
Thus G(n)x= x Li + f2Lxx with sin(nws) and f2 [_ which is evaluable in
((d) time via few inner products. By the minimal polynomial x LL expanding exp(1)L and regrouping yields the rank-2 update form used throughout
J SPECTRAL ANALYSIS OF GRAPE AND OTHER METHODS
In this section, we discuss eigenvalue-level results for GRAPE-M generatoons/expoeeniias and sum- marize the unipotent spectra of GRAPE-A/GRAPE-AL Throughout, L(a, b) ab -ba" so(d), and a = ||a|| 8 [[b]| 7 = a b, A a3 7 8 VA as in Section 2.
J.1 RANK-2 PLANE:: EXACT SPECTRUM AND GEOMETRIC INTERPRETATION
Lemma J.1 (Rank-2 spectrum). For L (a, b). the eigenvalues are {=is) U (0)d-?, and there exists B E (O(d) such that
sJ 0
L
J (i -.).
0
O4-2
Moreover, 8 = ||a|||]|| sin e, where Φb E [0, m] the angle between a and b.
Proof. From Section 2, L P with U spanfa, b), whenee the minimal polynomial is X(X² + =²) and (UU) = { 0}. Choosing an orthonormal basis aligned with 2 @ U- yields the claimed form. Finally, A a3
[b]| cos² Φ) [|||||[||| sin x))².
Corollary J.2 (Phase bounds and orthogonality). The per-step rotation angle of exp(1IL) on U equals = r7s and satisfies < < 17 r/||a|||bl| with equality when a 1 If = Ja (Section 2.4)
and ||||| = then and 0 77.
Exponential spectrum. For any n E Z.
of (exp(n))) tittn U (1)2-
Hence p(exp(nL)) = 1, the map is unitary (orthogonal), and all Lyapunov exponents are zero Periodicity holds with fundamental period T 2n when 8/T Q; otherwise, the trajectory is quasi-periodic on the unit circle.
J.2 MULTI-SURSPACE GRAPE-M AND ROPE
Let L E,L0, with mutually orthogonal planes (hence [L, L,] for i + j) and L, U.JUJ. Then
T7 B
L
J @ Od-2m-
oLL) { d0m
j
for some B (O(d). Consequently,
a (exp(nnL) {e ربm U (1)d-2m
This recovers RoPE when the planes are the coordinate pairs and {0,} follow the canonical log- uniform spectrum (Proposition 3.1).
21

---

Published as a conference paper at ICLR 2026
J.3 ADDTTIVE GRAPE
We now analyze the spectral properties of the additive lifts in GL introduced in Sections and 5. The key structural fact unipotency: all per-step factors are identity plus rank-1 (or few-rank) nilpotent update of index 2.
Setup. Let A gl(d+1) (or gl(d++) for ALiBi) satisfy A 0 as Eq. (4.1) and Eq. (4.7). For scalar path parameter R. define the unipotent factor
(() (sA I+
H(s) H(s)- -I- A,
det H(s)
For Additive GRAPE(GRAPE-A) with offset m = j-i, = m for GRAPE-PA. s = (t, = Eej44 Vh, E) from Eq. (5.2).
Proposition J.3 (Eigenvalues and Jordan structure of additive lifts). Let A E gl(D) satisfy A and A * 0. Then for every # O,
a (H(s)) = (1),
(H(s) - 1)2 112==0 0, det H (1)=1,
11 p (H(()) =
Hence, the minimal polynomial of H(s) is (A - 1 and the Jordan form consists of size-2 Jordan blocks for the 1-eigenspace, with the number of nontrivial blocks equal rank(A).
Proof. Since A O, exp(sA) + AA and (H(s) 1)a A² = 0. The characteristic polynomial is (A - 1)) for Hss), so all eigenvalues equal 1. The determinant equals the product of cigenvalues, hence 1; the spectral radius is therefore
Dictionary closure. If {A,}R satisfy A = O and A,A, = 0 for all r, s, then
= O.A. Σ -.
so the combined generator is also index-2 nilpotent and yields the same unipotent spectrum.
Singular values. Although Hs)) not orthogonal, its deviation from is rank-limited and exactly analyzable. We first give sharp, explicit formula for the canonical rank-1 case (ALiBi block), then a general bound.
Lemma J.4 (Exact singular-value pair for canonical rank-1 unipotent). Let E := ep eg with * g and define H(s) := I +E E DxD, Then D - 2 singular values equal 1, and the remaining two are
a+ (H(s) V 1+ of + 1+ (H()))) 1.
(J.1)
In particular,
(((s) 0+ 0 (H(s) a (H()))² 1+ o + 1 o 1 |s|+ as + 0.
Proof. The action of H(s) (()) is identity on span } In the basis {cg. Cp} it equals (1+s2x) 1), whose eigenvalues are + 2 + |s| |s|v 1 1. Taking square roots yields (J.1). The product equals / (e ( H det
Corollary J.5 (ALiBi and Additive GRAPEIGRAPE-AI conditioning numbers). For the exact AL- iBi gineraatoor in Eq. (4.7), let E 2 so that A B.E.. Then G I+ mA I-m 3g E = I+ s E with For the vector additive lift Eq. (4.1) with A (or uaii and ||uainn|| 1, the same for- mula holds with E replaced by orthogonally similar rank-1 update and
m 3h. and the only nontrivial singular values follow from Eq. (J.1).
22

---

Published as a conference paper at ICLR 2026
Lemma J.6 (General operator- norm bounds for index-2 unipotents). For any A with A 0 and any s E I, s | I minn (I+ + sA) S max (I+ sA) < 1 + ||| ||
These bounds are conservative but dimension-free. In the canonical rank-1 1 case of Lemma J.4 with || one has the sharper small-/s] behavior m (I sA) 1 + 1 Os²) and + Oss)).
Tmin (I + =A) =
Proof. Use the triangle inequality II(I + sA)x|12 S ||||| + |s] ||/|||| and its reverse form applied to (I + sA) = sA; also Weyl inequalities for singular values under rank-1 perturbations.
Cancellation in the relative logit. While H(s) can be anisotropic (Lemma J.4), the Additive GRAPE(GRAPE-A) scoring uses paired anverse-ranssoo (Eq. (4.2)), which cancels all multi- plicative distortions and yields pure additive term:
q: kj = q. iI+ (I [I-jwAY [I-jwA]),,
(I- (1- = ब. Gadd (j -k,,,
since (AT)² O. This reproduces the exact relative law Eq. (4.3) and the closed form Eq. (4.5) (e.g. Eq. (4.6),, independently (s)).
GRAPE-AP as path-integral unipotent. Fix head h and endpoint l. The per-row path product
in Section 5 is
II (I- I- Σ va(l, = I (n(1,) e-j+1
because E² = 0. Thus GRAPE-AP inherits the unipotent spectrum of Prop. J.3 with row-dependent = sh (t. & (since uh I O by construction). Its only two nontrivial singular values are exactly (J.1) with (); the rest equal 1. Consequently,
0+ (1,())))
Ki (PA factor)
(+(--s(t,)))
0
8t (1,)))
=1 (n(t,)) 2 + st (t, =a(l,f)² 1+ (sx(t)) + (ts
while the determinant remains and eigenvalues are 1. As in Additive GRAPE(GRAPE-A), the paired inverse-transposs transpose used in the bilinear scoring removes any multiplicative anisotropy. leaving the bounded additive term by. () in Eq. (5.2).
Implications. Now summarize the implications of previous results. For all H(s) is invertible with H(s) I sA; eigenvalues do not grow with offset length (spectral radius 1). The operator grows most linearly in |s| (Lemma J.6) and is exactly characterized in the rank-1 canonical cases (Lemma J.4).
Secondly, det (()) 1 implies net volume change; any expansion along direction is actly balanced by contraction along its paired direction (product a = 1). Despite anisotropy. the GRAPE-A and GRAPE-AP logits remain exactly relative because the key transform H(s)-T, algebraically climinating multiplicative distortion and yielding the closed-form additive bias (Eqs. (4.3), (4.5), (5.2)).
J.4 COMPARISON TO PATH ATTENTION
PaTH Attention (Yang et al., 2025b) proposes contextual multiplicative position map given by cumulative product of dentity-plus-rankee matrices
H Bg w.w." |well 1, 33 (0,2),
applied along the path between key position j and query position as I[ H. (see Section 2 of the PaTH paper). In contrast to GRAPE-M factors, each H, not orthogonal unless 3 E (0,2). This has immediate spectral consequences.
23

---

Published as a conference paper at ICLR 2026
Per-step spectrum. Since H, is symmetric rank-1 perturbation of the identity with projector Pa W.W,
o (H.) {1 Sc, 1,...,1h b
det(H, Be.
H. ll max 3e1 1.
Thus Hr is norm nonexpansive (operator norm 1) but not norm-preserving unless 3r E {0, 2}. Singular values equal the absolute eigenvalues because H, is symmetric; the component along Wr is scaled by |1 - Pel < for any 8 E (0, 2) \ {0, 2}. and flips sign when 3n (a design choice in PaTH allow negative eigenvalues for tracking))..
Path product contractive and near-singular. Let P II j+1 H, Submultipicaatiityy of singular values gives
mmax (P)+e)
H. 12 =2
mmin (P)t)
min (H
Bal.
8=)+1
=++
#=f+1
Hence Py-+ is (at best) nonexpansive, with worst-case exponential lower bound on the smallest singular value governed by the path-length product of B,). Whenever some (3 is close 1, H. is nearly singular (and exactly singular if 3, = 1), driving min (P toward zero. Volume contraction is quantified by
det(P
II B2),
s=j+1
which typically decays exponentially in i j unless 3, concentrates at the orthogonal endpoints {0, 2).
Aligned-plane special case. If the directions are time-invariant, W. = w, then P. ww is an idempotent projector and the factors commute:
II H. s=f+1
II
B.P) =1 -1 (1-
(1 (1)
s=f+1
s=j+1
so the eigenvalue along w is exactly IT:=/41(1 3.), making the contraction along w explicit and exponential in path length unless 3, E (0,2).
Implications for long-context modeling. Because the PaTH transport multiplies the Q/K bilinear by Pj-+i. any persistent deviation of 3e from {0, yields cumulative energy loss along a moving one-dimensional subspace. This concentrates mass in progressively fewer directions and can flatten or attenuate long-range logits q, Pj-skk as forget-gates are introduced. contrast, GRAPE-M maps lie in SO(d). so for both non-contextual and contextual types. all singular values are 1; volumes and norms are preserved, and Lyapunov exponents are 0, avoiding contraction-inducee degradation of long-range interactions.
grows, unless additional renormalizations or
Lemma J.7 (Orthogonality condition for РaTH factors). For H, I 3 w.,. with ||w:]] 1. H, is orthogonal iff 8 E 20,2}. For 3 (0,2) {0, 2}. H, is symmetric, diagonalizable with cigenvalues (-1, {1}. and strictly contractive on span (we).
24


Reviews:
[{"review_id": "mYK77N3gfx", "reviewer": "Reviewer_fTQk", "summary": "The paper proposes GRAPE, a unified group-theoretic framework for positional encoding in Transformers. It combines Multiplicative GRAPE (rotations in SO(d), generalizing RoPE) and Additive GRAPE (unipotent actions in GL, recovering ALiBi and FoX). GRAPE preserves exact relative relationships, supports streaming, and offers an extensible design space for long-context modeling.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper presents an elegant theoretical unification of multiplicative and additive positional mechanisms within a single group-theoretic framework. It offers closed-form and computationally efficient implementations, demonstrates strong compatibility with existing Transformer architectures, and provides extensibility toward contextual, learned-basis, and non-commuting variants for more expressive positional representations.", "weaknesses": "This paper utilizes Lie algebras. While unifying existing work with Lie algebras is natural, its drawback is that it makes the paper's contribution seem more like the superiority of Lie algebras themselves rather than the authors' contribution. I believe the authors should emphasize more on how introducing Lie algebras facilitates combining the strengths of various existing methods, explaining why each strength is beneficial, and then supplementing with corresponding ablation experiments.\n\nThe experiments in this paper are somewhat limited, lacking extrapolation experiments and comparisons with more metrics. I suggest at least supplementing with the experiments in Tab. 4 of RoPE.\n\nDue to the highly complex formulas, I cannot guarantee my complete understanding of this paper. The pseudocode in the appendix does not alleviate my concerns about reproducibility; I would appreciate to see the complete project code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F0BCGcsCSA", "reviewer": "Reviewer_dFym", "summary": "The paper proposes a framework for positional encoding based on group actions, dubbed as GRAPE (Group RepresentAtional Position Encoding). Specifically, the authors describe two families of positional encodings grounded in group actions: (1) Multiplicative\nGRAPE based in multiplicative rotations; (2) Additive GRAPE from unipotent actions in the general linear group. The authors describe how existing positional encodings such as RoPE, ALiBi, and FoX can be recovered as special cases within this framework. The authors also provide some empirical evidence supporting the advantages of GRAPE over existing positional encodings.", "questions": "1. Can the authors compare their Mul-GRAPE with the recently proposed LieRE in [1], which parameterizes the rotation as a sum of skew-symmetric matrices (followed by matrix exponential)? LieRE seems to provide a more general parameterization, so I am curious to see if this results in any computational or performance differences.\n\n2. In Prop 3.1: the equality of MS-GRAPE and ROPE only holds when the planes are the canonical coordinates pairs and the angles follow the log-uniform spectrum, right? If so, I suggest to make the statement more precise.\n\n3. The authors introduces GRAPE as a way to provide a group-theoretic view of PEs. Does GRAPE provide additional insights of the existing PEs, such as which PE one should choose over another given certain tasks in mind (e.g., length extrapolation)? \n\n References\n [1] Ostmeier et al., LieRE: Lie Rotational Positional Encodings, ICML 2025", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper grounds the design space of positional encoding (PE) in group actions, arriving at a general framework which can possibly motivate more expressive and useful PEs.", "weaknesses": "1. The message of the paper is confusing. For Multiplicative GRAPE, the authors stated the exact relative law in Section 2.2 which naturally leads to commuting Mul-GRAPE, but then describe non-commuting Mul-GRAPE in multiple places (e.g. abstract, related work, appendix) without any motivations.\n\n2. The paper devotes section 3 and 4 for describing Multiplicative GRAPE, but does not use it in the empirical experiment. This casts doubts on the practical utility of Multiplicative GRAPE.\n\n3. The empirical experiments are quite limited. The authors compare PI-Add-GRAPE with other baseline PEs only on their loss curves, without other metrics (e.g., perplexity) or downstream task performance, or ablations (e.g., context length, model size).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJod8GcscH", "reviewer": "Reviewer_MUmm", "summary": "This paper proposes GRAPE, a unified theoretical framework for positional encoding based on group theory. The authors categorize existing methods into two families: Multiplicative GRAPE (rotations, like RoPE) and Additive GRAPE (biases, like ALiBi). The paper claims that RoPE, ALiBi, and FoX are all exact special cases or instances of this framework. The authors then propose a new, endpoint-dependent variant called Path-Integral Additive GRAPE (PI-Add-GRAPE). In a minimal experiment, this new method is shown to achieve lower (or possibly comparable) loss than baselines on a language modeling task.", "questions": "Can you clarify the computational overhead of PI-Add-GRAPE during training and inference?\n\nGiven the complexity of PI-Add-GRAPE and validation of RoPE's unstable result, do you plan to release a reference implementation? This would be crucial for reproducibility and adoption by the community.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper provides a somehow novel viewpoint to design of positional encoding. The goal of unifying the two dominant (and seemingly different) positional encoding methods (rotations and biases) under a single mathematical framework is could be ambitious and interesting.\n\nThe proposed PI-Add-GRAPE mechanism, which introduces content-dependent biases, may be a novel concept. In theory, this dynamic approach could offer more expressive power than static position methods.", "weaknesses": "Unclear Practical Benefit of the Theory: The paper spends significant effort on group theory formalism. However, the practical benefit of this complex formalization is unclear. It seems obvious that rotational embeddings like RoPE can be described by group theory (e.g., SO(2)). The paper does not clearly explain what new, practical advantages this complex theory provides over a simpler understanding. The claim of offering a \"design space\" is abstract and its benefit is not well-supported.\n\nInsufficient Experimental Validation: The empirical evaluation is minimal and insufficient to support the paper's claims. It consists of a single set of training curves for one model configuration. The claim of a \"persistent edge\" also appears to be an overclaim; the validation loss for ALiBi looks very competitive with PI-Add-GRAPE.\nFurthermore, the paper is missing empirical analyses to understand the proposed method. For example, there are no ablation studies, no length extrapolation tests (a key feature of ALiBi), and no analysis of attention distributions to show how the dynamic bias works. Section 7 feels aimless; it shows a result but provides no insight into why the method is good or what its specific advantages are.\n\nComputational Cost: The PI-Add-GRAPE method (Section 6) is endpoint-dependent. This implies that during inference at step t, the bias for all t−1 previous keys must be recomputed relative to the current query, right? This likely introduces a significant O(t) computational overhead per step, which is a major drawback compared to the O(d) cost of RoPE or ALiBi. This trade-off is not benchmarked empirically.\n\nLogical Gaps and Confusing Terminology: The logical connection in the introduction (from \"These observations\" to \"motivate a unified formulation\" ) is a significant jump and is not well-justified for me. In addition, there is some confusing expressions (e.g., the interchangeable use of \"exact special case\" and \"exact instance\" is confusing).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVRRVzJ60D", "reviewer": "Reviewer_TbLJ", "summary": "This paper proposes GRAPE, a unified group-theoretic framework for positional encoding for 1D sequences that combines multiplicative rotations in SO(d) and additive unipotent actions in GL to recover and generalize methods like Rotary Position Embedding (RoPE), Attention with Linear Biases (ALiBi), and Forgetting Transformer (FoX).\nUnlike prior work that focuses only on rotations, it formalizes both multiplicative and additive mechanisms under a single algebraic structure and introduces path-integral additive biases for contextual, streaming-friendly position encoding.\nExperiments on FineWeb-Edu 100B with Llama show slightly improved stability and lower loss compared to RoPE, ALiBi, and FoX.", "questions": "- Is the training fully converged? Would you mind running for 50 more epochs? \n- You emphasize the importance of exact relativity and orthogonality for translation invariance in positional encodings. Could you comment on what a strict structure is enables?\n- What is the motivation for combining multiplicative and additive logit biases in your positional encoding design? Do they serve complementary roles (e.g., scaling vs shifting positional effects), and how does this impact learning stability or expressivity?\n\nMinor:\n- The abstract has undefined variables ($a$ and $b$ ... ) which are only later defined", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The generator construction L is smart and novel. \n- The combination of both multiplicative and additive mechanisms under a single algebraic structure is a contributing perspective for this field. \n- The formal description of GRAPE is complete. \n- The method is efficient and is a direct extension of Rope for 1D sequences.", "weaknesses": "The motivation of this work could be better described. Which practical problem does GRAPE solve? What can GRAPE encode what Rope and other variants cannot encode and why is that important in practice? \n- The comparison to prior works and the related work section is incomplete.\n-- For the multiplicative GRAPE, there are several prior works like i.e. LieRE (Ostmeier et al.), STRING (Schenck et al.) that have conceptually predescribed and evaluated Lie Group structured positional encodings, where rank-2 exponentials in SO(d) and learned basis. How does GRAPE compare to just learning the 2x2 basis generators for the block diagonal rotation matrix?\n-- How does GRAPE compare to YARN (Peng et. al)?\n-- How does GRAPE compare to CoPE? \n- The paper aims to present a unified framework for positional encoding based on group actions for transformer in general, but only focuses on 1D sequence encoding and not higher dimensional inputs.\n- Although it is valuable to present the learning curves, the experimental results could be better presented i.e. confidence intervals, at least two validation sets or a test set. \n- Ablations between multiplicative and additive would enhance the understanding of the practical contributions of each of them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)