Composing Linear Layers from Irreducibles
Authors: Travis Pence
Daisuke Yamada
Vikas Singh
Venue: NeurIPS 2025 poster ()
Keywords: Clifford Algebra
Compositions
Modular
Primary area: general_machine_learning
TL;DR: We show that linear layers, the primary player in large language models, can be composed from fundamental algebraic primitives with exponentially less parameters.

Abstract: Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. 
We investigate this compositional structure in linear layers by asking: \textit{can we identify/synthesize linear transformations from a minimal set of geometric primitives?} 
Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors---geometric objects encoding oriented planes---and introduce a differentiable algorithm that decomposes them into products of rotors. 
This construction uses only $\mathcal{O}(\log^2 d)$ parameters, versus $\mathcal{O}(d^2)$ required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.

Composing Linear Layers from Irreducibles
Travis Pence Daisuke Yamada
Vikas Singh
University of Wisconsin-Madison (tnpence, (aaaada2)@wiscc.. edu, veingh@biostat. wisc.
Abstract
Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality. but these fundamental building blocks remain poorly understood. We investigate this compo- sitional structure in linear layers by asking: can we identif//ynthesize linear trans- formations from a minimal set of geometric primitives? Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors geemeerric objects encoding oriented planes-and and introduce a differentiable algorithm that decomposes them into products of rotors. This construction uses only O (log" d) parameters, versus (Od²) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, rotor- based layers match the perfor- mance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.
Introduction
There is growing consensus Kozachkov et al. 2023 that, like biological systems, modern models may internally rely on low-level primitives that compose to form modules with more complex functionality (Weiss et al. [2021]. This compositional perspective
motivation behind capsule networks [Sabour et al. 2017]. which explicitly modeled part-whole relationships through vector- based capsules. But the task of localizing and characterizing such primitives remains challenging albeit interesting. A general capability to compose pre-trained modules in prescribed way could, in principle, lead larger model with predictable and more controllable functionality [Zou et al. 2025, (Schug et al.) 2024 Ghazi et al. 2019 Abnar et al. 2023 Press et al., [023]] with potential applications ranging from safety guardrails to interpretability.
Some recent results suggest some progress in this direction. In mixture-of-exppers architectures [Masoudnia and Ebrahimpour 2014 Riquelme et al., [021]. specialized sub-networks are condi- tionally activated and composed via routing [Büchel et al.) 2025). Model merging has evolved from simple parameter averaging more sophisticated alignment of different model latent representations
[Lähner and Moeller 2024]. Fine-tuning methods like LoRA (Hu et al.] [202T] implicitly assume that low-dimensional adjustment of a base network should suffice essentially a two-level composition. Each approach offers distinct perspective on composition (e.g., see Chytas et al. 2024) but are not focused on addressing how the mechanistic composition low-level primitives gives more complex behavior. To this end, mechanistic interpretability [Rai et al. 2024] and neurosymbolic methods Yang and Chaudhuri
2022 explore this space, but are still in a nascent stage of development.
Scope of this paper. Consider core module- millions of parameters-inn large model. Can we synthesize its functionality from its most basic primitives? How many such objects would we need? This casts our broader interest composition into a concrete problem: identifying minimal set of irreducibles that combine in specific ways to realize the full functionality of the module. We study this problem for linear layers-notine that linear layers make up a large portion of parameters in large
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

language models (LLMs). While prior work suggests various rrrameter-efficien approximations, our interest is not only function approximation, rather to build up the functionality by characterizing its algebraic structure. To formalize the above intuition, we use the language of Clifford algebra, where linear transformations naturally decompose into simple bivectors- geometric objects representing oriented planes. This view reveals how the functionality of a linear layer can be synthesized as structured composition of a few hundred geometric objects (parameters).
Our key contributions are: (a) We express linear transformations compositions of geometric primitiees-specficcllly specifically, bivectors in Clifford algebra-usinn rotor sandwich products acting on local subspaces of an input multivector. This requires O (log² d) scalar parameters, compared to (d²) for dense layers, where d is the input/output dimension. (b) We propose a differentiable invariant decomposition algorithm that maps bivectors to their corresponding rotors closed-form, which enables integration with autograd and gradient-based optimization. (c) Empirically, we replace the key, query, and value projections in LLM attention layers and show comparable downstream performance in accuracy and perplexity across various datasets.
Our goal is to show the feasibility of this algebraic decomposition approach. It is not a drop-in replacement yet, since practical benefits will require additional system-level integration beyond the scope of this work. The focus here is on the underlying algorithmic and mathematical foundations.
2 Preliminaries
We review some relevant concepts from Clifford algebra (also see Hestenes and Sobezyk (2012).
Clifford algebra. A Clifford algebra, Clp.o (R), is an associative algebra over R" equipped with quadratic form of signature (p. g). where = p q. The algebra admits two basic products: the inner product and outer (wedge) product, denoted by and A respectively. Their sum defines the geometric product. which for vectors u, v R takes the form:
wv - u. v + uu C.
The algebra is generated by orthogonal basis vectors 61,
Cn. called generators, satisfying:
= for 1 sis 25 ef =-1 for p < iiSm < es Aej Aa===== ====-0 -e Aei for i + j.
These relations encode the metric and orientation of the underlying space. The algebra has a canonical basis of 2" elements: the scalar and all distinct products of the basis vectors E Ex. In particular, basis bivectors are wedge products of two distinct basis vectors, ie.. e; cgey for j. More generally, basis k-vectors are wedge products of i distinct basis vectors (see Fig. D and there are (w) such elements for each k. The total number of such basis elements is the dimension of the algebra. The reversion, given by
Tea Co A
eress
i, reverses the order of basis vectors. For example, (e1e2) e261 and ( = E362E1.
ej A
The algebra decomposes into a direct sum of subspaces indexed by grade: scalars (grade 0), vectors (grade 1), bivectors (grade 2), and general k-vectors, which are linear combinations of corresponding basis elements. A multivector is general element of Clp (IR), expressed as a linear combination of components of multiple grades. For example, ej 22 is basis bivector; c,63 + 222e3 is a bivector; and + (2eg + E4) - eré3 is a multivector composed of elements of
e1 6g
grades 0, 1, and 2. We denote the subspaces of vectors and bivectors by CIR(RR and C12 (IR), and more generally, C1 (R) for grade-k elements. The algebra naturally splits into even and odd subalgebras based on grade parity. The even subalgebra (n) C Cl(n) consists of elements of even grade (scalars, bivectors, 4-vectors, etc), while odd elements include vectors, trivectors, and so on. Some well-known algebraic systems arise as special cases of Clifford algebras: the real numbers R (IR), the complex numbers C E Clo, the quaternions HI & (R)) and the hyperbolic numbers Cl CRR). This way, Clifford algebras naturally generalize familiar algebraic systems by incorporating geometric structure. To keep notations short, we write Clp.(( Cl(p, q) and only consider Cl(n, 0), denoted as Cl(n).
Figure 1: bivectors, and trivector for CI(3)
The basis vectors,

---

objects that encode both magnitude 3 Algebraic Structure of Rotor-based Transformations
We aim to describe standard linear transformations in terms of the algebraic structure of Clifford algebra. We begin by noting how any linear map can be expressed sum of multivector products, and then show how restricting to rotors in the Spin group helpful.
Clifford form of linear transformations. A linear map between two vector spaces is one that preserves additivity and homogeneity. Traditionally, such transformations are represented as dense matrices with independent parameters. In contrast, consider Cl(n) and write these transformations in terms of the geometric product between multivectors-algebee and orientation. We first restate textbook result.
Lemma 1. (Hestenes and Sobezyk (2022) Let a, and by denote multivectors in Cl(n). Any linear function F from CI (n) to Cl(n) can be written as the finite sum for some width w < co,
F(x) Σ ajab
(1)
This result shows that linear transformations Clifford algebra can expressed as products involving multivectors acting from both the left and right. Thus, Clifford algebra gives way to represent general, arbitrary linear map. But there is a cost: arbitrary multivectors have too much freedom and require all 2" parameters of the full Clifford algebra, which makes the representation inefficient. We will constrain a, and b, to preserve rotational symmetries, leading to the Spin group
Spin(n) _ {r CI+(n) | rrl and rort e CI for all v E CI'
where CI+(m) (m) c Cl(n) is the even subalgebra and f denotes wise reversion. The Spin group captures the set of ceenttation-prserving rotations within Cl(n). The elements E Spin(n), called rotors, act on multivectors x Cl(n) through the sandwich product as shown Fig. 2
T33
rrrl
rotor
exp
h rzr
(2)
c1 A C2
Applying multiple rotors in parallel instantiates Lem. where a, = Tt and b, = r!. We clarify that while the general Clifford algebra represent any linear map, restricting to Spin(n) Figure 2: The sandwich product rotat- ones. ing a vector 60' in the ej A e2 plane.
C1
limits transformations to orthogonal contation-preseviin Consequently, our construction does not capture arbitrary linear maps. In practice, expressivity is recovered by combining multiple rotor modules acting on different subspaces and aggregating their outputs (see Sec. 5). While this sandwich form is useful, we must efficiently parametrize these rotors. To do so, we first observe the relationship between Spin(n) and the more familiar rotation group, (O(n).
Fact 1 GGalleer and Quaintance 2020 ). Spin(n) is double cover of the special orthogonal group SO(n) for 2 3.
This relationship is important in that while Spin(n) and SO(n) are topologically different (hence the "double cover"), they share the same infinitesimal structure namely. their Lie algebra. This allows using parametrization techniques for SO(n) to represent elements of (pin(n). In particular, we will see that rotors can be generated from bivectors via the exponential map. just as rotation matrices arise from kkew-symmetrie matrices.
Remark. For vector inputs x, the sandwich product rxrt performs the same rotation as the corre- sponding matrix in SO(n). However, the full utility of rotors becomes clear when x is multivector: the transformation extends to higher-grade components within the Clifford algebra, going beyond the vector subspace. This makes rotors especially suitable for operating on the richer representations.
We return to the question: can we efficiently represent rotors so that ar b in Lem. I belong to Spin(n)?
Constructing rotors from bivectors. Rotors in Spin(n) can be parametrized via bivectors: grade- 2 elements of Cl(n) that encode oriented planes of rotation. To understand this parametrization, we examine the connection between rotors and rotation matrices via Lie groups and their Lie algebras.
3

---

Definition 1. A Lie group G is smooth manifold with the usual group properties along with smooth (infinitely differentiable) group operations. Its associated Lie algebra vector space equipped with an antisymmetric, bilinear operation [X, YY,Y, called the Lie bracket, satisfying the Jacobi identity.
Let us check examples. Spin(n) and (O(n) are different Lie groups with the same Lie algebra. Specifically, the Lie algebra of skew-symmetric matrices, so(n) _ { B TR=x"| -B") } un- derlies both Spin(n) and SO(n). de- spite their topological differences. This shared structure is important. The ponential map gives surjective corre- spondence between so(n) to SO(n) as
(12(
Spriny((1)
Space of Diiecttors
Spuce of Simple Divectors
Spuce Rators
CtPin)
inveriant
(TE(N)
closad--orm geumesric exponential
Spinyin)
gromeine product
Spanis)
decompusition
Most
Space of
Sulsspcces
(H)
Sprin,
Rators
B
(3) Figure 3: The [bivector + invariant decomposition + rotor de- composition rotor] process that enables exact parametrization. Note that a pure rotor is one that corresponds to simple bivector. showing that every rotation in SO(n) can be realized as exp(B) for some B E so(n) using only dim(so(n)) = ( independent parameters- than the n² entries of matrix [Lezcano-Cass
exp(B) B)
Σ 1=0
i!
and Martinez-Rubio 2019]. The bridge the Clifford algebra setting from another key isomor-
phism:
Fact 2 (Doran and Lasenby 2003)). ). The spuce of skew-symmetric matrices is isomorphic to that of bivectors, ie., so(n) = (n).
Given a skew-symmetric matrix B E (n). the corresponding bivector b E Cf is constructed as Bi, c, Aej.
Isicjen
Similar to the exponential map relating (n and SO(n) (i.e., exp( (B generating rotations in SO(n)), every rotor r Spin(n) is the exponential of some bivector b E CI²(n) given explicitly by the series
2
r r= exp(b)
2
where 6* is the k-fold geometric product of b with itself. This means that we can encode rotor with only dim (C)( = =() parameters-the dimension of the bivector space.
Main advantage. At first glance, our parametrization may seem to offer no advantage. We have parametrized both Spin(n) and SO(n) from the same (2) parameters from so(n). Notice that while SO(n) acts only on ne-dimensional vectors, rotors in Spin(n) act on subset of the full 2 dimensional space of multivectors. By utilizing rotors as the action on multivector imput and bivectors as our irreducible primitives, we will use exponentially fewer parameters.
Example 3.1. A x d dense matrix with d = 2048, common in self-attention blocks and projection layers (e.g.. in LLaMa), uses more than 4M parameters. If w (e.g., 22 3) is the width hyperparameter in Lem. 1 approximating with bivector irreducibles requires only (l0gg d) 55 w.
This reduction (to 55 x 3) comes from identifying bivectors as the primitive that generate rich linear maps through composition. Note that the infinite series in (4) problematic due to the need for approximation. We avoid this by presenting closed-form solution that preserves differentiability.
4 Algorithmic Implementation and Analysis of our Rotor-gadget
We discussed above how rotors can be parametrized through bivectors via the exponential map in 4 remaining challenge is the infinite series. Of course, we can truncate to a finite length and ineur approximation errors. However, for special class of bivectors, the exponential map admits an exact closed-form solution, which prevents any approximation errors. Moreover, this form remains fully differentiable. We describe the details of this alternative here.

---

Algorithm Differentiable Inv. Decomp.
Algorithm 2 GA Power Iteration
Require: b g (n), E CI (m)4=1
Require: b E (l))), (n).
Ensure: Inv. Decomp. {bi.... b.).
threshold é
singular vectors (Ui,..., Uk-1
Ensure: Approximate Proj simple (b)
1: Initialize decomp, vectors +- 0,0
Upra <
2: fori= = to k do
2: while ||v Upre > с do
3: 0,, f- simple (b, v4)
3: Upres <-
4: b-
4:
(b.v)
5: decomp + decomp U (b.}
V U 2/ ///l//|z
6: vectors +- vectors U {v;)
6: end while
7: end for
7: du bLU
8: decomp <- decomp U (b)
8: bg f- au A
return decomp, vectors
9: return b,, V
4.1 Closed-form differentiable computations of rotors
key observation is that when a skew-symmetric matrix E so(n) generates a rotation restricted to single 2-dimensional plane, the matrix exponential in (3) reduces to a finite closed-form expression- mirroring the simplicity of the classic Rodrigues formula for axis-angle rotations [Goldstein et al.) 2002]. An analogous simplification holds for bivectors. The exponential map in 44) admits closed form when b (I²()) is simple, meaning it can be written as b A for some vectors u E (n), equivalently b A b 0. This ensures that b represents single-plane rotation rather than composition of rotations. The resulting dlosed-form expression is:
exp(b) cos([]6]]) sin((1b1)) ||a||
This form is exact [Doran and Lasenby|
However, restricting simple bivectors is limiting, as simple bivectors span only a subset of (n), and thus generate only subset of Spin(n). To capture the full expressivity of rotor-baat based transformations, we wish extend to general bivectors.
2003
Building bivectors from simple bivectors. To utilize the closed-form exponential in (5) while retaining the full richness of (I²(n. we need way to express arbitrary bivectors in terms of simple ones. Fortunately, Roelfs and Keninck [202 show that any bivector b E (I²(n) admits an invariant decomposition
sum of at most l [n/2] mutually commuting, orthogonal, simple bivectors {b1, be...., ) This decomposition has two key advantages: (I) each component by admits an efficient closed-form solution exp(b.) in (5) since b, is simple, and (2) mutual commutativity
ensures exp EEin b.) = II exp(b:), via the standard Lie algebra identity exp(X + Y)
exp(X) exp(Y)) when their commutator [X, Y] = O. In so(n), the Lie bracket is [X, YJ XY YX, which vanishes when X and Y commute. With these benefits in place, the remaining question is how to construct the decomposition? A recent result provides a spectral formulation in terms of the eigenvectors and eigenvalues of b, stated below.
Lemma 2 (Thm. 4.8 Felbode al. (2024). ). Let b E (12(m) have as many eigenvectors as its effective pseudo-dimension. Then, we have
A3
A
b Σ 1=1
where a =b)= = ( " is the spectrum of b and onz and 1 are partner eigenvectors.
Differentiable invariant decomposition. While the invariant decomposition can be computed using eigendecomposition, standard algorithms pose challenges. The eigenvalues of a bivector come in conjugate pairs, and singular values come in positive pairs, making differentiation (and backpropagation) not as straightforward due to the numerical instability of eigen-deccomoosition for near-degenerate singular values. To address this, we introduce a Krylov mubspace-inspiree algorithm that iteratively extracts the simple bivectors without requiring explicit eigendecomposition. The
5

---

procedure is shown in Alg. 1 It includes subroutine for projecting bivector onto the manifold of simple bivectors, which we accomplish with a Clifford algebraic adaptation of the power iteration method shown in Alg. 2 This uses right contraction bLu, which extracts the components of b that lie in the direction of v, avoiding the need to construct explicit matrix representations.
The projection has the closed-form expression Projsimple (b) ( A v) where is the top singular value of b, and u, v are the corresponding left and right singular vectors. Note that because of sign symmetry of the paired singular vectors, we detect convergence by the sum, not their difference, and threshold e. Further discussion and proofs of Alg 2 are in Appendix B. We provide a full visualization of bivector to rotor in Fig.
Computation graph size. To control the size of the computation graph, we adapt DEQ (Bai et al. 2019]: we run the fixed-point iteration without gradient tracking, and then perform a single final forward pass with tracking enabled. This ensures that the graph scales only with (number of components in decomposition), and not the number of iterations in Alg. 2 which depends on the spectral gap and threshold c. Also, while one could initialize v in Alg. 2 using cinfeefferent SVD, this is unnecessary. Under small perturbations of b, the singular vectors vary smoothly. This follows from matrix perturbation results, which guarantees that in non-degenerate cases, eigenspaces vary analytically with the matrix entries (Ch. 2 of Kato 1980). Thus, warm-starting with the previous singular vectors yields fast convergence, if gradient steps are not too large.
4.2 A Generalized Rotor Gadget
Six rotors Preyino
Occasionally, may need mappings between arbitrary dimensional spaces. To do so, we now describe generalized rotor gadget. Instead of the standard sandwich product in (2). we can allow two different rotors on left/right for more expressiveness. Define the rotor- based transformation as:
pooling
(r,,(x) rass!
(6)
where s E Spin(n). This construction, how- ever, assumes the linear map acts data with dimension of the Clifford algebra, a power multiple rotor-
R
2 c Cl(n)
yP c Cl(n)
ic (L.2.3)
5C (1.2)
of 2. To address this, sandwich modules operating on different sub- spaces. For arbitrary input and output dimen- sions din and doun, we utilize the rotor-sandwich modules vr,, (x) building blocks construct map vb- Rd
Figure Rotor architecture with c
cz
An input x is split into { "elej"" each mapped to y via rotor maps Brgg.sy for each j E [ca]. The outputs {yos) are pooled and assembled into the final output y-
Rdin. Fix positive integers c1; C2, and n with 2n min(dm,, doat) and let [h] A 21, ........ 43. For each E [cı] and j E [c2], define 1; S [din] and O, S [dout] 2n subsets of input and output coordinates, respectively. We associate each pair with rotor map Pray.nn defined by rotors rij; 8iy E Spiin(n), parametrized by their corresponding bivectors azj, bij CCI (n). Then, each sub-module operates Cl(n) and computes the rotor-sandwich action Mrre RR - RO, The full output defined by aggregating all C102 rotor maps:
(x)) _ a (EWrgyes (x") i [cil,j E [e2])),
where a is a pooling operator on the outputs of Wry...yy Note that U3T, = [dml] and U,O, [dou]]
needed to fully cover the input and output dimensions. This gives a general raser- transformation from arbitrary input and output dimensions parametrized by bivectors which is visualized in Fig. We now analyze the number of learnable parameters requires.
Theorem (ub Parameter Count). Let vb Rd + Rdw be the mapping defined above, composed of rotor modules Vres.ess with E [c1] and j E [c2]. each acting in Cl(n) with 2n S mminddm door) A The total number of learnable parameters is upper bounded by
(m) -0 (log" d).
20102
2
Thm. 1 shows that rotor maps only O (log² min( (din dow)) parameters. In comparison, standard dense layer and rank-r factorization require O dout) and (r + dou))). respectively.

---

5 Experiments
Goals. We empirically evaluate rotors by replacing key, query, and value linear layers in pre- trained LLMs and measuring downstream performance on perplexity (PPL) and accuracy. Our experiments span multiple models and datasets. The main goals are to: (G-1) Demonstrate the feasibility of composing linear layers from bivector primitives by assessing whether rotors match baseline performance of Low-Rank and Block-Hadanne
approximations across diverse settings. (G-2) Quantify rotor parameter efficiency compared to dense and approximate alternatives. (G- 3) Analyze how rotor architectural choices-such as width and depth-affec affect performance.
We focus on linear layers in smaller pre-trained language models (up to 1.5B parameters), where reduced redundancy compared to larger models makes preserving performance harder. We do not attempt full model conversion- wwhich would require larger calibration datasets and a layerwise optimization scheme like GPTQ [Frantar et al. 2022]. Instead, we selectively replace 1-3 attention layers (key, query, and value projections) to isolate the effect, and assess whether rotor decomposition offers competitive PPL and accuracy relative to baselines.
End-to-end training with rotors. While the main LLM experiments train rotor layers mimic individual dense layers isolation (see Sec. Sec.5.T). we also assess their behavior when used throughout network trained jointly from scratch. To keep the setup lightweight, we replace all dense layers in a simple MLP (except the classification head) with rotor layers and train the model end-to-end on FMNIST Xiao et al., 2017] under identical conditions the dense baseline. Details and results are provided in Sec. 5.2 with experimental settings in Appendix C With this experiment, we aim to complement the layerwise LLM analysis by testing full-network training with rotor layers.
5.1 Experimental Setup
Substituting attention layers. Given x E IRd to transformer block, we have:
QKKT T
(ztn(z)= softmax
mask
V Wo.
where the query, key, and value linear projections are defined as Q = Wqz, K = WA.,, and V = W,,,, with W,, Ws, and W,, as dense learnable matrices. We jointly replace these linear layers in 1-3 selected attention layers with rotors or baseline approximations, keeping other parameters fixed except, for consistency, retraining the corresponding output linear layer Wo after each substitution.
Training protocol and architec- tural choices. To fit each sub- stitute layer (rotor, LR, BH), we extract hidden states from the pre-trained model and minimize MSE between the projected out- puts of the original and approx- imated layers. Each variant is trained independently using the Adam optimizer [Kingma and Ba 2017]. In our rotor architec- ture, depth refers to the number of stacked rotor maps v while width denotes the number of par- allel rotor maps within each layer. For example, the rotor map in Fig. 4 has both width and depth equal to 1 (i.e., one rotor map).
Dataset
Method
LLaMa-3.2 IB
Qwen-2.5 I.SB
three
three
Wikitex22
Original
2.575
2.287
LKI
2.688
3.455
4.956
2.350
2.402
2,591
LR4
2.658
2.729
2,850
2.342
2.372
2.548
BHI
2.636
2,700
2.779
2.323
2.388
2.558
Rotor
2.629
2.717
2.818
2:307
2,369
2,515
Original
3.151
2.834
Log- Log-PPi PPL
I.II
3.414
4.071
5.001
2.884
2.910
2.985
C4
LR4
3,390
3.315
3,504
2.874
2.905
2.980
BH1
3,343
3.262
3:404
2,865
2.897
2.975
Rotor
3.261
3,285
3,428
2.854
2,900
2.977
Original
3,260
2.985
PTB
LR4
LRI
3.358
3,316
4.684
3,400
3,466
6.904
3.034
3.046
3.127
3.151
3,192
3,225
3.355
3.395
3.101
3.168
Rotor
3.327
3,392
3,442
3.01
109
3,202
Are Challenge
Original LRI
58.37
66.09
50.78
50.44
44.26
$5.06
50.97
44.55
Accuracy (%)
LR4
53.39
54.51
BHI
54.83
54.25
49.61
60.11
49.27
60.68
Roter
55.31
54.50
49.64
61..4
$2.27
47.28
Hellaswag
Original
41.00
55.00
We also insert fixed permutations between rotors to enable grade mixing and add normalization layers to stabilize training; both are parameter-free All architec- tural details and hyperparameters are provided in Appendix
LR4
LRI
38.02
36.17
33.79
28.93
33.87
14.47
44.41
42.53
40.53
32.33
13.93
11.27
BHI 39.10
35.27
35.87
45.96
42.73
13.06
Rotor
39.33
34.94
37.52
50.20
40.60
6.868
Table 1: Log-PPL (4) and accuracy (t) using original, Low-Rank (r = or 4), BIII, and Rotor (ours) for 1-3 layer replacements. One-layer results are averaged over all layers; wootthreellaeer results are averaged over five random selections. Red indicates best, blue second- best per setting.
7

---

Dataset
Method
One layer replaced (Layer index)
3
8
13
14
LR1
3.620
2.750
2.754
2.703
2.714
2.695
2.628
2.612
Wikitex12 (U)
LR4
3.851
2.723
2.752
2.675
2.673
2.650
2.671
2.667
2,674
2.620
2.628
2.614
2
2.602
2,620
2.634
BHI
3.615
2.686
2.675
2.645
2.657
2.637
2.645
2.653
2.647
2.612
2.617
2.612
2.592
2,606
2.614
Rator
2.924
2.665
2.064
2.645
2.664
2.635
2,642
2.640
2.640
2.607
2.616
2.613
2.593
2.611
2.566
LR1
4.433
3.297
3,440
3.274
3.300
3.276
3,309
3,282
3.292
3.205
3.284
3,271
3.192
3.236
3.214
C4 (4)
LR4
4.432
3.292
3.404
3.250
3.266
3.250
3,264
3.260
3.281
3.203
3.204
3.191
3.187
3.219
3.208
BHI
4.293
3.288
3,276
3,212
3.230
3.215
3.241
3.230
3.276
3.196
3.193
3,184
3.174
3.194
3.194
Rotor
3,660
3,258
3.292
3.232
3.242
3.228
3,249
3.245
3.248
1.197
3.196
3,187
3.176
3.202
3.203
LRI
5.401
3,468
3.435
3,406
3.419
3,346
3.358
3.401
3.363
3. 3.322
3.281
3.308
3.271
3.322
3.292
PTB (4)
LR4
3.395
3.316
3.304
3.334
3.264
BH1
4.835
3.352
3.336
3.293
3.524
3.288
3.292
3.307
3,302
3.273
3.266
3.268
3.255
3.265
3.270
Rotor
3.412
3,403
3.356
3.369
3.338
3,336
3.326
3,300
3,266
3,300
3.284
LR1
53.22
46.75
46.35
51.07
52.79
51.93
56.65
55.
Are Challenge (1)
LR4
50.64
53.22
50.64
51.93
54,51
55.79
$4.08
$1.07
46.35
51.07
58,80
59.23
$7.51
56.22
57.51
BHI
53.22
54.51
52.79
54.94
57.08
54.08
53.65
52.36
50.21
57.94
57,
58.80
57.51
Rator
54.51
55.36
53.65
55.36
54.51
55.79
54.51
53.22
52.36
50.64
58.37
60.09
56.65
57.94
57.08
LR1
29.00
32.00
34.33
39.67
34,00
33.00
34,00
33,00
34.67
37,00
41.00
38.33
40.33
40,00
39.67
HellaSwag (t)
LR4
32.67
39.67
34.67
39.00
37.67
37.00
35.00
38.67
33.67
37.67
41.33
40.00
40.33
40.33
40,00
BH1
37.33
37.67
37.67
41.00
37.00
40.00
40.33
37.00
36.33
36.33
42.67
40.00
41.33
42.00
41.33
Rotor
40.00
39.67
35.67
42.33
41.00
38.67
36.67
38.67
36.00
36,67
42,00
39.67
40.33
42.00
40.67
Datasel
Method
Two layers replaced (Layer pairs)
10,11
10,12
10.13
10,14
10,15
11,12
11,13
11,14
11,15
12,13
12,14
12,15
13,14
13,15
14,15
LRI
2.724
2.702
2.695
2.697
2.718
2.757
2.731
2.756
2.757
2.768
2.723
2.741
2.729
2.863
2.862
Wikitex12 (4)
LR4
2.703
2.674
2.662
2,669
2.694
2.691
2.662
2.669
2.694
2.660
2.665
2.676
2.686
2.708
2.758
BHI
2.677
2.600
2.645
2,650
2.656
2.670
2.643
2.662
2.660
2.645
2.656
2.657
2.655
2,680
2.700
Rotor
2.679
2.670
2.654
2,660
2.676
2.662
2.652
2.667
2.677
2.667
2.667
2.675
2.662
2.688
2.745
3.273
3,258
3.296
3.276
3.315
3,278
C4 (L)
LR4
3,290
3,258
3,249
3,279
3.269
3.265
3,255
3,287
3,271
3.241
3,278
3,246
3.281
3,268
3,353
BHI
3,281
3.241
3.228
3.246
3.244
3.242
3.227
3,246
3.244
3.216
3.237
3,226
3.234
3.232
3.296
Roter
3.267
3:246
3,231
3,256
3:254
3:248
3.232
3,260
3.257
3.223
3,252
3.241
3:249
3,247
3.327
TRI
3.385
3.386
3.367
3,410
3.402
3.418
3.370
3,418
3.367
3,486
3.423
3.371
3,431
3,455
3.619
PTB (L)
LR4
3.345
3.319
3.320
3.348
3.336
3.326
3.315
3,353
3.332
3.300
3,317
3.313
3.367
3,335
3.419
BH1
3.315
3.307
3,297
3,304
3,310
3.309
3.287
3,332
3.301
3.281
3,293
3.298
3,294
3,300
3,345
Rotor
3,336
3.343
3.311
3.349
3,334
3.339
3,303
3.342
3,299
3.311
3,359
3,336
3.352
3.342
3,441
LRI
43.06
46.07
46.50
48.21
43.92
46.93
55.51
57.23
55.94
52.94
$4.65
54.65
54.22
55.51
$4.65
Are
LR4
42.92
48.50
48.07
51.07
49.79
59.23
56.65
59.23
57.94
58.80
56.65
57.51
56.22
55.36
$7.94
Challenge (1)
BHI
43.35
48.50
51.07
50:21
50.64
59.66
58.37
58.
56.65
57.94
59.23
57.51
57.94
55.65
58.37
Roter
45.49
48.93
53.65
52.79
51.50
57.94
58.
56.65
59.66
56.65
57.08
57.03
55.79
57.05
32.00
35.00
36.00
37.67
40.67
HellaSwag (t)
LR4
32.33
35.33
38.33
38.33 33
36.67
39.00
41.67
41.00
42.35
40.00
38.00
39
39.00
40.00
39.00
37.67
37.33
37.67
41.67
41.33
42.67
40.67
40.67
Rotor
36.00
37.67
38.67
38.67
37.67
39.33
41.67
41.33
41.33
39.33
41.67
39.00
41.00
40.33
40.33
Table 2: Performance on log-PPL (L) and accuracy (1) when replacing one attention layer (top) for layer indices 1-15 and two attention layers (bottom) for pains of indices from 10-15 of LLaMa-3. 2 1B. Methods are Low-Rank (r 1 and 4), BH1, and Rotor.
Models, datasets, and baselines. We evaluate on two pre-trained LLMs: LLaMa-3., 3.2 1B [Touvron
ct al. 2023) and Q 1.5B Qwen et al. 2025). Metrics include log perplexity (1) three
language modeling
Wikitext2, C4 [Dodge et al., 202T]. J. and PTB [Mrcuus et al. [993
and accuracy (+) on two multiple-choice benchmarks Arc Challenge [Clark et al. 2018 and
HellaSwag [Zellers et al. 2019]. We compare rotors to: (a) LRI and LR4: Low-rank projections
with rank 1 or 4, where a dense matrix W e B xd,, is approximated as XY, with X E Rd and Y e xda, (b) BHI:
Hadamaadd [Zeng et al., 2023] of depth 1, which approximate W by BH, where H is fixed Hadamard matrix and 73 is block block-diagonal and learnable. Parameter counts for all the methods are detailed in Tab. 4 reference LLM performance (with dense matrices) are shown in Tab. [ Additional experiments provided in Appendix D
5.2 Results and Discussion
Parameter efficiency. As shown in Tab. rotors require significantly fewer parameters than both dense and approximate baselines. For example, in LLaMa-3.2 2 1B, the query projection uses over 4.19M parameters in its dense form, compared to 16.4K in LR4, 32.7K in BHI, and just S 896 in rotors-47006 reduction over dense and 18x over LR4, Similar savings apply across key and value layers, directly supporting G-2.
8

---

#Epoch
2
3
4
5
6
7
8
9
10
Dense
85.05
86.23
87.17
88.07
87.90
88.54
89.07
89.36
89.53
89.67
Rotor
80.80
82.05
82.60
82.38
86.14
86.75
86.74
86.52
87.94
88.36
Table 3: Accuracy (%) on FMNIST over 10 epochs for dense and rotor-based MLPss
Compute cost. Rotor layers are currently slower in wall-clock runtime. Depending on rotor width (w) and depth (d). inference is roughly w times slower than dense layers our experiments. In terms of FLOPs, rotor layers require 35.8M X (wd) and 82.4M x (wd) on LLaMa-3.2 1B and LLaMa-3.2 3B, compared to 206. 1M and 515.3M for dense structure. These reductions arise naturally from the geometric formulation rather than from explicit sparsity constraints. During training, gradient storage and backpropagation through the decomposition (Fig. 3) add minor additional cost. Optimized kernels and hardware-awen aware implementations could exploit this structure much better, as well as offer benory-bandwidth benefits discussed in Sec
to their block-diagonal
Rotor performance vs. baselines. Despite their compact size, rotors match outperform LRI, LR4, and BH1 across most datasets and models. In Tab. [ on Wiki text2, rotors achieve 2.629 log-PPL in 1..Ma-3.2 2 1B (vs. 2.636 for second-best BHI), and 2.307 in Qwen-2.5 1.5B (vs. 2.323 for second-best BH1) when a single attention layer is replaced. On Arc Challenge, rotor layers yield 55.31% % accuracy in LLaMa-3. 2 1B (vs. 54.83% for BH1) and 61.34% in Qwen-2.5 1.5B (vs. 57.48% for LR4). Rotor projections are consistently either the best or second-best across most settings.
Tab. 2 confirms rotor robustness across layer combinations. For example, replacing layers 12 and 13 Qwen-2.5 1.5B yields 59.66% accuracy on Arc Challenge, outperforming LR4 (58.80%) and BHI (57.94%). In contrast, LRI, despite having 2-5x more parameters than rotors, shows significant performance drops (e.g., log-PPL on Wikitext2 for LLaMa-3.2 1B). These results confirm G-1 and show that rotors match downstream performance of other baselines only using far fewer parameters.
In Fig. 5 PPL consistently decreases with both increasing rotor width and depth. The strongest improvements occur from depth 1 to 2, after which gains taper off. This trend suggests that both stack- ing more layers (depth) and using more parallel rotor maps (width) contribute to better approxima- tion of linear soyers-showing G-3.
Width
12
Perplexity
11.5
10.5
10
2
3
Results of end-to-end training FMNIST. As shown in Tab.3] the dense network improves faster in early epochs, while the rotor-based model learns more gradually but reaches a comparable final accuracy (89.67% vs. 88.36%). This suggests that end-to-end optimization with rotors is slightly sloetr-potentially due the high compression but ultimately competitive with dense layers.
Depth
Figure 5: Effect of rotor width and depth. Replac- ing Layer- 13 in Qwen-2.5 1.5B with rotors of vary- ing depth and width. The dashed line (9.845) indicates convergence to the base model's 's perplexity.
Overall, the I.LM results support our claim: rotor layers consttueod from bivector primitives-off insight into how linear layers can be synthesized from a compact of building blocks. Moreover, the FMNIST experiment shows that rotor layers are feasible when trained jointly with the rest of the network end-to-end from scratch. Our full codebase, including datasets and hyperparameters for all experiments, is available at /ettss:/ggt thub. com/ com/vsingse /ss///iiinnggi//eeee group haahap/Cooooosnniie ComposingLinearLayrrr Key architectural and training details are also provided in Appendix C
Related Work
Compositions in machine learning. Building complex model behavior from composition of simpler compute primitives is an active research topic. Capsule networks [Sabour et al.) 2017] modeled phrt-whole relations, while mixture-of-experts [Riquelme et al. 2021 and model merging
[äähner and Moeller 2024] compose specialized sub-modules in conditional manner. Recent studies
show how internal circuits in LLMS organize around functional units (Wesss et al. 202T]. In NL.P,
Tree-structus struttrruc models [Tai et al. 2015) have been used to encode syntactic composition, while works

---

on compositional semantics [Mitchell and Lapata 2008 investigate how word meanings compose to form sentence meanings. The ability to understand new combinations of familiar components remains a challenge (Zhou et al. 2024, Lake and Baroni, 2018) and few strategies are being studied [Li et al. 2022 Chen et al. | Chytas et [0.4) Other directions include modular approaches
Das et al., 2018, Pieiffer et al. 2023
Approximating linear layers. To reduce the cost of dense layers, various structured approximations are com- mon. Low-rank factorization such as LoRA [Hu et al.]
Method
Key
Query
Value
Dense
1048576
4194304
1048576
2021] constrains weights rank-r matrices. Other matri- ces such as circulant Yu et al. 2014 Cheng et al. 2015].
LR1
2560
4096
2560
LR4
10240
16384
10240
Toeplitz (Sindhwani et al. 2015 Walsh-Hadamard Alam et al. 2024]. and Block- Hadamard (Zeng et al. 2023].
BHI
8192
32768
8192
Rotor
S 1080
8.896
S 1080
among others, have been shown to be resource efficient.
Table 4: Summary of the number of pa- rameters used for key, query, and value projections in a single attention layer of 3.aMa-3.2 1B.
Our goal is to understand the compositional structure of the linear transformation itself. Resource efficiency is the result, not the key motivation.
Clifford/Geometric algebra in ML. Recent works have explored Clifford algebra for encoding
geometric structure in ML models. GCANs [Ruhe al. 20236] and CGENNs [Ruhe et al. 2023a
construct equivariant layers by combining Ga-based transformations. Clifford neural layers Brand
stetter et al. 2023] was applied to physical modeling tasks. GA has been connected to convex training of ReLU networks [Pilanci] [024] and randomized methods for multivector representations (Wang
et al.
2024 are available. GA has also been used for time-series models (Chen et al. 2025
7 Conclusions and Future Work
We show that the functionality of standard linear layers can be expressed with exponentially fewer parameters, O (log² d) versus O (d²). while maintaining competitive performance when applied to attention mechanisms in modern L.Mss. In particular, our experiments on 1-1.5B parameter models demonstrate that rotor-based modules when trained in isolation, while our end-to-end experiment on FMNIST shows their feasibility when trained jointly with the rest of the network from scratch. The underlying algebraic structure reveals a rich compositional hierarchy, where geometric primitives combine through rotor operations to form expressive yet compact transformations. This is achieved by mapping bivector parameters to their corresponding rotor operations. These insights open several promising directions, including the development of interpretable architectures and parameter-efficient models that leverage this compositional structure. Beyond that, our framework connects to statistical models involving interaction decomposition, such as ANOVA (St et al., 1989]. multivariate analysis [Mardia al. 2024), and mechanisms to approximate Shapley features CCen et al., 2003)). For example, modeling high-order dependencies in statistical learning often suffers from exponential parameter growth. While ideas such as Tucker tensor decomposition [Li et al. [2018] factorize the full coefficient tensor, our framework offers an alternative by mapping k predictor variables to a multivector X and modeling the response variable via a rotor transformation where (a, b) are learned bivectors from our algorithms. This constructs interaction spaces through the composition of 2-way primitives.
effectively reproduce the behavior of dense linear layers
= (exp(a)X exp(b)t)o learned from Okk)),
Practical considerations. Our current implementation remains computationally demanding, since it does not yet exploit the inherent sparsity algebraic structure of rotor decompositions (see Appendix A for a discussion of limitations). In parallel, a promising direction is to leverage these ideas mitigate memory bandwidth bottlenecks that dominate large-model inference. As noted in Davies et al. 20025]. (i) memory capacity requirements for frontier LLMs exceed hundreds of gigabytes, and (ii) memory bandwidth, rather than compute, is the primary bottleneck for inference throughput. Our approach directly addresses (ii): instead of loading millions of dense-layer weights from memory, rotor layers can, in principle, synthesize them on-chip from a small set of geometric parameters, effectively trading compute for memory bandwidth. Future work will focus on developing sparse and hardware-aware Clifford kernels to realize these gains in practice and on extending the framework to full-model training and dynamic rotor composition for scalable deployment.
10

---

Acknowledgments
We thank the anonymous reviewers for their constructive feedback. We are very grateful to Prof. Karthikeyan Sankaralingam for numerous valuable discussions and working out the link to Davies et al. 2025)_ TP, DY and VS were all partly supported by NIH RO1 AG092220.
References
Samira Abnar, Omid Saremi, Laurent Dinh, Shantel Wilson, Miguel Angel Bautista, Chen Huang, Vimal Thilak, Etai Littwin, Jiatao Gu, Josh Susskind, and Samy Bengio. Adaptivity and modularity for efficient generalization over task complexity, 2023. URL https:
08866
Mohammad Mahmudul Alam, Alexander Oberle, Edward Raff, Stella Biderman, Tim Oates, and James Holt. A walsh hadamard derived linear vector symbolic architecture. arXiv preprint arXiv:2410.222 222.22669. 2024.
Francesco Alesiani. Pytorch geometric algebra. UKLL/ttgge//gtthh...m////llaaa//
ga
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models, 2019. URL https
7/arxiv. org/ aabs/999..oe.. 01377
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K. Gupta. Clifford neural layers for pde modeling, 2023. /ttp://tps:://rii..rrrrg www./oo/////aaaaee.ee 222/ 04934
Julian Büchel, Athanasios Vasilopoulos, William Andrew Simon, Irem Boybat, Hsin Yu Tsai, Geof- frey W Burr, Herman Castro, Bill Filipiak, Manuel Le Gallo, Abbas Rahimi, et al. Efficient scaling of large language models with mixture experts and 3d analog in in-memory computing. Nature Computational Science, pages 1-14, 2025.
Hui Chen, Viet Luong, Lopamudra Mukherjee, and Vikas Singh. Simpletm: simple baseline for multivariate time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025.
Lu Chen, Siyu Lou, Keyan Zhang, Jin Huang, and Quanshi Zhang. Harsanyinet: Computing accurate shapley values in single forward propagation. arXiv preprint arXiv:2304.01811, 2023.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou, Compositional generaliza- tion via neural-symbolie stack machines. Advances in Neural Information Processing Systems, 33: 1690-1701, 2020.
Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE international conference computer vision, pages 2857-2865, 2015.
Sotirios Panagiotis Chytas, Hyunwoo J Kim, and Vikas Singh. Understanding multi-coomositional learning in vision and language models via category theory. In European Conference Computer Vision, pages 324-341. Springer, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. https::/t::/arrt
org/ abs/ 10t1/1000000077
M. Cohn. Basic Algebra: Groups, Rings and Fields. Springer, London, 2003. doi: 10.1007/ 978-0-85729-428-9
Keith Conrad. Exterior powers. Lecture notes, Department of Mathematics, University of Connecticut, n.d. Available at httttts://kcooraat math, uconn. www.rbs/linnmll wwt.ttiagggg..eaaaoeoooo eetmodttooooooo////eeeeee//ooo pdf
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference robot learning, pages 53-62. PMLR, 2018.
11

---

Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Christos Kozyrakis. Efficient Ilm inference: Bandwidth, compute, synchronization, and capacity are all you need, 2025. URL
https://arxiv..rrt www.://.//////b/tto hbt///07./////...//// 2507 14397
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021. URL hhttts://aartat org h/s///b///// /204 08758
Chris Doran and Anthony Lasenby. Geometric algebra for physicists. Cambridge University Press, 2003.
David Felbode, Martin Roelfs, and Steven De Keninck. Outer and eigen: Tangent concepts, 2024.
URL btttps://arxiv... www.riiii..oaoaaooooo httgg/bbs2411..22/2
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 10..2220..12222 2022.
JEAN QUAINTANCE Gallier and Jocelyn Quaintance. Differential geometry and lie groups. volume 12. Springer, 2020.
Badih Ghazi, Rina Panigrahy, and Joshua Wang. Recursive sketches for modular deep learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2211 2211-2220. PMLR, 09-15 Jun 2019. URL https::/ppooeeedngssss wwnllr.pees////e www.s/eesssvee / ghazi 19a. html
Herbert Goldstein, Charles P. Poole, and John L. Safko. Classical Mechanics. Addison Addison-Wesley, 3rd edition, 2002. ISBN 978-0201657029.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, MD, 4th edition, 2013. ISBN 978 47144444079
David Hestenes and Garret Sobczyk. Clifford algebra geometric calculus: a unified language for mathematics and physics, volume 5. Springer Science & Business Media, 2012.
Eckhard Hitzer. Introduction to clifford's geometric algebra. arXiv preprint URL https://arxiv. org/ /000////3..... 1660
2013.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https
//arxiv. waaxiv..rg//a aba////6..... 09685
Zijian Hu, Jipeng Zhang, Rui Pan, Zhaozhuo Xu, Shanshan Han, Han Jin, Alay Dilipbhai Shah, Dimitris Stripelis, Yuhang Yao, Salman Avestimehr, Tong Zhang, and Chaoyang He. Fox-1: Open small language model for cloud and edge, 2025. URL/https: //tp:v/aavvvvvv// www.aii.g./aaaa.eeea.eaae /tw.s///4110022999
Tosio Kato. Perturbation Theory for Linear Operators, volume 132 of Grundlehren der mathematis- chen Wissenschaften. Springer- Berlin, Heidelberg, 2nd edition, 1980. ISBN 3-540-07558-5 URL https::/ maths ed.
pdf
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv. org/ abs/ 11/2. 6980
Lco Kozachkov, Ksenia V. Kastanenka, and Dmitry Krotov. Building transformers from neu-
rons and astrocytes. Proceedings of the National Academy of Sciences, 120(34))e221111501200 doi: 10.1077/paas2221115000 pnas. 2219150120
2023.
URL https:: htmp:::/www... /www. pnas http://///a////...00000/0
Zorah Lähner and Michael Moeller. On the direct alignment of latent spaces. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, pages 158-169. PMLR, 2024.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of senuencceto-sequene recurrent networks. In International conference on machine learning, pages 2873-2882, PMLR, 2018.
12

---

Mario Lezcano-Casado and David Bartinez-Rubio Cheap orthogonal constraints in neural networks: simple parametrization of the orthogonal and unitary group, 2019. URL https://aaxx org/ abs/1901. has/1990.08428,
Qing Li, Yixin Zhu, Yitao Liang, Ying Nian Wu, Song-Chun Zhu, and Siyuan Huang. Neural- symbolic recursive machine for systematic generalization. arXiv preprint arXiv:2210.0100, 2022.
Xiaoshan Li, Da Xu, Hua Zhou, and Lexin Li. Tucker tensor regression and neuroimaging analysis. Statistics in Biosciences, 10(3):520 145520055,,11 2018.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):13-330, 1993.
Kanti V Mardia, John T Kent, and Charles C Taylor. Multivariate analysis. John Wiley & Sons, 2024.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey. Artificial Intelli- gence Review, 00:275-293. 2014.
Jeff Mitchell and Mirella Lapata. Vector-based models of semantic composition. In proceedings of ACL-08: HLT, pages 236-244, 2008.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. Modular deep learning. arXiv preprint arXiv:2302. 11599, 2023.
Mert Pilanci. From complexity to clarity: Analytical expressions of deep neural network weights via clifford's geometrie algebra and convexity, 2024. URL httttttts//axiiiv org/ Bbrrg//bs/// hbs//baa////// 16512
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models, 2023. URL htttp:///aatt::i org/ abs/2210.0333 03350
Qwen,
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.. org/
www.g////////o..oe/oo///.eeeeee/ abs/ 2412 15115
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. practical review of mechanistic interpretability for transformer-base language models. arXiv preprint arXiv:2407.0266 2024.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Martin Roelfs and Steven De Keninck. Graded symmetry groups: Plane and simple, 2021. URL https://arziv.org/abp
03771
David Ruhe, Johannes Brandstetter, and Patrick Forré Clifford group equivariant neural networks, 2023a. Btt.hhttp::/arrxv..orr 2b3/ 11141
David Ruhe, Jayesh K. Gupta, Steven de Keninck, Max Welling, and Johannes Brandstetter. Geomet- ric clifford algebra networks, 2023b. URL hnttp:://arxii http//axxivvorrre (ttp://33200659
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules, 2017. URL
09829
Simon Schug, Seijin Kobayashi, Yassir Akram, Maciej Wolczyk, Alexandra Proca, Johannes von Oswald, Razvan Pascanu, João Sacramento, and Angelika Steger. Discovering modular solutions that generalize compositionally, 2024. URL https: httpt:tttttsiiiiiii :/appii///xxvv.. org/abs/ htgg/bb///11.... 15001
13

---

Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for small-footprint deep learning, Advances Neural Information Processing Systems, 28, 2015.
Lars St, Svante Wold, al. Analysis of variance (anova). Chemometrics and intelligent laboratory systems, 6(4):259-272) 1989.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint
2015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL URL httth:tttaxiiiiii wwwpaawwwww...a////////////////e httg/aab//202..39//
Yifei Wang, Sungyoon Kim, Paul Chu, Indu Subramaniam, and Mert Pilanci. Randomized geometric algebra methods for convex neural networks, 2024. URL https://arxiv..t org abs////s////// 028066
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers, 2021. URL https
//arxiv. wrg/ wwtpaa/......e//ee/..//...eeeeee 06981
James H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Press, Oxford, 1965. /tt.:///.../...///00000000000000
URL
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017. UKL/bttpss//ardy...or////// wttp://rr.wabb..........
Chenxi Yang and Swarat Chaudhuri. Safe neurosymbolic learning with differentiable symbolic execution, 2022.
Felix Yu. Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang. Circulant binary embedding. In International conference on machine learning, pages 946 954.954. PMLR, 2014.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL htl.htttt://arcwvorrr////// wtt.:/r///a/aa//////////// 1905. 07830
Zhanpeng Zeng, Michael Davies, Pranav Pulijala, Karthikeyan Sankaralingam, and Vikas Singh. Lookupffn: making transformers compute-lin lite for cpu inference. In International Conference on Machine Learning, pages 40707-40718. PMLR, 2023.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017. /ttp
01083
Yanli Zhou, Reuben Feinman, and Brenden M Lake, Compositional diversity in visual concept learning. Cognition, 244:105711, 2024
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson. J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2025. URL httpsttp:: / /ttp///axvvv//ee///e//eeeeeaeev/ org/ wbgg/bb//a.... abs/ 01405
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claim made in the abstract is that express linear layer in O (log" d) parameters with algebraic primitives. The framework is detailed in Sections and g with supporting experimental results in Section and Appendix D.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification:
The proofs of Alg. 1 and 2 are in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detail the experimental setup in Section and give hyperparameters and training details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We include codebase and instructions the code in supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
mublicc/qudees/Codeuubbss www.sssinoooiiillooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttp:/cccppplclguuiieeeeaec
Stteee///ooSSSeeissSiSi.a wsw.sssoiccgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details are given in Section 5 with hyperparameters and additional details on data splits and experiments given in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported for accuracy because it would be too computation- ally expensive. See Appendix C for GPU hours and resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix for GPU hours and memory requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is technical paper that has no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets, models, and pre-existing code bases used are cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: None used
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: hate:p///eerrippc.cc////ffeerees for what should or should not be described.
21

---

Appendix
the Appendix, we provide additional discussions, formal proofs, and further experimental details that support the main paper: Section A outlines the limitations of this work. Section B presents the proofs of the algorithms and theorems, along with general results for certain special cases. Section descrrbes the hyperparameters and training setups used in the experiments. Section D includes additional experimental results.
A
Limitations
While our results demonstrate feasibility, they remain theoretical this saae--though rotor layers achieve competitive performance when replacing attention layers, the benefits are primarily in radical parameter count reduction rather than immediate compute speedup in practice. Realizing such gains will require dedicated systems-level optimizations beyond the scope of this work.
Our current implementation relies on dense matrix representations rotors and does not yet exploit the inherent sparsity of the rotor decomposition. Leveraging this sparsity will require new backpropa- gation schemes and software libraries, particularly when training from scratch. We discuss the matrix representation and sparse structure of rotors in detail in Section B
Overall, this work is intended to highlight the feasibility and promise decomposing linear layers key building blocks in modern large models- smaller (potentially geometric) modules, such as bivectors and rotors, to facilitate more compact and interpretable architectures.
B Proofs
B.1
Correctness Proofs for Main Algorithms
We prove the correctness of Alg. D and in the theorems below.
Theorem Given bivector CI²(n), random vector Cl- that has non-zero component in the direction of the dominant simple component, and a threshold e E IR, Alg. returns an approximate simple projection of in a differentiable way.
Proof. Let B E so(n) be the skew-symmetric matrix corresponding to b. We begin by showing that bLu Bu for any vector u. Fix an orthonormal basis {c1....; en of RR" and
b =
Σ IsiciSn
bigle and
UkCk:
noting that [B]i bij for 1 < <i<js and [B]g = bbe for < <i n. It is easy verify that (c, A e2)LU = Dye; U,eg. By bilinearity of the right contraction, we have
bu Σ biy(Ufe= uzc,)
bi(t)e)))
Σ biy()))))))
(is(tee))) Σ [bsi(UEEi)
swapping i and
jzi
(1Bl(1) - Σ [B)e)()e))
Σ Bluuy es Σ
(Bu)ee: Bv.
It follows that
(bLw) b Bv B²v - Bv. Bv.
Since 2 is symmetrie, may apply the power iteration method. Observe, by skew-symmetry. the non-zero eigenvalues of come in conjugate pairs
{ .i.
22

---

with 2 02 2 ... d2 and k I |n/2]. An associated orthonormal set of complex eigenvectors can be written as {u, = ét; jısk- Since dt the dominant eigenvalue for B², has multiplicity 2, the power method on 8² will not converge in a single direction. Instead, it will oscillate within the two-dimensional span of {u1, f1 Wilkinson [1965].
Let be one of the singular vectors on which the algorithm terminates. Since singular vectors are unique up to rotation, WLOG we choose e to be the right singular vector. Then, the left singular vector u and singular value a satisfy b_t = Be ou by definition.
By the Eckart- Y'oung-Mirsks theorem [Golub and Van Loan 2013). the best rank-2 approximation to B is its projection onto its simple wwo-dimensional subspace spanned by top two singular directions:
Projsimple (b) dinie + 02u2222
As the singular values come pairs for B, this reduces to σi (urof + uzvu ). But for B, the two right singular vectors v1, U2 and the two left singular vectors u1, lie in the same two-plane, coming from the dominant eigenvalue conjugate pair, =id Thus, we can choose U2 U1 and t2 = 11 (or vice versa), which ensures U1 1 U1 and U2 U2. It follows that
Projsimple (b) a1 (wref vnu{) on (wz A v1)-
Since Alg 2 solves only approximately for (u A v) ~ (w) A vi), we have that Projsimple (b) = (1 A v1).
Note that since the right contraction implementation is differentiable, Alg 2 is differentiable.
Theorem 3. Given b (n) and k 1 many vectors CI (n). Alg. decomposition in a differentiable way.
returns the invariant
Proof. The heavy lifting done by Eelbode et al. 2024 in Lem. 2 which states that b can be written as the sum of at most k commuting, orthogonal, simple bivectors
k
M
1=1
where {=p1, +/22, ... =/4e} the spectrum of b and "nJ and U, are partner eigenvectors.
To obtain the full decomposition, it suffices to iteratively extract each term. If we have a differentiable subroutine to find the first term, then we can subtract it from apply the same routine to the residual, and repeat. Thus, it suffices to prove that the first term in the sum is equal to the projection of b onto the simple bivectors.
From the proof of Alg. we know that
Projetmple (b) al
where a is the largest singular value and u, are the corresponding left and right singular vectors. We wish to show that
(u = 141 1ut A UN Unj 4 iv. as cited in the proof of Alg.
The partner eigenvectors take the form Unt iv and Un Then, the numerator reduces to
0 put A E
(u + iv) A (u iv)
= =uA iu A + uutuuu
=-iu +
-iiu A v.
as the wedge product is antisymmetric.
The denominator reduces to
(u + iv) (u iv)
= u u iu 0 + in u + v
00.0
2
23

---

as singular vectors are orthonormal. Since μr = id, we have that
P4
UT
-2iu A o(u A v) Projsimple(O)
2
as desired. Subtraction differentiable and as Projsimmle can be found in differentiable way. this algorithm is differentiable.
B.2 Parameter Count
We revisit the main theorem the parameter count for rotor maps and provide proof of the statement.
Theorem (vb Parameter Count). Let ab Rd + Rda be a linear map composed of rotor modules Vrgge with E [ci] j E [02], each acting in Cl(m). Let 2n S 2 min(d (dins dow)). Then, the total number of learnable parameters is upper bounded by
n2 2
20102
Olog' d).
Proof. Each rotor is the exponential of bivector. As general bivectors are a linear combination of basis bivectors, of which there are (2). parametrizing bivector takes (?) scalar parameters. Hence, a single satdr-sandwich map Mra is parametrized using 2(²) scalar parameters. The result is then immediate, since there C2 such modules, corresponding to all input and output pairs.
Tab. 7 summarizes the number of parameters required for each key, query, and value projection per attention layer using different replacement methods across all LLMs used in our experiments.
B.3 Technical Analysis of Matrix Representation of Rotor Application
implement the sandwich product (2), along with other operations such as grade-restricted wedge and inner products, using the torch_ga Clifford algebra package for PyTorch, available at Alesiani In our implementation
(2). the rotor action is represented by a matrix M. this section, describe the construction of M and introduce some of its key properties. These results highlight the special orthogonality and block structure M. While the two-rotor map v (x) in (6) does not satisfy these properties exactly, it appears to share similar properties and structure; we do not discuss here.
We compute the sandwich product using a change-of- of-bas- basis matrix.
Definition 2. Let Spin(n), and let {ej) denote the canonical basis of Cl(n) (where J is an ordered multi-index). Define
N =
where (x) = rzr and T is the canonical vector-space isomorphism T Cl(n) - BR?". That is, each row of N, f the coefficients of reptt expressed in the basis. refer N, as the change-of- basis matrix for r.
Lemma 3. Let x Cl(n), Spinnn),, and N, be defined as in Def 2 Then, the two mappings
x x xN, and x r rxrl
are the same linear map up to isomorphism.
Proof. Write as
Σ JC
ZJeJ.
where xy is the real coefficient of the basis element ej. Let : Cl(n) - 2² be the canonical vector-space isomorphism that gives the row vector coordinates of the multivector, Φ.(x)) = rer! for x E Cl(n), and 5(y) N, for y E RR2" To prove and are the same up to isomorphism, we must prove that (x)) 5(7(x)) for all x Cl(n).
24

---

For each basis element e,, the sandwich product gives some multivector vi(e)) reer!, which we express as
reari- Σ
[N.],rer,
M
since (N stores the coefficient of eg reyrt by definition. Then,
[(0r(x)) = (rIT²)
Σ
IjeJ
rt
|C
Σ E.rerrt
J 15 ml
Σ JJ
IN-loeer
J ml
x
61
n
[C
x. [N,N,
T(er)
by linearity
IMM
JC(M]
T()) = =(T(z))
where the second to last step follows as the inner sum is the dot product of r(x) with the /th column of Np.
This allows us to compute the rotor sandwich product via matrix multiplication. However, this gives us very little insight into the structure of N, itself. For one, N, must respect the saade-preservng property of rotors, meaning it is a block diagonal matrix. The following alternative view provides more insight into its structure.
A more revealing view of Np- Let E CI²(n) be the bivector associated with E Spin(n) by exp(b), and let B E so(n) be the corresponding SO(n) via the matrix exponential map. For each grade
matrix. Let R cxp(2B)
n, set
M₁ C&(R) E ( (t))
where Ca(R) is the kth compound matrix of R. Algebraically, CR(R) is the kth exterior power of R. i.e. the unique linear map A (R) such that
(m) A m2 A ... A m&) Rm, A Rmz A... A Rmy
for all m, R for i [k] [Conrad n.d.j.
Definition 3. Let My be the kih compound, or exterior power, of R. Stack the Mt on the diagonal to form
M A diag(Mo, Mi...., Mn) E x2"
We first show that N, and M are in fact the same matrix.
Theorem 5. Let N, and M be defined as in Def. 2 and Then, M N..
25

---

Proof. The rotor sandwich product is grade-preserving automorphism, implying e (CI = (n). Thus, N, is biock-diagonal, one block per grade. First, we show (,) = Rv for vector where = ex)(2B). Observe
rerl ce") =c've" = ce
where the second equality follows reversion is anti-automorphism and the third equality follows as -b. Now, consider the adjoint operator in an associative algebra defined as (V) - [x XY XY YX. The Hadamard lemma and Taylor expansion together imply
eev
Σ A20
(ads)* (v) - exp(ado)v.
We also note the following identity:
adg(v) by vb (b + 64 v) (u- + A) 26 2BB,
where B is the skew-symmetric matrix for b. Putting these together, it follows that rb-(w) - rort exp(2B) v= Rv. Now, consider any k-veccor v Ci ... A Cig- We have
(v) vi(nn) A ... (ew) (Re,, A... A (Rea)
, is sude--peserring automorphism from above
(OR)(O).
Since the mapping is unique, are done.
Note that this block-diagonal structure highlights the grade-preserving behavior of rotors. This motivates our design choice to allow grade-mixing in our gadget for more expressivity. In particular, we insert random permutations between rotor layers so that information can mix across different grade components of the input multivector. Now that we have an alternative characterization of the sandwich product, we analyze its structure and the space in which it lives. Below, we examine several structural properties the matrix M.
Property 1. The matrix M in Def. 3 is block diagonal matrix with at most (?r) non-zero entries.
Proof. M = N, by Thm. 5 and so M is block diagonal. There are n + 1 blocks, each of size x for 0 k S n. Thus, the total number of non-zero entries is most
2n
k=0
which is far less than the 2 of dense 2n 2n matrix.
Property 2. For every grade k I n, the block Ms CR( lies in SO ((3)) Consequently, M E SO(2").
Proof. By assumption, B E so(n), and so exp(2B) = R E SO(n). Taking the kth exterior power preserves orthogonality and determinant of 1. For each the leth compound matrix M₁ = C&(R) the matrix representation of the kth exterior power A * (RR) with respect to the standard basis of k- vectors. Since R is orthogonal, A (It) preserves the induced inner product on k-vectors, making Mr. orthogonal. Moreover, since det (R) 1, we have det (M+) = det (A* ((())) = (det( 1. Therefore, M₁ E SO (()) for all 0 < k < n. To prove M E S2(2"), we note that since M is block diagonal, MT diag (M] .... MM and so M M I the M, are special orthogonal. det(M) = 1 as the determinant of block diagonal matrices is the product of the determinant of the blocks, meaning det(MM) = Ii det(MM) = II
26

---

Note that under this view, we obtain a map from so(n) to SO(2") by first applying the surjective exponential map from so(n) to SO(n). and then lifting SO(n) SO(2") via the exterior powers. This transformation enables represent rotor layers. We conclude by identifying the class of matrices which M belongs. The following is simple but useful characterization. The group of invertible elements of Cl(n) is
exponential reduction in the number of parameters required to
CI (n) = {a e CI(n)]Ba E Cl(n) st. aa a 1-1
Let
C1 (m) + GL((2"R))
be the mapping which assigns to each a E C1 (n) its change-of-basin matrix No. Then for any N, corresponding to Spin(n).
N, ((Spin(n)
We now give more refined version of Prop. 2
Property 3. Let be defined above. Then,
(
n
= 5o (0 n (c)) (n)) n
km
Proof. The first direction, ((Spin(n)) S SO(2") (CI* (n)) n (0 R())(()) is immediate.
r Spin(n), then by definition is an invertible element in CI+(n) which satisfies rr 1, (where 1 denotes the scalar identity). Thus, Spin(n) S CI* (n), which implies ((Spin(n) C (n)). Furthermore, Prop. 2 and Thm state N, E SO((2) for r Spin(n), which ensures ((Spin(n)) S SO(2"). As rotors are grade preserving, ((Spin(n)) S to (())
For the other direction SO(2") no (m)) (Di. (m)) (())() C ((Spin(n)) we start by taking an arbitrary element from the set the LHS and show it must also be in the set the RHS. Thus, we assume g E CI* (n) (an invertible multivector) such that its corresponding matrix N₂ =((g) is in S.(2"). We wish to show Spin(n). To do this, it suffices to prove three conditions: gg = 1, (i.e., 9 is unit-norm),
C1 (n) (i.e., is an even multivector), and is grade preserving.
The matrix Ng represents the linear transformation (X) = gg The condition Ng E S0(2")
means that Ng is orthogonal matrix with det(Ny) 1,, We know that due to orthogonality. Tg preserves the canonical inner product Cl(n). We define this inner product as (X, (XtY) where (-)o denotes the scalar part of multivector.
We will first show that g'g 1g. Since Ta is orthogonal transformation w.r.. (X,Y) (XiY),, it must satisfy T, = I, where is the identity map and T the adjoint of T. w.r.t. the inner product. The adjoint Т is given by TV (Y) = gtYg. This can be verified by checking the main (Y),
property of an adjoint, (Y,TT(())
(X,T,(X)) (Y" x))
((g\rig) x). (cyclically permute multivectors inside
scalar part operation, (MNP)o (PMN)))
= ( xx) = (T)
(since (ABC)+ CtBtAt and (At)* =A)
Now, applying the condition To means TH (Ty(X)) X for all X E Cl(n). Substituting the explicit forms for T7 and T we get
9¹ (gXgg) g=
(g'9) (g'g) ()) = X.
Let A = g.g. The equation becomes AXA A X for all E Cl(m). By the Wedderburn-Arrs theorem Cohn [2003]. the Clifford algebra Cl(n, 0) is isomorphic to matrix algebra (or direct
27

---

sum of two such algebras) over R, C, or H. In such matrix algebras (and hence in CI(n, 0) or its relevant simple components where lives, noting A is even), if an element satisfies AXA X for all elements X of the algebra, then A must be a scalar multiple of the identity, specifically +1,- Therefore, we must have = g = +,
Fix an orthonormal basis e, (where J is an ordered multi-index). Note that efes = 1, for every es- Write g = Σ, CJéJ for scalar coefficients çj E I. Then, the scalar part of g'g is (g'9) ( -, ej. Since g is invertible, we have 9 + O, so at least one cJ & O, making this sum strictly positive. Since g' g = A- 1, (where = 1,), its scalar part is X. As this scalar part (g) 9), must be positive, we must conclude that À = 1.. so g g 1,,
1. From the previous result, g g 1s. This means, g is invertible Next, will show that 99¹
and its unique inverse is g = g¹ Then, we can write gg" 99 = 1,- This helps us establish the first condition required for g to be an element of Spin(n).
The block-diagonal structure of Di (() ensures that the matrix representation o(g) acts independently on each grade subspace CI (n). Consequently, the induced inner automorphism (X) gXggt is
mapping each k-vector space to itself.
Finally, we will need to show that g E CI+ (i.e., g is an even multivector). Since N₂ E SO(2") have g), 1.2- Since gg¹ 1,, it follows that g¹ g_ The transformation can now be written (X) g.gg- which is inner automorphism of Cl(n) induced by gi such transformations are algebra automorphisms, meaning that they preserve the algebraic product structure (i.e., [x(X)) Y) = Tg (V)) Since 9 preserves grade (and in particular preserves the vector subspace CI² (n)), the element g can be expressed as product of invertible vectors. Combined with gg L.. this means g is unit versor, i.c., Pin(n). Versors who are products of an odd number of vectors (odd versors) are elements of Pin(n) \ Spin(n), while those that are products of an even number of vectors (even versors) are elements of Spin(n). Note that Pin(n) is the group of all such versors, and Spin(n) (in(n) n C1+ (n).
We know that if g were indeed an odd versor (an element of Pin(n) \ Spin(n)), the induced inner automorphism X + gXg when restricted to the vector subspace C1 (n) is an orthogonal transformation with determinant -1, (as it includes reflection, reversing orientation). It is known result from the representation theory of Clifford algebras that the determinant of the full automorphism Ng X + 9Xg- acting on the entire algebra Cl(n) corresponds to the parity of the versor g. Specifically, det(Ng) +1, if is an even versor (i.e., g E Spin(n)). and det(N,) = -1, if g is an odd versor (i.e., g E Pin(n) \ Spin(n)). This directly links the parity of the versor g the determinant of the full transformation matrix Ng. Since we know det (N₂) 1,, g must be an even versor, i.e., g E C1+ (n).
Since we have shown ggt = 1,, E (n), and that is grade preserving, we conclude that g E Spin(n) by its definition. Thus, So(2") (CI (m)) ( (1)) S ((Spin(n)) O
Prop. 3 provides an alternative characterization of the group of matrices associated with rotor conjugation, i.c., Nr, lives in the intersection of the special orthogonal group, the subgroup of matrices defined by the group of units of Cl(n), and those that are grade preserving.
C Hyperparameters and Experiment Details
We detail the training configurations for all experiments involving Rotor, Low-Rank (LR), and Block-Hadaaard (BH) projections used to approximate attention layers.
C.1 Training Details
Learning approximation layers in attention. To approximate the linear projections in attention layers of an I.M., we train all replacement modules (Rotor, IR, or BH) by minimizing mean squared error (MSE) loss between the predicted and true projection outputs, based on latent representations extracted from LLMs.
Formally, let W E Xdu be the dense projection matrix (i.e., query, key, or value) we want to approximate within a transformer block. Given hidden input E R. the projection computes
28

---

Wz E R To train an approximate layer. we collect a dataset D = { Bi) by prompting the LLM with a set of prompts bs (e.g,, Arc Challenge) and extracting the relevant hidden states at the target layer. Since hidden states are available for each token position in sequence for self-attention, we have N nT, where 7 is the average prompt length. We then learn an approximate layer Hg (rotors, LR, or BH) by minimizing
min Σ (Hoz - yy))
where 0 denotes the set of trainable parameters (e.g.. bivector coefficients for rotors). Optimization is done via gradient descent using the Adam optimizer (Kingma and Ba 2017].
We jointly replace the query, key, and value projections within an attention layer of a transformer replace up to three such attention layers and evaluate the resulting
block. In our experiments, model on downstream tasks of perplexity and accuracy metrics across various prompt datasets. When replacing multiple attention layers, say layers I, then J. and finally K. For each layer, we first replace all earlier trained layers (c.g.. I before J), and then extract the input-output data for training the new layer under this modified model. This is to ensure that each replacement layer is trained with respect to the distribution induced by preceding replacements. Also, whenever we replace layer L, we retrain the output linear projection W within the same attention block, using the same MSE and Adam optimizer for consistency.
J < K, we train them sequentially in order: first
Model
Key
Query
Value
LLaMa-3.2 1B
2048 + 512
2048
2048
2048 + 512
LLaMa-3.2 2 3B
3072 -> 1024
3072 > 3072
3072 > 1024
Qwen-2.5 1.5B
1536 -> 256
1536 >> 1536
1536 -> 256
Fox-1.0 1.6B
2048 512
2048 2048
2048 + 5122
Table 5: Input/output hidden dimensions for key, query, and value projections in single attention layer of different LLMs.
In Tab. 5 summarize the input and output hidden dimension for different LLMs.
Rotor networks.
We define the rotor-based transformation as:
thr, (x) A rass where r, E Spin(n). Each rotor map is then composed as
(xx a (f (x/t) i E e
where a is a pooling operator applied over the outputs of individual rotor transformation "ry More details are given in Subsection 4.2 As discussed, each rotor is parameterized by a small numher of bivector coefficients that encode geometric rotations, leading to significantly fewer learnable parameters compared to dense or baseline layers. rotor layer is constructed by stacking multiple rotor maps in depth and arranging them in parallel across width. For example, with width 2 and depth the layer contains 6 rotor maps, each parameterized independently. In our experiments, to reduce computational cost, we use at most width 2 and depth 3 for a rotor layer, which along with any up and down projection needed that is done by the generalized version in 4.2 sums up to roughly 1000 scalar parameters per projection (i.e., per Q/K/V) layer for LLaMa-3 3. 2 1B, compared to 1 4M in original dense layers.
Each rotor map is followed by a sequence of fixed (i.e., parameter-free permutations, normalizations, and a nonlinearity. Since rotor sandwich products are grade-preserving (see Section (B). we apply fixed permutations to enable interaction across grades and increase expressivity. We found that normalization improves training stability.
Hyperparameters such as depth, width, learning rate, and weight decay are selected via grid search; the final values along with the values we explored are listed in Tab. 6 All Clifford algebraic operations, including exponentiation of simple bivectors and sandwich products, are implemented entirely in Py PyTorch using the torch_8 library Alesiann]] which supports differentiation. We modified several methods in this package to reduce memory usage. For example, the original package computes the
29

---

Method
Hyperparameter
Values Explored
Final Value
Chunk Size
1024, 2048, 4096
2048
Depth
1,2,3
or
Width
1,2,3 ReLU, PReLU, GELU
or 2
Nonlinearity
PReLU
Normalization
true, false true, false 0.001, 0.005, 0.01, 0.05
true
Rotor
Permutations
true
Learning
0.05
é weight decay
0.001, 0.01, 0.1, O 16, 32, 64, 128, 256
0
Batch size
64
Cosine anncaling
true, false
true
Learning
0.001, 0.005, 0.01, 0.05
0.01
Low-Rank (LR)
e weight decay
0.001, 0.01, 0.1,
0
Batch size
16, 32, 64, 128, 256 true. false
256
Cosine annealing
true
Block number
dacck-Hadamard (BH)
Learning
0.001, 0.005, 0.01, 0.05
0.01
e weight decay
0.001, 0.01, 0.1.
0
Batch size
16, 32, 64, 128, 256
256
Cosine anncaling
true, false
true
Table 6: Hyperparameter settings used for each method.
geometric product using very sparse three dimensional Cayley table of shape 2 x 2n x 2n (Hitzer 2013). However, since we only require the geometric product between pure rotor and multivector when computing the sandwich product, we discard all but + (o) parts of the third dimension, rather than keeping the full 2".
Low-rank approximations. We use low-rank (LR) approximation as one of our baselines, follow- ing prior work such as Hu et al. [2021]. Given dense matrix W E Rdui we approximate it as the product of two lower-dimensional matrices: X R and Y Rexxu, such that
W XY,
where < dour- This decomposition effectively constrains the rank of the approximation at most r, capturing a low-rank subspace of the original operator. Low-rank approximations have been shown be effective in downstream tasks, especially when applied additive fine-tuning modules to frozen large pre-trained weights. They are also computationally efficient, requiring only Or + dia)) parameters and operations. our experiments, we choose r 1 and r and LRI requires roughly 3 5x parameters than rotor layers. Hyperparameters are listed in Tab. 6
Hadama- approximations. We adopt the Boock-Hadamard (BH) projection as another baseline. The idea is to alternate Hadamard transforms with learnable block alock-diagonal matrices, enabling a trade-off between expressivity and efficiency through the block size [Zeng et al. 2023
Formally, is defined
W I B.H.
where each B; is learnable Mlck-diaggnnl matrix with block size b, and H is fixed Hadamard transform. This approximation requires only O(b- max(dm: (ou)) parameters, and can interpreted as analogous to grouped convolutions followed by channel shuffling (Zhang et al.) 2017 The parameter m controls the depth of the transformation. our experiments, we use m (i.e., BHI) to reduce parameter count much as possible and match the scale of rotor-based models. Even with this minimal configuration, BH1 has 8-40x more parameters than rotors the case of LLaMa-3.2 1B. Specifically, use the approximation W BH, where B is block-diagonal matrix composed
30

---

of n. rectangular blocks, each of height h = dow/n and width = dm/n for the number of blocks. Hyperparameters are listed in Tab. 6
Hyperparameters and model architecture of FMNIST experiments. We trained both dense and rotor-based MLPs under identical conditions. Each model consists of hidden layers (either rotor dense), followed by ReLU activations. All dense layers were replaced with rotor layers in our rotor variant, except for the final classification head. The dense layers had hidden dimension of 512, while the rotor layers used (1(1 with width and depth 1. We performed search over learning rates r) E (0.001, 0.1) and selected 17 = 0.005 for the rotor- based model and = 0.002 for the dense baseline based validation accuracy.
C.2 Computational Resources for Experiments
The experiments for each LLaMa-3.2 1B and Qwen-2.5 1.5B each took around 1500 GPU hours, Fox-1.0 1. 6B around 1000 GPU hours, and LLaMa-3. 3.. 3B around 500 GPU hours for total of around 4500 GPU hours. This was spread across 8 NVIDIA A100 PCIe GPUs with 40 GBs of HBM2 memory. Total time of execution was around weeks run-time across all 8 GPUs.
A note the execution time of the rotor gadget. We have not optimized the current code for speed memory. At inference time, rotor layers are implemented as dense matrix multiplications. Therefore, its runtime scales with the depth of the rotor layer used in the replacement. In our experiments, used hyperparameters of at most depth and width 3; with an optimized implementation, this will result in at most 2x slowdown, since width is trivially parallelizable
is important to note that this corresponds to a direct implementation we currently do not leverage the block sparsity structure described in Section B.3 current software support is quite limited. Custom kernels have recently become available to batch matrix-matrix multiplication of different dimensions, such those available in weroaaaeeeerroueeeaattliioo vector-matrix multiplication. We found that this is slower than performing the dense implementation. though performance is expected to improve significantly once specialized vector-matrix versions become available. While custom kernels such as cublasSgemmStzz iedBBtcchdd support batching matrix-mi mattrixmmarix multiplication of the same dimension, their adaptation to rotors remains limited: even when applied to the parallel portions of the rotor gadget, they must operate sequentially on each block of the block sparse matrices. As result, this gives modest speedup, and there remains substantial room for further optimization.
but our setting requires
D Additional Experimental Results
In Fig. we report the number of iterations required by Alg. to converge within tolerance of T0 as function of the number of gradient updates applied to the rotors (i.e., bivector coefficients). Synthetic data was generated with random input and having output come from the rotation corresponding to a random bivector. Each gradient update step is towards learning that random bivector with MSE loss. As expected, higher-dimensional projections require more iterations to converge. Notably, warm-starting from previously learned singular values significantly reduces the convergence speed across all dimensions, supporting claim in Section 4.1
In Tab. 8 9 and TO we provide additional experimental results on Qwen-2.5 1.5B, LLaMa-3. 3B, and Fox-1 1.6B [u et al.) 2025), where single attention layer is replaced by Rotor, LR1, LR4, or BHI approximations. In Tab. we provide averages for single and two-layer replacements for Fox-1-11 6B. The trends observed in the main paper persist: our rotor-based method consistently matches the basclines with significantly fewer parameters (see Tab. 77. These results further support central claim: linear layers be synthesized from small number of geometric primitives encoding rotations.
31

---

dim=64
dim=128
dim=256
dim=512
dim=1024
dim=2048
dim=4096
d
# Iterations to Converge
4
10
Gradient Update Steps
Figure Number iterations required by Alg 2 converge within tolerance of 10 plotted against the number gradient updates applied parameters rotors (i.e., bivector coefficients). Results are averaged over 50 runs and simple bivectors the invariant decomposition, with one standard deviation shown error bars. The results demonstrate that warm-starting with previously learned singular vectors significantly accelerates convergence.
Model
Method
Key
Query
Value
Total
Dense
1,048,576
4,194.304
1,048.576
6,291.456
LR1
2,560
4,096
2,560
9,216
LLaMa-3.2
LR4
10,240
16,384
10,240
36,864
8,192
32,768
8,192
49,152
Rotor
$1,080
<896
$1.080
$3,056
Dense
3,145,728
9,437,184
3,145,728
15,728,640
LR1
4,098
6,148
4,098
14,344
LLaMa-3.2 3B
LR4
16,384
24,576
16,384
57,344
BHI
32,768
98,304
32,768
163,840
Rotor
$1,080
$1,120
31,080
$3,280
Dense
393,216
2,359,296
393,216
3,145,728
LRI
1,792
3,072
1,792
6,656
1we 1.5B
LR4
7.168
12,288
7.168
26.624
BHI
4,096
24,576
4,096
32,768
Rotor
3904
<896
<904
<2,704
Dense
1,048,576
4,194.304
1,048,576
6,291,456
LRI
2,560
4.096
2,560
9.216
Fox-1.0 1.6B
10,240
16,384
10,240
36,864
BHI
8,192
32,768
8.192
49,152
Rotor
<1,080
<896
<1,080
<3,056
Table 7: Number of parameters for key, query, and value projections in single attention layer of each model, with the rightmost column showing their sum.
32

---

Dataset
Method
One layer replaced (Layer index)
LRI
2.669
2.330
2.351
2.354
2.332
2.316
2.350
2.340
2.326
2.315
2.381
2.311
2.514
2.306
Wikitext2 (L)
LR4
2.635
2.320
2.334
2.343
2.320
2.310
2.335
2.331
2.318
2.310
2.368
2.307
2.456
2.306
2.323
2.323
2.308
2.314
2.307
2.417
2.307
Rotor
2.312
2.299
2.305
2.312
2.308
2.298
2.301
2.315
2.303
2,300
2.321
2.304
2.320
2.301
LRI
3.129
2.880
2,891
2.867
2,863
2,846
2.875
2.862
2.870
2.854
2.891
2,854
2.992
2.848
C4 (L)
LR4
3.007
2.880
2.890
2.869
2.862
2.846
2.873
2.862
2.867
2.854
2.888
2.853
2.939
2.848
BHI
2,943
2.872
2.878
2.864
2.856
2.844
2.870
2.858
2.863
2.849
2.879
2.851
2.925
2.847
Rolor
2,892
2.863
2.861
2.856
2,852
2.841
2.846
2.854
2.852
2.848
2.861
2.848
2.858
2.845
LRI
3.115
3.034
3.063
3.073
3.011
3.023
3.081
3,031
3,033
3.023
3.102
3.019
3.173
3.018
PTB (L)
LR4
3.110
3,021
3.063
3.055
3,006
3,015
3.076
3.030
3,028
3.012
3.078
3.018
3.150
3.017
BHI
3.091
3.007
3.046
3.044
3.004
3.007
3.076
3.020
3.022
3.007
3.064
3.015
3.101
3.012
3.045
3.025
3.001
3,003
3,028
3/012
3,034
3.009
LRI
62.23
63.95
59.23
47.21
60.52
66.52
52.36
58.37
57.51
50.21
40.34
58.80
22.32
57.51
Are Challenge (f)
LR4
59.66
62.66
64.81
54.94
60.94
66.09
56.22
62.66
66.09
58.37
55.79
61.37
9.01
58.80
63.52
63.52
60.94
50.21
Rolor
64.81
67.38
65.61
62.23
62.66
64.38
61.80
63.95
65.24
66.09
62.23
64.81
60.09
$6.22
LR1
50.00
20.00
HellaSwag (f)
LR4
38.00
54.00
46.33
48.00
51 10
55.00
56.33
51.00
26.00
43.67
49.00
54.33
31.33
20.00
BHI
45.67
48.00
46.00
48.33
48.67
50.67
51.67
52.33
38.33
44.67
48.00
51.33
24.33
32667
Rolor
50.67
53.00
53.67
52.33
52.00
55.33
57.00
54.67
48.67
55.33
51.00
57.00
51.33
32,00
Dataset
Method
One layer replaced (Layer index)
15
16
17
18
20
21
22
25
26
27
LRI
2.317
2.304
2.314
2.317
2.323
2.316
2.331
2,316
2.309
2.301
2.409
2.300
2,316
Wikitex12 (4)
LR4
2.317
2.304
2.314
2.317
2.323
2.316
2.331
2.316
2.309
2.301
2.409
2.300
2.316
BHI
2.313
2.301
2.314
2.316
2.320
2.311
2.311
2.315
2.309
2.300
2.398
2.297
2.309
Rotor
2,307
2.299
2:302
2,309
2.308
2,307
2.306
2:31
2.304
2.296
2.358
2.295
2.308
LRI
2,866
2,854
2.900
2.860
2.869
2,866
2,898
2.883
2.861
2.866
2.872
2,854
2,860
C4 (4)
2,851
2,889
BHI
2.862
2,847
2.856
2.858
2,867
2,862
2,872
2.857
2.854
2,849
2.852
2,850
2,853
Roter
2.855
2.849
2.858
2,862
2.852
2.851
2,849
LRI
3.025
3.003
3.020
3.021
3.036
3.019
3.086
3,010
3.014
3.012
3.019
3.061
3.098
PTB (4)
LR4
3.024
3.003
3,026
3.018
3.034
3.017
3.017
3,010
3.015
2.997
3,017
3.012
3,074
BIII
3.017
3,000
3.008
3.014
3.029
3.012
3,010
3.009
3.011
2.995
3.015
3.005
3.016
Rotor
3.009
2.998
3,003
3.006
3.019
3.008
3.008
3,001
3.007
2.993
3.012
3.001
3.011
LRI
41.20
65. 24.24
27. 90
65.24
68.24
16.31
47.64
67.38
66.95
65.67
65 24
66.09
66.52
Are
LR4
31.76
69.1
58.37
65.24
68.24
16.31
47.64
67.38
66.95
65.67
65.24
66.09
66.52
Challenge (f)
BHI
49.79
68.67
48.50
62.23
62.24
42.49
48.93
65.24
67.81
64,39
65.31
65.79
65.24
Rotor
56.65
70.82
61.37
60.09
69.53
51.07
47.21
67.81
66.52
66.09
65.67
66.09
65.24
LRI
28 67
49.67
32.67
45.67
51.00
5.67
36.33
52.67
57.33
53,33
53.67
53.67
54.67
HellaSwag (1)
49.67
32.67
51.00
5.67
36.33
52.67
57.33
53.33
53.67
53.67
54.67
B111
33.67
56.67
45.67
45.67
53.33
22.67
37.67
53.67
53.67
47.67
$2.33
56.00
51.67
49.33
52.67
35.00
51.00
54.33
54.00
56.67
Table 8: Performance log-PPL (L) and accuracy (1) when replacing one attention layer for layer indices 1-27 Qwen-2.5 1.5B. Methods are Low-Rank (r and 4), BHII and Rotor. Original log-PPL and accuracy are: Wikitext2 2.287, C4 2.834, PTB 2.985, Arc Challenge 66.09, Hellaswag 55.00.
Dataset
Method
One layer replaced (Layer index)
5
6
7
9
10
11
12
13
14
LRI
3.867
2.539
2.525
2.521
2.517
2.509
2.557
2.499
2.507
2.519
2.510
2.505
2.550
2.519
Wikitext2 (4)
LR4
3.389
2.514
2.512
2.501
2.510
2.492
2.531
2488
2.494
2.497
2.501
2,500
2,510
2.503
BHI
2.552
2.505
2.492
2,502
2.502
2.492
2.166
2
2.486
2.497
2.493
2.488
2.505
2.494
2.594
2.499
2.517
2.498
2.511
2.499
2.492
Dataset
Method
One layer replaced (Layer index)
16
21
26
2.501
2.487
2.487
2.484
2.477
2.479
2,490
2.479
2.519
Wikitext2 (4)
LR4
2.497
2.485
2.477
2.471
2
2.479
2,471
2.467
2.400
2.474
2.471
2.471
2.503
2.484
2.479
2.475
2.466
2.469
2.482
Rotor
2,488
2.481
2.477
2.471
2
2,478
2,466
2.471
2.482
2.484
2.482
2.477
2.496
Table 9: Performance on log-PPL (L) and accuracy (1) when replacing one attention layer for layer indices 1 1-27 of LLaMa-3. 3.2 2 3B. Methods are Low-Rank (r = 1 and 4), BHI, and Rotor. Original log-PPL is 2.460.
33

---

Dataset
Method
One layer replaced (Layer index)
12
14
16
LRI
2.645
2.573
2.549
2.535
2.531
2.546
2.550
2.533
2.581
2.500
2.557
2.
2.546
2,538
2,538
2.536
2.546
Wikitext2 (4)
LR4
2.614
2.568
2.538
2.535
2.530
2.544
2.545
2.531
2.570
2.538
2.552
2.535
2.538
2.538
2.536
2.534
BH1
2.633
2.548
2.545
2.534
2.530
2.545
2.544
2.531
2.577
2.536
2.550
2.538
2.537
2.537
2.534
2.532
Rotor
2.544
2.542
2.541
2.532
2.529
2,538
2.540
2.530
2.537
2.533
2.549
2.536
2.534
2.531
2.531
2.531
LRI
2.997
2.915
2.879
2.877
2.877
2.891
2.884
2.878
2.889
886
2.876
2.884
2,874
2,881
2,878
2,875
C4 (4)
2.8
2.874
BH1
2.991
2,901
2,875
2,878
2,876
2.891
2.881
2.878
2 889
2.879
2.874
2.876
2,872
2,879
2,877
2.872
2.924
3.231
3.226
3.255
3.231
3.233
3.268
3.237
3.279
PTB (1)
LR4
3.317
3.259
3,215
3.231
3.225
3,260
3.224
3,230
3.276
3.232
3.229
3.235
3.230
3.242
3.229
3,226
BHI
3.318
3.230
3.214
3,230
3.224
3,259
3.224
3,229
3.277
3.231
3.228
3.234
3,230
3.241
3.227
3.127
Rotor
3,283
3.229
3.212
3,229
3.224
3.257
3.223
3.228
3.255
3.229
3.228
3,232
3.228
3.234
3.225
3.225
LRI
31.76
31.33
33.05
42.06
42.49
49
47.64
36.48
44.64
22.75
41.63
24.03
39.48
39.91
20.17
21.03
18
Arc
LR4
27.47
26.61
27.90
36.91
39.06
48
36.05
41.20
15.45
35.19
18.88
31.33
42.06
16.74
21.03
23.61
Challenge (1)
BH1
34,33
21.46
28.33
32.19
38
45.06
26.61
42.49
11.59
36.48
14.16
21.89
46.78
24.89
25.75
9.44
Rotor
27.04
21.03
26.18
23.61
30.47
36.48
29.
39.91
22.32
31.33
13.30
22.32
31.76
28
24.46
18:45
31.33
45.33
17.33
27.67
42.67
39.00
HellaSwag (f)
LR4
35.67
36.67
25.00
33.00
42.67
41.33
43.00
44.67
6.33
19.00
4.33
26.67
31.13
41.33
18.67
39.33
Rotor
40.00
37.33
28.33
36.67
44.67
44.33
41.00
44.33
17.33
32.33
18.33
35.67
34.00
42.67
26.33
38.33
Dataset
Method
layer replaced
index)
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
LR1
2.536
2.530
2.564
2.553
2.534
2.544
2.546
2.523
2.537
2.522
2.535
2.617
2. 2,540
2.544
2.571
Wikitext2 (L)
LR4
2.534
2.528
2.563
2.553
2.534
2.543
2.541
2.523
2.536
2.523
2.534
2.558
2.531
2.543
2.541
BHI
2.532
2.528
2.558
2.550
2.530
2.538
2.537
2.521
2.530
2.521
2.532
2.556
2.529
2.539
2.532
Rotor
2.531
2.527
2.531
2.548
2.529
2.538
2.534
2.521
2.529
2.521
2.531
2.553
2.528
2.537
2.531
LRI
2.879
2.878
2.891
2.884
2.879
2,888
2.879
2.872
2,885
2.872
2.875
2.875
2,870
2,876
2.904
C4 (U)
LR4
2,879
2.877
2.884
2.880
2,889
2.879
2.872
2.884
2.872
2.874
2.875
2,870
2,875
2,890
BHI
2.878
2,876
2,883
2,883
2.878
2.886
2,878
2,871
2.879
2.870
2.872
2.873
2,868
2.874
2.886
PTB (4)
LR4
3.224
3.221
3,239
3.240
3.214
3.226
3,222
3,213
3.230
3.219
3.218
3.222
3,212
3.233
3.235
BHI
3.222
3,222
3.238
3.239
3,222
3,210
3.230
3,222
3,212
3.226
Rotor
3.221
3,220
3.232
3:238
3:213
3.225
3,220
3.210
3.228
3.216
3.217
3,220
3.211
3.220
3:223
LR1
4.72
6.87
39.91
44.21
18:45
33.91
34.33
21.46
7.30
27.47
18.88
29 18
47.64
10,30
7.73
Are
LR4
8.15
41.63
18.88
30.90
34.33
21.46
8.58
27.04
18.88
31,76
48.07
12.45
13.73
Challenge (1)
BHI
10.30
5.58
35.62
42.49
20.17
30.04
32.19
24.03
9.87
25.32
19.31
25.32
46.78
18.88
24.46
Rotor
20.17
18.88
25.32
33.91
21.03
30.04
26.61
24.46
11.59
26.18
21.46
24.89
45.49
19.74
24.46
LR1
27.33
26.33
42.33
47.33
41.33
37.00
37.67
32.00
23.67
40.00
34.67
46.67
47.33
28.67
12.00
HellaSwag (t)
LR4
25.00
28.33
41.00
46.33
40.00
37.00
38.67
32
26.67
40.67
36.33
44.33
46.00
31.33
28.00
BH1
28.67
27.67
42.33
47.67
44.00
40.67
38.33
34.67
27.33
39.67
36.67
42.33
47,00
32.33
36.67
Rotor
33.67
34
34.67
38.33
34.33
29.0
33.67
40.67
46.33
31.67
Table 10: Performance on log-PPL (L) and accuracy (1) when replacing one attention layer for layer indices
1-31 results of Fox-1.0 1. Methods are Low-Ranks (r and 4), BHI, and Rotor. Original -Pg and accuracy are: Wikitext2 2.517, C4 2.862, PTB 3.205, Arc Challenge 24.89, He Hellaswag 38.33.
34

---

Dataset
Method
Fox-1.0
one
Original
2.517
Wikiexx2
T.I
2,550
2,589
LR4
2.540
2.578
BHI
2.538
2.573
Rotor
2.534
2.576
Original
2.862
Log-PPI.
LRI
2,881
2,907
BHI
2.878
2.898
Rotor
2.877
2,901
Original
3.205
LRI
3.238
3.299
PTB
3,230
BIll
3.228
3.275
Original
24.89
Are Challenge
29.03
Accurcy (%)
LR4
27.40
27.12
Rotor
25.82
30.22
Original
38.33
Hellaswag
LRI
32.33
25.49
IR4
32.41
24.40
Rolor
35.14
32.87
Table 11: Log-PPL. (L) and accuracy (1) using original, Low-Rank (r = 1 or 4), BHI, and Rotor for 1-2 layer replacements on Fox-1.0 1B One-layer results are averaged over all layers; two-layer results are averaged five random selections. Red indicates best, blue second-best per setting.
35


Reviews:
[{"review_id": "Q2H8Cmmj4S", "reviewer": "Reviewer_tqhq", "summary": "The authors show that replacing linear layers with clifford algebra layers can improve interpretability while maintaining comparable performance and reducing parameter count. To do this, the authors use an existing theoretical result stating that any linear function can be written as a sandwich product with clifford algebra multivectors. Naively designing an architecture based on this result would introduce a huge computational burden. Instead, the authors claim any linear function can be written as the composition of rotors which have many fewer parameters. While this theoretical result is not proved in the main body, it is empirically validated in the case where 1-3 attention layers in a small LLM are replaced with rotor layers.", "questions": "*Questions*\n- (line 111) It isn’t clear to me how choosing to consider the subalgebra Spin(n) still allows you to represent a general, arbitrary linear map.\n- (line 256) Do the authors expect that performance would remain comparable if all layers were replaced with clifford algebra layers?\n- (line 287) Are the clifford algebra layers harder to train than conventional linear layers? Are they more computationally expensive?\n\n*Comments*\n- It would be easier to read Table 2 if the best performing methods were highlighted as they are in Table 1", "limitations": "The authors do not acknowledge limitations or potential negative societal impacts of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The submission appears technically sound, with the exception that a main claim (any linear function can be written as a sandwich product with rotors) is not proved. The claim is, however, supported in a narrow context by experimental results.\nThe submission is very well written and organized.\nThe results appear significant, potentially providing a mechanism to improve interpretability in LLMs.\nThe work is part of a growing body of research leveraging clifford algebra representations for interpretability and expressivity in neural networks. As far as I know, the insights presented in this work are original and seem to expand our understanding of the utility of clifford algebra representations.", "was_revised": true, "final_justification": "My perception of the work was relatively unchanged by the rebuttal, however, I am more confident in my assessment.", "rebuttal": "We thank the reviewer for their constructive comments and thoughtful questions. Below we respond in detail to all questions.\n\n**Q1 (line 111): It isn’t clear to me how choosing to consider the subalgebra $\\text{Spin}(n)$ still allows you to represent a general, arbitrary linear map.**\n\nThank you for raising this subtle point. As you noted, we use a known result (Lemma $1$ by Hestenes and Sobczyk) which states that a general linear map can be written as a finite sum of the conjugations of multivectors in $\\text{Cl}(n)$. This is useful, but directly using multivectors is computationally expensive. To make this practical, we must impose additional structure on the multivectors that can be exploited. The subset that we choose, the Spin group $\\text{Spin}(n)$, does not capture arbitrary maps in $\\text{GL}(d)$, but rather *a subset of maps defined by rotors*. In Appendix B.3 (Property 3), we fully characterize the matrices isomorphic to the image of rotor sandwich products as a subgroup of $\\text{SO}(2^n)$ induced by the Clifford algebra. \n\nWhat helps in recovering expressivity is our parallel rotor gadget: using rotor modules acting on different subspaces and aggregating their outputs through pooling, we approximate more general linear maps through superposition. Each rotor contributes an orthogonal transformation on its subspace, and their weighted combination approximates non-orthogonal transformations. While we cannot represent all of $\\text{GL}(d)$ exactly (unless we go back to Lemma 1), empirically this is sufficient. We find that this subspace is expressive enough to approximate LLM projection layers.\n\n\n**Q2 (line 256): Do the authors expect that performance would remain comparable if all layers were replaced with clifford algebra layers?**\n\nYes, for smaller models performance does remain comparable to the dense linear layer. We ran additional experiments on FMNIST, where we trained an MLP end-to-end from scratch, replacing *all* dense layers with rotor layers (except the classification head). We performed a hyperparameter search for both and trained each for the same number of epochs. In this setting, the conventional dense layers and rotor layers achieved 89.67% and 88.36% accuracy respectively.\n\nFor large models such as LLMs, full end-to-end training is currently infeasible due to the nascent state of Clifford algebra packages and their limited hardware integration. For this reason, we chose layer-wise replacement, keeping all other weights frozen. When multiple layers are replaced with any approximation method under this scheme, small errors can accumulate across layers.\n\nTo our knowledge, there are only two PyTorch-compatible packages that implement Clifford operations. On line 764, we discuss one of the many optimizations we made to the *torch_ga* package (which we provide in the supplementary material). This allows us to execute key Clifford algebra operations (e.g., wedge product, left contraction, sandwich product) for the larger algebras required in LLMs, but significantly more engineering effort is needed to make full-network training efficient at scale.\n\n\n**Q3 (line 287): Are Clifford layers harder to train or more computationally expensive?**\n\nYes, Clifford (rotor) layers are currently more computationally intensive to train, but more efficient to apply at inference time (with a suitably optimized low-level implementation). For example, on LLaMA-1B and 3B, rotor layers require 35.8M and 82.4M FLOPs (multiplied by a factor $K$ which ranges from 1 to 6 depending on the configuration of rotor gadget), compared to 206.1M and 515.3M FLOPs for standard dense layers. Thus, even in the worst case, it works out favorably. \n\nHowever, we emphasize that this comparison is not our primary goal. Our core contribution is to **synthesize dense linear layers from exponentially fewer geometric primitives, rather than to minimize FLOPS of linear layers directly.**\n\nThe FLOP reduction is coincidental: the synthesized matrices simply happen to have fewer non-zero entries due to their block-diagonal structure. We were not targeting sparsity; in this specific construction, it is a byproduct of our geometric formulation. It is possible that alternative compositional approaches could synthesize the exact dense matrix with no sparsity (and FLOP) benefits.\n\n**The real promise going forward lies in eliminating memory bandwidth bottlenecks**. The recent LIMINAL paper (arXiv:2507.14397) from NVIDIA Research identifies three critical findings for LLM inference:\n\n1) Memory capacity requirements are massive (385GB+ for Llama-405B)\n2) Memory bandwidth is the primary bottleneck for user tokens/second\n3) DRAM-based designs offer fundamental efficiency advantages\n\nThe reviewer will see that **our approach directly addresses finding #2**: instead of loading millions of parameters from memory to GPU (the dominant cost in LLM inference), with the appropriate engineering effort, we can synthesize these weights on-chip from ~1000 geometric parameters! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary bottleneck, as noted in the LIMINAL work (and other industry reports).\n\nAs an aside, this reminds us of the Fast Fourier Transform: recognizing that the DFT matrix can be factored didn't immediately yield compute benefits, but this compositional structure later enabled $O(n \\log n)$ algorithms compared to $O(n^2)$. Similarly, our decomposition reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest.\n\nTo summarize, we match expressivity with exponentially fewer parameters, which coincidentally achieves modest FLOP reduction, and more importantly, opens a path to eliminate memory bandwidth constraints (on the practical side) and to exploit the compositional structure (on the model development side).\n\n\n**Statement about limitations: The authors do not acknowledge limitations or potential negative societal impacts of their work.**\n\nWe discuss limitations in Appendix A, but have now added a reference to this section in the main paper’s conclusion. Thanks for the suggestion."}, {"review_id": "vfpZ9nTLwi", "reviewer": "Reviewer_7rjk", "summary": "The paper posits a way to rewrite linear transformations in deep learning frameworks in a nonlinear algebraic form with fewer parameters, based on composition of simple primitives. These primitives are based on bivector products in Clifford algebra. The expression is differentiable via chain rule and it applies to LLMs (on which experiments are done to compare to other model compression algorithms), and the compression goes from scaling as d^2 of number of learnable parameters to that number scaling like log^2d in which d is the layer dimension.", "questions": "I'm impressed by the experimental results, such as 4700x reduction of number of parameters, and I would like to understand possible limitations of the method. So I try to ask some questions in that direction below\n\n1) In theorem 2 and in the LLM layer experiments you have a number of parameters going down by a huge factor. This means that the models become highly compressed by this approach. Does it mean that the learning rate on the compressed models become comparatively slower as the parameters get more entangled? Or else, how do the learning rates compare?\n\n2) How do you explain the fact that your rotor based composition works well with 1 or 2 layers in experiments but then stops improving? (as mentioned in line 318 and following ones)\n\n3) Do you think that the composition of primitives as given by your result can be interpretable in some setting (maybe some LLM case) or in some toy example? You say in line 349-350 that \"Our goal is to understand the compositional structure of the linear transformation itself\" so can you please state in more detail what do we understand now with your method precisely?\n\n4) Do you know, besides drop in parameters, what is a drop in Kolmogorov Complexity incurred during the learning dynamics by your method compared to others? I'd be curious to know how much information is lost due to gradient descent not being able to fit the models, and I'm wondering if there is a metric or a toy case in which you can give more insights on that, besides computing the accuracy or PPL as you do.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: the method seems interesting and it performs well compared to the presented alternatives, therefore it is a good advance as far as I could check. \n\nI feel that possible limitations of the method are not discussed, which can mean two things: either this is a cornerstone paper which will revolutionize LLM and general model compression, or this lack of discussion of limitations is a weakness of the paper. Unfortunately I can't tell which one is the case, as I have never worked with Clifford algebra and thus this is just marginal to my domain of expertise", "was_revised": true, "final_justification": "Authors replies cover my doubts and I'm raising score from 4 to 5 to quantify this.", "rebuttal": "We thank the reviewer for the insightful questions and positive feedback. Below we respond in detail to all questions.\n\n**Q1: In theorem 2 and in the LLM layer experiments you have a number of parameters going down by a huge factor. This means that the models become highly compressed by this approach. Does it mean that the learning rate on the compressed models become comparatively slower as the parameters get more entangled? Or else, how do the learning rates compare?**\n\nThank you for the interesting question. Full end-to-end training on LLMs is currently infeasible due to limitations in Clifford algebra packages and hardware integration. To study this question more carefully, we ran additional experiments with a MLP trained on FMNIST, where we replace all dense layers (except the final classification head) with rotor layers. The accuracy as epochs increase are shown below:\n\n**Accuracy (%)**\n| #Epoch | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Dense  |$85.05$ |$86.23$ |$87.17$ |$88.07$ |$87.90$ |$88.54$ |$89.07$ |$89.36$ |$89.53$ |$89.67$ | \n| Rotor |$80.80$ |$82.05$ |$82.60$ |$82.38$ |$86.14$ |$86.75$ |$86.74$ |$86.52$ |$87.94$ |$88.36$ | \n\nFrom these experiments, we observed: \n\n1. Accuracy improves more quickly during training with dense layers than with rotor layers\n2. Both methods ultimately reach a final accuracy of 88-89%\n\nYour intuition is correct. In this setting, the learning rate on the compressed model is comparatively slower than the dense approach.\n\n\n**Q2: How do you explain the fact that your rotor based composition works well with 1 or 2 layers (depth) in experiments but then stops improving (as mentioned in line 318 and following ones)?**\n\nThank you for pointing this out! We realize that Figure 5 can be made clearer. Our current setup can be viewed as fitting to a fixed target functionality of the original dense matrix, where we train only to replace that layer while keeping all other weights frozen. This is different from end-to-end training, where more abstract representations across layers co-adapt. Therefore, the best possible perplexity we can reach is that of the original LLM. \n\nFrom Table 1, we see this lower bound is 9.85. Looking at Figure 5, we see that after a depth of 2, the PPL is already 10.4--somewhat close to the limit, with a depth of 5 reaching 10.1. The PPL doesn't improve as much beyond depth 2 or 3 because the model is already nearing the lower bound. For clarity, we have added a horizontal line at 9.85 in Figure 5 to emphasize this convergence. \n\nThis behavior can be understood more precisely as follows. Since we use pre-trained LLMs, the attention projection matrices (i.e., query, key, and value) may already lie close to the subspace representable by our rotor construction. As we are approximating the functionality of fixed matrices rather than training from scratch, 1-2 rotor layers may capture most of the structure needed to match the pre-existing functionality. Additional layers then provide only marginal refinement, rather than letting us access new transformations.\n\n\n**Q3: Do you think that the composition of primitives as given by your result can be interpretable in some setting (maybe some LLM case) or in some toy example? You say in line 349-350 that \"Our goal is to understand the compositional structure of the linear transformation itself\" so can you please state in more detail what do we understand now with your method precisely?**\n\nThank you for asking this important question.\n\nFirst, we now understand that the function space learned by linear layers in LLMs can be represented using objects with $O(\\log^2 d)$ geometric degrees of freedom, rather than $O(d^2)$ arbitrary parameters. This suggests that the linear transformations (via dense layers) live near highly structured objects that preserve similar invariances in the data. This gives us a *clearer perspective on their intrinsic geometric simplicity*.\n\nSecond, each bivector-rotor pair in our decomposition operates independently on a $2^n$-dimensional subspace. We now understand that the seemingly monolithic $d\\times d$ transformation factors into smaller, independent modules--a structure that is very different from low-rank or block-diagonal approximations. This modularity is discovered rather than imposed. As a result, these transformations can be generated from compact geometric primitives instead of being stored as large matrices. \n\nTaken together, this understanding suggests that the underlying computations in LLMs may exhibit hidden geometric simplicity, and that the models could be implicity discovering and exploiting such structure in the data. While we do not yet claim semantic interpretability of the geometric primitives in large models, this can open promising directions for both interpretability research and the design of more efficient architectures. For an example of how this compositional structure can be exploited for efficieny gains, please see our response to Q3 for Reviewer tqhq.\n\n\n**Q4: Do you know, besides drop in parameters, what is a drop in Kolmogorov Complexity incurred during the learning dynamics by your method compared to others? I'd be curious to know how much information is lost due to gradient descent not being able to fit the models, and I'm wondering if there is a metric or a toy case in which you can give more insights on that, besides computing the accuracy or PPL as you do.**\n\nWe much appreciate this suggestion to explore the information-theoretic implications of our construction. Our background in this topic is somewhat limited, and certainly not at a level to provide a fully satisfying answer.\n\nFrom our understanding, Kolmogorov Complexity is upper bounded by the number of parameters. For our general rotor gadget (see Appendix B.2), this corresponds to $2c_1c_2{\\log_2 d \\choose 2}$, where $c_1$ and $c_2$ are hyperparameters and $d$ is the hidden dimension. For the models used in our experiments, we provide parameter counts of the dense linear layer, rotor layer, and baselines in Table 6, Appendix D.\n\nWe would be grateful for any guidance or references to theoretical or empirical results that could help us explore this direction more rigorously. If we are able to formalize a meaningful result, we will include it in the final version and credit the reviewer for the suggestion."}, {"review_id": "0AxKYQ24iI", "reviewer": "Reviewer_EgXM", "summary": "The paper investigated how linear transformations in neural networks and LLMs can be synthesized from geometric primitives via Clifford algebra. \nThe authors showed that linear layers can be represented as compositions of rotors, which are parameterized by bivectors (geometric objects encoding oriented planes).\nThey introduce a differentiable invariant decomposition algorithm that breaks general bivectors into another components with closed-form exponential. The authors replaced the key, query, and value projection layers in the attention mechanism of two LLMs with their proposed layers to evaluate their method.", "questions": "Can the authors add more on the runtime of their proposed method and does it compare to the baselines?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- Using Clifford algebra to represent linear layer: The paper provided an algebraic lens on linear layers, showing that they admit a compact factorization into bivector irreducibles and rotors, with less number of parameters. \n- Empirical results: The authors showed that replacing the key, query, and value projections in LLaMA-3.2 1B and Qwen-2.5 1.5B with rotor-based layers achieved competitive performance aganist low-rank and block-Hadamard baselines on Wikitext2, C4, and PTB datasets and Arc Challenge & HellaSwag benchmarks.\n\n\n**Weaknesses**\n\n- Practical Computational Cost: The paper focuses on parameter efficiency, but I think the practical computational cost is an important consideration. The authors mentioned that the current implementation relies on dense matrix representations and does not yet exploit the sparsity of the rotor decomposition.", "was_revised": true, "final_justification": "The rebuttal addressed some of my questions and I already have a positive score.", "rebuttal": "Thank you for the insightful question and positive feedback! We answer it below. \n\n**Q: Can the authors add more on the runtime of their proposed method and does it compare to the baselines?**\n\nCurrently, rotor layers are slower than dense layers in wall-clock inference runtime due to the overhead of the unoptimized Clifford algebra operations. Inference is about $(\\text{width} \\times \\text{depth})/4$ times slower, where width and depth of our rotor gadget are at most 3 and 2 in our experiments with minimal use of the structure of our matrices.\n\n**We believe we could not only close this gap, but rotor layers may outperform dense layers**, and here is why. The primary goal of our work was to *synthesize dense linear layers from exponentially fewer geometric primitives.* **The real opportunity unlocked by our geometric compotition lies in eliminating memory bandwidth bottlenecks**, as also described in our answer to Reviewer tqhq. NVIDIA Research's recent LIMINAL paper (arXiv:2507.14397) identifies three critical insights for LLM inference:\n\n1) Memory capacity requirements are enormous (385GB+ for Llama-405B)\n2) Memory bandwidth is the primary bottleneck for user tokens/second\n3) DRAM-based architectures offer fundamental efficiency advantages\n\nThe reviewer will see that **our approach directly addresses finding #2**: rather than reading in millions of parameters from memory to GPU (the dominant cost in LLM inference), we can, with enough engineering, generate these weights on-chip by reading only ~1000 geometric parameters and constructing rotors (see Figure 3) on the fly! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary constraint as noted in the LIMINAL work and other industry reports. \n\nAs an aside, this decomposition reminds us of the Fast Fourier Transform: factoring the DFT matrix didn't immediately reduce runtime, yet this compositional insight later enabled $O(n \\log n)$ algorithms compared to $O(n^2)$. Likewise, our construction reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest, allowing for potential new algorithms to take advantage.\n\nIn summary, this direction is highly promising, particularly for scaling up inference efficiency in large models. However, realizing the full benefit would require substantial engineering work such as implementing sparse-aware Clifford libraries and optimizing GPU memory layouts, which is beyond the scope of this paper. For these reasons, we do not compare wall-clock runtime to other baselines."}, {"review_id": "bVM2FSxzzd", "reviewer": "Reviewer_dZ69", "summary": "This paper presents a method for synthesizing linear layers from a small number of geometric primitives (irreducibles) encoding rotations. \nFor this, the authors utilize the framework of Clifford (geometric) algebra and introduce a differentiable algorithm that\ndecomposes them into products of rotors. \nThe proposed construction, in which linear layers can be expressed as compositions of bivectors, uses only $\\mathcal{O}(\\log^2 d)$ parameters, significantly reducing the number required by dense matrices — $\\mathcal{O}(d^2)$, with $d$ being the input/output dimension.\nTo demonstrate the effectiveness of the method, the authors use their rotor-based construction and substitute the K, Q, and V dense matrices in LLM attention layers, exhibiting comparable performance (in accuracy and perplexity) to strong baselines such as block-Hadamard and low-rank approximations, across different datasets.\nOverall, the paper demonstrates that bivectors in Clifford algebra can effectively compose linear layers.", "questions": "Q1. Pertaining to W1 and with the current implementation, how does the computational speed (both training and inference) of the proposed bivector method compare to the related methods (Dense, BH, LR)?\n\nQ2. With regards to W2, could the authors run a simple experiment, e.g., an MNIST classification with a feed-forward model, using their layer construction method and the baseline/related ones, training the model from scratch (as opposed to using a pre-trained LLM)?\n\nQ3. Suggestion: It would improve the paper if the limitations were stated in the main paper, or at least, if it were mentioned that they are given in the Appendix.\n\nThe final rating depends on the author’s rebuttal.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "First of all, I appreciate seeing the powerful toolbox of Clifford algebra being further explored in neural processing. The strengths of this paper are pretty straightforward to identify:\n\nS1. The paper is well-written. The storyline is clear. The claims are supported by experiments in a well-defined scope and framework.\n\nS2. Novelty and significance: The idea of expressing linear transformations as compositions of bivectors using sandwich products results in an exponential reduction of the required number of parameters for linear layers.\n\nS3. Compatibility: The proposed differentiable invariant decomposition algorithm enables integration with autograd and gradient-based optimization.\n\n----\nAs for the weaknesses, I have identified the following, despite the formulated scope and goals of the experiments and the current implementation limitations clearly stated by the authors:\n\nW1. No comparison of the computational time of the related methods (Dense, BH, LR) is presented.\n\nW2. No experiment in which the proposed bivector-composed linear layers are learned in a network end-to-end, along with a computational cost comparison against dense-matrix linear layers. Such an experiment would serve the paper well in that it would help a broader audience of readers to understand better the scope of the current limitations and the applicability.\n\nW3. Minor: \n- Currently, this work is more of a feasibility study.\n- Equation 3 is missing its number; besides, all the equations should be numbered for consistency.", "was_revised": true, "final_justification": "My questions are well answered, and I will maintain the original score. I strongly recommend that the authors update their manuscript with the responses provided in the rebuttal, to the extent possible. This includes the FMNIST experiment.", "rebuttal": "We thank the reviewer for taking the time to carefully engage with our work, and for the thoughtful suggestions and questions. We answer each question below.\n\n**Q1: Pertaining to W1 and with the current implementation, how does the computational speed (both training and inference) of the proposed bivector method compare to the related methods (Dense, BH, LR)?**\n\nCurrently, rotor layers are slower than dense layers in wall-clock inference runtime due to the overhead of the unoptimized Clifford algebra operations. Inference is about $(\\text{width} \\times \\text{depth})/4$ times slower, where width and depth of our rotor gadget are at most 3 and 2 in our experiments. Training has the extra cost of storing the gradients, but the backpropagation through decomposition (see Figure 3) from applied weights to actual parameters, which involves $\\log_2 d$ block-sparse matmuls, where $d$ is the model dimension, can be done on a different GPU while backpropagation proceeds through the rest of the network.\n\n**We believe we could not only close this gap, but rotor layers may outperform dense layers**, and here is why. The primary goal of our work was to *synthesize dense linear layers from exponentially fewer geometric primitives.* **The real opportunity unlocked by our geometric compotition lies in eliminating memory bandwidth bottlenecks**, as also described in our answer to Reviewer tqhq. NVIDIA Research's recent LIMINAL paper (arXiv:2507.14397) identifies three critical insights for LLM inference:\n\n1) Memory capacity requirements are enormous (385GB+ for Llama-405B)\n2) Memory bandwidth is the primary bottleneck for user tokens/second\n3) DRAM-based architectures offer fundamental efficiency advantages\n\nThe reviewer will see that **our approach directly addresses finding #2**: rather than reading in millions of parameters from memory to GPU (the dominant cost in LLM inference), we can, with enough engineering, generate these weights on-chip by reading only ~1000 geometric parameters and constructing rotors (see Figure 3) on the fly! This trades compute for memory bandwidth—an extremely favorable exchange given that memory, not compute, is the primary constraint as noted in the LIMINAL work and other industry reports. \n\nAs an aside, this decomposition reminds us of the Fast Fourier Transform: factoring the DFT matrix didn't immediately reduce runtime, yet this compositional insight later enabled $O(n \\log n)$ algorithms compared to $O(n^2)$. Likewise, our construction reveals that dense linear layers have far fewer degrees of freedom than their raw parameter counts suggest, allowing for potential new algorithms to take advantage.\n\nIn summary, this direction is highly promising, particularly for scaling up inference efficiency in large models. However, realizing the full benefit would require substantial engineering work such as implementing sparse-aware Clifford libraries and optimizing GPU memory layouts, which is beyond the scope of this paper. For these reasons, we do not compare wall-clock runtime to other baselines.\n\n**Q2: With regards to W2, could the authors run a simple experiment, e.g., an MNIST classification with a feed-forward model, using their layer construction method and the baseline/related ones, training the model from scratch (as opposed to using a pre-trained LLM)?**\n\nYes, thank you for this excellent suggestion! \n\nWe ran additional expeirments with an MLP architecture on FMNIST, where we replace all dense layers (except the final classification head) with rotor layers and trained both end-to-end from scratch. We performed a hyperparameter search for both and trained each for the same number of epochs. In this setting, the conventional dense layers and rotor layers achieved 89.67% and 88.36% accuracy respectively.\n\nAs you noted, full end-to-end training on large models like LLMs is currently infeasible due to limitations in Clifford algebra libraries and their hardware integration. These results on FMNIST suggest that rotor layers can be competitive with dense layers when trained from scratch, giving us encouraging evidence for scalability with more engineering effort.\n\n**Suggestion: It would improve the paper if the limitations were stated in the main paper, or at least, if it were mentioned that they are given in the Appendix.**\n\nWe have now added a reference to this section in the main paper’s conclusion. Thanks for the suggestion!"}]

Author rebuttal:

Meta-review: The authors present a method for parameterizing linear layers in neural networks in terms of compositions of geometric objects from Clifford algebra known as bivectors, and an algorithmic approach for decomposing these into simple factors in a differentiable way. The approach leads to an effective reduction in the number of parameters relative to the $d \times d$ baseline of a dense matrix, and empirical results in which a rotor-based decomposition is done to approximate the K, Q, and V dense matrices in a subset of LLM attention layers demonstrates comparable performance (in accuracy and perplexity) to strong baselines such as block-Hadamard and low-rank approximations, across different datasets. Initial reviews of the paper were positive -- authors were appreciative of the clear presentation, the convincing experimental results, and the potential for the technique to improve interpretability -- and increased further after clarifications and additional results during the rebuttal and discussion phase. I concur with the reviewers and recommend acceptance. The authors should carefully incorporate feedback from the reviewers during the rebuttal into the final revision, especially the promised additional experiment.
Decision: Accept (poster)