Title: Composing Linear Layers from Irreducibles
Venue: NeurIPS 2025
Accepted: yes
Keywords: Clifford Algebra

Compositions

Modular
Abstract: Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. 
We investigate this compositional structure in linear layers by asking: \textit{can we identify/synthesize linear transformations from a minimal set of geometric primitives?} 
Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors---geometric objects encoding oriented planes---and introduce a differentiable algorithm that decomposes them into products of rotors. 
This construction uses only $\mathcal{O}(\log^2 d)$ parameters, versus $\mathcal{O}(d^2)$ required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for synthesizing linear layers from a small number of geometric primitives (irreducibles) encoding rotations. \nFor this, the authors utilize the framework of Clifford (geometric) algebra and introduce a differentiable algorithm that\ndecomposes them into products of rotors. \nThe proposed construction, in which linear layers can be expressed as compositions of bivectors, uses only $\\mathcal{O}(\\log^2 d)$ parameters, significantly reducing the number required by dense matrices — $\\mathcal{O}(d^2)$, with $d$ being the input/output dimension.\nTo demonstrate the effectiveness of the method, the authors use their rotor-based construction and substitute the K, Q, and V dense matrices in LLM attention layers, exhibiting comparable performance (in accuracy and perplexity) to strong baselines such as block-Hadamard and low-rank approximations, across different datasets.\nOverall, the paper demonstrates that bivectors in Clifford algebra can effectively compose linear layers.\n\nSTRENGTHS AND WEAKNESSES:\nFirst of all, I appreciate seeing the powerful toolbox of Clifford algebra being further explored in neural processing. The strengths of this paper are pretty straightforward to identify:\n\nS1. The paper is well-written. The storyline is clear. The claims are supported by experiments in a well-defined scope and framework.\n\nS2. Novelty and significance: The idea of expressing linear transformations as compositions of bivectors using sandwich products results in an exponential reduction of the required number of parameters for linear layers.\n\nS3. Compatibility: The proposed differentiable invariant decomposition algorithm enables integration with autograd and gradient-based optimization.\n\n----\nAs for the weaknesses, I have identified the following, despite the formulated scope and goals of the experiments and the current implementation limitations clearly stated by the authors:\n\nW1. No comparison of the computational time of the related methods (Dense, BH, LR) is presented.\n\nW2. No experiment in which the proposed bivector-composed linear layers are learned in a network end-to-end, along with a computational cost comparison against dense-matrix linear layers. Such an experiment would serve the paper well in that it would help a broader audience of readers to understand better the scope of the current limitations and the applicability.\n\nW3. Minor: \n- Currently, this work is more of a feasibility study.\n- Equation 3 is missing its number; besides, all the equations should be numbered for consistency.\n\nQUESTIONS:\nQ1. Pertaining to W1 and with the current implementation, how does the computational speed (both training and inference) of the proposed bivector method compare to the related methods (Dense, BH, LR)?\n\nQ2. With regards to W2, could the authors run a simple experiment, e.g., an MNIST classification with a feed-forward model, using their layer construction method and the baseline/related ones, training the model from scratch (as opposed to using a pre-trained LLM)?\n\nQ3. Suggestion: It would improve the paper if the limitations were stated in the main paper, or at least, if it were mentioned that they are given in the Appendix.\n\nThe final rating depends on the author’s rebuttal.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

comments: SUMMARY:
The paper investigated how linear transformations in neural networks and LLMs can be synthesized from geometric primitives via Clifford algebra. 
The authors showed that linear layers can be represented as compositions of rotors, which are parameterized by bivectors (geometric objects encoding oriented planes).
They introduce a differentiable invariant decomposition algorithm that breaks general bivectors into another components with closed-form exponential. The authors replaced the key, query, and value projection layers in the attention mechanism of two LLMs with their proposed layers to evaluate their method.

STRENGTHS AND WEAKNESSES:
**Strengths**:

- Using Clifford algebra to represent linear layer: The paper provided an algebraic lens on linear layers, showing that they admit a compact factorization into bivector irreducibles and rotors, with less number of parameters. 
- Empirical results: The authors showed that replacing the key, query, and value projections in LLaMA-3.2 1B and Qwen-2.5 1.5B with rotor-based layers achieved competitive performance aganist low-rank and block-Hadamard baselines on Wikitext2, C4, and PTB datasets and Arc Challenge & HellaSwag benchmarks.


**Weaknesses**

- Practical Computational Cost: The paper focuses on parameter efficiency, but I think the practical computational cost is an important consideration. The authors mentioned that the current implementation relies on dense matrix representations and does not yet exploit the sparsity of the rotor decomposition.

QUESTIONS:
Can the authors add more on the runtime of their proposed method and does it compare to the baselines?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper posits a way to rewrite linear transformations in deep learning frameworks in a nonlinear algebraic form with fewer parameters, based on composition of simple primitives. These primitives are based on bivector products in Clifford algebra. The expression is differentiable via chain rule and it applies to LLMs (on which experiments are done to compare to other model compression algorithms), and the compression goes from scaling as d^2 of number of learnable parameters to that number scaling like log^2d in which d is the layer dimension.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the method seems interesting and it performs well compared to the presented alternatives, therefore it is a good advance as far as I could check. \n\nI feel that possible limitations of the method are not discussed, which can mean two things: either this is a cornerstone paper which will revolutionize LLM and general model compression, or this lack of discussion of limitations is a weakness of the paper. Unfortunately I can't tell which one is the case, as I have never worked with Clifford algebra and thus this is just marginal to my domain of expertise\n\nQUESTIONS:\nI'm impressed by the experimental results, such as 4700x reduction of number of parameters, and I would like to understand possible limitations of the method. So I try to ask some questions in that direction below\n\n1) In theorem 2 and in the LLM layer experiments you have a number of parameters going down by a huge factor. This means that the models become highly compressed by this approach. Does it mean that the learning rate on the compressed models become comparatively slower as the parameters get more entangled? Or else, how do the learning rates compare?\n\n2) How do you explain the fact that your rotor based composition works well with 1 or 2 layers in experiments but then stops improving? (as mentioned in line 318 and following ones)\n\n3) Do you think that the composition of primitives as given by your result can be interpretable in some setting (maybe some LLM case) or in some toy example? You say in line 349-350 that \"Our goal is to understand the compositional structure of the linear transformation itself\" so can you please state in more detail what do we understand now with your method precisely?\n\n4) Do you know, besides drop in parameters, what is a drop in Kolmogorov Complexity incurred during the learning dynamics by your method compared to others? I'd be curious to know how much information is lost due to gradient descent not being able to fit the models, and I'm wondering if there is a metric or a toy case in which you can give more insights on that, besides computing the accuracy or PPL as you do.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that replacing linear layers with clifford algebra layers can improve interpretability while maintaining comparable performance and reducing parameter count. To do this, the authors use an existing theoretical result stating that any linear function can be written as a sandwich product with clifford algebra multivectors. Naively designing an architecture based on this result would introduce a huge computational burden. Instead, the authors claim any linear function can be written as the composition of rotors which have many fewer parameters. While this theoretical result is not proved in the main body, it is empirically validated in the case where 1-3 attention layers in a small LLM are replaced with rotor layers.\n\nSTRENGTHS AND WEAKNESSES:\nThe submission appears technically sound, with the exception that a main claim (any linear function can be written as a sandwich product with rotors) is not proved. The claim is, however, supported in a narrow context by experimental results.\nThe submission is very well written and organized.\nThe results appear significant, potentially providing a mechanism to improve interpretability in LLMs.\nThe work is part of a growing body of research leveraging clifford algebra representations for interpretability and expressivity in neural networks. As far as I know, the insights presented in this work are original and seem to expand our understanding of the utility of clifford algebra representations.\n\nQUESTIONS:\n*Questions*\n- (line 111) It isn’t clear to me how choosing to consider the subalgebra Spin(n) still allows you to represent a general, arbitrary linear map.\n- (line 256) Do the authors expect that performance would remain comparable if all layers were replaced with clifford algebra layers?\n- (line 287) Are the clifford algebra layers harder to train than conventional linear layers? Are they more computationally expensive?\n\n*Comments*\n- It would be easier to read Table 2 if the best performing methods were highlighted as they are in Table 1\n\nLIMITATIONS:\nThe authors do not acknowledge limitations or potential negative societal impacts of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}