Title: DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Venue: NeurIPS 2025
Accepted: yes
Keywords: linear rnn

linear attention

state-space models

state-tracking

negative eigenvalues

householder

formal languages

regular languages

stability

finite-state automata
Abstract: Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference. However, existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices. Diagonal matrices, used in models such as Mamba, GLA, or mLSTM, yield fast runtime but have limited expressivity. To address this, recent architectures such as DeltaNet and RWKV-7 adopted a diagonal plus rank-1 structure, which allows simultaneous token and channel mixing, 
improving associative recall and, as recently shown, state-tracking when
allowing negative eigenvalues in the state-transition matrices.
Building on the interpretation of DeltaNet's recurrence as performing one step of online gradient descent per token on an associative recall loss, we introduce DeltaProduct, which instead takes multiple ($n_h$) steps per token. This naturally leads to diagonal plus rank-$n_h$ state-transition matrices, formed as products of $n_h$ generalized Householder transformations, providing a tunable mechanism to balance expressivity and efficiency. 
We provide a detailed theoretical characterization of the state-tracking capability of DeltaProduct in finite precision and how it improves by increasing $n_h$.
Our extensive experiments demonstrate that DeltaProduct outperforms DeltaNet in both state-tracking and language modeling, while also showing significantly improved length extrapolation capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the fundamental trade-off between expressiveness and computational efficiency in State Space Models (SSMs). While diagonal SSM blocks are computationally tractable but limited in expressive power, and unstructured transition matrices offer full expressiveness at prohibitive computational costs, the authors propose a novel middle-ground approach. Their solution involves taking several rank-one steps for each token, effectively using low-rank transition matrices with rank equal to the number of steps.\n\nThe paper provides theoretical results proving the well-posedness and expressive power of the new SSM on state tracking tasks, supported by extensive experiments comparing against state-of-the-art competing architectures.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. **Thorough Theoretical Foundation**: The paper provides comprehensive theoretical backing for the proposed approach, with formal proofs of well-posedness and expressive power.\n    \n2. **Extensive Experimental Validation**: The experimental section offers comprehensive comparisons with state-of-the-art architectures.\n    \n3. **Clear Problem Motivation**: The paper tackles an important and well-defined problem in the SSM literature with a principled approach.\n    \n4. **Transparency**: The authors are transparent about both the limitations and strengths of their approach.\n    \n\n## Major Weaknesses\n\n1. **Errors in Main Proofs**: There are errors in key theoretical results that need correction, though these appear to be fixable without changing the main conclusions.\n    \n2. **Inconsistent Proof Quality**: Some proofs are hastily written and inconsistent with the rest of the paper, making them very difficult to follow and verify.\n    \n3. **Experimental Fairness Concerns**: The use of identical hyperparameters and training setups across different models without justification raises questions about fair comparison.\n\nQUESTIONS:\n### Notation and Consistency Issues\n\n- **Line 97**: The definition \"yi = x1 · x2 · · · xi\" is inconsistent with parts of the appendix where \"yi = xi ... x1\". For non-abelian products, this distinction is crucial. Please maintain consistency or clearly indicate when different conventions are used.\n\n### Direct Parametrization Questions\n\n- **Lines 184-188**: You have a couple of lines on direct parametrization of the low-rank structure. It is true that in the presented case (line 188) the matrix is restricted to be symmetric, but is there something blocking you from choosing the second ki to be different from the first? In that case there would not be any imposed symmetry. Is there some reason why you did not consider this?\n\n## Appendix Issues\n\n### Proposition 1\n\n- **Line 1034 (CRITICAL)**: In computing D explicitly, you appear to have omitted the term -4det(A). Fortunately, since (2-b1-b2)² - 4(1-b1)(1-b2) = (b1-b2)², the conclusion still holds. With this correction, you could derive explicit bounds on cos(θ).\n\n### Unclear Statements and Missing Information\n\n- **Line 1064 (Pedantic)**: Can this really model ANY product of Householders, or do restrictions on operator norm apply? This may be inconsistent with later comments.\n- **Line 1069**: \"The assumption on H0\" - which specific assumption are you referring to?\n- **Line 1073**: In the RMS formula, the xi terms need to be squared.\n- **Lines 1078-1082**: Please clarify this section as it's currently difficult to follow.\n- **Line 1084**: \"H_t^j = ei\" - the indices i and j are not quantified. Are they related? Can they be any possible tuple?\n\n### Theorem 3 Issues\n\n- **Lines 1117-1119**: Please expand for clarity. This is where the definition of m is crucially used as $m \\cdot n_h \\geq n - 1$.\n    \n- **Lines 1120-1127 (CRITICAL)**: This section requires substantial clarification as it's a crucial part of the proof but appears hastily written:\n    \n    - Line 1120: What do you mean by \"with enough information in xi about previous tokens\"?\n    - Lines 1121-1122: Please place this within your established framework (A, B, k, v, etc.). Why is x bold now? What are the matrices \"P_{lm+i}\" - these haven't been defined. Aren't the \"G_{l,m}...G_{l,i+1}\" functions on tokens coming after the current one? How is this information encoded in the tuple (Hi,xi)?\n- **Line 1128**: The 'a' should not be bold here.\n### Lemma 2 Concerns\n\n- **Line 1141**: Use clearer set notation: write \"2-d ≤ i ≤ 0\" instead of \"i ∈ {2-d, ..., 0}\".\n- **Line 1142**: Where does the bold 'a' come from? Is this the tuple (t mod d, a_{t-d+1}, ..., at)?\n- **Line 1145**: There is no j>1 since nh = 1.\n- **Line 1146 (Important)**: While claiming that the system can learn any function on the specified domain is technically correct, it seems somewhat disingenuous. What you actually prove is that your recurrence can memorize input up to a certain time, while learning is performed by a universal approximator in the decoder (e.g., MLP). Do you believe this occurs in practice, or can your system perform meaningful computation independently of decoder universality? This deserves discussion.\n\n### Theorem 4 Issues\n\n- **Line 1152**: Which specific isomorphism are you referring to?\n- **Line 1153**: \"that\" appears to be a typo.\n- **Lines 1159-1161**: Please cite this result - it's unlikely the intended audience is familiar with it.\n- **Lines 1162-1164**: Why recover y only at the last step? Shouldn't it be recovered at all steps according to your state tracking problem definition?\n\n### Lemma 3\n\n- **Line 1198**: \"exist\" should be \"exists\".\n- **Lines 1215-1229**: Same critical issues as found in Theorem 3.\n\n### Theorem 6\n\n- **Lines 1247-1248**: There are two (ii) labels and no (i); line 1256 should be (i).\n- **Line 1252**: Consistency issue with \"yi = x1 · x2 · · · xi\" as discussed above.\n- **Line 1265**: Is the angle clockwise or counterclockwise? To be pedantic the line passes from the origin to the POINT (1,0), this line in some sense is the vector (1,0) itself.\n- **Lines 1268-1269**: Mismatch between j and i indices in the last two θ expressions.\n- **Line 1270 (Important)**: When referencing \"standard identities of products of 2D rotations and reflections,\" clarify that these are matrix representations of the group as given in equation (5). You should consider placing this paragraph (1270-1275) at the beginning for clarity, then introduce the system. In general I believe this theorem lacks a paragraph giving intuition, explaining what d_0 and c_0 are supposed to represent, etc.\n- **Lines 1271-1272**: First line - d0 is not bold. Second line - need to prove \"H(π/m)d0 = R(π/m)d0\"; avoid ambiguous ± notation.\n- **Lines 1273-1274**: \"d0 = H(π/m)c0\" appears true but unproven. \"R(-2iπ/m)d0 = H(-2iπ/m)d0\" is not proven.\n- **Lines 1274-1275 (Critical Error)**: There appears to be an error in the last line. θ(sj, 0) - θ(s{m-i}, 0) = (-2m + 2(i+j))π/m since (2+2j)-(2+2(m-i)) = 2j - 2m + 2i = -2m + 2(i+j). The expression in R should be (-2m + 2(i+j))π/m instead of (2 + 2(i+j))π/m.\n\n### Additional Issues\n\n- **Line 1292 (Pedantic)**: \"When this property is not satisfied, the norm of the state will diverge exponentially fast to infinity.\" Does this not depend on how fast the norms diverge?\n- **Lines 1375-1377 and 1407-1411**: Using identical hyperparameters for different models without justification raises fairness concerns. How do you ensure fair comparison?\n\n\n## Recommendation\n\nThis paper tackles an important problem with substantial theoretical and empirical evidence supporting it. The approach is well-motivated and the experimental work is comprehensive. However, the seeming mistakes in the main proofs (particularly in Proposition 1, where it is not a typo) and unclear arguments in Theorem 3 and Lemma 3 prevent acceptance of the paper in its current form.\n\n**Path to Acceptance**: Please correct the identified errors in the proofs (or prove me wrong) and clarify the murky arguments in Theorems 3 and Lemma 3. With these corrections, I would raise my recommendation to at least a 5, as the underlying contribution is valuable and the work is otherwise very well-executed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDeltaProduct extends DeltaNet by performing rank-n updates instead of rank-1, yielding state-transition matrices as products of generalized Householder transformations. This creates a principled mechanism to interpolate between diagonal matrices (efficient) and more expressive transformations while maintaining stability through an operator norm less than 1. The authors prove that DeltaProduct can solve symmetric group word problems with improved trade-offs and that Gated DeltaProduct recognizes any regular language with finite layers. \n\nExperimentally, DeltaProduct demonstrates superior performance on state-tracking tasks involving permutation groups and achieves dramatically improved length extrapolation in language modeling compared to DeltaNet, with minimal performance degradation on sequences beyond training context. The effective rank analysis reveals that multiple Householder products enable more efficient state forgetting mechanisms. The work provides both theoretical insights and practical improvements in synthetic reasoning and real-world language modeling tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper demonstrates exceptional scientific rigor with a clear, meaningful contribution that advances linear RNN research. The theoretical foundation is mathematically sound yet accessible through intuitive geometric explanations. The experimental validation is comprehensive, effectively combining synthetic state-tracking tasks with realistic language modeling benchmarks at an appropriate scale. The work exemplifies proper incremental research, extending DeltaNet in a principled direction that yields genuine improvements. The related work section is particularly thorough and well-organized, while the authors' commitment to reproducibility strengthens the work's impact. I consider the presentation quality to be outstanding throughout.\n\nQUESTIONS:\nI'm curious about a FLOPs comparison in the scaling analysis in Figure 10. To match parameters, the number of heads was reduced, but how do these variants compare w.r.t. FLOPs?\n\nDid DeltaProduct display any training instabilities? \n\nIn Figure 5 bottom 2nd and 4th plots, an increased number of layers seems to hurt performance on the state tracking tasks. You addressed it, but isn't this unexpected? Is there an explanation? Also, the number of layers is not consistent in that row, which reduces the readability of the figure as a whole.\n\nIn Figure 9, the Gated DeltaNet effective ranks are extremely jagged. Why is this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DeltaProduct, a novel linear Recurrent Neural Network (RNN) architecture that improves upon existing models like DeltaNet by enhancing state-tracking and expressivity. While DeltaNet's recurrence is interpreted as a single online gradient descent step per token, DeltaProduct performs multiple (nh) steps, resulting in a more expressive state-transition matrix formed by the product of nh generalized Householder transformations. This creates a diagonal plus rank-nh matrix structure, providing a tunable mechanism to balance expressivity with computational efficiency. The authors provide theoretical analysis showing that increasing nh improves the model's capacity for complex state-tracking, such as solving group word problems. Through extensive experiments, they demonstrate that DeltaProduct outperforms DeltaNet in both state-tracking and language modeling tasks, and exhibits significantly improved length extrapolation capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :-\n\n- The paper constructs complex, non-symmetric state-transition matrices by composing products of simple, symmetric Householder reflections. This allows the model to learn rotational and oscillatory dynamics, crucial for advanced state-tracking, while mathematically guaranteeing stability since the operator norm remains bounded. \n- The paper demonstrates nice scientific rigor by not just showing that the model solves a task (S4 permutations), but by verifying how it solves it. Through model introspection, it provides strong evidence that the network learns the underlying geometric structure of the problem (the isomorphism to a cube's rotation group) exactly as predicted by theory.\n\nWeaknesses :-\n\n- The model's core design using Householder products enforces a strict norm ||A(xi)|| <= 1 on its state-transition matrix. While this guarantees stability, it is also a fundamental expressive bottleneck that prevents the model from performing norm-increasing operations, which are crucial for amplifying signals and are a key reason why competing models like RWKV-7 achieve stronger theoretical guarantees.\n\n- Increasing the model's expressivity via the hyperparameter nh (the number of Householder products) is inefficient. It leads to a linear growth in both model parameters and, more critically, the sequential computation time of the recurrence, making it nh times slower per token. This architectural choice limits the practical scalability of using a large nh to achieve higher performance.\n\n- The number of operations, nh, is a fixed, global hyperparameter. The model applies the same amount of costly computation to every token, regardless of its importance. This is wasteful for simple tokens (e.g., punctuation) and may be insufficient for complex ones (e.g., resolving a long-range dependency), representing a missed opportunity for adaptive computation that could allocate resources more efficiently.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}