Title: Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: recurrent neural networks

bi-linear models

state tracking

length generalization
Abstract: The role of hidden units in recurrent neural networks is typically seen as modeling memory, with research focusing on enhancing information retention through gating mechanisms. A less explored perspective views hidden units as active participants in the computation performed by the network, rather than passive memory stores. In this work, we revisit bilinear operations, which involve multiplicative interactions between hidden units and input embeddings. We demonstrate theoretically and empirically that they constitute a natural inductive bias for representing the evolution of hidden states in state tracking tasks. These are the simplest type of tasks that require hidden units to actively contribute to the behavior of the network. We also show that bilinear state updates form a natural hierarchy corresponding to state tracking tasks of increasing complexity, with popular linear recurrent networks such as Mamba residing at the lowest-complexity center of that hierarchy.


Peer reviews:
comments: SUMMARY:
This paper analyzes bi-linear operations in RNNs (where hidden states and inputs interact multiplicatively) in the context of state tracking tasks. By parameterizing the hidden state matrices of SSMs in bilinear form, this theoretically show how to solve the state tracking task. The paper also introduces a hierarchy of task complexity, with simple linear RNNs (like Mamba) at the base.

STRENGTHS AND WEAKNESSES:
The paper is very well-written, and the motivation is clear. I learned a lot reading this work! 

One weakness is the lack of experiments on "real" benchmark sequential processing tasks, like those found in the [Long Range Arena](https://arxiv.org/abs/2011.04006).

QUESTIONS:
1. How do bi-linear RNNs compare to the other RNNs in Figure 1 on "real" sequential processing tasks? E.g., sequential MNIST? 
2. Is there any connection to associative memory models, which also have multiplicative connections (albeit between hidden units, and not between inputs). For example, see Figure 1 [here](https://arxiv.org/pdf/2008.06996). There is also a (very) recent [paper](https://www.science.org/doi/epdf/10.1126/sciadv.adu6991) which contains multiplicative interactions between hidden states and external inputs. It would be interesting to know if there is some theoretical connection between bi-linear RNNs and associative memory.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the properties of bilinear RNNs, where state transitions are governed by a tensor that is contracted against both the current state and the current input. They prove limitations on the state-tracking capabilities of several variants of standard SSMs and linear RNNs on state tracking tasks, and further prove that bilinear RNNs can simulate any finite state machine. Finally, they conduct a series of experiments on parity, modular arithmetic, and simulating random state machines to demonstrate that bilinear RNNs with more practical parameterizations (in terms of parameter efficiency) perform well on state tracking tasks when compared to other architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well-written, and to the best of my knowledge, clarifies important limits on the expressivity of various parameterizations of linear RNNs on state-tracking tasks. The analysis appears mathematically tight, and covers a wide range of interesting subproblems in state tracking (commutative vs noncommutative tasks, parity, automata). \n\nWeaknesses:\n\nWhile the authors do address the parameter-inefficient nature of bilinear RNNs by considering low-rank or factorized parameterizations, their experiments do not compare different model classes in a parameter-matched way; instead, hidden dimension is held fixed, giving a potentially massive parameter count advantage to the bilinear RNNs. A more complete set of experiments comparing bilinear RNNs and their factorized variants to other architectures with conserved parameter count would be needed to understand whether bilinear RNNs are actually practically useful on state-tracking tasks, and would be a useful addition to round out the paper regardless of what the results end up showing. I am open to bumping up my score to \"Accept\" if this can be done.\n\nQUESTIONS:\n- Can the authors comment further on why including biases degrades performance of bilinear RNNs? Why can't the model learn to simply ignore the biases, and why does this lead to instabilities?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on bi-linear RNNs where the transition matrix is a bi-linear function (a multiplicative interaction) of current input and hidden activations of the previous time-step. The paper showed that bi-linear RNNs are highly effective at learning state tracking tasks by mainly removing any additive components, s.t. the hidden state is a true bi-linear (not affine function) of the previous time-step hidden state and input. This leads to a scale-invariance which allows for keeping hidden activations stable during training and inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Bi-linear RNNs showed OOD generalization on synthetic state-tracking tasks \n- The models show similar sample efficiency to LSTM\n\nWeaknesses:\n\nSeveral theoretical, conceptual, and experimental similarities with prior art (A. Terzic, et al., \"On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages\", AAAI 2025) have not been mentioned in the paper. These should be clarified and elaborated:\n\n> We show that linear RNNs with diagonal transition matrices are a special case limited to learning state tracking tasks with commutative structure. This restriction is true even for complex-valued diagonal transition matrices. Hence, linear RNNs with block-diagonal transition matrices of size 2 × 2 are not able to learn general state machines (negative result).\n\nTheoretically, it has also been shown in the prior art that under a described mapping of FSA to a single-layer selective (i.e., input-dependent) state-space model (SSM) whose transition matrices are simultaneously diagonalizable, the selective SSM can only emulate commutative automata.\n\nMoreover, the prior art had already noted that models that do not utilize the B matrix (i.e., additive interactions) tend to learn solutions that exhibit better length generalization than their counterparts (which include the B matrix).\n\nConceptually, the proposed state transition matrix A is fully parametrized through a linear transformation of the input x. In the prior art, the A matrix is a convex combination of a set of transition matrices based on the current input.\n\nExperimentally, a single recurrent cell is followed by a linear classification head over the hidden states. This is also similar to what has been used in the prior art, with further ablation that shows the inefficiency of non-linear readouts.\n\nQUESTIONS:\nQ1) The empirical evaluations in Table 1 are limited to only three synthetic tasks. How about the non-commutative automata with solvable transformation semigroups, such as the dihedral group? A pointer for these tasks: B. Liu, et al. \"Transformers Learn Shortcuts to Automata\", ICLR 2023.\n\nQ2) In Table 1, what is the reason for the drop in the OOD accuracy of RNNs as the Module increases? RNNs have sufficient expressiveness to learn any regular language.\n\nQ3) The best-performing model is reported (out of three using different learning rates). It is also good to report the min-max, or average-max, performance across models.\n\nQ4) Table 2 (right) shows the detrimental effect of additive terms on the parity automata, which is consistent with the prior art. How about benchmarking when learning a non-commutative automaton (e.g., D_30)? Could the model provide better length generalization once the B matrix is introduced? See Table 4 of the prior art.\n  \nQ5) Is there any sample efficiency comparison with RNNs in Figure 2?\n\nQ6) What are the potential limitations and implications of applying this method towards making LLMs on real-world datasets?\n\nLIMITATIONS:\nWhat are the potential limitations and implications of applying this method towards making LLMs on real-world datasets?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper investigates the expressivity of bi-linear recurrent models theoretically and empirically by studying their performance on three formal language tasks: modular addition, random state machine, and modular arithmetic.
The paper investigates different factorizations and parameterizations of the state transition tensor.
In the experiments bi-linear recurrent neural networks perform best on the state tracking tasks (with LSTM being close).

STRENGTHS AND WEAKNESSES:
Strengths:
- Very well written, clear and easy to understand
- Table 1: Good performance of Bilinear models (even though LSTM is close) 

Weaknesses:
- The paper mentions that the stability problems of bi-linear recurrent models were the main reason for them not being used, and seems to solve this issue by dropping any additive terms in the state update. However, further (details on) experiments on this issue are missing.
- There are no „real world“ experiments, e.g. on language or vision. It could be at least explained if these models are applicable to a more large scale training setup?

QUESTIONS:
- What are the exact parameter counts of the models? Since it is mentioned in the discussion section that the parameter count is larger due to the cubic scaling with embedding dim, it would make this point more clear if the parameter counts are added to Table 1, too.
- L 271: Why do you need to normalize during inference and not during training?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01