Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling
Authors: Mónika Farsang
Radu Grosu
Venue: NeurIPS 2025 poster ()
Keywords: state space model
non-linear RNN
liquid neural network
parallelization
sequence modeling
Primary area: deep_learning
TL;DR: LrcSSM is a nonlinear RNN with parallel prefix-scan updates, formal gradient stability, and compute-optimal scaling. It outperforms LRU, S5, and Mamba on long-range tasks - matching SSM speed without attention or FFT overhead.

Abstract: We present LrcSSM, a $\textit{non-linear}$ recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel, giving $\mathcal{O}(TD)$ computational work and memory and only $\mathcal{O}(\log T)$ sequential depth, for input-sequence length $T$ and a state dimension $D$. Moreover, LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 and Mamba do not provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized to other non-linear recurrent models, demonstrating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba.

Parallelization Non-iinear State-Space Models: Scaling Up Liquid-Resistane Liquid-Capacitace Networks for Efficient Sequence Modeling
Mónika Farsang
Radu Grosu TU Wien Vienna, Austria
TU Wien Vienna, Austria farkan..ars
radu. www.srurwwuo..acccooom
Abstract
We present LrcSSM, non-linear recurrent model that processes long sequences as fast as today's linear state-space layers. By forcing its Jacobian matrix to agonal, the full sequence can be solved in parallel, giving 0 (TD) computational work and memory and only & (log 7') sequential depth, for input-sequence length T and state dimension D. Moreover, LreSSM offers formal suudient-stabillie guarantee that other provide. Importantly, the diagonal Jacobian structure of our model results in no performance loss compared to the original model with dense Jacobian, and the approach can be generalized other non-linear recurrent models, demon- strating broader applicability. On a suite of long-range forecasting tasks, we demonstrate that LrcSSM outperforms Transformers, LRU, S5, and Mamba.
systems such as Liquid-Se and Mamba do
Introduction
With the advent linear structured state space models (SSMs), more and more architectures have emerged, with increasingly better accuracy and efficiency. While they can be efficiently parallelized, for example with the aid the parallel scan operator, this is considerably more difficult for traditional, non-linear models. This led to decreasing interest in non-linear recurrent neural networks (RNNs), although these should arguably capture input correlations in a more refined way through their state.
Fortunately, recent work has shown how to apply the parallel scan operator to non-linear RNNs, by linearizing them in every time step, and by implementing this idea in their DEER framework [27]. Unfortunately, the ptatetrranstton matrix (the Jacobian of the model) was not diagonal, which precluded scaling it up to very long sequences. Subsequent work however, succeeded to scale up non-linear RNNs by simply taking the diagonal the Jacobian matrix, and stabilizing the DEER updates with trust regions. They called this method ELKK (evaluating Levenberg- Marquardt via Kalman)
In this paper, we propose an alternative approach to scaling up non-lineat RNNs. Instead of disregarding the non-diagonal elements of the model's Jacobian, which might contain important information about the multistep interaction among neurons along feedback loops, we learn model whose Jacobian matrix constrained be diagonal, and whose entries depend on both the current state and current input. As for linear SSMs, our main intuition is that intricate loops of the non-lineat SSM state (neural connectivity) matrix can be well summarized by its complex eigenvalues. After all, the synaptic parameters define constant matrices that can be themselves diagonalized.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

(0)
(1)
Xy
Input sequence (pxT)
Input encoder
Pre-Normalization
dependent LRC
lterativv parallelizabb updates
Post-Normalization
Decoder
Non-linear inputt
(DxT)
Output
MLP
X2
connection
LrcSSM block
repeat
Figure 1: AqquidRReiistncee Liquid- Sauuddd-aaciaancee SSM (LreSSM) architecture. The input sequence of length T and input dimension p is first passed through an input encoder, followed by normal- ization layer. The core component a non-linear, mate-and-inpur dependent I.RC with hidden dimension D and sequence length T. This SSM is computed by a parallelizable iterative lineariza- tion method. The final state values are then processed by an MLP, with a skip connection added to preserve information flow. The LrcSSM block can be stacked and repeated an arbitrary number of times (we use 2, 4, 6 layers in our experiments). A nost-normalization layer is applied before the output is passed to the decoder, which produces the final output.
To test our idea, we modified and scaled up LRCs (liquid-resistance) cruud-capacitance neural net- works), bio-inspired non-linear RNN that considerably increased LTCs (liquid networks) accuracy while also decreasing their convergence time [4], by capturing saturation effects in biological neurons, and accounting for the sate-and-input dependent nature of their capacitance. Most importantly, we introduce an inherent diagonal form to LRCs, forcing the system's Jacobian be diagonal. This modification enables exact updates during parallelization rather than approximations.
Our experimental results the Heartbeat, SelfRegulationSCPP, centration, Motorlmagery, and FigenWorms httpapguaanoosssspe.efeeeeeeaerr RthanolCon- benchmarks show that LrcSSMs are either par outperform state-of-the-art SSMs such NRDE, NCDE, Log-NCDE, LRU, S5, S6, Mamba, LinOSS- -MEES and LinOSS-IM and Transformer variants.
In summary, our main contributions in this paper are the following ones:
We discuss in detail how to scale LRCs to LrcSSMs having a diagonal non-lineat state- and-input dependent state matrix, resulting an inherently diagonal Jacobian matrix, which allows exact computations via efficient parallelization.
We demonstrate that LrcSSMs can capture fashion set of standard benchmarks used to assess accuracy and efficiency of SSMs.
tasks in a very competitive
We show that LreSSMs consistently outperform many of the state-ot of- the-art SSMs, in- cluding LRU, S5, S6, and Mamba, especially on the EthanolConcentration benchmark.
We show how our diagonal model approach can be generalized to other non-linear recurrent models, broadening the impact of its design.
Background
Here introduce the necessary background for understanding LrcSSMs: Firstly, the bio-inspired non- non-linear liquid networks LTCs, STCs, and LRCs known for their dynamic expressivity. Secondly, the parallelization techniques enabling efficient training of traditionally sequential non-linear models.
Our code is available athttps: http://hww....oooooaaaaaaaaaaaa/

---

2.1 Bio-inspired Liquid Neural Networks
Electrical Equivalent Circuits (EECs) are simplified models defining the dynamic behavior of the membrane potential (MP) of a postsynaptic neuron, as a function of the MP of its presynaptic neurons and external input signals 19 In ML, EECs with chemical synapses are termed liquid time-constan networks (LTCs) 26 14 For neuron i with m presynaptic neurons of MPs x and
inputs of value u, the forget conductance (x, u) and update conductance 2 (x, u) are defined as:
#
fi(, = Σ
8ji
max
DajiiV +jja) leak
(1)
man
(x =
Σ km oaa;; bji) +g; leak
(2)
where y= (x, ul concatenates the MP (state) of all neurons and the inputs. In Equation (T. max represents the maximum synaptic channel conductance, aj and by parameterize the sigmoidal activation governing channel openness, and g; g!rak is the leaking conductance. In Equation ( kmax = =j= 8ji
8i
max / le/eak, where ej is the synaptic reversal potential (equilibrium membrane po-
ji
tential) and eleak is the leaking potential. Since g)| g/ the sign of kijt depends on ejf" leleak
LTC-Equation 4 states that the rate of change of x; of neuron i, is the sum of its forget current xi and its update current 2; e eleak LTCs ignore saturation aspects, which were introduced in saturated LTCs (STCs) in 44. As (x, u) is positive, it is saturated with a sigmoid, and as (x, u) is either positive negative, it is saturated with tanh. Saturation is captured in STC-Equation (5)
Finally, both LTCs and STCs assume that the membrane capacitance is constant, and for simplicity equal to in Equations However, this assumption does not hold in biological neurons. reality, the capacitance has non-linear dependence on the MP of presynaptic neurons and the external input as they both may cause the neuron to deform
4 5) as the capacitance can be learned jointly with the other parameters.
18
38
23
This behavior can be modeled by the following elastance:
de; u)) = = Wjyyj + vj),
where y |x, ul, as before. LRCs incorporate this biological behavior of neurons, by introducing the elastance (which is the reciprocal of the capacitance) multiplicative term in IRC-Equation 66)
LTC: x;
21 u) ej
leak
(4)
STC: i; -a(ff(
+ (2, deak
(5)
LRC: X = =-
u e leak (x,
(6)
The time constant of LRCs is मेट = o(f which factors into liquid resistance u)). While the resistive liquidity is the core both LTCs and STCs, the capacitive liquidity acts as an additional control in LRCs.
R = 1/o u)) and liquid capacitance C 1/o(e(x,
The states x of LRCs at time can be computed using the explicit Euler integration scheme as:
LRC: = +
2.2 Parallelization Techniques
The DEER method 27 formulates next-state computation in non-linear RNNs as a fixed-pon -pint problem and solves it using a parallel version of the Newton's method. At each iteration step, DEER linearizes the model. This approximation is widely effective across many domains, and often yields accurate estimates and fast convergence. The main limitation of DEER is the use of a square Jacobian, which does not scale up to long sequences when included in the parallel scan. The second limitation is its numerical instability, which arises from the nature of Newton's method. In particular, the undamped version lacks global convergence guarantees and often diverges in practice 43 B.
3

---

As an improvement, 8 introduces quasi-DEER, which scales DEER by using the diagonal of the Jacobian, only. This is shown to achieve convergence comparable to Newton's method while using less memory and running faster. Nevertheless, quasi-DEER still suffers from limited stability.
To stabilize its convergence, the connection between the Levenberg -arauurrdda algorithm and Kalman smoothing is leveraged in the ELK (Evaluating Levenberg- Marquurdd with Kalman) algo- rithm This stabilisation of the Newton iteration by constraining the step size within trust region prevents large and numerically unstable updates. As a result, updates are computed using parallel Kalman smoother, with running time that is logarithmic in the length of the sequence. Algorithm below, presents these methods 8
Algorithm 1 DEER/ELK method with optional quasi approximation 8
1: procedure PARALLELIZERNN( init guess, tol, method, quasi)
2: diff -0
3: states iniiinntigp guess
4:
while diff tol do
5:
shifted_statee statestttts---ss (tates|:
6:
f. - fihiftedd____dtee states)
J, GETJAAAABIINS((FFF Saa.ooiiissssssssteddssss. chifted_ssatees
8:
if quasi: Js - DIAG(A.)
b, - J=-J= shifted _ttatteed__ttessssseeeeeee states
10:
new stw_sttaes-t - method(/s b,, states, sg]
diff states new see_stateslloo
12:
states - nee__stttes end while
13:
14:
return states
15:
end procedure
3 Scaling Up Non-linear LRCs
A scalable DEER or ELK approximation, first computes the dense Jacobian of the non-linear RNN, shown in Line 7 of Algorithm and then extracts its diagonal as shown in Line 8. This results in a quasi approximation of the original DEER/ELK technique, called quasi-DEER and quasi-ELK g
Our Parallelization. Instead of following this approach, we directly modify the underlying non- linear LRC- EqC-Eiuaiion 6 such that its Jacobian is diagonal by the model formulation itself. The main idea of this modification is that the state- connectivity submatrices a", w² and g mm are constant parameter matrices that are themselves diagonalizable. Consequently, all cross terms are zeroed out in the LRC through diagonalization. Accordingly, we learn the complex diagonal matrices (vectors) directly, instead.
As a result, non-diagone non-diagonal entries of the Jacobian during parallelization. Instead, we inherently learn their contribution to the dynamics the model, within the complex eigenvalues of the diagonal. Conse- quently, Line 8, J, - Diag(a) of Algorithm is not needed anymore, and the update computations become more efficient. this way, we retain the best of both approaches: much more precise, more stable, and more scalable, parallelization technique by model design.
algorithm is longer quasi-approximation,
do not explicitly remove
3.1 Proposed Model
In order achieve diagonal Jacobian for the LRCs by model design, we first modify the Equa- tions 3, by splitting their summation terms into state-dependen and an input-dependen group, respectively. For the former, we only keep the self-loop synaptic parameters, and zero out all the cross-state synaptic parameters in the associated matrices. For the latter we keep the influence all external inputs u through their cross-input synaptic parameters, as this part zeroed out anyway in the Jacobian. To highlight the separation of the terms, we include an extra superscript for the learnable parameters in the -tateddependent part, and the superscript for the parameters in the ieput-dependent part. This separation results in Equations 8
10

---

As a consequence, instead of keeping cross sonss-iy activations, where each individual synapse between neuron j and has its own ğji mmax b and ji was in Equation (I) and (2) we now only keep the self-loop neural activations, where the synaptic parameters from the same neuron equal. Note that instead of the ij indices, we have only the i index for g gmax and kmmxx, and
for b in Equations and
We denote the modified equations of the LRCs with asterisk. This gives us the following equations for the f (x₃, u), 27 (x,, u), and e7 (x), terms:
f (x u)
gi Biaak (a Xi b,) g;
max,
o E u
leak
(8)
5 state dependennddnt
input-dependent
27 (x;, & x by) + k kmasa of E 2j bj)) + look
(9)
xj state-dependent
u input-depender
ef (x,,u) W
(10)
Xj state-dependent
unnput-dependent
LreSSM:
X = (x,, x))x + T(2, (x,, ()),))))ee (xi, (x) e
(11)
For the final form our proposed LRC model, Equation [1] can be formulated into the form of SSMs, by taking the vectorial form of the states x of size m and input vector of size
LrcSSM: X = A(x, u)x b(x, u)
(12)
where
OOf (Xi (n,u))o (x)
A(x, u) = diag
off (xr, (i)o (Xy,
(13)
o -f w))a (Xmi u))
and
r(2 (X (n, (Xi u)) ef efeak
b(x,)=
T(2 (x), u))o(r (x), u)) eleak
(14)
I T(2 ()) (Xm, u)) Jeak
This reduced diagonal A(x, form of Equation and the reduced version of b(x, u) in Equa- tion [4] results in a diagonal Jacobian matrix which makes the parallelizable iterative state updates exact and efficient, that is, this is not anymore a guasi-approximation of the Jacobian.
3.2 Comparison Linear State Space Models
State-of-the of-the-arttttt art
linear SSMs typically take the following general form:
* Ax Bu
(15)
y= Cx + Du
(16)
The main differences between LrcSSM and time-invariane linear SSMs are the following:
There is no non-linearity in the recurrent state and input update (A and B, respectively) in time-invariant linear SSMs, which allows them be parallelized over the time dimension. Here, we investigate non-lineat recurrent update and non- non-linear input update too.

---

There are two key aspects of the matrices that state- of-the-are linear SSMs usually follow:
(1) First, matrix A is generally innerinarr (constant), although recent work has intro- 15 10.. In our model however, this matrix A(x, u). Second, instead of using traditional B
duced an input-dependent variant A(u)
is both state- and matrix that is simply multiplied by the input u, we adopt the form b(x, u), allowing the inpur to have more embedded influence on the state update.
(2) Modern linear SSMs typically require special initialization, such as diagonal plus low rank parameterization of the state matrix of the linear SSMs via higher-order polynomial projection (HiPPO) matrix [ or only diagonal state matrices with specific parameterization 12 32) In our case, we calculate the entries of A and from grouuded equations of and 14
3.3 Comparison to SquuidResssttanee Slquid-Capacitanne Networks (LRCs)
In summary, our approach of LrcSSM differs from LRCs 5 in the following ways:
Learning in LRCs, like in traditional non-linear RNN models, is inherently sequential. In contrast, we aim for an efficient, parallelizable version in LrcSSMs.
We modified entries of A and b to only depend on the self states, rather than on all other states, while still allowing them to depend on the full input. This change yields diagonal Jacobians, exact solutions, and improved efficiency in the update computations.
While the original LRCs use single computation layer (a single computation block), we have restructured the LRC architecture into a block-wise design in LreSSMs, similar to the linear SSM-styled models such as LRU and S5. This design is illustrated in Figure
3.4 Theoretical Insights
The LreSSM architecture enjoys three important theoretical properties. Firstly, by forcing the state-transitee transition matrix of LrcSSMs to be diagonal and learned at every time step, the full sequence can be solved in parallel, giving (TDD) time and memory and only 0 (log 7) sequential depth, where T is the input-sequence length, and D is the state dimension.
Secondly, LreSSMs offer formal gradient-stabie such as Liquid-S4 and Mamba do not provide. Lastly, because LreSSM forward and backward passes cost ( D L) FLOPs, where is the network depth of the LreSSM architecture, for low sequential depth and parameter count O L), the model follows the compute-optime scaling law regime (B = 0.42) recently observed for Mamba, outperforming quadratic-attentee Transformers at equal compute while avoiding the memory overhead of FFT-based long convolutions.
guarantee that other input-varying systems
The full proof of all these properties is given in Appendix In particular we provide details about LreSSMs stability in and scalability in A.2
4 Related Work
Linear Structural State-Space Models. Since the introduction of S4 13 linear SSMs rapidly evolved in sequence modeling. S4 used FFT to efficiently solve linear recurrences, and inspired sev- eral variants, including S5 (39), which replaced FFT with parallel scans. Liquid-S4 [5] introduced input-dependent state matrices, moving beyond the static structure but relied on FFT. Recent work, such as 56 and Mamba 10, adapted the concept of inpur-dependency and continued to push linear SSMs more efficient computation with hardware-aware parallel algorithm.
Recurrent Neural Networks. While traditional non -on-ernear RNNs have been favored for their memory efficiency, their major limitation lies the lack of parallelizability over the sequence length. This has led to the development of parallelizable alternatives, such as 28 32 g One notable example is the Linear Recurrent Unit (LRU) 32 which uses complex diagonal recurrent state matrices with stable exponential parameterization, achieving comparable performance with SSMs. While LRUs argue that linear recurrence sufficient, in this work we show that incorporating non non-lineanee linearity in the transition dynamics can offer significant advantages.
Parallelizing

---

Importantly, these approaches achieve parallelism through entirely new architectures, without addressing how to parallelize existing non-linear RNNs. Techniques like DEER (27] and ELK & fill this gap by enabling parallel training and inference for arbitrary non-linear recurrent models.
Positioning LrcSSM in Recent Advances. Our LrcSSM aligns with the structured state-space duality (SSD) framework introduced by 3 as its main focus is on designing an RNN that behaves almost like SSM (diagonalizable and parallelizable). In addition, recent work on parallel can be also combined with LreSSMs to further enhance their efficiency.
state-free inference
33
5 Experiments
We follow the same classification evaluation benchmark proposed in 41 and then used by 37 These tasks are part of the UEA Multivariate Time Series Classification Archive (UEA-MTSCA), All of these datasets consist of biologically or physiologically grounded time-series data, derived from real-world measurements of dynamic systems, which can be human, animal, or chemical. They capture continuous temporal signals such as neural activity, bodily movements, or spectroscopic for benchmarking models that need to learn complex
readings, making them very temporal dependencies.
5.1 Short- and Long- Horizon Sequence Tasks
We compare LrcSSMs against eleven models representing the state of the art for a range of long-
sequence tasks. These include the Neural Controlled Differential Equations (NCDE) 22 Neural
Rough Differential Equations (NRDE) 30 and Log-NCDE [41, Linear Recurrent Unit (LRU) 32]
$5 39, MAMBA 110, S6 10, Linear Oscillatory State-Space models with implicit- time
integration (LinOSS IMEX (7) and with implicit time integration (LINOSS-IN IM (37), Transform-
40 and RFormers
24
We followed the exact same hyperparameter-tunin protocol of 37 using a grid search over the validation accuracy. More details on these experiments are given in Appendix B.2 After fixing the
hyperparameters, we compare the average test set accuracy over five different random splits of the dataset. As we are reporting the most of the model results from Rusch et. al (7), we also used the exact same seeds for the dataset splitting as well. When presenting the results, we highlight the top three performing models.
Short-Horizon Sequence Tasks. In the first block of Table 1 we report results on datasets with sequence lengths shorter than 1,500 elements. These datasets include the Heartbeat dataset (Heart) 7 which contains heart sound recordings, as well as SelfRegulationSCPI (SCP1) and SelfRegulationSCP2 (SCP2) M1, which include data on cortical potentials. We report the test accuracy results. We found that our LrcSSM model performed average on these tasks, and we suspect that these datasets lack interesting input correlations.
Long-Horizon Sequence Tasks. Next, we focus on the tasks that require learning interactions, especially those with sequence length above 1,500, up to 18,000. These include the EthanolConcentration dataset (Ethanol) 25 which contains spectroscopic recordings of solutions, the Motorlmagery dataset (Motor) 24 which captures data from the motor cortex, and the EigenWorms (Worms) ! dataset of postural dynamics of the worm C. elegans, As shown in the second block of [ our LrcSSM model outperforms all other state-of-ther thtte- SSM methods on the EthanolConcentration task and achieves second-best -bet performance on the Motorlmagery and FigenWorms datasets too. We believe that EchanolConcentration contains interesting input correlations, which LreSSMs can capture through its input- and state-dependence.
Average Performance Across Datasets. In the rightmost column of Table we also report the av- erage accuracy across all six datasets considered from the UEA-MTSCE archive. LreSSM achieved an accuracy of 66.3%, placing it at the forefront alongside the LinOSS- IM model, outperforming all other tfate-of-the-are models, including LRU, S5, S6, Mamba, and LinOSS-IMEX, The implicit integration scheme of the LinOSS-IM model, seems to have played an important role, and we plan to investigate a similar integration scheme for LreSSM, too. Our current scheme just a simple explicit Euler.
7

---

Table 1: Test accuracy comparison of different models across relatively short-horizon datasets (<1,500) in the left block and long-horizon datasets (>1,500) the middle block. Average test accu- racy with standard error (%) across all datasets is in the rightmost block. The performance of the models marked by is reported from [37], and those with from (29). The same hyperparameter tuning protocol and dataset splitting over the same 5 seeds were used.
Heart
SCP2
Ethanol
Motor 3,000
Worms
Average
Sequence length
405
896
1,152
1,751
17,984
Input size
7
6
#Classes
2
2
4
5
NRDE!
76.7 5.6
48.1 11.4
4.5
7.8
77.2 7.1
60.2 7.7
NCDE
5.8
2.0
49.1
2..0
6.2
62.2 2.2
Log-NCDE
7.0
8
64.0
6.1
5.6
82.8 + 2.7
64.4 -4.47..6
LRU
7.66
4.6
4.0
2.88
8.6
85.0 6.2
61.8 10.1
S5T
3.11
8.11
3.3
3..6
5.99
83.9
63.1
Mamba'
3.8
8.4
48.2
4.5
4..7
70.9 15.8 15.8
58.6 8.4
S6T
76.5
8..7
19.9 9.4
26.4
5.73
85.0 + 16.1
62.0
LinOSS-IMEX
75.5
87.5
58.9 8.1
1.0
5.3
80.0 2.7
65.0 8.5
LinOSS-IM!
3.77
2.6
58.2 6.9
0.6
6..0
95.0
67.8
Transformer*
0.1
84.3
49.1 2.5
40.5
3.00
0OM
59.0 8.0
RFormert
7.1
81.2 11.2222 2.8
3.7
34.7 + 4.1
55.8
90.3 0.1
64.5 8.4
LrcSSM (Ours)
5.7
85.2 2.1
53.9
36.9 5.3
58.6
90.6 1.4
66.3 8.3
5.2
Generalizing the Technique of Diagonal Model Design
Other non-linear RNNs can be transformed into an efficient, parallelizable form by following steps similar those outlined in Section 3 In Appendix [ we provide sketch of how this transformation of any non- non-linear recurrent model can be achieved, and in Table we present results comparing LreSSM with other non non-linear SSM models constructed using the same diagonal model design. In the line of this work, we extend our method to build up other non non-linear SSMs from Minimal Gated Units (MGU) 45 Gated Recurrent Units (GRU) 22, and Long Short- Term Memory (LSTM) [6), which we refer to MguSSM, GruSSM and LstmSSM, respectively. Our experiments show that the LreSSM model outperforms the other non-linear models across the six classification benchmarks.
Table Experimentation with different non- RNN models formulated into efficient SSMs. For these experiments, SSMs with 64 units. For each dataset, the average test accuracy and standard deviation (%) are reported, and the last presents the mean and standard error aggregated across datasets.
fixed configuration with input encoding of 64 and blocks of
MguSSM
GruSSM
LstmSSM
LrcSSM (ours)
Heart
4.80
7..7
7.50
75.0 2.6
SCP1
6.6
8..2
7..8
84.8
SCP2
49.6
52.5+3.3
51.1
55.4 7.7
Ethanol
31.1
3..5
32.6=5.6
36.1 +.1110 1.1
Motor
4.74
49.647.3
54.3+3.3
4.1
Worms
90.0
86.146.3
8..2
85.6 5.4
Average
6.8
6..1
6..3448.0
6..4
5.3 Ablation Studies
In Appendix E| we present ablation studies to analyze the contributions different design choices. First, we compare liquid capacitance (as in LRCs (6) with constant capacitance (as in STCs (5) when scaled up to SSMs. Second, we show that simplifying the model enforce a diagonal Jacobian does not degrade performance, based on experiments against the original full LRC model with dense Jacobians. Finally, we evaluate the impact input- and state-dependency, as well as the use of real versus complex camplex-valued valued matrices.

---

Discussion
Competitive Long-Horizon Performance. Our experimental evaluations show that the Lre- SSM model performs moderately well on short-horizon datasets, while demonstrating highly competitive performance on datasets with long input sequences, as shown in Table ] In those long-sequence tasks, LrcSSMs outperform LRUs, Mamba, and S6, and also achieve better average performance across all datasets.
The only model that LrcSSMs generally does not outperform is the LinOSS-IM model, except on the EchanolConcentration dataset (for both LinOSS- IMEX and LinOSS-IM versions), and on MotorImagery and EigenWorms (in the case of LinOSS-IMEX), This may be attributed to the fact that LinOSS is based on forced linear second -oderr ODEs, whereas LrcSSMs are built upon LRCs, which are non- non-linear first-order ODEs. Another possible reason lies in the integration technique: while we were able outperform the implicit-expiice (IMEX) integration scheme, we did not surpass the fully implicit one (IM) in average test accuracy. This suggests that more sophisticated integration schemes for LreSSMs (which currently use explicit Euler) may be worth investigating.
Biological Inspirations in Sequence Modeling. We find it particularly interesting that the LinOSS model also exhibits biological relevance, as models cortical dynamics through harmonic oscilla- tions. In contrast, approach models information transmission through chemical synapses, which a different biological phenomenon. The strong performance of both approaches, de- spite being grounded in different aspects of neuroscience, highlights the significant potential of biologically inspired models as a foundation for future research in sequence modeling.
Efficient Sequence Modeling with Diagonalized Jacobians. In this paper, we focused on the biologically inspired non non-linear LRC model, and demonstrated how this model can be made more efficient for sungseeueence modeling, by redesigning its underlying state-recurrene matrix and its input-trame trant-transtionn vector b, such that the resulting Jacobian is diagonal matrix, for the state-update iterations. This matrix can then be directly used in the parallelizable methods of ( which gives an exact parallel update, and not an approximation. As we have shown, this approach can also be applied to other non-linear RNNs of interest.
Limitations. As pointed out in Section A.2 this parallelized version holds good promise towards efficient non non-linear RNNs compared to sequential computation costs. Linear SSMs have also the same costs. However, also have take into account that LRCss need multiple Newton steps to converge at each iteration, which linear SSMs do not require. The number of iterations depends the convergence of the state updates, which stops once the difference between the consecutive state updates gets below a defined threshold (see Line of Algorithm The number of iterations per datasets for LreSSM is shown in Figure 2
Conclusion
In this work, revisited the potential of non-linear RNNs in the of efficient, scalable linear SSMs. While linear SSMs have seen remarkable success due to their parallelizable structure and computational efficiency, non-linear RNNs have largely been sidelined due to their inherently sequential nature. However, recent advances, particularly the DEER (27] and ELK methods, and their quasi-variants g have opened the door to parallelizing non-linear RNNs, thus challenging their long-standing scalability limitation.
Building these developments, we introduced the resuiddreeistance cauddd-apacitance non- linear state-space model (LreSSM), a novel SSM architecture that combines the expressive power of bio-inspe inspiiepprd non-linear RNNs with the scalability of modern SSMs. By adapting the parallelization methods and carefully redesigning the internal structure of LRCs, we enable efficient parallel computation by inherently learning diagonal Jacobian matrices, while still preserving the dynamic richness of non non-linear state updates biological neurons. Our design allows for exact parallel updates, rather than relying on quasi-approximattios.
Our experiments demonstrate that LreSSM not only matches but often exceeds the performance of leading linear SSMs such as LRU, $5, S6, and Mamba, particularly in hong-horizon sequence
9

---

modeling tasks. These results suggest that non non-lineat -el---inn---a KINN-based SSMs are not only feasible solution but can also be competitive, offering promising direction for future research in sequence modeling.
In summary, this work bridges the gap between the expressive flexibility of non-linear dynamics and the computational advantages of parallelism, within the LreSSMs architecture, opening new pathways for scalable, biologically inspired architectures in modern deep learning.
Acknowledgements
We thank Ramin Hasani for his feedback and contributions and Daniela Rus for her support in this project. M.F. has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska- Curie grant agreement No 101034277. Experiments were performed on the datalab cluster at TU Wien, whose support and infrastructure contributed significantly to these research results. We also thank Liquid Al for providing extra computational resources. Finally, we thank the anonymous NeurIPS reviewers for their valuable feedback and comments, which helped to improve our work.
References
[1] Niels Birbaumer, Nimr Ghanayim, Thilo Hinterberger, Iver Iversen, Boris Kotchoubey, Andrea Kübler, Juri Perelmouter, Edward Taub, and Herta Flor. A spelling device for the paralysed. Nature, 39866725) 1999.
[2] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empir- ical evaluation of gated recurrent neural networks on sequence modeling, 2014.
10.5000///aaaiiv 141..355511
[3] Tri Dao Albert Gu. Transformers ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.2106/ 2024.
[4] Mónika Farsang, Mathias Lechner, David Lung, Ramin Hasani, Daniela Rus, and Radu Grosu. Learning with chemical versus electrical synapses does it make a difference? In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 15106-15112. IEEE, 2024.
[5] Mónika Farsang, Sophie A Neubauer, and Radu Grosu. Liquid resistance liquid capacitance networks. In The First Workshop on NeuroAI@ NeurIPS2024, 2024.
(6] Daniel Y Fu, Hermann Kumbong, Eric Nguyen, and Christopher Ré. Flashfftconv: Efficient convolutions for long sequences with tensor cores. arXiv preprint arXiv:231 1,05908, 2023.
[7] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark. Joseph Mietus, George B Moody, Chung-Kann Peng, and H Eugene Stan- ley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215 e220, 2000.
[8] Xavier Gonzalez, Andrew Warrington, Jimmy Smith, and Scott Linderman. Towards scalable and stable parallelization of nonlinear rnns. Advances in Neural Information Processing Systems, 37:5817-5849, 2024.
(9] Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-trau -tacketrackkng in linear rnns through negative eigenvalues, 2025.
[10) Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference Language Modeling, 2024.
[11] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1483 2020.
[12] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 15:35971-35999 2022.
10

---

[13) Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with struc- tured state spaces, 2022. 110004/85500/xxxv..11111
(14) Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time-constant networks. In Proc. of the AAAI Conference on Artificial Intelligence, volume 35(9), pages 7657-7666, 2021. doi T0. v3ai.v3559. 16366
(15) Ramin Hasani, Mathias Lechner, Daniela Rus. Liquid structural stace models. In The Eleventh International Conference on Learning Representations, 2023.
Wang, Makram Chahine, Alexander Amini, and
[16] Sepp Hochreiter and Jürgen Schmidhuber. LSTM can solve hard long time lag problems. In Advances in Neural Information Processing Systems 9, NIPS, Denver, CO, USA, December 2-5, 1996, pages 473 479. MIT Press, 1996.
[17] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training coamnptee-ppimman large language models. arXiv preprint arXiv:2203 2022.
(18) B. Howell, L.E. Medina, and W.M. Grill. Effects of frequency- membrane capaci- tance on neural excitability. Neural Engineering, 12(5) October 2015.
[19] Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, James Hudspeth, Sarah Mack, et al. Principles of neural science, volume 4. McGraw-hhe New York, 2000.
[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, 2020. Version Number: 1.
[21] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXIv:2001.08361. 2020.
[22] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in neural information processing systems, 33:6696-6707, 2020.
[23] Jitender Kumar, Patrick Das Gupta, and Subhendu Ghosh. Effects of nonlinear membrane capacitance in the hodgkin-huxley model of action potential the spike train patterns of single neuron. Europhysics Letters, 1 14200), jun 2023.
[24] Thomas Lal, Thilo Hinterberger, Guido Widman, Michael Schröder, Hill, Wolfgang Rosen- stiel, Christian Flger, Niels Birbaumer, and Bernhard Schölkopf. Methods towards invasive human brain computer interfaces. Advances in neural information processing systems, 17, 2004.
(25) James Large, E. Kate Kemsley, Nikolaus Wellner, Ian Goodall, and Anthony Bagnall. Detecting forged alcohol non-invasively through vibrational spectroscopy and machine learning, In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 298-309. Springer, 2018.
[26] Mathias Lechner, Ramin Hasani, Alexander Amini, Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelli- gence, 2(10):642-652, 2020. doi 10.11...0380 wtt.l103///22555
[27] Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non- linear sequential models over the sequence length. In The Tweifth International Conference on Learning Representations, 2024.
[28] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. arXiv preprint arXiv:1708 (afe:1179.040555 04057, 2017.
11

---

(29) Fernando Moreno-Pino, Álvaro Arroyo, Harrison Waldon, Xiaowen Dong, and Álvaro Cartea. Rough transformers: Lightweight and continuous time series modelling through signature patching. In A. Globerson, I. Mackey, D. Belgrave, A, Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 00624-1062944 Curran Associates, Inc., 2024.
[30] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
[31] Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone, and Antonio Orvieto. Fixed-poine tooitt rnns: From diagonal to dense in few iterations. arXiv preprint arXiv:2503.10793 2025.
[32] Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670-26698. PMLR, 2023.
[33] Rom N Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy TH Smith, Ramin Hasani, Mathias Lechner, An, Christopher Ré, Hajime Asama, Stefano Ermon, al. State free infer- ence of state-space models: The transfer function approach. arXiv preprint arXiv:2405.06144 2024.
[34] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043-28078.
PMLR, 2023.
[35] Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint 14t.:::403.1184444 2024.
[36] Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors, 19(14):3079, 2019.
[37] T Konstantin Rusch and Daniela Rus. Oscillatory state-space models. In The Thirteenth International Conference on Learning Representations, 2025.
[38] Daniel Severin, Sofia Shirley, Alfredo Kirkwood, and Jorge Golowasch. Daily and cell type- specific membrane capacitance changes in mouse cortical neurons. bioRviv, 2022.
[39] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. In ICLR, 2023.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
(41) Benjamin Walker, Andrew Donald McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. Log neural controlled differential equations: The lie brackets make difference. In Forty-first International Conference on Machine Learning, 2024.
[42) Stephen R Wicks, Chris] J Roehrig, and Catharine H Rankin. A dynamic network simulation of the nematode tap withdrawal circuit: predictions concerning synaptic function using behavioral criteria. Journal of Neuroscience, 16(12)4001-44031, 1996.
[43] Stephen J Wright. Numerical optimization, 2006.
(44) Eviatar Yemini, Tadas Jucikas, Laura J Grundy, André EX Brown, and William R Schafer. A database of caenorhabditis elegans behavioral phenotypes. Nature methods, 10(9):877-879, 2013.
[45] Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, and Zhi-Hua Zhou Minimal gated unit for recurrent neural networks. International Journal of Automation and Computing, 13(3):226- 234, 2016. 10010.0007//111133-00665
12

---

Technical Appendices and Supplementary Material
Theoretical Insights
A.1 Stability
We analyze single hidden dimension because every recurrence is diagonal, all dimensions behave independently and identically. Recall the discrete tineette--mmeeeee update:
x = X X b,,
0 0< s<
(17)
where is a user-chosen radius (typically 0.9-0.99) enforced by either the tanh-clamp or the negattie--ofpplu--exppeentt
parametrisation. patameetttnnttataaatttettttoooo
One step is contractive.
Lemma (p-contraction). For any x, e ERD we have ||x++1 yr+1 11 y)ll p plxx yll--
Proof. A, is diagonal with all entries p, hence its operator (spectral) norm is p; multiplying by can only shrink Euclidean distances.
Forward states stay bounded.
Iterating Lemma r times yields
1-p²
xxlll xlxlolll
B,
B max b.ll2.
(18)
1-p
Therefore the hidden state never blow up, irrespective of sequence length.
Back- phekkpropagated gradients never explode.
Theorem 1 (Gradient stability). Let a loss L depend only on the final state xy. Then for anyO ST
v[, 1-112 s p V
hence the Jacobian product norm is = 1 and cannot explode.
Proof. The Jacobian of one step is J, = A, so Wellz s Back propagation multiplies T --1 such Jacobians: V= J Xxy Sub-muttiplicativityy of the spectral norm gives the result.
Controlled vanishing. Because is tunable, gradients decay most geometrically: choosing = 0.99 keeps long-range signals alive; smaller values add regularisation.
Deep stacks. For L stacked layers with radii Pe the bound becomes || (layyyr L1|2
(1 ",,,,,,,,,, Keeping every pe close to therefore preserves stability in depth.
How others models handle forward/gradien stability. S4/S6 keep Re(A) 0 and collapse the recurrence into single convolution kernel. In this setting, forward activations are bounded and back-propagater Jacobians never appear. Mamba re- re-introduces recurrence via gate e [0, }]; that gate is clipped the same p-Lipschitz bound as ours holds, but no proof is given. LinOSS discretizes non-negative diagonal ODE with symplectic IMEX step, proving both state and gradient norms stay s 1. Liquid-S4 adds an input term B u, without clamping the spectrum, so stability relies on empirical eigenvalue clipping. Thus, among truly recurrent models, only LrcSSM (and LinOSS under its specific integrator) enjoy formal guarantee that both forward trajectories and full Jacobian chains remain inside the unit ball.
LreSSM has a stronger guarantee than Liquid-S4 or Mamba, and-unlike 54-type convolutions, can propagate gradients through actual recurrent steps while remaining provably safe from explosion. This makes training deep, long- segg-eeeuenee stacks straightforward: set p= 1, forget about gradient clipping, and tune itself single parameter to trade off memory length versus regularization.
13

---

Table 3: Per-layer asymptotic complexity (sequence length T, width D).
Architecture
F/B FLOPs
Memory
Parallel depth
Mamba
O(TD)
O(D)
(log 7')
LinOSS 37
((TD)
((D)
(logg]
Liquid-S4
@(TD)
6(D)
O(T)
S4/Hyyna K 34
O(TlogT
((T)
Ø T)
Transformer 21
O(T²D)
O(T + TD)
O(1)
LreSSM (ours)
O(TD)
6(TD)
Øllog
A.2 Scalability
Let T denote the input sequence length and D the state dimension. Sequential methods inherently cannot be parallelized, requiring (D) memory complexity and @(TD²) computational work. Compared this, the DEER 27 method is parallel but comes with major drawback, it requires G(TD²) memory complexity and (TD*) computational cost.
The ELK technique introduced in ( achieves fast and stable parallelization by incorporating diagonal Jacobian computation for scalability. This reduces both memory and computational complexity significantly (TD). Our approach achieves the same complexity-@(C) for both memory and computation, thanks to the use of inherently diagonal Jacobians.
Now let's assess formal complexity and compute-optimal scaling laws for LrcSSM:
Compute, throughput, and memory. Let FLOPs = ey BTDDL be the dominant training cost, where B is the batch size, T the sequence length, D the hidden width, L the network depth, and cj an architecture specific constant we define (lower for SSMs and higher for Transformers). The single-GPU throughput (tokens s GPU-I) ) is throughput wall-clnck time is the sum of peak activations and model parameters.
TB
The memory footprint
Scaling-law
20 17
scaling law is any asymptotic or empirical relation of the form
Loos(C)) AC F,
C = compute (FI.Pss), B > 0,
(19)
or a dlosed-form complexity identity such as FLOPs & T..
Recent large-scale studies like 35 show that / depends on the operator's per-token cost: Dense
attention: = 0.48-0.50 200. Linear-time RNN/SSM (Mamba, Hyena): B = 0.42-0.45 in 70 M-7 B
runs 10 34
Hybrid (recurrence sparse attention): & can reach 0.41 (MAD pipeline) 35
Table 3 summarizes the per-layer cost of the main long- song-sequence architectures in terms of for- ward/ hard/backward FLOPs, peak activation memory, and parallel depth over the sequence length Because LrcSSM shares the same (TD) compute curve as Mamba but with smaller constant cf (no low-rank gate, no FFT), we expect it to at-or or slightly below-the 0.42-0.45 band. The claim is compatible with existing data: Mamba-3B matches 6-B Transformer at the same FLOPs [TO] and LinOSS shows 2x lower NLL than Mamba on 50 k-token sequences at equal compute 37 Hence, f 0.42 is a defensible prior for LrcSSM; hybrid LreSSM local-attention block could plausibly move f toward 0.41.
Sequence-length scaling. For single-GPU throughput K(T), LreSSM inherits the near- wear-perfect linear behaviour ((7) of the scan primitive, with practical speed-ups obtainable through width-w windowing and double buffering that saturate 12 cache bandwidth. Liquid-S4 degrades linearly in latency because remains sequential, whereas FFT-based S4/Hyena layers incur G log T) compute and become memory-bound beyond T 64k tokens. Hence, for contexts up to 64k, LrcSSM (and Mamba) are the compute winners; at larger T the EFT models may overtake them in raw FLOPs but pay a significant activation cost.
Sequence-length scaling. Let K(T) be the wall-clock time for single forward pass of length T
T on one GPU. LrcSSM: K(T) = SMs T (linear) but can drop to = SMs w
with a width-u w scan 1-cccchh reuse, where SM the number of CUDA Streaming
and www.aabeinggna----eeerreeee Multiprocessors on the GPU, and bardware-and-Ke wawraar--ann-eeeeeeeeeeeeeeeeeee constant (e.g., time per
14

---

token per SM). Mamba [0]: same asymptotic, but the fused CUDA kernel shows = 5x higher throughput than Transformer on 4k tokens; on shorter sequences the constant of its scan kernel dominates. S4/Hyena (FFT): Ø (Tlog 7); cross-over with linear methods occurs around T=8-16k on A100s FlashFFTConv reduces the constant 4x-8x Liquid-S4 15: remains sequential; throughput degrades linearly without remedy.
Thus, for 7 k, LrcSSM and Mamba compute winners; beyond 64k, Hyena/S4 win in pure flops but can be memory-bound.
B Experimental Details
B.1 Training Setup
We used A100 GPUs with 80 GB of memory. Training time ranged from less than up to 2-3 hours per data split, depending on the dataset and model. Early stopping was used to prevent overfitting, which varies the training time.
B.2 Hyperparameters
We performed grid search over the following set of hyperparameters:
Table 4: Hyperparameter grid. Same values as in
Parameter name
Value
learning rate
10 5 100000t,00 100-3
hidden dimension
16, 64, 128
state-space dimension
16, 64,256
number of blocks (#blocks)
2, 2,4,6
Using the grid shown in Table 4 we selected the best configuration for each dataset based on the average validation accuracy across five data splits. The splits were generated using the same random seeds as in 37 to ensure full comparability. The final hyperparameters used to report the test accuracies are listed in Table
Table Optimized hyperparameters used for LreSSM per dataset.
Ir
hidden dim.
state-space dim.
#blocks
include time
Heart
10
64
64
4
False
SCP1
10-3
64
16
2
False
SCP2
10-3
128
64
2
False
Ethanol
10-4
128
16
2
False
Motor
10-4
16
16
4
False
Worms
10
64
16
4
False
We found that, in general, LreSSMs benefit from higher learning rates and are not particularly sensitive to the hidden dimension of the encoded input. However, a lower state-space dimension and fewer layers tend to be advantageous.
B.3
Dataset Sources
The datasets can be downloaded from the following links:
Short-horizon tasks:
Heartbeat (Heart)
SeI/eegllationSCPI (SCP1)
SellRegulationSSC22 (SCP2)
15

---

Long-horizon tasks:
EthanolConccentaation (Ethanol)
Motorlmagery (Motor)
EigenWorms (Worms)
B.4 Additional Remarks on the Model Design
Integration Scheme. As pointed out in Section B we used the explicit Euler integration scheme. This is a simple and straightforward solution, but it might be worth investigating more sophis- ticated and computationally expensive integration methods. In fact, we conducted some pre- liminary experiments with a hybrid explicit- iittiiit solver but did not observe any performance improvement, although did not explore it across the full hyperparameter grid.
Integration Timestep. For the integration step, we used timestep of Ar= 1 in all our experi- ments. As 37 investigated different År values across the datasets and observed no substantial gain in performance, they also continued with Ar I for all their experiments. However, it might still worth investigating this in our case as well.
B.5 Run Times and Number of Iterations
Table 6: Run time in seconds for the considered models for 1000 training steps. Values for the models other than LreSSM are taken from 37
NRDE
NCDE
Log-NCDE
LRU
S5
Mamba
S6
LinOSS-IMEX
LinOSS-IM
LreSSM
Heart
9539
1177
826
8
11
34
4
7
23
SCP1
1014
973
635
9
17
7
3
42
38
12
SCP2
1404
1251
583
9
9
32
7
55
22
15
Ethanol
2256
2217
2056
16
9
255
4
48
8
15
Motor
7616
3778
730
51
16
35
34
128
11
31
Worms
5386
24595
1956
94
31
122
68
37
90
33
Number of Iterations per Dataset
Heartbant
SCPI
DhanelCon enaaaiConeettttii entration
SCP2
Muterimapery
Epenttorms
Figure Iterations needed for convergence for LrcSSM on average per dataset.
In 6 we present the average run time of the best models for 1000 training steps for each dataset. The values for the models other than LreSSM are reported from B7] Figure the number of iterations needed for convergence for LrcSSM on average per dataset. As one can see, we do not lose significant runtime due to the extra iterations that are needed for the states to converge through the Newton-steps.
16

---

C
Additional Experiments
conducted additional experiments on the IPG-DaLiiA dataset 36 and report the results (baselines are taken from 37 in T We ran the hyperparameter tuning following the same protocol as before. LrcSSM achieves performance comparable to other models, demonstrating its competitiveness on this benchmark.
Table 7: Mean squared error (MSE 10 2) for different models on PPG-Dal.iA As before, the performance of the models marked by f is reported from 37 Results are averaged over 5 seeds.
Model
MSE 10-2
NRDE
9.90 + 0.97
NCDE
13.54 0.69
Log-N Log-NCDEt
9.56 = 0.59
LRU
12.17 0.49
S5*
12.63 + 1.25
S6*
12.88 + 2.05
Mamba
10.65 + 2.20
LinOSS-IMEX IMEX
7.50 + 0.46
LinOSS-IM'
6.40 + 0.23
LreSSM
10.89 = 0.96
Generalized Model Design
Any non-linear RNN can be reformulated by considering its underlying differential equations using an explicit first-ordes Euler integration scheme with a unit time step. We assume the non-linear RNN has the following general form:
x, = qı (x X, S, uc)
where q, (x, and 8, = o (x, up)) are non-linear, potentially state- and input- dependent gated (o) functions, which aim express it as:
x A(x, x b(x, u).
To see this, consider:
X (q(x, u) +
which yields (using Euler integration with Ar= 1.0) the original equation:
X = x + = X (q,(,,1, - 1),,, 8, (x,,1, u,) = qı (x(xx-1, (x,,1,,
Thus, in this general formulation: A(x, u) q(x u) - 1, b(x, u) = u).
To obtain the desired efficient model representation, we define:
A(x, u) ( (x1, u)) 1,..., off, (x), u)) 1,..., (fam (X u)) 11)
b(x, u) [a (x) u), (fs, u))..., (fs W)]
D.1 Example of GRU
The GRU equations are:
o (Wg [X,-1> U,] + (f2))
(W, [x,-1, u,]+b,,) (f,))
C, tanh (W, -,,x,,,,,,,1 b,) = (f,,)
X; = (1 X,-1 + 2,
Rewriting the update using the differential form:
17

---

x= -X Zc to check, we get back to the original form (using Euler integration with A/= 1.0):
x; = x A X 2,x,-1 +2,0 = (1 7,2) X C,
To express this in the form x A(x, u) b(x, u), we set: x x/x -2x+ zc, thus A(x, u -2, and b(x, u) = z2
Using our proposed formulation for constructing A(x, u) and b(x, u):
A(x, u)
(x1, -0( (x,, -0 (Xm u))
b(x, u (X, (iih,)))(fa Tfp, u)), u)), .... of (x,, ()) (x), u)), ... affzm (Xms u)) Tf (Xm u))
This reformulation results in a diagonal Jacobian matrix, enabling more efficient and exact (rather than quasi) computations for the parallelization method.
Ablation Studies
For the ablation studies below, due the extensive hyperparameter search required, fixed the architecture to 6 layers of SSM blocks, each with 64 states, an input encoding dimension of 64, and a learning rate of 10
Motivation for LRC and Model with Constant Capacitance. We considered models of non- spiking neurons, such as those based on electrical and chemical synapses. Electrical synapses can be modeled using continuous- time recurrent neural networks (CT-RNNss, but they tend to show degraded performance compared to models based on chemical synapses. In such models, part models (SSMs) before, and lacks input or
of the transition matrix is fixed, as in many state dependency, which offers nothing novel in this regard.
Instead, we focused on models of chemical synapses, the so-called liquid neural networks, where the term liquid refers the imput- and state-dependent dynamics of the transition matrix. These dynamics are non-linear. Within this family, Liquid Time-Constall Networks (LTCs) do not have saturation (or gating) terms in their dynamics of Eq. which is desirable for achieving stable behavior. Fortunately, their saturated (gated) counterpart, Saturated LTCs (STCs), does include such terms in Eq. 5
We modified the model of STCs the same way as LRCs, achieving a diagonal Jacobian matrix (without the elastance term indicated in orange in the equations) and ended up with StcSSMs. This model has constant capacitance, as assumed in models derived from chemical synapse dynamics.
We compared this model against LrcSSM across the datasets, using fixed setup. We found that LrcSSM outperforms StcSSM, highlighting the need for the non-ling non-linear membrane capacitance as shown in T
Table 8: Comparison of StcSSM with LreSSM (proposed model) across datasets.
SteSSM
LrcSSM
Heart
75.7 5.0
75.0 +: 2.6
SCP1
78.8 4.3
84.8 +
SCP2
50.4 + 6.9
55.4 7.7
Ethanol
36.8 4.0
36.1 1.1
Motor
53.9 4.4
55.7 4.1
Worms
85.0 + 4.8
85.6 + 5.4
Average 63.4 +: 7.8
65.4 1: 8.0
Impact Model Simplification. evaluated the LrcSSM model with a dense Jacobian matrix well, where A and b have all input- and sateedependencces in the model, to compare performance and assess any potential trade-offs. used the same architectural design as before to ensure fair comparison.
18

---

Table 9: Performance comparison of LrcSSM with the full model resulting in a dense vs. our proposed model with inherently diagonal Jacobian matrix.
LreSSM-full dense Jacobian
LrcSSM diagonal Jacobian (ours)
Heart
72.3 + 4.2
75.0 + 2.6
SCP1
83.6 = 1.9
84.8 + 2.8
SCP2
49.6 5.0
55.4 +: 7.7
Ethanol
33.9 + 1.5
36.1 + 1.1
Motor
55.7 + 5.7
55.7 + 4.1
Worms
84.4
85.6 + 5.4
Average
63.3 + 8.3
65.4 + 8.0
These results in Thol that empirically, we do not lose the expressive capabilities of LRCs by constraining the Jacobian matrix to be diagonal. We hypothesize that this because the strict structure encourages the model to learn more efficient representations, while also enabling exact (rather than approximate) computations due the design.
Input- and State- dentte-deendencc. We conducted ablation studies to assess the importance of incor- porating state-dependency in the saate-transition matrix A and input-transition transition vector b. As shown in Ta the average results indicate that learning both input- and state-dependen transitions yields better performance. We also suggest that future work could treat these dependencies as tunable hyperparameters, some datasets may benefit from both forms of dependency, while others may perform well with deput-deeendeccy alone.
Table 10: Experimentation with different input and state- suate-dependent matrices. Here, we use fixed configuration with input encoding of 64 and blocks of SSMs with 64 units. For each dataset, the mean and standard deviation are reported, and the last row presents the mean and standard error aggregated across datasets. We found that excluding state dependency from A and then from too, downgrades performance on average.
LrcSSM (ours)
LrcSSM
LrcSSM
A dependence
A(x, u) b(x, u)
A(u)
A(u)
b dependence
b(x, u)
b(u)
Heart
75.0
75.0 1.8
73.0
SCP1
84.8 2.8
85.0 : 2.9
83.1 + 1.4
SCP2
55.4 7.7
4.5
51.4 2.9
Ethanol
36.1 1 1.1
37.6 1.6 3.9
34.2 2.9
Motor
55.7
57.9 +
+ 6.0
Worms
85.6 2 5.4
85.0
86.7 + 5.4
Average
65.4 8.0
65.0 8.0
63.8 + 8.4
Please note that results reported here for LrcSSM, do not match the results of the previous tables because we used fix setup without hyperparameter tuning, only focus the importance of state-dependency and changed the underlying matrix A and b of x = A(x, u)x + b(x, u). This results in having even better test accuracies reported here for Heartbeat and SelfRegulationSCCP.
Complex-vales valued with complex-vale valued learnable parameters, focusing on those interacting directly with the state In particular,we experimented with the parameters mmaxx of f (x;, and k kmmax, of z (x;, u) as defined Eq. g and (9) respectively, as well as their shared sigmoidal channel parameters at and bz. These were gradually converted to complex values, and experiments were conducted using fixed configuration of SSM blocks, each with state dimensions and 64-dimensional encoded input, and learning rate of 10 As shown in Table 11 found no significant performance
Matrix and Input- Transition Vector. We also experimented
19

---

gains average from using complex-valued valued parameters. As result, we opted to use real- valued learnable parameters in our main experiments. Nevertheless, we also evaluated the tuned models with their complex-valuee valued counterparts, The only notable improvement occurred on the Motorlmagery dataset, where accuracy increased from 54.3 = 3.1 58.6 + 3.1. Substituting this result into the average accuracy reported in Table woold yield 65.6%, which still ranks our model as the second- best overall.
would
Table 11: Experimentation with complex valued parameters. Here, we use a fixed configuration with input encoding of 64 and 6 blocks of SSMs with 64 units. For each dataset, the mean and standard deviation are reported, and the last row presents the mean and standard error aggregated across datasets. We found very similar average performance between real-valued and complex- valued parameters.
LrcSSM (ours)
LreSSM with
LrcSSM with
LrcSSM with
8 maxxx mmax,
E IR
EC
EC
E C
E E R E R
EC
EC
E
a
E R
EC
E C
by
E IR
ER
E C
75.0 + 2.6 Heart
74.3 5.2
73.75 + 3.2
73.0 + 4.0
SCP1
8
82.9 2.7
83.1
84.8
SCP2
4.4 7.7
3.6
58.6 + 3.5
Ethanol
1.11
41.8222
4..
42.1 + 3.6
Motor
5.1
2..6
3.5
52.5 4.3
Worms
85.6
85.0 65.0
88.3 5.7
86.1 + 6.3
Average
65.4 8.0
64.6 7.5
65.4 4.8
66.2 + 7.3
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that we scale up LRCs for efficient sequence modelling (LrcSSMs), which we show throughout the paper, first with the model design, then with experimen- tation results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes)
Justification:
See limitations Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I.mmitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model specificciii,,onn, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to- system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applliaa,,,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the com- munity, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes)
Justification: Yes, they are in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results the paper to the extent that affects the main claims and/ or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes)
Justification: We provide details on the datasets, the hyperparameter grid used, the final hyperparameters, the hardware setup, etc. in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibil- ity. In the case of closed-soures models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in sup- plemental material?
Answer: [Yes]
Justification: We make the code publicly available on GitHub https: MoniParsang/LreSS)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines [https://nips.ce.
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines mnttps:/niips.. http://ippsccccplliiccpccccc wwtpcccpc.ggp..cceeooooooo httpiii/ggiide/Cdeeeeeee
wwt.:Suwbmssooooll111ooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/detaíls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appro- priate information about the statistical significance the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide the standard deviations in each table we report (over 5 seeds).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, con- fidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1 -sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Yes, we specified the hardware with the memory capacity and average runtime in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https httttt::////eietii h/teer/is.......e
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics. Our research does not involve human subjects or participants. Related to the data, we use publicly available datasets. We not see any societal impact or potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
24

---

Justification:
We work on scaling non non-linear state space models, leading to better effi- ciency. We are not tied any particular applications and we do not see any direct path to any negative applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g,, disinformation, generating fake profiles, surveillance), fairness consid- erations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mit- igation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any pretrained generative model or scraped dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite Gonzalez et al. 8, whose code we build on top of for the paral- lelization, and Farsang et al. for the model 5 with the combination for the experimental design of Rusch et al. 37..
Guidelines:
The answer NA means that the paper does not use existing assets.
25

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g,, CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
New assets
Question: Are assets introduced in the paper well documented and is the documen- tation provided alongside the assets?
Answer:
[Yes]
Justification: We include an anonymized version of the code now. Later, we will make publicly available on GitHub.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/ htt:://dde/oooeee model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
26

---

Justification:
We do not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, standard components.
Please refer our LLM policy httt://///ourppss.c/o/ofeereeee for what should or should not be described.
27


Reviews:
[{"review_id": "wUAIIPHaSw", "reviewer": "Reviewer_dhK1", "summary": "This paper introduces LrcSSM, a switching linear system (given by equations 12-14), where the diagonal transition matrix A depends on both input u **and state x** (i.e., the system is still nonlinear), They show that LrcSSM can outperform Mamba on three long time series classification tasks  drawn from sensor recordings, the longest of which (T=18K) is eigenworms.", "questions": "The actionable items that I would like to see to give this paper an accept center on the experiments.\n\n1. I need to see Long Range Arena (LRA) (https://arxiv.org/abs/2011.04006) included as a second experiment to advocate for publication of this paper. I don't know if LrcSSM will do better than LRU on this dataset or not---and I don't think it's critical for publication that LrcSSM do better on LRA---but I think it's very important for publication for this benchmark dataset to be included. LRA is much more standard than UEA-MTCSA and I think it's really important for readers to be able to see the results. If LrcSSM crushes on LRA---great!! If it has weaker performance, I think this opens up a great opportunity to dive deep into the differences between these datasets and comment on use cases for linear vs nonlinear RNNs.\n2. The abstract explicitly claims that the model follows compute-optimal scaling law regime. However, the evidence for this is give only in appendix A.2, which states \n> Hence, $\\beta \\approx 0.42$ is a defensible prior for LrcSSM\nTo me, it seems like a claim made in the abstract should be demonstrated empirically or LrcSSM itself, not handwaved in the appendix based on mamba experiments.\n3. Something is odd with the ablations reported in Table 7. What are the $\\pm$ terms and how are they computed? And why are they so much higher for average than for the original experiments. It seem like based on the very large $\\pm$ terms, the differences between the different architectures are not statistically significant---is this a correct interpretation? In essence, it looks like input dependent only is on par with state dependent (and will run faster as it actually will converge in the 1 pscan). Could the authors comment on this interpretation of Table 7?\n4. Would it be possible to report the wallclock times for the different architectures? I'm curious how their training time compares.\n5. Would it be possible to add Liquid-S4 to the baselines? It is strange that Liquid-S4 is mentioned by name in the abstract, but not included in the experiments.\n6. Do we really believe the mamba performance on eigenworms? it seems unbelievably low, as if a real effort on getting strong performance was not made. I see that it is reported from another paper, but would it be possible to try to train your own mamba baseline, to genuinely try to make it as strong as possible, and to document your approach? For example, in Table 7, in the input dependent only (not state dependence) LrcSSM is reported at 86.7% on eigenworms. Mamba, which is also input dependent, is reported at 70%. How can this be? Can the authors comment on why there is such a big difference between these two input dependent switching linear systems? This is extremely hard for me to believe, I would have thought that Mamba and input-dependent-only LrcSSM would be extremely similar.\n7. Could the authors clarify whether they use ELK or quasi-DEER in their experiments? If they use ELK, how did they set the hyperparameter $\\lambda$, and could they report its value?\n8. Would it be possible to include a fair comparison against a transformer baseline, such as in Amos et al \"Never train from scratch\" (https://arxiv.org/pdf/2310.02980)", "limitations": "The abstract appears to contradict the limitations section.\n\nThe abstract reads (lines 2-4)\n> By forcing the state-transition matrix to be diagonal and learned at every step, the full sequence can be solved in parallel with a single prefix-scan.\n\nLimitations section says (lines 268-9)\n> However, we also have to take into account that LRCs solved by ELK needed more Newton steps to converge at each iteration, which linear SSMs do not require.\n\nFor publication, I would really like to see graphs/tables showing the number of Newton steps needed for LrcSSM to converge, and a comparison of wall clock time (to answer the question: is LrcSSM much slower in practice than its linear counter parts?).", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Very clever way to make quasi-DEER and quasi-ELK work at scale---simply make the nonlinear RNN diagonal!\n* interesting choice of benchmark with UEA-MTCSA, using real-world measurements of dynamical systems\n\nWeaknesses:\n* Limited experimental validation: the experiments are pretty slim and should be built out\n* Based on this paper, I really don't know if this approach works at scale any better than existing approaches", "was_revised": true, "final_justification": "I raised my score because of the strengthened experimental evaluation. In particular, their wall clock time results are extremely exciting, indicating the speed of LrcSSM. Also exciting are the fast convergence rates of quasi-DEER. These results indicate that LrcSSM has tremendous promise to work at large scale.\n\nThis innovative paper has opened up new perspectives for me on the parallelization of nonlinear recurrent architectures. It has already begun to affect how I think about future research directions. Papers like these should definitely be discussed at conference like NeurIPS. I have strong conviction and confidence that this paper should be accepted for publication.", "rebuttal": "We thank the reviewer for their highly constructive feedback. We appreciate it a lot and will incorporate it into our revision. Here are our replies:\n\n> 1. LRA benchmarks and comparing linear vs. non-linear models\n\nThank you for the suggestion. We would like to point out that, particularly in recent work, researchers have increasingly shifted from the LRA benchmarks to the UEA-MTSCA archive. Our decision to focus on UEA-MTSCA aligns with this trend, as reflected in several recent top publications [1]-[3]. To further support this direction, we also cite here additional examples published on arXiv after the NeurIPS submission, in May/June [4]-[8].\n\nThat said, we completely agree that including LRA would provide valuable context and comparison. We are very interested in exploring how our non-linear LrcSSM performs on LRA, especially in relation to recent linear models such as LinOSS, which also don't report performance on these benchmarks. However, we believe such a broader investigation would be more out of the scope of the current paper.\n\nNonetheless, one can address this point in future work by exploring which types of datasets benefit more from Transformer-like, linear-SSM-like, or non-linear-SSM-like architectures.\n\nFor now, as also pointed out by other reviewers, we ran additional experiments on the PPG-DaLiA dataset [9], and can report the following results (baselines are taken from [3]):\n\n| Model                   | MSE × 10⁻²       |\n|----------------------------|----------------------|\n| NRDE      | 9.90 ± 0.97         |\n| NCDE     | 13.54 ± 0.69        |\n| Log-NCDE  | 9.56 ± 0.59         |\n| LRU      | 12.17 ± 0.49        |\n| S5        | 12.63 ± 1.25        |\n| S6             | 12.88 ± 2.05        |\n| Mamba          | 10.65 ± 2.20        |\n| LinOSS-IMEX                     | 7.5 ± 0.46          |\n| LinOSS-IM                       | 6.4 ± 0.23          |\n| LrcSSM (ours)                         | 10.89 ± 0.96        |\n\nWe ran the hyperparameter tuning following the same protocol as before. LrcSSM achieves performance comparable to other models, demonstrating its competitiveness in this benchmark.\n\n>2. Compute-optimal scaling law regime\n\nThank you for raising this point. We agree that claims in the abstract should be clear and well-supported. Our intention was not to present a new empirical measurement of the scaling exponent $\\beta$ for LrcSSM, but rather to argue based on properties of LrcSSM that it is expected to follow the same compute-optimal scaling regime. We will revise the abstract to clarify this and note that a full scaling study for LrcSSM is a valuable direction for future work.\n\n>3. Table 7 and input dependency\n\nIn Table 7, we report the mean and standard deviation of accuracy across the 5 different dataset splits. We observed that performance can vary substantially from split to split, which naturally leads to higher standard deviations.\n\nThe deviations appear larger in Table 7 compared to earlier tables because these results are not fully tuned over the entire hyperparameter grid. Instead, we use a standardised configuration across ablations (6 layers, 64-dimensional input encoding, and 64 hidden states) to ensure comparability across the model variants.\n\nRegarding statistical significance: yes, the standard deviations are relatively large, which makes the differences between variants less statistically robust. However, this is a common observation in Table 3 as well, where many baseline models show similar variance. This stems from the inherent variability in achievable accuracies across the datasets.\n\nDespite the high variance, we focused on identifying configurations with the highest mean accuracy. Your interpretation is correct: the input-dependent variant, which converges in 1 iteration, but performs slightly worse than the state-dependent version. Given this trade-off, we choose the better performing model, as it does not impose a significant computational overhead, please check our reported results in the next point.\n\n> 4. Wall-clock times\n\nWall clock time (in seconds) for 1000 training steps of the best performing (tuned) models for each dataset, same as it was reported in [3]. \n\n| Dataset     | NRDE  | NCDE  | Log-NCDE | LRU | S5 | Mamba | S6 | LinOSS-IMEX | LinOSS-IM | LrcSSM (ours) |\n|-------------|-------|-------|----------|-----|----|--------|-----|--------------|------------|--------|\n| Heartbeat   | 9539  | 1177  | 826      | 8   | 11 | 34     | 4   | 4            | 7          | 24     |\n| SCP1        | 1014  | 973   | 635      | 9   | 17 | 7      | 3   | 42           | 38         | 12     |\n| SCP2 | 1404  | 1251  | 583      | 9   | 9  | 32     | 7   | 55           | 22         | 16     |\n| EthanolConcentration | 2256  | 2217  | 2056     | 16  | 9  | 255    | 4   | 48           | 8          | 17     |\n| MotorImagery | 7616  | 3778  | 730      | 51  | 16 | 35     | 34  | 128          | 11         | 29     |\n| EigenWorms  | 5386  | 24595 | 1956     | 94  | 31 | 122    | 68  | 37           | 90         | 33     |\n\nIterations needed for convergence for LrcSSM on average per dataset in the same order: 3.4, 2.1, 4.4, 4.6, 3.0, 3.8.\nTo make it explicit, this number is 1.0 for all other models across all datasets. \nAs one can see, we do not lose significant runtime due to the extra iterations that are needed for the states to converge.\n\n> 5. Liquid-S4 baseline\n\nInstead of reporting results for Liquid-S4 directly, we include S6, which builds on the same core idea, which is an input-dependent \n$A$ matrix, but incorporates parallel scan operations for improved efficiency. In this sense, S6 can be viewed as a more efficient version of Liquid-S4, and we include it as a representative of that model class.\n\n> 6. Mamba results\n\nWe would like to clarify that Mamba is an overall architecture design that uses S6 (or other SSM layers) as its core building block. Therefore, the more appropriate comparison is between S6 and LrcSSM. For the Eigenworms dataset, we report an accuracy of 85.0% for S6, which aligns closely with our results of the input-dependent version of LrcSSM in Table 7.\n\nThe difference in performance between S6 and the input-dependent-only LrcSSM likely stems from the linear nature of S6 vs the non-linear model of LrcSSM. This distinction can explain the performance gap observed despite both being input-dependent systems.\n\n>7. ELK\n\nThank you for pointing this out. We use (quasi)-ELK, which in our case is equivalent to ELK because our Jacobian matrices are diagonal. Thus, it is not an approximation but an exact computation for our model. We used the default parameter setting, with lambda=1/10^8. We will include this information explicitly in the paper.\n\n>8. Comparison against Transformers\n\nGood point. Before, we missed this recent publication from last NeurIPS, which introduced a new Transformer model called Rough Transformer (RFormer) [1] and evaluated it on the same benchmarks. LrcSSM achieves better performance compared to the Transformer and competitive/better than the Rough Transformer baselines across the evaluated datasets. We will incorporate these results into our paper. \n\n| Dataset    | Transformer       | RFormer          | LrcSSM (ours)     |\n|------------|-------------------|------------------|-------------------|\n| Heartbeat  | 70.5 ± 0.1        | **72.5 ± 0.1**       | **72.7 ± 5.7**        |\n| SCP1 | 84.3 ± 6.3        | 81.2 ± 2.8       | **85.2 ± 2.1**        |\n| SCP2 | 49.1 ± 2.5        | 52.3 ± 3.7       | **53.9 ± 7.2**       |\n| EthanolConcentration | **40.5 ± 6.3**        | 34.7 ± 4.1       | 36.9 ± 5.3        |\n| MotorImagery | 50.5 ± 3.0        | 55.8 ± 6.6       | **58.6 ± 3.1**        |\n| EigenWorms | OOM               | **90.3 ± 0.1**       | **90.6 ± 1.4**        |\n| **Average**| 59.0       | 64.5      | **66.3**       |\n\n> Limitations:\n- What we meant by \"parallel with a single prefix-scan\" is that each iteration of our iterative algorithm processes the entire sequence of length $T$ simultaneously, rather than processing the sequence elements one at a time as in non-linear RNNs. We'll revise our text to make this clear.\n- See point 4.\n\nReferences:\n\n[1] Moreno-Pino, Fernando, Alvaro Arroyo, Harrison Waldon, Xiaowen Dong and Álvaro Cartea. “Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching.” Neural Information Processing Systems (2024).\n\n[2] Benjamin Walker, Andrew Donald McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. \"Log neural controlled differential equations: The lie brackets make a difference.\" Forty-first International Conference on Machine Learning (2024).\n\n[3] T. Konstantin Rusch and Daniela Rus. \"Oscillatory State-Space Models\". Thirteenth International Conference on Learning Representations (2025).\n\n[4] Walker, Benjamin, Lingyi Yang, Nicola Muca Cirone, Cristopher Salvi, and Terry Lyons. \"Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models.\" arXiv preprint arXiv:2505.17761 (2025).\n\n[5] Boyer, Jared, T. Konstantin Rusch, and Daniela Rus. \"Learning to Dissipate Energy in Oscillatory State-Space Models.\" arXiv preprint arXiv:2505.12171 (2025).\n\n[6] Pourcel, Guillaume, and Maxence Ernoult. \"Learning long range dependencies through time reversal symmetry breaking.\" arXiv preprint arXiv:2506.05259 (2025).\n\n[7] Nzoyem, Roussel Desmond, Nawid Keshtmand, Idriss Tsayem, David AW Barton, and Tom Deakin. \"Weight-Space Linear Recurrent Neural Networks.\" arXiv preprint arXiv:2506.01153 (2025).\n\n[8] Karuvally, Arjun, Franz Nowak, Anderson T. Keller, Carmen Amo Alonso, Terrence J. Sejnowski, and Hava T. Siegelmann. \"Bridging Expressivity and Scalability with Adaptive Unitary SSMs.\" arXiv preprint arXiv:2507.05238 (2025).\n\n[9] Reiss, Attila, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. 2019. \"Deep PPG: Large-Scale Heart Rate Estimation with Convolutional Neural Networks\" Sensors 19, no. 14: 3079. doi: 0.3390/s19143079"}, {"review_id": "QLe9O8ZFnG", "reviewer": "Reviewer_YHNE", "summary": "Inspired by biological neurons, this paper proposes LrcSSM, a nonlinear state space model that yields a diagonal Jacobian matrix. Despite its nonlinearity, the model adopts a linearization-based parallelization strategy using the Jacobian, ensuring numerically stable gradient computation. The proposed approach is validated on both short- and long-horizon time-series classification tasks, demonstrating competitive accuracy compared to baseline models.", "questions": "1. Is it appropriate to refer to the nonlinear update as a state-transition matrix, considering that the term conventionally implies linearity in control theory? Wouldn’t this cause confusion?\n2. Why are key nonlinear SSM/recurrent baselines such as NSSM, xLSTM, or Quasi-DEER not included in the comparison?\n3. Would it be helpful to provide a more detailed explanation of LinOSS when introducing LSSM, given their conceptual similarity?\n4. There seems to be some inconsistency in how the proposed model is categorized—it is referred to as both an LSSM and an NSSM in different parts of the paper. Would it be possible to clarify and unify it? It would also be helpful if the authors could explicitly state which class of models their approach belongs to, and highlight in which aspects it offers advantages over existing methods.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths\n    1. A novel design that clearly separates state-dependent and input-dependent dynamics.\n    2. Well-conducted ablation study analyzing both input/state dependency and real-/complex-valued parameterizations.\n- Weaknesses\n    1. The Introduction contains a lot of awkward wording and phrasing, and suffers from a lack of proper citations. Abbreviations are confusingly defined throughout the paper.\n    2. On the UEA benchmark, the proposed model does not show significant performance gains over baseline methods. In particular, it does not clearly outperform LinOSS.\n    3. It is unclear whether the authors claim SOTA in  recurrent models. The experimental comparison lacks strong baselines from this category (e.g., xLSTM, Quasi-DEER, other NSSMs).", "was_revised": true, "final_justification": "While concerns about writing clarity and performance margins remain, the revised perspective on the technical novelty of the proposed approach justifies a more favorable assessment. I am thus raising my score to Weak Accept.", "rebuttal": "We thank the reviewer for their comments. In the responses below, we aim to clarify the concerns raised.\n\n> 1. State-transition matrix\n\nThe reviewer is correct, this term usually refers to linear systems. To incorporate this feedback, we will explicitly highlight that we mean *non-linear* state transition matrix in the text when we talk about non-linear SSMs.\n\n> 2. Comparison with nonlinear RNNs\n\nTo the best of our knowledge, there are currently no other NSSMs. However, the reviewer raises an excellent point. To enable comparison with other types of non-linear models, we formulated a general framework that allows scaling any non-linear RNN in the same way as we showed by constructing a diagonal Jacobian.\n\nUsing this approach, we extend our method to build NSSMs from Minimal Gated Units (MGU) [1], Gated Recurrent Units (GRU) [2], and LSTM [3], which we refer to as MguSSM, GruSSM and LstmSSM, respectively. We report their performance in the following table:\n\nTest accuracy across *non-linear* models:\n\n| Dataset     | MguSSM      | GruSSM       | LstmSSM      | LrcSSM (ours)      |\n|-------------|---------------------|----------------------|----------------------|----------------------|\n| Heartbeat   | 74.00 ± 4.78        | **75.67 ± 4.67**         | **75.00 ± 3.50**        | **75.0 ± 2.6**           |\n| SCP1        | 78.33 ± 6.62        | 80.24 ± 4.16         | 78.81 ± 3.05         | **84.8 ± 2.8**           |\n| SCP2        | 49.64 ± 9.87        | 52.50 ± 3.31         | 51.07 ± 9.49         | **55.4 ± 7.7**          |\n| EthanolConcentration | 31.05 ± 4.21        | 34.47 ± 1.93         | 32.63 ± 5.61         | **36.1 ± 1.1**           |\n| MotorImagery | **56.43 ± 4.74**        | 49.64 ± 7.27         | 54.29 ± 3.31         | **55.7 ± 4.1**           |\n| EigenWorms  | **90.00 ± 5.15**        | 86.11 ± 6.33         | 82.22 ± 4.51         | 85.6 ± 5.4           |\n| **Average** | 63.2        | 63.1          | 62.34        | **65.4**         |\n\nHere, similar to our ablation studies, we used 6 layers with a 64-dimensional input encoding and 64-dimensional state. The results show that LrcSSM outperforms the other NSSMs built from MGU, GRU and LSTM.\n\n> 3. LinOSS relation to LSSMs\n\nGood point. LinOSS is a special type of LSSM, which is based on forced linear second-order ODEs. Meanwhile, the other LSSMs have first-order linear dynamics.\n\n> 4. Categorization\n\nThank you for pointing this out. We will go through the paper and clarify the terminology to ensure consistency.\n\nFor this and the previous point, we plan to include visualizations to illustrate how these models form categories. However, due to rebuttal restrictions requiring text-only responses, we are unable to provide them at the moment.\n\nIn our paper, we refer to NSSMs (*non-linear* state-space models) as those with *non-linear* transition matrices $A$, while LSSMs (linear state-space models) have linear transition matrices, which makes them inherently parallelizable. Our work focuses on the NSSM category, which has traditionally been avoided due to the belief that such models could not be solved in parallel and required sequential processing over the sequence length.\n\nHowever, recent work [4] introduced the ELK algorithm as a way to parallelize these models efficiently and approximately. Building on this, we propose a model architecture with a diagonal Jacobian structure, enabling exact (not quasi) parallel computation for NSSMs.\n\nSpecifically, we explore LrcSSM, a model that leverages this structure alongside the new scaling technique of ELK. As shown in our experiments, LrcSSM performs strongly compared to other NSSMs (see above) and even many leading LSSMs (see the paper).\n\nThe key advantage of LrcSSM is that it allows for *non-linear* dynamics within a parallelizable framework, something not achieved before. While this introduces a small number of additional iterations per step, it does not significantly increase runtime and yields performance competitive with, or superior to, strong baselines like LRU (which argued non-linearity was unnecessary), S5, Mamba, and S6.\n\nWhile LinOSS remains a strong baseline due to its rich second-order dynamics, which we plan to explore further in future work, our LrcSSM model introduces a novel direction by incorporating a *non-linear*, input- and state-dependent transition matrix $A$, which we believe adds a meaningful contribution to the field and opens a new category of exploring NSSMs.\n\nReferences:\n\n[1] Zhou, Guoxiang, Jianxin Wu, Chen-Lin Zhang and Zhi-Hua Zhou. “Minimal gated unit for recurrent neural networks.” International Journal of Automation and Computing 13 (2016): 226 - 234.\n\n[2] Cho, Kyunghyun, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio. “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.” Conference on Empirical Methods in Natural Language Processing (2014).\n\n[3] S. Hochreiter and J. Schmidhuber, \"Long Short-Term Memory,\" in Neural Computation, vol. 9, no. 8, pp. 1735-1780, 15 Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.\n\n[4] Xavier Gonzalez, Andrew Warrington, Jimmy Smith, and Scott Linderman. Towards scalable\nand stable parallelization of nonlinear rnns. Advances in Neural Information Processing\nSystems, 37:5817–5849, 2024"}, {"review_id": "3On4JmFdTI", "reviewer": "Reviewer_3zsp", "summary": "The paper presents a variation of the liquid-resistance, liquid capacitance recurrent neural network. A closely related method, which is ELK (and their algorithm 1) involves the computation of a Jacobian (on line 7) followed by the extraction of the diagonal of that Jacobian, which is done to facilitate a computationally tractable approximation. The contribution here involves modifying the underlying recurrent neural network architecture in such a way as to obtain a Jacobian that is diagonal by construction. This makes things tractable while avoiding the approximation. The resulting algorithm is demonstrated to be highly competitive in terms of accuracy, while being argued (if not quantitatively) to be efficient. There is also an interesting theoretical stability result that is claimed not to exist for the related methods.", "questions": "appendix. Please move the definition forward or otherwise drop it.\nQ2) Is it possible to compute the diagonal of the jacobian efficiently without simply computing the full dense jacobian first? I am referring here to the ELK algorithm, not the proposed method which obviates the question by having a diagonal JAcobian by construction.\nQ3) Can you share some timings? There are theoretical big O complexities in the appendix, but as these are not actually better than the related methods, and since efficiency is a big point in the paper, it would be helpful to get an idea of run times.\nQ4) Unless I missed it, there are no discrete sequences here. Is this possible? Why isn’t it included?\nQ5) (related to Q4) Is it possible to compare with a transformer?\nQ6) Why not include the analogous discretizations for STC and LRC in equation (7)? Apologies if I am missing something here.\nQ7) Why did you not use the PPG-Dalia dataset from [32]? The omission is odd given you use everything else.\nQ8) In figure 1, wouldn’t it make sense to move the pre-normalization block outside the dotted box?\nQ9) Why only 2, 4 and 6 layers? This seems super shallow compared to transformers.\nQ10) What was the actual objective / loss function for training?\n\nMinor points:\nM1 Bad comma on line 97\nM2 Save space by transposing table 3?\nM3 Use \\mathrm for the max in (1) and (2) and similarly throughout the paper.\nM4 Line 284 - quasi approximation is unclear.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper attacks an important problem, which is long range dependencies and efficient sequence modelling. The contribution is rather elegant - avoiding an approximation by cleverly designing the transition model to not need that approximation, namely the diagonal approximation of the aforementioned Jacobian. It is also cool to see a theoretical stability result, although I have not verified this result in detail (which is in the appendix).\n\nThe empirical results are pretty strong. Given that table 1 was created in [32] it is impressive to match it, since presumably that work chose a setup that is good for their method.\n\nWeakness:\n\nThe actual timing gains are not demonstrated in detail. I understand that this is difficult when a lot depends on the details of the implementation, and it is not reasonable to reimplement all related methods.\n\nIt would be good to at least include the theoretical statement in the main paper, if not the proof.\n\nThere are a number of potential minor weaknesses in the questions section, though the authors may please resolve my misunderstanding for a number of them if possible.", "was_revised": true, "final_justification": "I have considered everything already, thanks.", "rebuttal": "We thank the reviewer for their thoughtful feedback, which has helped us improve our work.\n\nSee our responses below:\n\n> appendix. Please move the definition forward or otherwise drop it.\n\nNoted. Thank you.\n\n> Q2) Efficient computation of the diagonal of the jacobian\n\nNo, unfortunately, it is not straightforward.\n\n> Q3) Run times\n\nHere are our wall-clock times for the best models for each dataset, in seconds for 1000 training steps:\n\n| Dataset     | NRDE  | NCDE  | Log-NCDE | LRU | S5 | Mamba | S6 | LinOSS-IMEX | LinOSS-IM | LrcSSM (ours) |\n|-------------|-------|-------|----------|-----|----|--------|-----|--------------|------------|--------|\n| Heartbeat   | 9539  | 1177  | 826      | 8   | 11 | 34     | 4   | 4            | 7          | 24     |\n| SCP1        | 1014  | 973   | 635      | 9   | 17 | 7      | 3   | 42           | 38         | 12     |\n| SCP2        | 1404  | 1251  | 583      | 9   | 9  | 32     | 7   | 55           | 22         | 16     |\n| EthanolConcentration | 2256  | 2217  | 2056     | 16  | 9  | 255    | 4   | 48           | 8          | 17     |\n| MotorImagery | 7616  | 3778  | 730      | 51  | 16 | 35     | 34  | 128          | 11         | 29     |\n| EigenWorms  | 5386  | 24595 | 1956     | 94  | 31 | 122    | 68  | 37           | 90         | 33     |\n\nResults for the other models are from [1]. We would like to point out that our efficiency claims are *not against* other linear models, because they are really efficient. We aimed to bring up *non-linear* models to the same/comparable efficiency.\nIterations needed for convergence for LrcSSM on average per dataset in the same order: 3.4, 2.1, 4.4, 4.6, 3.0, 3.8. \nTo make it explicit, this number is 1.0 for all other models across all datasets. \nAs one can see, we do not gain significant runtime due to the extra iterations that are needed for the states to converge.\n\n> Q4) Discrete sequences \n\nWe are not sure if we get the question. The datasets we are using are discrete sequences.\n\n> Q5) Transformers \n\nGood point. Before, we missed a recent publication [2] from last NeurIPS, where Transformers and Rough Transformers (RFormer) were evaluated on the same datasets. We report their results [2] here, which show that LrcSSM achieves better performance than these Transformer-based models. We will incorporate these new model comparisons into our paper.\n\n| Dataset    | Transformer       | RFormer          | LrcSSM (ours)     |\n|------------|-------------------|------------------|-------------------|\n| Heartbeat  | 70.5 ± 0.1        | **72.5 ± 0.1**       | **72.7 ± 5.7**        |\n| SCP1       | 84.3 ± 6.3        | 81.2 ± 2.8       | **85.2 ± 2.1**        |\n| SCP2       | 49.1 ± 2.5        | 52.3 ± 3.7       | **53.9 ± 7.2**       |\n| EthanolConcentration | **40.5 ± 6.3**        | 34.7 ± 4.1       | 36.9 ± 5.3        |\n| MotorImagery | 50.5 ± 3.0        | 55.8 ± 6.6       | **58.6 ± 3.1**        |\n| EigenWorms | OOM               | **90.3 ± 0.1**       | **90.6 ± 1.4**        |\n| **Average**| 59.0      | 64.5      | **66.3**       |\n\n> Q6) Analogous discretizations for STC \n\nIndeed, the same discretization approach can be applied to STCs. One might naturally ask why a model similar to LRC was not constructed for STCs. To address this, we present an alternative formulation of StcSSM, following the same steps used to achieve a diagonal Jacobian matrix (without the elastance term indicated in orange), in order to compare the performance of these two bio-inspired models.\n\nTest accuracy across the same model architecture of 6 layers, 64-dimensional input encoding and 64-dimensional states:\n\n| Dataset     | StcSSM (also ours, but not the main proposed model)        | LrcSSM (ours)  |\n|-------------|--------------------|----------------------|\n| Heartbeat   | **75.7 ± 5.0**         | **75.0 ± 2.6**           |\n| SCP1        | 78.8 ± 4.3         | **84.8 ± 2.8**           |\n| SCP2        | 50.4 ± 6.9         | **55.4 ± 7.7**           |\n| EthanolConcentration | **36.8 ± 4.0**         | **36.1 ± 1.1**           |\n| MotorImagery | 53.9 ± 4.4         |**55.7 ± 4.1**           |\n| EigenWorms  | **85.0 ± 4.8**         | **85.6 ± 5.4**           |\n| **Average** | 63.4        | **65.4**          |\n\nThese results show that our originally proposed LrcSSM model performs better than StcSSM, and this performance gain comes from the extra expressiveness of the underlying LRC model, which has an input- and state-dependent membrane capacitance in its bio-inspired model, which is a constant term in the case of the STC model.\n\n> Q7) PPG-Dalia\n\nThank you for pointing this out. We ran this additional experiment on the PPG-DaLiA dataset [4], and report the following results. As before, baselines are taken from [1]:\n\n| Model                   | MSE × 10⁻²       |\n|----------------------------|----------------------|\n| NRDE      | 9.90 ± 0.97         |\n| NCDE     | 13.54 ± 0.69        |\n| Log-NCDE  | 9.56 ± 0.59         |\n| LRU      | 12.17 ± 0.49        |\n| S5        | 12.63 ± 1.25        |\n| S6             | 12.88 ± 2.05        |\n| Mamba          | 10.65 ± 2.20        |\n| LinOSS-IMEX                     | 7.5 ± 0.46          |\n| LinOSS-IM                       | 6.4 ± 0.23          |\n| LrcSSM (ours)                         | 10.89 ± 0.96        |\n\nWe ran the hyperparameter tuning following the same protocol as before. LrcSSM achieves performance comparable to other models, demonstrating its competitiveness in this benchmark.\n\n> Q8) Moving the pre-normalization block\n\nOur goal was to apply a normalization layer before each SSM block. Moving it outside would only normalize the original input to the first layer, rather than each layer's input. This approach is consistent with other models, where normalization is similarly applied before the SSM layer.\n\n> Q9) Number of layers \n\nWe used the same hyperparamater tuning grid as it was used for the other models in [1],[3] to have full comparability. This grid search includes layers={2,4,6}.\n\n> Q10) Loss function for training\n\nWe used softmax cross-entropy for the classification tasks.\n\n\nReferences:\n\n[1] T. Konstantin Rusch and Daniela Rus. \"Oscillatory State-Space Models\". Thirteenth International Conference on Learning Representations (2025).\n\n[2] Moreno-Pino, Fernando, Alvaro Arroyo, Harrison Waldon, Xiaowen Dong and Álvaro Cartea. “Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching.” Neural Information Processing Systems (2024). \n\n[3] Benjamin Walker, Andrew Donald McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. \"Log neural controlled differential equations: The lie brackets make a difference.\" Forty-first International Conference on Machine Learning (2024).\n\n[4] Reiss, Attila, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. 2019. \"Deep PPG: Large-Scale Heart Rate Estimation with Convolutional Neural Networks\" Sensors 19, no. 14: 3079. doi: 0.3390/s19143079"}, {"review_id": "3WTPvXyTrX", "reviewer": "Reviewer_H8n5", "summary": "The paper introduces LrcSSM, a non-linear state-space model designed for efficient long-sequence modeling. The model is based on the bio-inspired Liquid-Resistance Liquid-Capacitance (LRC) network, a type of non-linear RNN. The core contribution is a modification to the underlying LRC dynamics to enforce that its state-transition Jacobian is inherently diagonal. This \"diagonal-by-design\" approach allows the model to be parallelized over the sequence length using the iterative ELK (Evaluating Levenberg-Marquardt via Kalman) framework without the need for approximation. The authors demonstrate through experiments on long-sequence classification benchmarks that LrcSSM achieves competitive, and in some cases, state-of-the-art performance compared to linear SSMs like Mamba, S5, and S6.", "questions": "**Practical Efficiency:** The central claim of the paper is achieving efficiency comparable to modern SSMs. Could you please provide wall-clock training and inference time comparisons against key baselines like Mamba and S5 on a long-sequence task like EigenWorms? Crucially, how many ELK iterations are typically required for the states to converge during training, and how does this number affect the overall runtime?", "limitations": "Yes, the authors have a discussion section that touches on some limitations, such as the simple integration scheme used. However, the discussion could be significantly strengthened by explicitly addressing the practical computational overhead of the iterative ELK solver compared to the single-pass nature of key SSM baselines. This is a critical limitation of the proposed approach that directly impacts the paper's central claims of efficiency, and it warrants a more prominent and detailed discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.  **Strong Empirical Performance on some Long-Horizon Tasks:** The experimental results on datasets with very long sequences (e.g., EthanolConcentration, EigenWorms) show that the model outperforms several highly-regarded baselines like Mamba and S5 on these specific tasks, suggesting it is effective at capturing certain types of long-range dependencies.\n2.  **Formal Stability Guarantees:** The paper provides a theoretical analysis in the appendix showing that the model's gradients are stable and will not explode, a property not formally guaranteed by all input-dependent SSMs. This is a valuable theoretical contribution that adds to the model's robustness.\n3.  **Clarity:** The paper is generally well-written and easy to follow. The motivation, background, and proposed method are explained clearly.\n\n**Weaknesses:**\n\n1.  **Potentially Misleading Efficiency Claims:** The primary motivation is to scale non-linear RNNs to be as \"efficient\" as linear SSMs. However, the model relies on the ELK solver, which is an *iterative* algorithm. Each training step requires multiple forward/backward passes within the solver to converge on the hidden states. In contrast, baselines like Mamba or S5 require only a single, non-iterative parallel scan. The paper presents complexity in terms of a single pass (`O(TD)`) but omits the crucial factor of the number of iterations, `k`. The true cost is closer to `O(k * TD)`. Without wall-clock time comparisons, the claim of comparable efficiency to single-pass SSMs is unsubstantiated and a significant weakness.\n2.  **Ambiguous Attribution of Performance Gains:** While the model performs well, it is difficult to disentangle the source of the performance gains. Is it the bio-inspired LRC formulation, the state-and-input dependent non-linearity, or simply that the overall architecture (stacked blocks, normalizations, etc.) is well-tuned for these specific tasks? The paper argues for the superiority of its non-linear approach, but the evidence is not conclusive. For example, the model performs very well on EthanolConcentration but is average on others. A more thorough analysis is needed to isolate the impact of the core non-linear mechanism.\n3.  **Limited Significance and Scope:** The proposed method is essentially a clever application of the existing ELK framework to a specific, modified RNN. While successful, it does not represent a new paradigm for sequence modeling. It is an incremental improvement for parallelizing a particular class of non-linear models, and its broader impact on the field may be limited, especially given the practical efficiency concerns.\n4.  **Significant Model Simplification:** The \"diagonal-by-design\" approach requires removing all direct cross-neuron interactions within the state update, a major simplification of the original, more biologically-plausible LRC model. The paper does not adequately discuss the implications of this trade-off in expressivity.", "was_revised": true, "final_justification": "Since the reviewers addressed all of my comments, I will update my rating from 3 to 4.\n\nI expect that they will introduce all of this in the new revision of the manuscript, which will strongly improve the paper.", "rebuttal": "We would like to sincerely thank the reviewer for their thoughtful and detailed feedback. Many of the points raised were insightful and important, which we addressed in the following way:\n\n> Weakness 1 and Question on practical efficiency, runtime\n\nHere we report the wall-clock times for the best models for each dataset, in seconds for 1000 training steps:\n\n| Dataset     | NRDE  | NCDE  | Log-NCDE | LRU | S5 | Mamba | S6 | LinOSS-IMEX | LinOSS-IM | LrcSSM (ours) |\n|-------------|-------|-------|----------|-----|----|--------|-----|--------------|------------|--------|\n| Heartbeat   | 9539  | 1177  | 826      | 8   | 11 | 34     | 4   | 4            | 7          | 24     |\n| SCP1        | 1014  | 973   | 635      | 9   | 17 | 7      | 3   | 42           | 38         | 12     |\n| SCP2        | 1404  | 1251  | 583      | 9   | 9  | 32     | 7   | 55           | 22         | 16     |\n| EthanolConcentration | 2256  | 2217  | 2056     | 16  | 9  | 255    | 4   | 48           | 8          | 17     |\n| MotorImagery | 7616  | 3778  | 730      | 51  | 16 | 35     | 34  | 128          | 11         | 29     |\n| EigenWorms  | 5386  | 24595 | 1956     | 94  | 31 | 122    | 68  | 37           | 90         | 33     |\n\nNumber of iterations needed for convergence for LrcSSM on average per dataset in the same order: 3.4, 2.1, 4.4, 4.6, 3.0, 3.8.\n \nResults for the other models are from [1]. As one can see, we do not lose significant runtime due to the extra iterations that are needed for the states to converge.\n\nBy efficient sequence modeling, we mean that we have taken a step toward scaling nonlinear RNNs to nonlinear state-space models (NSSMs), which can process long sequences in parallel. As the reviewer rightly pointed out, NSSMs require $k$\niterations to converge to the desired state dynamics, unlike linear SSMs which don't require this. However, compared to traditional nonlinear RNNs, which require sequential computation over the full sequence length $L$, our method overcomes this limitation through parallelization, at the cost of a small number of additional iterations.\n\nIn practice, this is highly favorable: the number of iterations $k$ is typically based on experiments very small (between 2 and 4), while sequence lengths $L$ range from 400 up to 18,000. We observed that the number of iterations $k$ is independent of both the sequence length $L$ and the state dimension $D$. Since $k$ is effectively a constant, we believe it is reasonable to omit it from the $O$-notation.\n\nThat said, we agree with the reviewer that this should be clearly discussed in the limitations or discussion section of the paper to make our claims more transparent.\n\n> Weakness 2. and 3. Performance gains and broader impact\n\nThe reviewer raised a critical point that we aim to address. Although in the submission we only demonstrated how to construct the model from LRC to LrcSSM, this approach can, in theory, be applied to any nonlinear RNNs of interest to obtain its SSM counterpart.\n\n**Sketch of the generalized method:**\n\nAny RNN can be reformulated by considering its underlying differential equations using an explicit first-order Euler integration scheme with a unit time step. We assume the non-linear RNN has the following general form:\n\n$x_t = q_t(x_{t-1},u_t) x_{t-1} + s_t(x_{t-1},u_t)$, \n\nwhere\n$q_t=\\sigma(f_q(x_{t-1},u_t))$ and $s_t=\\sigma(f_s(x_{t-1},u_t))$ are non-linear potentially state- and input-dependent gated functions, which we aim to express it as:\n\n $\\dot{x} = A(x,u)x+b(x,u)$. \n\nTo see this, consider:\n\n$\\dot{x} = (q(x,u)-1)x+s(x,u)$, \n\nwhich yields (using Euler integration with $\\Delta t=1.0$) the original equation:\n\n$x_t = x_{t-1} + \\dot{x} \\Delta t = x_{t-1} + (q_t (x_{t-1},u_t) -1)  x_{t-1} + s_t(x_{t-1},u_t) = q_t (x_{t-1},u_t)  x_{t-1} + s_t(x_{t-1},u_t) $.\n\nThus, in this general formulation:\n$A(x,u)=(q(x,u)-1)$, \n$b(x,u) = s(x,u)$.\n\nTo obtain the desired efficient representation, we define: \n\n$A(x,u) = \\mathrm{diag}([\\sigma(f_{q1}(x_1,u)) - 1, \\sigma(f_{qi}(x_i,u)) - 1, \\ldots, \\sigma(f_{qm}(x_m,u)) - 1])$\n\n$b(x,u) = [\\sigma(f_{s1}(x_1,u)), \\sigma(f_{si}(x_i,u)), ..., \\sigma(f_{sm}(x_m,u))]$.\n\n\n**Example: GRU:**\n\nThe GRU equations are:\n\n$z_t = \\sigma\\left(W_z \\left[ x_{t-1}, u_t \\right] + b_z \\right) = \\sigma(f_z)$\n\n$r_t = \\sigma\\left(W_r \\left[ x_{t-1}, u_t \\right] + b_r \\right)= \\sigma(f_r) $\n\n$c_t = \\tanh\\left(W_h \\left[ r_t x_{t-1}, u_t \\right] + b_h \\right) = \\tau(f_p) $ \n\n$x_t = (1 - z_t)  x_{t-1} + z_t  c_t $\n\nRewriting the update using the differential form:\n\n$\\dot{x} = z(-x + c) = -z x + z c$, to check, we get back to the original form (using Euler integration with $\\Delta t=1.0$)\n\n$x_t = x_{t-1} + \\dot{x} \\Delta t = x_{t-1} + -z_t x_{t-1} +  z_t  c_t   = (1 - z_t) x_{t-1} + z_t c_t $. \n\nTo express this in the form $\\dot{x} = A(x,u)x+b(x,u)$, we set:\n$\\dot{x} = A(x,u)x+b(x,u) =-z x + z c$, thus\n$A(x,u) = -z$, and \n$b(x,u)=z c$. \n\nUsing our proposed formulation for constructing $A(x,u)$ and $b(x,u)$:\n\n$A(x,u) = \\mathrm{diag}([-\\sigma(f_{z1}(x_1,u)), -\\sigma(f_{zi}(x_i,u)), ..., -\\sigma(f_{zm}(x_m,u))]$,\n\n$b(x,u) = [\\sigma(f_{z1}(x_1,u))\\tau(f_{p1}(x_1,u)), \\sigma(f^*_{zi}(x_i,u))\\tau(f_{pi}(x_i,u)), ..., \\sigma(f_{zm}(x_m,u))\\tau(f_{pm}(x_m,u))]$.\n\nThis reformulation results in a diagonal Jacobian matrix, enabling more efficient and exact (rather than quasi) computations for ELK.\n\nIn the line of this work, we extend our method to build up other nonlinear SSMs from Minimal Gated Units (MGU) [2], Gated Recurrent Units (GRU) [3], and LSTM [4], which we refer to as MguSSM, GruSSM and LstmSSM, respectively. We report their performance in the following table:\n\nTest accuracy across these *non-linear* models:\n\n| Dataset     | MguSSM      | GruSSM       | LstmSSM      | LrcSSM (ours)      |\n|-------------|---------------------|----------------------|----------------------|----------------------|\n| Heartbeat   | 74.00 ± 4.78        | **75.67 ± 4.67**         | **75.00 ± 3.50**        | **75.0 ± 2.6**           |\n| SCP1        | 78.33 ± 6.62        | 80.24 ± 4.16         | 78.81 ± 3.05         | **84.8 ± 2.8**           |\n| SCP2        | 49.64 ± 9.87        | 52.50 ± 3.31         | 51.07 ± 9.49         | **55.4 ± 7.7**          |\n| EthanolConcentration | 31.05 ± 4.21        | 34.47 ± 1.93         | 32.63 ± 5.61         | **36.1 ± 1.1**           |\n| MotorImagery | **56.43 ± 4.74**        | 49.64 ± 7.27         | 54.29 ± 3.31         | **55.7 ± 4.1**           |\n| EigenWorms  | **90.00 ± 5.15**        | 86.11 ± 6.33         | 82.22 ± 4.51         | 85.6 ± 5.4           |\n| **Average** | 63.2        | 63.1         | 62.34        | **65.4**         |\n\nHere, similar to our ablation studies, we used 6 layers with a 64-dimensional input encoding and 64-dimensional state. The results show that LrcSSM outperforms the other NSSMs built from MGU, GRU and LSTM using the same architecture. \n\nNote: The stacked blocks (2,4 or 6 are part of the hyperparameter search grid) and the normalization blocks are also present in other models reported too.\n\nOverall, with this, we show the performance gains in the same architecture across different non-linear models and we also show the broader impact of our paper, as this diagonalization technique can be applied to many RNNs.\n\n\n> Weakness 4: Model simplification\n\nAgain, this is a valuable point that we wanted to address more thoroughly. To this end, we evaluated the LrcSSM model with a dense $A$ matrix as well, to compare performance and assess any potential trade-offs. We used the same architectural design as before to ensure a fair comparison.\n\n| Dataset     | LrcSSM with dense $A$     | LrcSSM (diagonal $A$, proposed) |\n|-------------|------------------|-------------------------|\n| Heartbeat   | 72.3 ± 4.2       | **75.0 ± 2.6**              |\n| SCP1        | 83.6 ± 1.9       | **84.8 ± 2.8**              |\n| SCP2        | 49.6 ± 5.0       | **55.4 ± 7.7**              |\n| EthanolConcentration | 33.9 ± 1.5       | **36.1 ± 1.1**              |\n| MotorImagery | **55.7 ± 5.7**       | **55.7 ± 4.1**              |\n| EigenWorms  | 84.4 ± 3.8       | **85.6 ± 5.4**              |\n| **Average** | 63.3  | **65.4**         |\n\nThese results show that empirically, we do not lose the expressive capabilities of LRCs by constraining $A$ to be diagonal. We hypothesize that this is because the strict structure encourages the model to learn more efficient representations, while also enabling exact (rather than approximate) computations due to the design.\n\nReferences:\n\n[1] T. Konstantin Rusch and Daniela Rus. \"Oscillatory State-Space Models\". Thirteenth International Conference on Learning Representations (2025).\n\n[2] Zhou, Guoxiang, Jianxin Wu, Chen-Lin Zhang and Zhi-Hua Zhou. “Minimal gated unit for recurrent neural networks.” International Journal of Automation and Computing 13 (2016): 226 - 234.\n\n[3] Cho, Kyunghyun, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio. “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.” Conference on Empirical Methods in Natural Language Processing (2014).\n\n[4] S. Hochreiter and J. Schmidhuber, \"Long Short-Term Memory,\" in Neural Computation, vol. 9, no. 8, pp. 1735-1780, 15 Nov. 1997, doi: 10.1162/neco.1997.9.8.1735."}, {"review_id": "aqq42PbqoL", "reviewer": "Reviewer_hQD2", "summary": "The authors propose a diagonalisation strategy to speed up the training of nonlinear SSMs. Linearisation is replaced by an adaptive diagonalisation step, which guarantees that the Jacobian of the state-space transition matrices is diagonal without any information loss. The scheme is applied to bio-inspired Liquid-Resistance Liquid Capacitance networks.", "questions": "- Is comparing with quadratic-attention Transformers at equal compute the standard practice? Was this principle used to set the sequence lengths in the experiments?\n- Why does neglecting the off-diagonal entries of the Jacobian lead to information loss, but enforcing the state-transition matrix to be diagonal does not? What is the difference between the first approach and the inherent diagonalisation proposed on page 2?\n- You say that *synaptic parameters define constant matrices*. Is this true for general NSSMs or models with input-dependent transition matrices? \n- How strong is the effect of assuming the capacitance to be constant? Is this a standard assumption in the EEC literature? Would the proposed method work if the *capacitance has a nonlinear dependence* (possibly with reduced computational efficiency)?\n- Is Euler integration needed at each step? How many iterations were run on average?\n- According to the argument that any constant matrix can be diagonalised, it is reasonable to assume that a single state-space matrix is diagonal. When three matrices are involved, it requires the existence of a joint diagonalizer. Is this always the case?", "limitations": "I appreciate that the authors explicitly address the paper's limitations. In my understanding, the work does not have any potential negative societal impact.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Bio-inspired models are often praised for their energy efficiency. As traditional deep learning systems become increasingly unsustainable, developing alternative methods is an urgent need. \n- The proposed approach demonstrates how enforcing structural task-specific constraints can improve the performance of learning models. \n- The idea of an adaptive complex diagonalisation is inspiring. \n\n**Weaknesses**\n- The author should spend some more words explaining why they chose Liquid-Resistance Liquid Capacitance networks, why they are relevant, and whether the computational gains are expected to translate straightforwardly to other bio-inspired models.\n- The problem with the DEER method is computing the Jacobian of the transition matrix. It is unclear why forcing this by construction is better than approximately updating a full matrix. The authors may explain if this requires a trade-off between speed and accuracy.\n- The model requires learning a diagonal matrix at every time step. The authors should comment on its computational costs.\n- The models' performance over different datasets has a large variance. Extracting a clear winner from Table 3 is not straightforward. The computational complexity of the methods is not reported.", "was_revised": true, "final_justification": "I confirm my positive judgment and will vote to accept the paper.", "rebuttal": "We thank the reviewer for their constructive feedback. Many good points were raised that we address below.\n\nFor clarity, we numbered the questions to make them easier to follow. Please note that we moved Questions 2 and 4 to the end, where we also address points raised as weaknesses.\n\n> Q1: Equal compute and sequence lengths in the experiments\n\nYes, considering equal compute is becoming standard practice for fair comparison. In our experiments, the sequence lengths were determined by the characteristics of the datasets themselves, using their original length.\n\n> Q3: Synaptic parameters are constant matrices, how general...\n\nWhat we meant is that the synaptic parameters, the learned values $a_{ji}, b_{ji}$, $g_{ji}$ and $k_{ji}$ are not state- or input-dependent, they remain constant after training and define the fixed connectivity between neurons (neuron $j$ and $i$). This is generally true for any NSSM formulation based on this structure.\n\n> Q5: Euler integration and number of iterations\n\nWe follow the integration scheme from LRCs, which requires only a single Euler integration step. As a result, the system can be solved immediately without the need for multiple Euler iterations.\n\n> Q6: Matrix diagonalisation, joint diagonalizer\n\nThank you for bringing this up, we will clarify it in the revision. We explicitly constrain the model to learn diagonal matrices (which can be viewed as vectors). Therefore, the need for a joint diagonalizer does not arise, as the matrices are inherently diagonal by design.\n\n> Weakness 1 and Q4: Motivation for LRC and model with constant capacitance \n\nWe considered models of non-spiking neurons, such as those based on electrical and chemical synapses. Electrical synapses can be modeled using continuous-time recurrent neural networks (CT-RNNs), but they tend to show degraded performance compared to models based on chemical synapses. In such models, part of the transition matrix is fixed, as in many state-space models (SSMs) before, and lacks input or state dependency, which offers nothing novel in this regard.\n\nInstead, we focused on models of chemical synapses, the so called liquid neural networks, where the term liquid refers to the input- and state-dependent dynamics of the transition matrix. These dynamics are non-linear. Within this family, Liquid Time-Constant Networks (LTCs) do not have saturation (or gating) terms in their dynamics, which is desirable for achieving stable behavior. Fortunately, their saturated (gated) counterpart, Saturated LTCs (STCs), does include such terms. \n\nWe modified the model of STCs the same way as LRCs, achieving a diagonal Jacobian matrix (without the elastance term indicated in orange in the equations) and ended up StcSSMs. \nThis model has *constant* capacitance. Models coming from modeling chemical synapses assume this.\n\nWe included a comparison of this model against LrcSSM across the datasets, using a fixed setup of 6 layers, 64-dimensional encoded input and 64-dimensional states. We found that LrcSSM outperform StcSSM, highlighting the need for the non-linear membrane capacitance - pointed out by the reviewer. We will include this observation in our ablation section.\n\n| Dataset     | StcSSM       | LrcSSM (proposed model) |\n|-------------|-------------------|--------------------|\n| Heartbeat   | **75.7 ± 5.0**        | **75.0 ± 2.6**         |\n| SCP1        | 78.8 ± 4.3        | **84.8 ± 2.8**         |\n| SCP2        | 50.4 ± 6.9        | **55.4 ± 7.7**         |\n| EthanolConcentration | **36.8 ± 4.0**        | **36.1 ± 1.1**         |\n| MotorImagery | 53.9 ± 4.4        | **55.7 ± 4.1**         |\n| EigenWorms  | **85.0 ± 4.8**        | **85.6 ± 5.4**         |\n| **Average** | 63.4   | **65.4**    |\n\n\n> Weakness 2 and Q2: Comparing diagonal and full matrix results\n\nThis is a valuable point that we wanted to address more thoroughly. To this end, we evaluated the LrcSSM model with a dense $A$ matrix as well, to compare performance and assess any potential trade-offs. We used the same architectural design as before to ensure a fair comparison.\n\n| Dataset     | LrcSSM with dense $A$     | LrcSSM (diagonal $A$, proposed) |\n|-------------|------------------|-------------------------|\n| Heartbeat   | 72.3 ± 4.2       | **75.0 ± 2.6**              |\n| SCP1        | 83.6 ± 1.9       | **84.8 ± 2.8**              |\n| SCP2        | 49.6 ± 5.0       | **55.4 ± 7.7**              |\n| EthanolConcentration | 33.9 ± 1.5       | **36.1 ± 1.1**              |\n| MotorImagery | **55.7 ± 5.7**       | **55.7 ± 4.1**              |\n| EigenWorms  | 84.4 ± 3.8       | **85.6 ± 5.4**              |\n| **Average** | 63.3  | **65.4**         |\n\nThese results show that empirically, we do not lose the expressive capabilities of LRCs by constraining $A$ to be diagonal. We hypothesize that this is because the strict structure encourages the model to learn more efficient representations, while also enabling exact (rather than approximate) computations due to the design.\n\nThis is another important part pointed out by the reviewer, which we will include in the paper.\n\n\n\n> Weakness 3: Computational costs\n\nLet $T$ denote the input sequence length and $D$ the state dimension (i.e. the dimension of the diagonal of the matrix).  Traditional sequential methods are difficult to parallelize and require $O(D)$ memory and $O(T D^2)$ computational work. The ELK technique [1] addresses this by enabling efficient parallelization through diagonal Jacobian computation, reducing both memory and computational complexity to $O(T D)$. Our approach achieves the same complexity: $O(T D)$ for both\nmemory and computation. Since our model learns inherently diagonal matrices (i.e., vectors), Jacobians remain diagonal throughout, enabling efficient and scalable computation without additional overhead, making previously quasi-approximations exact.\n\n> Weakness 4: Large average variance in the datasets\n\nWe reported the variances in the submission to provide a more complete and transparent view of model performance, but actually, this is something omitted in all prior work using these datasets (only reporting mean values). The high variance is consistent across all models and reflects only the inherent variability in achievable accuracy on these datasets, which shifts the attention away from the more meaningful mean values. We decided now to only keep them for each dataset separately, because it is much more relevant there and shows the reliability of the models across 5 seeds.\n\nReferences:\n\n[1] Xavier Gonzalez, Andrew Warrington, Jimmy Smith, and Scott Linderman. Towards scalable and stable parallelization of nonlinear rnns. Advances in Neural Information Processing Systems, 37:5817–5849, 2024"}]

Author rebuttal:

Meta-review: The authors present a novel deep SSM architecture that has nonlinear dynamics within each layer. The authors leverage recent advances in parallelization of nonlinear recursions to render their architecture computationally tractable. As Reviewer dhK1 said, 
> This innovative paper has opened up new perspectives for me on the parallelization of nonlinear recurrent architectures. It has already begun to affect how I think about future research directions. Papers like these should definitely be discussed at conference like NeurIPS.

I agree with this assessment and I congratulate the authors on their nice work.
Decision: Accept (poster)