Oscillatory State-Space Models
Authors: T. Konstantin Rusch
Daniela Rus
Venue: ICLR 2025 Oral (2025)
Keywords: state-space models
sequence models
oscillators
long-range interactions
time-series
Primary area: learning on time series and dynamical systems
TL;DR: Oscillatory state-space models are provably able to learn long-range interactions of arbitrary length, are universal, and achieve state-of-the-art performance in practice.

Abstract: We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x on a sequence modeling task with sequences of length 50k.

Published as a conference paper at ICLR 2025
OSCILLATORY STATE-SPACE MODELS
T. Konstantin Rusch
Daniela Rus MIT
MIT
ckrusch@mit.ees
ABSTRACT
We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learn- ing on long sequences. Inspired by cortical dynamics of biological neural net- works, we base our proposed LinOSS model on system of forced harmonic os- cillators. A stable discretization, integrated over time using fast associative paral- lel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive param- eterizations. Moreover, we rigorously show that LinOSS is universal, i.e., can approximate any continuous and causal operator mapping between vare-na functions, to desired accuracy. In addition, we show that an implicit-explicit dis- cretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-howe morizon forecast- ing. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outper- forms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x sequence modeling task with sequences of length 50k. Code: https: /egiggttuss com/ tk rus crusch/aee linoss
INTRODUCTION
2022 Smith et al. 2023 Orvieto et al. 2023) have recently emerged as a powerful tool for learning on long sequences. These models posses the state- fullness and fast inference capabilities of Recurrent Neural Networks (RNNs) together with many of the benefits of Transformers [Vaswani] 2017 Devlin 2018), such as efficient training and competi- tive performance on large-scale language and image modeling tasks. For these reasons, space models have been successfully implemented as foundation models, surpassing Transformer- -ased counterparts several key modalities, including language, audio, and genomics (Gu & Dao 2023).
State-space models (Gu al. 2021 Hasani et al.
Originally, state-space models have been introduced to modern sequence modelling by leveraging specific structures of the state matrix, i.e., normal plus low-rank HiPPO matrices (Gu et al.. 2020) 2021), allowing to solve linear recurrences via Fast Fourier Transform (FFT). This has since been simplified to only requiring diagonal state matrices (Gu et al., 2022a: Smith et al.) 2023 Orvi- eto al. 2023) while still obtaining similar even better performance. However, due to the linear nature of state-space space models, the corresponding state matrices need to fulfill specific structural prop- erties in order to learn long-range interactions and produce stable predictions. Consequently. these structural requirements heavily constrain the underlying latent feature space, potentially impairing the model's expressive power.
In this article, we adopt a radically different approach by observing that forced harmonic oscillators, the basis of many systems in physics, biology, and engineering, can produce stable dynamics while the same time seem to ensure expressive representations. Motivated by this, we propose to construct state-space models based on stable discretizations of forced linear second-order ordinary differential equations (ODEs) modelling oscillators. Our additional contributions are:
we introduce implicit and implicit-explicit associative parallel scans ensuring fast training and inference.

---

Published as a conference paper at ICLR 2025
show that proposed state-space model yields stable dynamics and is able to learn long-range interactions only requiring nonnegative diagonal state matrix.
we demonstrate that a symplectic discretization of our underlying oscillatory system con- serves its symmetry of time reversibility.
we rigorously prove that our proposed state-space model is universal approximator of continuous and causal operators between time-series.
we provide an extensive empirical evaluation our model on wide variety of sequential data sets with sequence lengths reaching up to 50k. Our results demonstrate that model consistently outperforms or matches the performance of state-of-the-an state-space models, including Mamba, $4, S5, and LRU.
THE PROPOSED STATE-SPACE MODEL
Our proposed state-space model based on the following system of forced linear second-order ODEs together with linear readout,
y
(t) (t)
= At(() + Bu(t) + b,
x(t) Cy(() Du(t).
with hidden state y(t) R", output state x(t) E IR4, time-dependent input signal u(() E IR,, weights A E mR B E RRxxp, C Rexm, D E Reex,, and bias b E RT. Note that A is diagonal matrix, i.., with non non-zero entries only on its diagonal. We further introduce an auxiliary state z(t) E R", with y'. We can thus write (I) (omitting the bias b and linear readout x) equivalently as,
21 (t Bu(t),
(2)
y'(1) (t).
y(t) z(1)
y(t) z(t)
Input Sequence (t))
Output Sequence
time t
LinOSS layer nonlinear layer
LinOSS layer nonlinear layer
LinOSS block
LinOSS block
Figure Schematic drawing of the proposed Linear Oscillatory State-Space model (LinOSS). The input sequences are processed through multiple LinOSS blocks. Each block is composed of
LinOSS layer (2) (y(t) plotted using orange solid lines and z(t) using blue dashed lines) followed by nonlinear transformation, specifically Gated Linear Units (Dauphin et al.] (017)) (GLU) layer our case, After passing through several LinOSS blocks, the latent sequences are decoded to pro- duce the final output sequence.
2.1 MOTIVATION AND BACKGROUND
To demonstrate that the underlying ODE I models a network of forced harmonic oscillators, we begin with the scalar case by setting m in (1). Choosing B b O. obtain the classic
2

---

Published as a conference paper at ICLR 2025
ODE, y" + Ay O, which deseribes simple harmonic motion with frequency A > 0, such that of a simple pendulum (Guckenheimer Holmes, 1990). Now, allowing B o 0 introduces external forcing proportional to the input signal u(t), where B modulates the effect of the forcing. Finally, setting p. m > yields an uncoupled system of forced harmonic oscillators.
Neuroscience inspiration. Our approach is inspired by neurobiology, where the periodic spiking and firing of action potentials in individual neurons can be observed and analyzed oscillatory phe- nomena. Furthermore, entire networks of cortical neurons exhibit behavior-dependen oscillations, reviewed in Buzsaki & Draguhn (2004). Remarkably, despite their complexity, these neural oscil- lations share characteristics with harmonic oscillators, as described in Winfrre (1980). Building on this insight, distill the core essence of cortical dynamics and aim to construct machine learn- ing model following the motion of harmonic oscillations. Finally, we note that our focus is on the theoretical and empirical aspects of the proposed oscillatory state-space model in this manuscript. However, our model can be further used in the context of computational neuroscience emulating characteristic phenomena of brain oscillations, such frequency-varying oscillations, transient syn- chronization or desynchronization of discharges, entrainment, phase shifts, and resonance, while at the same time being able learn non-trive coivirivial input-output relations.
2.2 COMPARISON WITH RELATED WORK
State-space models have seen continuous advancements since their introduction to modern sequence modeling in Gu et al. (2021). The original S4 model (Gu al., (02I). along with its adaptations (Gu et al. 2022a Nguyen et al. 2022 Goel et al.] (022). utilized FFT to solve linear recurrences. More recently, simplified variant, S5 (mmihh al. (2023). was introduced, which instead employs associative parallel scans to achieve similar computational speed. Another advancement to S4 have been Liquid Structural State-Space models (Hasami et al., 2022) which exchanged the static state matrix with an input-dependent state transition module. While all aforementioned models rely on the HiPPO parameterization (Gu al., 2020p, Linear Recurrent Units (LURs) (Orvieto et al. (023) demonstrated that even simpler parameterization yield state-of-the-an performance. Finally, selec- tive state spaces have been introduced in Gu & Dao (2023) in order to further close the performance gap between state-space models and Transtormers utilized in foundation models.
Oscillatory dynamics as a neural computational paradigm has been originally introduced via Cou- pled Oscillatory RNNs (CoRNNs) (Rusch & Mishra 2021a). In this work, it has been shown that recurrent models based on nonlinear oscillatory dynamics are able to learn long-range interactions by mitigating the exploding and vanishing gradients problem (Pascanu 2013). This approach was later refined for handling very long sequences in Rusch & Mishra (20216). which utilized uncoupled nonlinear oscillators for fast sequence modeling by adopting diagonal hidden-to-hhidee weight matrix. Interestingly, this method resembles modern state-space models but distinguishes itself by employing nonlinear dynamics. Since then, this concept, generally termed as neural oscillators (Lanthaler et al. (2024), has been extended to other areas of machine learning, e.g., Sraph-Coupled Oscillator Networks (GraphCON) (Rusch et al. 2022) for learning on graph-structured data, and RNNs with oscillatory dynamics combined with convolutions leading to locally coupled oscillatory RNNs (Klller& Welling, (203)). Our proposed LinOSS model differs from all these methods by its explicit use of harmonic oscillators via a state-space model approach.
2.3 DISCRETIZATION
Our aim is approximately solve the linear ODE system time being able to guarantec stability over long time-scales. To this end, we suggest to leverage the following two time integration schemes.
(2) as fast as possible, while at the same
Implicit time integration. We fix timestep 0 < At < and define proposed state-space model hidden states at time in = nAt as the following implicit discretization of the first order system 2
Zn Zn-1 Al(-Ayy Bu.),
Yn Yn-1 + Atz
3

---

Published as a conference paper at ICLR 2025
This can be written in matrix form by introducing X, Vnl'
MX, = x, + Fn:
with
AAA -AIT T Fn
Au
M
We can now solve our discretized oscillatory system by simply inverting matrix M and computing the induced recurrence. Note that inverting matrices typically requires O(m²) operations (where m is the number of rows or columns of the matrix) using methods like Gauss-Jordan elimination, making it computationally expensive. However, by leveraging the Schur complement, we can obtain an explicit form of M that can be computed in (m), thanks to the diagonal structure of A. More concretely,
I AS AtS
-ASS
S
AAS
M
=
AIS
(3)
with the inverse of the Schur complement S (I At A) which itself a diagonal matrix and can thus be trivially inverted. We point out that among other choices, straightforward condition that ensures S is well-defined is A₁ 2 for all l....,m m. The recurrence of the proposed model is then given as,
X M IM
FIM
(4)
with MM M and FIN = M M Fn. As we will show in the subsequent section, this discretiza- tion leads to a globally asymptotically stable discrete dynamical system.
explicit-explicit time integration (IMEX). Another discretization yielding stable dynamics that, however, do not converge exponentially fast to steady-state can be obtained by leveraging symplec- tic integrators. To this end, we fix again a timestep 0 < At S 1 and define our proposed state-space model hidden states time t, nAt the following implicit-explicit (IMEX) discretization of the first order system (2)
2n = 2n + Al(- A((AAy-- Ayn-1 + Bun),
(5)
Yn yn-1 + Atz
The only difference compared to the previous fully implicit discretization is the explicit treatment of the hidden state y (i.e., using Ya-1 instead of yn) in the first equation of (5). As before, we can simplify this system in matrix form.
X IEX + FIMEX
(6)
with
AA I A-A A
AiBu, At Bu,
M IMEX
IMEX
AtI
Interestingly, ODE system represents Hamiltonian system Amold 1989), with Hamiltonian,
1
H(y, 2,t) 2 k=1
A 2 + 2 2 Σ BRru(t) yk.
The numerical approximation of this system using the previously described IMEX discretization is symplectic, i.e., it preserves a Hamiltonian close to the Hamiltonian of the continuous system. Thus, by the well-known Liouville's theorem (Sanz Serma Calvo 1994), we know that the phase space volume of (2) as well as of its symplectic approximation 66) is preserved. This gives rise to invertible model architectures leading to memory efficient implementations of the backpropagation through time algorithm, similar as in (Rusch & Mishra 2021b). This denotes the most significant difference between the two different discretization schemes, i.e., the IMEX integration-based model is volume preserving, while IM integration- based model introduces dissipative terms. We will see in subsequent sections that both models have their own individual advantages depending on the underlying data. Finally, we note that stable higher-order time integration schemes (such as higher- order symplectic splitting schemes) can used in this context as well. An example of leveraging the second-order symplectic velocity Verlet method can be found in Appendix Section D
4

---

Published as a conference paper at ICLR 2025
2.4 FAST RECURRENCE VIA ASSOCIATIVE PARALLEL SCANS
Parallel (or associative) scans, first introduced in Kogge & Stone 1973) and reviewed in Bleiloch (1900), offer a powerful method for drastically reducing the computational time of recurrent op- erations. These scans have previously been employed to enhance the training and inference speed of RNNs (Martin & Cundy 2017; Kaul 2020). This technique was later adapted for state-space models in Smith et al. (2023), becoming crucial component in the development of many state-of- the-art sequence models, including Linear Recurrent Units (LRUs) (Orvieto al., 2023) and Mamba
models (Gu & Dao 2023).
parallel scan operates on a sequence [21....,.7, Fx]] with a binary associative operation o, i.e., an operation satisfying (x y)
I (y 2) for instances 3/ and to return the sequence T2 JNJ. Under certain assumptions, this operation can be performed in computational time proportional to [log (N)]_ This is in stark contrast to the computational time of serial recurrence that is proportional to N. is straightforward to check that the following operation is associative:
[#1, #1 T22...
(a), a2) (b), b2) = (b, aa, bj ag + b2),
where e, e are the matrix-matris and matrix-vector products for matrices a1, b2 and vectors ag, b Note that both products can be computed in (mm) time leveraging 2 block matrices with only diagonal entries in each block, e.g., matrices M and M in (6) Clearly, applying parallel scan based on this associative operation on the input sequence [M, F.F, 1). F2) F2),...] yields an output sequence [M, F1), (M2, MF; + F2),.... where the solution of the recurrent system X,, = M + F, (with initial value Xg = 0) is stored in the second argument of the elements of this sequence. Thus, we can successfully apply parallel scans to both discretizations (4) of our proposed system in order to significantly speed up computations. We refer to the application of this parallel scan to implicit formulations implicit parallel scans. When applied to implicit-explicit formulations, we describe as implicit-explicit parallel scans.
We term our proposed sequence model, which efficiently solves the underlying linear system of har- monic oscillators ( using fast parallel scans, as Linear Oscillatory Space (LinOSS) model. To differentiate between the two discretization methods, plicit discretization as LinOSS-IM, and the model based on the implicit-explicit discretization as LinOSS-IMEX. A schematic drawing of proposed state-space model can be seen in Fig. [ Moreover, a full description of a multi-layer LinOSS model, including specific nonlinear building blocks, can be found in Appendix A
refer to the model derived from im-
THEORETICAL INSIGHTS
3.1 STABILITY AND LEARNING LONG- RANGE INTERACTIONS
Hidden states of classical nonlinear RNNs are updated based on its previous hidden states pushed through a parametric function followed by a (usually) bounded nonlinear activation function such as tanh or sigmoid. This way, the hidden states guaranteed not blow up. However, this is not true for linear recurrences, where specific structures of the calden-to-hidded weight matrix can lead to unstable dynamics. The following simple argument demonstrates that computing the eigenspectrum (i.e., set of eigenvalues) of the hidden-to-hidden weight matrix of a linear recurrent system suffices in order analyse its stability properties. To this end, let consider a general linear discrete dynamical system with external forcing given via the recurrence X,, Mx 1 + Bu,, and initial value Xy = Assuming M diagonalizable (if not, one can make similar argument leveraging the Jordan normal form), i.e., there exists matrix S such that M = SAS where A is diagonal matrix with the eigenvalues of M on its diagonal. The dynamics of the transformed hidden state x, = S-1x, x, evolve according to X, S- x, = S- + S- B Ax, Bu, with initial value X = S Xg, where B S Unrolling the dynamics (assuming X, = 0) yields,
n-1
x = Bu, X2 = ABu, + Bug
+
Σ A A
k=0
Clearly, if all eigenvalues A have magnitude less or equal than the hidden states will not blow up. In addition stability guarantees, eigenspectra with unit norm allow the model to learn long-range
5

---

Published as a conference paper at ICLR 2025
interactions (Arjovsky et al. 2016 Gu et al. 2022b Orvieto et al. 2023) by avoiding vanishing and exploding gradients (Pascanu 2013). Therefore, it is sufficient to analyse the eigenspectra of our proposed LinOSS models in order to understand their ability to generate stable dynamics and learn long-range interactions. To this end, we have the following propositions.
Proposition 3.1. Let M Rmxm be the hidden-tee colden-tt-hiidee to-hidden weight matrix of the implicit model LinOSS-IM IM 4 We assume that Ask 2 0 for all diagonal elements k = ....., of A, and further that At > Then, the complex eigenvalues of MMM are given as,
VAkk
X 1 1+ AH (((-1))=1A/
for all -1,.... 2m,
1 1+ Ask
with k mod Moreover, the spectral radius p(M ) is bounded by 1, ie., 12yl S for all j = 1,...,22 2m.
Proof.
S - XI AtS
-ASS
S XI
AIAS
det(M AI)
S AI
S- AI + A/ AS 21--1
At As
( ck - A)
SEL X+
II [S - 2)2 + AP A St
Sak
k=1
Setting 1o = mod m, the eigenvalues of MM are thus given as,
A Sak + i( i(-1) 1 AIS Aiks for all 1,... 2m.
In particular, assuming As 2 for all k: = .,..., m, the magnitude of the eigenvalues X, are given as,
|y = S + At S Aus = S1 At = Sak S 1,
for all j 1, 2m, with & mod m.
This proof reveals important insights into our proposed LinOSS-IM model. First, the magnitude of the eigenvalues at initialization can be controlled by either specific initialization of A. or al- ternatively through a specific choice of the timestep parameter At. Moreover, LinOSS-IN IM yields asymptotically stable dynamics for any choices of positive parameters A. This denotes significant difference compared to previous first-order system, where the values of A (and thus its eigenval- ues) have to be heavily constrained for the system to be stable. We argue that this ficxibility in the parameterization of our model benefits the optimizer potentially leading to better performance in practice.
Remark 1. A straightforward adaptation of Proposition 3.1 can also be derived for the implicit- explicit version of the model, LinOSS-IMEX (6). The detailed proposition is given in Appendix Proposition E.1 In particular, the analysis shows that all eigenvalues X, of M in (6) satisfy 1 This result underscores the key distinction between the two models: LinOSS-IN incor- porates dissipative terms, whereas LinOSS-IMES denotes a conservative system. Following the argument at the beginning of Section 3.1 one can interpret the dissipative terms in LinOSS-IM as forgetting mechanisms, which are considered crucial for expressive modeling of long sequences. This makes LinOSS- IM more flexible model version compared to LinOSS-IMER Finally, note that explicit discretization schemes lead to exploding hidden states returning NaN output values in practice during training. For this reason, we focus solely stable methods in this context.
1.
Initialization and parameterization of weights. Proposition 3.1 and Remark reveal that both LinOSS models exhibit stable dynamics long the diagonal weights A in ① are non-negative. This condition can casily be fulfilled via many different parameterizations of the diagonal matrix A. An obvious choice is to square the diagonal values, i.e., parameterization A ÁA, with diagonal matrix Å E Rp Another straightforward approach is to apply the element- ReLU nonlinear activation function to A, i.e.. A ReLU(Á), with ReLU(z) max(0, 2) and A as before. The latter results in LinOSS model where specific dimensions can be switched off completely by setting the corresponding weight in Á to a negative value. Due to this flexibility, we decide to focus on the ReLU Rel.U-parameter --paammmteriitooon of the diagonal weights A in this manuscript.

---

Published as a conference paper at ICLR 2025
After discussing the parameterization of A, the next step is to determine appropriate method for initializing its weights prior to training. Since we already know from Remark that all absolute eigenvalues of the hidden matrix M of LinOSS-IMEX are exactly one, the specific values of A are irrelevant for the model to learn long-range interactions. However, according to Proposition 3.1 A highly influences the eigenvalues of the hidden matrix M of LinOSS-IM 4.. As discussed in the previous section, high powers of the absolute eigenvalues of the hidden matrix are of partic- ular interest when learning long-range interactions. To this end, we have the following Proposition concerning the expected powers of absolute eigenvalues for LinOSS-IM.
Proposition 3.2. Let { be the eigenspectrun of the hidden-to-hiddes matrix M of the LinOSS- IM model (4). We further initialize Agk ~ U(10, Amar) with Amax > for all diagonal
elements k- = 1...... m of A in ( Then, the N-th moment of the magnitude of the eigenvalues are given as,
)(AA)** = (At2 Amar + 11)1=²=-
(8)
At Amax 2
for all j = 1,..., 2m, with k j mod m.
The proof, detailed in Appendix Section 1..2 is straight-forward application of the law of the unconscious statistician. Proposition might be much smaller than 1, it is still sufhciently large for practical use-cases. For instance, even considering A/ = Amax
demonstrates that while the expectation of [
and an extreme sequence length of N = 100k still yields B((2,1*) 1/49999 2 x 10-5_ Based this and the fact that the values of A do not af- we decide to initialize A according to A ~ 2(10, 1J) for
fect the eigenvalues for LinOSS-IMER both LinOSS models, while setting At
3.2 UNIVERSALITY OF LINOSS WITHIN CONTINUOUS AND CAUSAL OPERATORS
While trainability is an important aspect of learning long-range interactions, it does not demonstrate why LinOSS is able to express complex mappings between general (i.e., not necessarily oscillatory) input and output sequences. Therefore, in this section we analyze the approximation power of our proposed LinOSS model. Following the recent work Lanthaler et al. (2024) we show that LinOSS is universal within the class of continuous and causal operators between time-series. To this end, we consider a full LinOSS block,
z(t) Wo Wo((W(()) b),
(9)
with weights W E Rexa E Raxm bias b E RR elem nonlinear activation function a (e.g., tanh or ReLU), and solution y(L) of the LinOSS differential equations (
Based on this, we are now interested in approximating operators + T]: Co,(O,T):RR)))
Co,, T): IR9) with the full LinOSS model 9. where
Co((o, 7]: RP)= == {u: [0, 7] - TR. < u(t) is continuous and u(0) = 0};
i.e., operators between continuous time-van varying functions with values TRP As pointed out in Lanthaler et al. (2024), the condition u(0) = 0 is not restrictive and can directly be generalized to the case of any initial condition u(0) = un E R2 simply by introducing an arbitrarily small warm- up phase [-to, 0] with to > 0 of the oscillators to synchronize with the input signal In addition to these function spaces, we pose the following conditions on the underlying operator Φ.
1. & is causal, i.e., for any t E [0, if u, B E ([, are two input signals, such that u/ u//0.1| = Bl then (())(t) = (B)()).
2. t is continuous an operator
Φ (Co R") 112~)~)))))0 + (C.((0,T):: RT; 11
with respect to the - -norm the input-/output-sinnal.
Theorem 3.3. Let ((0, T); IR) + Co((0, T): RR) be a causal and continuous operator. Let
K Co((O, IPP) be compact. Then for any 0, there exist hyperparameters m. m. diagonal
weight matrix A E Rmxm weights B E R Rm W Roxm W E Roxi and bias vectors b Rmm
b E R such that the output z [0,T] R of the LinOSS block 4 satisfiee,
sup ((()((t) z(t) $ e, VuE K. tE|0,7]

---

Published as a conference paper at ICLR 2025
The proof can be found in Appendix Section 3.3 The main idea of the proof is encode the infinte-dimeeniionll operator & with a finteeddimensional operator that makes use of the structure of the LinOSS ODE system ( and that further be expressed by This theorem rigorously shows that LinOSS can approximate any causal and continuous operator between continuous time-varying functions with values in IR? to any desired accuracy.
function.
EMPIRICAL RESULTS
In this section, we empirically test the performance of our proposed LinOSS models on variety of challenging real-world sequential datasets, ranging from scientific datasets in genomics to practical applications in medicine. We ensure thereby fair comparison to other state-of-the-an sequence models such as Mamba, LRU, and S5.
4.1 LEARNING RANGE INTERACTIONS
In the first part of the experiments, we focus on a recently proposed long-range sequential benchmark introduced in Walker et al. (2024). This benchmark focuses six datasets from the University of East Anglia (UEA) Multivariate Time Series Classification Archive (UEA-MTSCA) (Bagnall et al.) 2018). selecting those with the longest sequences for increased diffieulty. The sequence lengths range thereby from 400 to almost 18k. We compare our proposed LinOSS models to recent state-of- the-art sequence models, including state-space models such as Mamba and LRU. Table I shows the test accuracies averaged over five random model initialization and dataset splits of all six datasets for LinOSS as well as competing methods. We note that all other results are taken from Walker al. (2024). Moreover, we highlight that we exactly follow the training procedure described in Walker al. (2024) order ensure fair comparison against competing models. More concretely, the same pre-selected random seeds for splitting the datasets into training, validation, and testing parts (using 70/15/15 15 splits), as well as tune our model hyperparameters only on the same pre-described grid. In fact, since LinOSS does not possess any model specific hyperparameters unlike competing state-space models such as LRU or S5 our search grid is lower-dimensional compared to the other models considered. As result, the hyperparameter tuning process involves significantly fewer model instances.
Table Test accuracies averaged over 5 training runs on UEA time-series classification datasets. All models are trained based on the same hyper-parane pprrr-parreeter tuning protocol in order to ensure fair com- parability. The dataset names are abbreviations of the following UEA time-series datasets: Eigen- Worms (Worms), SellRegulationSCPI (SCP1), SellRegulationSCP2 (SCP2), EthanolConcentration (Ethanol), Heartbeat, MotorImagery (Motor). The three best performing methods are highlighted in red' (First), blue? (Second), and violet (Third).
Worms
SCP1
SCP2
Ethanol
Heartbeat 405 2
Motor
Avg
Seq. length 17,984
896
1,152
1,751
3,000
#Classes
5
2
2
NRDE
83.9 7.3
80.9 + 2.5
53.7 : 6.9
25.3 +
72.9 4.8
47,0
60.6
NCDE
75.0 3.9
83.1 + 79.8 5.6
53.0 1 2.8
29.9 1 6.5
2.6
49.5.12.
60.2
Log-NCDE
85.6 5.13
53.7
4.13
34.4
6.4
75.2 4.6
53.7 5.3
64.3
LRU
87.8 + 2.8
82.6
51.2 +1.2+3.6 3.6
21.5 2.1
78.4
6.71
48.4+
61.7
S5
81.1
4.69
50.
24.1
77.7
5.52
47.7 5.5
61.8
S6
85.0 16.1
2.7
49.9
2.44
7
6.7
62.0
Mamba
70.9 15.8
1.4
48.2 012133 3.9
27.914.5
7..8
47.7 1.
58.6
LiBOSS-IMEX
80.0
87.5 + 4.0"
58.9
8.1
29.9 1.0%
75.5
37.9
65.02
LinOSS-IM
95.0 4.44
87.8
1.6.
58.2
6.9
29.9
0.6
7..8
7.55
67.81
We can see in Tableethar on average both LinOSS models outperform any other model we consider here by reaching an average (over all six datasets) accuracy of 65.0% for LinOSS-IMEX and 87% for LinOSS-ING IM. In particular, the average accuracy of LinOSS IM is significantly higher than the two next best models, i.e.. NogE reaching average accuracy of 64.3%, and S6 reaching accuracy of 62.0% on average. is particularly noteworthy that LinOSS-IN IM yields state-of the-art results on the two datasets with the longest sequences, namely Eigen Worms and MotorImagery,
8

---

Published as a conference paper at ICLR 2025
4.2 VERY LONG-RANGE INTERACTIONS
In this experiment, we test the performance of LinOSS in the case of very long-range interactions. To this end, we consider the PPG-DaLiA dataset, a multivariate time series regression dataset de- signed for heart rate prediction using data collected from a wrist-worn device (Reiss et al. (2019). It includes recordings from fifteen individuals, each with approximately 150 minutes of data sampled at maximum rate of 128 Hz. The dataset consists of six channels: blood volume pulse, electro- dermal activity, body temperature, and three-axis acceleration. We follow Walker et al. (2024) and divide the data into training, validation, and test sets with 70/15/15 split for each individual. After splitting the data, a sliding window of length 49920 and step size 4992 is applied. As in previous experiments, we apply the exact same hyperparameter tuning protocol each model consider here to ensure fair comparison. The test mean- -aaannd error (MSE) of both LinOSS models as well other competing models are shown in Table We can that both LinOSS models significantly outperform all other models. In particular, LinOSS-IM outperforms Mamba and LRU by nearly a factor of 2. This highlights the effectiveness of proposed LinOSS models sequential data with extreme length.
Table 2: Average test mean -eean-d error over training runs the PPG-DaLiA dataset. All models are trained following the same hyper-parameter tuning protocol in order to ensure fair comparability. The three best performing methods are highlighted in red" (First), blue² (Second), and violet (Third).
Model
MSE 10 22
NRDE (Moriill et al. 2021
9.90 + 0.97
NCDE (Kidger
2020
13.54 + 0.69
Log-NCDE (Walker et al.
2024
9.56 + 0.593
LRU (Orvicoo et al. 2023
12.17:00.9
S5 (Smith el al. 2023)
12.63 + 1.25
S6
(Gu & Dao
2023)
12.88 + 2.05
Mamba (Gu & Dao
2023
10.65 1665:220 2.20
LinOSS-IMEX
7.5 + 0.46
LinOSS-IM
6.4 0.23
4.3 LONG-HORIIONN FORECASTING
Inspired by Gu et al. (202T), we test our proposed LinOSS on its ability to serve as a general sequence-to-sequencc model, even with weak inductive bias. To this end, focus time-series forecasting which typically requires specialized domain-specific preprocessing and architectures. We do, however, not alter our LinOSS model nor incorporate any inductive biases. Thus, we simply follow /uu et al. (2021) by setting up LinOSS as a general sequence-to-sequee forecasting as masked task introduced in Zhou et al. (202T). In this task, several climate variables are predicted into the future based on local climatological data. Here, we focus on the hardest task in Zhou et al. (202T) of predicting the future 720 timesteps (hours) based on the past 720 timesteps. Table Table the mean absolute error for both LinOSS models as well as other competing models. We can see that both LinOSS models outperform Transformers baselines well as the other state-space models.
model that treats ggquence-to-squenee transformation. We consider a weather prediction
4.4 ADDITIONAL EXPERIMENTS AND ABLATIONS
Our proposed LinOSS model is the result of several design choices, such as the state matrix pa- rameterization, state matrix initialization, and the numerical value of the discretization timestep At. In this section, empirically analyze how these choices affect the performance of LinOSS by providing ablations and sensitivity studies.
We start by evaluating the performance of LinOSS under different parameterizations and initializa- tions of the state matrix A. Specifically, we examine LinOSS-I IM within the experimental framework described in Section 4.1 For this analysis, we parameterize A using one of the two approaches pro- posed in Section 3.1 A ÄẮ A Re). where Á Rmxm is diagonal matrix. The

---

Published as a conference paper at ICLR 2025
Table 3: Mean absolute error on the weather dataset predicting the future 720 time steps based on the past 720 time steps. The three best performing methods are highlighted in red! (First), blue? (Second), and violet (Third).
Model
Mean Absolute Error
Informer (Zhou et al. 2021
0.731
LogTrans Li et al. 2019)
0.773
Reformer (Kitavv al.
2020
1.575
LSTMa Bahdanau et al.
2016
1.109
LSTnet (Lai et 2018)
0.757
S4 (Gu et 2021
0.5783
LinOSS-IMEX LinOSS-IM
0.508
0.528
results, presented in Appendix Table @ that the ReLU parameterization leads slightly better performance on average over all six datasets. However, the squared parameterization yields better performance on three out of six datasets. Thus, including different state matrix parameterization choices in the hyperparameter optimization can help achieve improved performance. We further test the performance of LinOSS- IM using a standard normal random initialization of the state matrix instead of the uniform initialization. The results of the standard normal initialization of the state matrix are shown in Appendix T We can see that this initialization yields similar performance almost all considered datasets, except the Higen Worms dataset, where
uniform initialization obtains a lower mean test accuracy and a much higher standard deviation. This indicates that the performance of LinOSS-IM with a standard normal initialization for the state matrix is highly sensitive to the random seed used during initialization. This sensitivity arises because normal dis- tributions do not have bounded support, allowing for the possibility of large matrix entries. These large entries result in small eigenvalues, which in turn lead to vanishing gradients.
Another natural question in this context concerns the sensitivity of performance variations in the timestep At used in the underlying discretization scheme. To investigate this, we train several LinOSS-IM models using different values of At spanning three orders of magnitude, i.e., ranging from 10 to 1. The average test error, along with the standard deviation, is plotted for three different datasets in Appendix C.3 We can see that while the choice of At does influence performance, the variations are not substantial.
Finally, we empirically analyze the roles of dissipation and conservation in LinOSS- and LinOSS- IMEX. As rigorously demonstrated in Section LinOSS- IM introduces dissipative terms, whereas LinOSS- IMEX denotes a conservative system. While our earlier experiments examined the differ- ences between these models using real-world data, energy-conserving dynamical systems to highlight their contrasting behaviors. To this end, we sim- ulate the energy-conserving simple harmonic motion with various initial positions and velocities. The test error over time for both models is plotted in Appendix Fig, 2 The results show that the er- in LinOSS-IM grows over time, whereas the error in LinOSS-IMEX remains constant. Notably, by the end of the time interval, LinOSS -MEX outperforms LinOSS- IM by factor of more than This stark contrast underscores the fundamental difference between the two models: LinOSS-IM IM introduces dissipative terms, making it more effective for dissipative systems, while LinOSS-IMEX, being fully conservative, excels in energy-conserving systems.
focus their performance for predicting
5 CONCLUSION
In this paper, we introduce LinOSS, a state-space model based on harmonic oscillators. We rigor- ously show that LinOSS produces stable dynamics and able to learn long-range interactions only requiring a nonnegative diagonal state matrix. In addition, we connect the underlying ODE system of the LinOSS model to Hamiltonian dynamics, which, discretized using symplectic integrators, per- fectly preserves its symmetry of time reversibility. Moreover, we show that LinOSS is a universal approximator of continuous and causal operators between time-series. Together, these properties en- able efficient modeling of long-range range interactions, while ensuring stable and accurate Long-horizon forecasting. Finally, we demonstrate that LinOSS outperforms oftte-oftthe--e art state-space models, such Mamba, LRU, and S5.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank Dr. Alaa Maalouf (MIT) and Dr. T. Anderson Keller (Harvard University) for their insightful feedback and constructive suggestions on an earlier version of the manuscript. This work was supported in part by the Dostdoc.Mobility grant P500PT-217915 from the Swiss National Science Foundation, the Schmidt AI2050 program (grant G-22-63172), and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Coop- erative Agreement Number FA8750-199 1900-111-----00-------0-0------0 The views and conclusions contained in this docu- ment are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the Department of the Air Force or the U.S. Government. The U.S. Gov- ermment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pp. 1120-1128, PMLR, 2016.
V. Amold. Mathematical methods of classical mechanics. Springer Verlag, New York, 1989.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075. 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016.
Guy E Bielloch. Prefix sums and their applications. 1990.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//qithub. com jacaan/jjxaaa jax-mmjjjammaaaaaa
Gyorgy Buzsaki and Andreas Draguhn. Neuronal oscillations in cortical networks. science, 304
(5679):1926-1929, 2004.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv.1810.04805, 2018
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state- space models. In International Conference on Machine Learning, pp. 7616-7633, PMLR, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint ar Xrii:2212..00722, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint a 1..0096, 2021.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:3597 1- 35983, 2022a.
11

---

Published as a conference paper at ICLR 2025
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How train your hippo:
State space models with generalized orthogonal basis projections. arXiv:2206.12037, 12037, 2022b.
arXiv preprint
Guckenheimer and Holmes. Nonlinear oscillations, dynamical systems, and bifurcations of vector fields.. Springer Verlag, New York, 1990.
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209. 12951, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.08411 08405, 2023.
S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Zheyuan Hu, Nazanin Ahmadi Daryakenari, Qianli Shen, Kenji Kawaguchi, and George Em Kar- niadakis. State-space models are accurate and efficient neural operators for dynamical systems. arXiv preprint arXiv:2409.03233. 2024.
Shiva Kaul. Linear dynamical systems as a core computational primitive. Advances in Neural Information Processing Systems, 33: 6808--6688 2020.
Anderson Keller and Max Welling. Neural wave machines: learning spatiotemporally structured representations with locally coupled oscillatory recurrent neural networks. In International Con- ference on Machine Learning, pp. 16168-16189, PMLR, 2023.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in neural information processing systems, 33:6696-6777 6707, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445 04451, 2020.
Peter Kogge and Harold S Stone. A parallel algorithm for the efficient solution of general class of recurrence equations. IEEE transactions computers, 100(8):786-733, 1973.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018.
Samuel Lanthaler, Konstantin Rusch, and Siddhartha Mishra. Neural oscillators are universal. Advances in Neural Information Processing Systems, 36, 2024.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length, arXiv preprint 17Xiv:7709.000 04057, 2017.
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dy- namics with mnss Advances in Neural Information Processing Systems, 1297-11133,,,, 2022.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference Machine Learning, pp. 7829-7838.
PMLR, 2021.
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. $4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2866 2022.
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando. Caglar Gulcehre, Razvan Pas- canu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference Machine Learning, pp. 26698. PML.R, 2023.
12

---

Published as a conference paper at ICLR 2025
R Pascanu. On the difficulty of training recurrent neural networks, arXiv preprint arXiv: 2013.
Attila Reiss, Ina Indlekofer, Philip Schmidt, and Kristof Van Laerhoven. Deep ppg: Large-scale heart rate estimation with convolutional neural networks. Sensors, 19(14):3779, 2019.
Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies. In Interna- tional Conference on Learning Representations, 2021a.
Konstantin Rusch and Siddhartha Mishra.
Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning, pp. 9168-9178. PMLR, 2021b.
T Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bron- stein. Graph-coupled oscillator networks. International Conference Machine Learning, pp- 18888-18909, PMLR, 2022.
J.M. Sanz Serna and M.P. Calvo. Numerical Hamiltonian problems. Chapman and Hall, London, 1994.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for se- quence modeling. In The Eleventh International Conference Learning Representations, 2023. URL /attps://opsnreviews www.nee//ooore wttp://wwwweeeeeeteeeemeeeeeea id=Ai A1t 8Hw3AXqks
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. Log neural controlled differential equations: The lie brackets make a difference. International Conference Machine Learning, 2024.
Arthur T Winfree. The geometry of biological time, volume 2. Springer, 1980.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
13

---

Published as a conference paper at ICLR 2025
Supplementary Material for: Oscillatory State-Space Models
A FULL ARCHITECTURE DETAILS
In this section, outline the detailed architecture of full multi-layer LinOSS model. To this end, the full LinOSS model starts by encoding an input sequence u [ui, U2,..., uv] with u, I29 for all I....,NV N via an affine transformation. After that, several blocks are ap- plied consisting of LinOSS layer (i.e., solving (I) with either IM or IMEX associative parallel scans) directly followed by nonlinear layer using the Gaussian error linear unit activation function (GELU) (Hendrycks & Gimpel 2023). the Gated Linear Unit (GLU) (Dauphin et al.) 2017). i.e., GLU(x) sigmoid( (Wrnidd(WWx]]] ) o W 2x, where W1.2 1.2 are learnable weight matrices, and skip connec- tion. Finally, the output of the final LinOSS block gets decoded by an affine transformation. The full LinOSS model is further presented in Algorithm Note that weight matrices and bias vectors applied parallel in time whenever possible (i.e., outside the recurrence). We are thus omitting subscript in Algorithm
Algorithm Full LinOSS model
Input: Input sequence
Output: T.blook LinOSS output sequence o n Wene u + bene
Encode input sequence
for | = 1...., L...,,Lu. do
y < solution of ODE in with input u'-1 via parallel scan
x' + Cy' + Du'
b Linear readout in
x' + GELU(x!)
u' + GLU(x') +u'
end for
<- Wdecy + bitec
b. Decode final LinOSS block output
A.1 SEQUENCE-TO-SEQUREE LINOSS ARCHITECTURE FOR TIME-SERIES FORECASTING
While in sequence regression and sequence classification tasks we simply take the final output the full LinOSS model at final time 7 as the model prediction, we have to slightly adapt our architecture to handle time-series forecasting problems. To this end, we follow common practice for state-space models suggested in Gu al. [22II) and generate train, validation, and test sequences of length L1 + L2, where L1 is the number of the past sequence entries used for forecasting and Le are the number of future steps we aim predict. Note that for the input sequences, simply mask out the last L2 entries of the sequence (i.e., with zero entries). Moreover, for the LinOSS output sequence, only the final La entries for the future predictions.
B TRAINING DETAILS
The code to run the experiments is implemented using the JAX auto-differentiation framework (Bradbury et al. 2018). All experiments were conducted on Nvidia Tesla V100 GPUs and Nvidia RTX 4090 GPUs, with the exception of the PPG experiment, which was run on Nvidia Tesla A100 GPUs due higher memory demands.
B. HYPERPARAMETERS
The hyperparameters of the models were optimized with the same grid search approach from
Walker et al. (2024) for the six datasets in Section 4.1 and the PPG dataset of Section 4.2
to ensure perfect comparability with competing methods, i.e., using the grid: learning rate (0.00001, 0.0001, 0.001), number of layers = {2, 4, 6), number of hidden neurons = {16, 64, 128), state-space dimension weather dataset, we performed a random search instead of grid search using the same hyperparame- ter bounds as before, except that increased the maximum number of LinOSS blocks from 6 to 8.
{16,6 64, 256). include time dimension (True, False). Note that for the
14

---

Published as a conference paper at ICLR 2025
The hyperparameters yielding the best performance can be seen in Table for both LinOSS-IM IM and LinOSS- -MEX- for each experiment in the main paper.
Table 4: Best hyperparameters
Model
hidden dim
state dim
#blocks include time
IM
0.001
128
64
True
Worms
IMEX
0.0001
64
16
2
False
IM
0.0001
128
256
6
True
SCP1
IMEX
0.0001
64
256
6
False
IM
0.0001
128
64
6
False
SCP2
IMEX
0.00001
64
256
True
IM
0.00001
16
16
4
False
Ethanol
IMEX
0.00001
16
256
4
False
IM
0.001
16
16
6
True
Heartbeat
IMEX
0.00001
64
16
2
True
IM
0.00001
128
16
2
False
Motor
IMEX
0.0001
16
256
6
True
IM
0.001
16
64
6
True
PPG
IMEX
0.0001
64
16
True
IM
0.0006
64
32
True
Weather
IMEX
0.0007
256
128
5
False
B.2 MEMORY REQUIREMENTS, RUNTIMES, AND NUMBER OF PARAMETERS
In this section, we present the number of parameters, GPU memory usage (in MB), and run time (in seconds) for every considered model on all datasets from Section 4.1 The GPU memory and run time results for the other models are taken from Walker et al. (2024). Note that we used exactly the same GPU architecture well as the ensure fair comparability, i.e., GPU memory usage and run time was measured on an Nvidia RTX 4090 GPU for all models. The run time was measured the average run time for 1000 training steps. 5 comprehensively shows the number of parameters, GPU memory usage, and run time for all models. Note that we further follow Walker al. (024) and report these results for the best performing model identified during the previously described hyperparameter optimization process. Both LinOSS models exhibit comparable GPU memory usage and run time performance other state-space models. Notably, LinOSS achieves the fastest runtime on two out of six datasets and ranks as the second fastest on another two datasets.
code and python libraries as in Walker et al. (2024) to
C ADDITIONAL EXPERIMENTS
C.1 ON THE STATE MATRIX PARAMETERIZATION AND INITIALIZATION
In the main paper, we have focused on parameterizing the state matrix A in 1 according to A = (Á), with diagonal matrix E Rmxm This was due to the fact that LinoSS requires the state matrix to be nonnegative order to produce stable dynamics. However, another viable parameterization choice would be A ÅÁ. In this section, we test how the squared parameteri- zation influences the performance of LinOSS on six long-range datasets taken from Section 4.1 of the main paper. ization well as using a squared parameterization together with the same baselines taken from the main paper. We can see that on average, the ReLU parameterization performs better. However, since the squared parameterization performs better on SCP2, Ethanol and Heartbeat (i.e., three out of six datasets), we conclude that including the two parameterization choices in the hyperparameter optimization process will lead to even better performance.
shows the average test accuracies of LinOSS-IM using a ReLU parameter-
15

---

Published as a conference paper at ICLR 2025
Table 5: Number of parameters, GPU memory usage (in MB) and run time (in seconds) for every
considered model on all long-range datasets from Section 4.1
NCDE
Log-NCDE
85
Mamba
LinOSS-IMEX
LinOSS-IM
#parameters
105110
166789
37977
101129
22007
27381
15015
26119
13-279
Worms
GPU memory (MB)
2506
2484
2510
10716
6646
13486
7922
6556
10654
run (s)
24595
31
68
aparameters
117187
166274
91557
25892
226328
184194
24898
447944
91240
SCP1
GPU memory (MB)
716
694
724
960
1798
1110
901
4768
4772
run (5)
Aparameters
200707
182914
36379
26020
5652
356290
26018
448072
399112
SCP2
GPI. memory (MR)
712
692
714
954
762
2460
1222
4772
2124
run (s)
*parameters
93212
133252
31452
76522
76214
1032772
5780
70088
6728
Ethanol
GPU memory (MB)
712
692
710
1988
1520
4876
938
4766
1182
run (s)
2217
2056
*parameters
15657742
109811
168520
338820
158310
1034242
6674
29144
10936
Heartheat
GPU memory (MB)
6860
1462
2774
1466
1548
1650
606
922
928
(S)
1177
11
#parameters
1134395
186962
81391
107544
17496
228226
52802
106024
9184
Motor
GPU memory (MB)
4552
4534
4566
8646 51
4616
3120
4056
12708
4510
time
7616
3778
730
16
35
34
128
Table 6: Test accuracies averaged over 5 training runs on UEA time-series classification datasets. All models are trained based on the same hyper-parameter tuning protocol in order to ensure fair comparability. The dataset names are abbreviations of the following UEA time-series datasets: EigenWorma (Worms), SeliRegulationSCPI (SCP1), SelfRegulationSCP2 (SCP2), EthanolConcen- tration (Ethanol), Heartbeat, Motorlmagery (Motor).
Worms
SCP1
SCP2
Ethanol
Heartbeat 405
Motor 3,000 2
Avg
Sog. length
17,984
896
1,152 2
1,751 4
#Classes
5
2
2
NRDB
83.9 +: 7.3
80.9 2.5
53.7 + 6.9
25.3 : 1.8
72.9
1 4.8
47.0 + 5.7
60.6
NCDE
75,0 3.9
5.6
5..
29.9 6.5
73.9
2.6
19.5
2.8
00.2
Log- NCDE
85.6 1: 5.1
83.1
2.
53.7
0.7=240 4.1
3.44
75.2
4.6
53.7
13.7:53.
64.3
LRU
87.8 1 2.8
62.6
5.6
21.
7.74
3.0
61.7
81.1
4.6
2.6
2.31
47.7 5.5
61.8
S6
85.0 - 16.1
2.7
9.4
2.44
8.3
51.3447
62.0
Mamba
70.9 = 15.8
50.7 1.4
48.2 3.9
27.9 11.9:00
76.2 3.8
47.7
1 4.5
56.6
LM (Rel.U)
95.0
87.8
58.2+66.9
29.9 0.6
75.8
60.0 7.5
67.8
LinOSS-IM IM (squared)
88.9 2.5
86.6 1.8
59.3 7.8
6.2
76.8 2,2
58.2 8.4
67.1
LinOSS-IM (squared Gaussian init)
74.4-+331.3
87.3
1.7
60.7 4.1
31.4 4.8
73.9
4.0
56.5 + 3.0
64.0
Another important question arises in the context of the state matrix initialization. While we argue in the main paper that initializing the state matrix using a simple random uniform distribution in [o. 1] leads to models that are able to learn long-range interactions, we are interested in exploring other choices in this context. To this end, we train LinOSS-IM models and initialize their state matrix using a standard normal distribution. Note that we need to use the squared parameterization in this case, as ReLU would lead to switching off approximately half of the dimensions. The results for the six datasets from Section of of the main paper are shown in Table 6 We can see that initializing the state matrix using standard normal distribution leads to competitive results on almost all datasets except Eigen Worms. Note that the standard deviation is very high on this dataset, suggesting that the performance is very sensitive to the random seed of the initialization. Therefore, the subpar performance FigenWorms can be explained by the possibility of this initialization produce large matrix entries that lead to small eigenvalues and thus vanishing gradients.
C.2 DISSIPATIVE VS CONSERVATIVE LINOSS MODELS
In this section, we empirically analyze the dissipative behavior of LinOSS-IM and compare it to the conservative behavior of LinOSS-IMEX. To this end, we aim to predict an energy-conservin dynamical system. More concretely, we train our two LinOSS models predict simple harmonic motion for different initial positions and velocities, i.e., the solution of,
(t) = y(t,,
(10)
s(0) = A, 5(0) = B,
16

---

Published as a conference paper at ICLR 2025
for A, B [0, 1]. We construct train, validation, and test sets by solving (TO) for uniform randomly chosen A, B. with 2000 train sequences, 500 validation sequences, and 500 test sequences. We set the stepsize for solving (t) to Åt 0.1 and predict y)() for 1000 steps, i... for the time interval [0, 100]. Clearly, (TOp is energy- conserving with the Hamiltonian given as H 5/) +/2
Since state-space models are struct the input sequences as two-dimensional sequences of length 1000, where all entries of the first dimension are set to A and the second dimension is set to B. The resulting test mean-squared error (MSE) shown in Fig- 2 for each point in time. We can see that LinOSS-IMES keeps the error constant, while the error for LinOSS-IM grows over time. This can be explained by the dissipative nature of LinOSS-IM, which forces the predicted trajectories to slowly converge to a steady-state, ie., LinOSS-IM would go to zero in the asymptotic case of t +
to-suueeee-ee models, we follow common practice and con-
LinOSS-IMEX
LinOSS-IM
0.0066
MSE
0.0022
0.0000
20
40
60
80
100
Time
Figure 2: Mean squared error over time of LinOSS-IMEX and LinOSS-IM on predicting simple harmonic motion for different initial positions and velocities.
C.3 ON THE SENSITIVITY OF At IN LINOSS
While we set the timestep At of the underlying time integration schemes to At for all our experiments in this paper, it is natural to ask whether different choices of At will lead to different performance. To analyze this, we train LinOSS-IM models on three datasets from Section 4. of the main paper, i.e., SellRegulationSCPP.. Heartbeat, and Motorlmagery dataset. We further vary At between 10-3 and 1, i.c., spanning three orders of magnitude. We plot the average test accuracy (with standard deviation) in Fig. for all three datasets. From this, we can conclude that while the choice of At does influence performance, the variations are not substantial.
90
Test accuracy in %
600
30
SelffegulationSCPP1
Hearteatt
20
Motorlmagery
2
10
Time step
Figure 3: Test accuracies (mean and standard deviation over five different seeds) of LinOSS-IM on three different datasets from Section of the main paper, i.e., SelfRegulationSCPI, Heartheat, and Motorlmagery, for varying values of the timestep At of the underlying implicit time integration scheme of LinOSS (4)
17

---

Published as a conference paper at ICLR 2025
D HIGHER-ORDER INTEGRATION SCHEMES
As outlined in the main text, higher-order discretization schemes can readily applied the context of oscillatory state-space models. Here, we provide example how leverage the second-order velocity Verlet method as the underlying discretization method of LinOSS. To this end, applying the velocity Verlet integration scheme
underlying system of ODEs 2 yields,
A/
Yn = yn-1 + Alzn +
Ayn-1 + +Bu));
2
At (- Ayn + Bu Ayn-1 Bun),
Z, = Zn +
This can be rewritten in matrix form as,
X
M VV
Xn-1 + FVV
(1I)
with X.,
[yn, Zn
and,
MV 1- A
A
Atl A 2
AC Bun
2
FVV
AIA A A)
I-
Ar A ABu,
2 B u U.
Equation can be efficiently solved using fast associative parallel scans, as described in Section 2.4 leading to an alternative model architecture we refer to as LinOSS- VV.The The key distinction from lies in the discretization order: LinOSS-VV employs second-order scheme, yielding more accurate approximations of the underlying ODE system compared to the first-order IMEX approach. However, this comes at the cost of greater computational complexity. We aim to explore the role of higher-order discretization schemes within the LinOSS framework in future research.
the symplectic LinOSS-
E SUPPLEMENT TO THE THEORETICAL INSIGHTS
E.1 EIGENSPECTRUN OF LINOSS-MEX
Proposition E.1. Let IMEX mm be the model LinOSS-IMEX (6). We assume that Ask >> 0 for all diagonal elements k 1...., m of A, Then, the eigenvalues of MIMEX are given as,
weight matrix of the implicit-explict
and further that < Al < max VAns
k=
X 2 (2 Ap) +
1 APP APA for all == 11,..., :....,m,, 2
V
with & mod m. Moreover, all absolute eigenvalues of MIXX are exactly 1, i.e., |Agl 1 for all 2m.
j
Proof. Following the same procedure outlined in the proof Proposition 3.1 the eigenvalues of M IMEX are given as,
>, (2 At A + i(-1)(
2 At A4 Att) for all = 1,...., ,22,
with & mod m. To calculate their absolute value, must consider two distinct cases.
At2 A = 4: it follows directly that |Ay|² = (=)42+2))² =1
2. A With this assumption, compute the absolute value of A as,
1 = - 2t A
2
A
A (4 At?A))
2
4 4At A kk
At
A2
4At2
At A2
kk
Akkk
4
1.
18

---

Published as a conference paper at ICLR 2025
E.2 PROOF PROPOSITION 3.2
Proposition. Let {>,] be the eigenspectrimn of the hidden-to-hidden matrix M of the LinOSS- IM model (4) We further initialize Ark ~ U{[0 Amar) with Amax > O for all diagonal elements ,..., of A in (2) Then, the N-th moment of the magnitude of the eigenvalues are given as,
(At² Amax + 1 -
B(IA)M)
(12)
At Amar (1 N 7
for all j =,.... 2m, with &o j mod
Proof. By the law of the unconscious statistician together with the identity 1yyl /S from the proof of Proposition 3.1 it follows that,
At Amat
E(1,,
(1 A())) 2x
1
u du
Amax J
A12 Amax
(At² Amax
1)
At² Amax
(1
N
where we substituted At +
E.3 PROOF OF THEOREM 3.3
Theorem. Let + Co( T]; TRP) + Co((O, T); R2) be a causal and continuous operator. Let
K C Co((o, TRPP) be compact. Then for any > 0, there exist hyperparameters mn, diagonal
weight matrix A E R weights B E Rm W Rūxx W E Raxm and bias vectors
b R", b R such that the output [0, R of the LinOSS model (9) satisfies,
sup (t) z(f) S e, Vue K. 1E|0,21
Proof. We begin by considering the simple forced harmonic oscillator, y"(() Aty((t) + u(t),
(13)
where u(t) IRP is an external forcing and assume A + O. We further introduce the time- windowed sine transform for the input signal u()).
Lu((A) = S( u(t T)
Then, straightforward calculation shows that the solution y(l) computes (up to a constant) a windoweddowed sine transform, i.e.,
(()) = A- / ((())) ((n)stt((((t d))dT.
(14)
This can easily be verified by differentiatinggy y,
(t) / u(7) cos(A(t T) T))dr (- (()--)
((T) cos(A(t -))d
Differentiating more time yields,
y"(1 A / u(T) ((7)sin(A()) - ((F))in(A(1-7))) + [u(r) cos( (os(A(() r)))
A (' u(T) +))dr u(t)
--PP((t) uu())
19

---

Published as a conference paper at ICLR 2025
We will now make use of the fundamental Lemma in Lanthaler et al. (2024) that provides a finite- dimensional encoding of the operator @ we wish to approximate, i.c., for any e 0, there exists N N, weights A...... Av and continuous mapping v/ R [0, T /4] + I2, such that
(())(()) (CC.u(A))..... Lu((N))
E
for all u E K and t [0,T].
Using the result in (14) we can then construct the input vector [C,u(Ai)..... C,
RON [0, T /4] based the ODE system ( underlying LinOSS model:
L.u(Ai) (A1) (u((A2)
Ay(():
(15)
L.u(AN) t² /4
where y() E RRN solves the system,
y") A + Bu(t) b,
(16)
with
= diag(
A
A
12
A2
AN AN.0): O))
times
p-times
times
0=,1..,0,111) T
p
N -imes
where Ip RPXP is the identity matrix and À equals A, except that ApN.pN = Thus, Ay(t) computes exactly the input to the mniee-immensionll operator By the universal approximation theorem for ordinary neural networks there exist weight matrices W, W and bias b, such that,
(C, .... Liu(AN); ? /4) Wo(W[C. C.l...,LNANNNNN...C /4] + +b] <e
Thus, for every u E K we have,
(u((t)) z(t) < CCCCCWCC)))....
+ (.CCUAA))..,,CAAAA)))) -W)())())) + b) 2e,
with W- WÀ. Since was arbitrary, we conclude that for any causal and continuous operator Co TY: RP) + (o, T]; H2), compact set K C ( [[,,,]]] 7'; RRP) and e > O, there exists LinOSS model of form ( which uniformly approximates & accuracy for all K. This completes the proof.
Remark 2. A natural question arises regarding the performance of LinOSS and other state-space models when learning chaotic dynamical systems. A key issue in the context of learning chaotic systems with recurrent models is the inevitability of exploding gradients during training, rigor- ously demonstrated in Mikhaeil et al. (2022). While our universality result, Theorem 3.3 still holds assuming the assumptions stated fulfilled, it can be seen that the Lipschitz constant of the read- out MLP (i.c., approximation of V in the proof of Theorem 3.3) would blow up, thereby enabling exploding gradients. However, it is crucial to emphasize that this issue persists with all recurrent models and is not unique to LinOSS or other state-space models. Interestingly, it has already been empirically shown (Hu al. (2024) that state-space models (i.e., models based on linear dynam- ies) vastly outperform chaotic RNNs such as LSTMs (Hochreiter 1997) and GRUs (Chung et al. 2014) for learning chaotic dynamical systems.
20


Reviews:
[{"review_id": "0kU14BMTLi", "reviewer": "Reviewer_o2RA", "summary": "This paper introduces a new continuous time recurrent network in the family of state space models. The architecture is proposed as an ODE that is discretized in two ways using first order implicit and implicit-explicit integrators. The terms of the ODE are further constrained to induce two different computational tricks to speed up computation; fast matrix inversion to make the implicit methods tractable, and parallel scans for faster sequence processing. The LinOSS architecture is empirically compared to other Neural ODEs and state-space models, and to transformers, on three different time domain problems for time series classification and prediction.", "questions": "- What are the runtimes of the different methods in the experiments? While the accuracy metrics are strong, one of the purported examples of SSMs is the efficiency. What are run times when enabling and disabling different introduced optimizations:\n  - Run time when not using the equation for matrix inversion?\n  - Run time when not using the parallel scan?\n- Line 74: Why is A diagonal? Is it only to induce the matrix inversion trick later?\n- Some aspects of the architecture are unclear. How exactly are the LinOSS layers stacked into each other and within the network? \n  - In Figure 1, which parts of  the figure are u, y, z?\n  - What effect do the nonlinearities have on the stability analysis?\n  - How is the nonlinear layer in the figure a part of the model?\n- In the experiments in Section 4, what are the exact hyperparameters and model graphs?\n- The exploitation of Formula 3 to speed up the matrix is clever. How does this differentiate during training, though? Did you pass this formula through autodiff, or define a special differentiation rule?\n- Does the parallel scans apply to only using first-order IM or IMEX integrators? Or does the ODE formula in general allow for the parallel scans with other integrators, such as a simple forward Euler, or a higher order IMEX?\n- Table 1: s/UAE/UEA/g.\n- Table 1: What does UEA stand for? Define abbreviations and add citations. (Is Walker 2024 supposed to be the citation?)\n- Color highlighting in the tables is not colorblind friendly, nor BW printer/ereader friendly. Use symbols instead.\n- Equation 7: Why is it important that equation 2 is a Hamiltonian system in this section? If the underlying ODE is indeed Hamiltonian, doesn’t that suggest that the IM discretization is not appropriate?\n- Line 222: What are a, b supposed to be? Describe the specific case of operator & tuple shown here.\n- Line 280: Why is it possible to assume that A_kk>0? Couldn’t they diverge from that assumption during training? What initialization is required?\n- Line 289: s/constraint/constrained/\n- Line 292: The steps in the proof are not obvious. More steps of proof should be presented. In the appendix would be sufficient.\n- Just a remark: The title of section 4.2 is overly extravagant for the claims. These days, “extreme long ranges” would be context lengths of millions of tokens :)\n- Appendix A: What are the parameter counts? What is the architecture of the “nonlinear layers”? What are the hyperparameters for the forecasting problem?\n- Why does the PPG model have higher memory demands, when the models actually seem small?\n- Appendix A: What ML library was used?\n- Could you describe the loss functions and the complete architecture for the 3 different types of problems? How is time series classification grafted onto the LinOSS network?\n- The importance of section 3.2 is unclear to me. It is nice to prove universality of a model, but what is special about LinOSS that other more general proofs of universality would not apply? I would not have questioned it. Is any aspect of the proof specific to LinOSS, or could it apply more broadly to more SSMs?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper demonstrates a cross cutting expertise from dynamical systems analysis through implementation optimization. The design of the ODE is crafting three advantages at three different levels of abstraction simultaneously: enforce theoretically proven stabilization, allow for efficient matrix inversion, and allow for parallel scans of the sequence recurrence. The experiments on time series problems are a good set of problems, and the results of LinOSS stand out against the broad set of comparisons. The proofs are sound, however, it is not clear if they apply to what actually is going on: see below.", "weaknesses": "One issue with the paper is highlighted in the core claim of pre hoc controlling for “forgetting” versus stability by choosing between LinOSS-IM and IMEX (Line 298).  Line 205 claims to demonstrate different advantages between the two methods, but this is not actually evident in the experiments. What characteristics of the problems in Table 1 lead to IM vs. IMEX performing differently? If anything, there is no difference between IM and IMEX in all examples but Worms. Is there something special about Worms, or is it a fluke? The result of Table 2 seems to contradict the claim that IMEX is better at long ranges and IM is better at forgetting: why does IM perform better on a problem where memory over a long sequence should be important? \n\nIt is odd to rely on the integrator choice to enforce stability vs. forgetting. Using an ODE framework, it would seem more natural to change the ODE with dissipative terms, for example, instead of changing the integrator. By relying on the integrator choice, it is unclear if those properties would actually hold after training, if the discretization is thought to add new properties that the ODE did not have. Given the unclear results of the experiments, is it possible that results of the theoretical analysis do not apply after the model is trained? Did you try inspecting if the parameters of A and S still hold the assumptions / initializations assumed in Section 3, after training?\n\n\nConsider the case where the system being learned is actually stable, but the backward euler IM formula is applied. Backward Euler is dissipative even when the dynamical system does not want to dissipate. What happens when you try to learn a model that should be energy conserving using  the LinOSS-IM architecture? The authors could try this by just trying to learn to forecast a simple oscillatory system with LinOSS-IM. I would expect that the model would learn “through” the IM discretization, and converge to an parameterization of an “unstable” ODE that is stable after being discretized by IM. See Krishnapriyan, “Learning continuous models for continuous physics” for a discussion on overfitting on learning through ODE discretizations. \n\nOne idea to start to tackle this problem: Try LinOSS-FE “forward euler”. The architecture would be similar and the tricks in parallel scans would still work. This would be an ablation that would illustrate why stabilization of the implicit and imex integrators is important. If there is no performance difference with the Forward euler discretization in the experiments, or if the IM method can forecast a stable system, then perhaps the theoretical results do not represent what actually happens after training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5ZbPUVrJJ", "reviewer": "Reviewer_su1u", "summary": "The paper provides a novel state-space model. They have two different versions of it, in which they rigorously show the power of their algorithms and also experimentally verify it. The method outperforms SOTA methods in many tasks.", "questions": "To the best of my understanding, the model cannot produce chaotic dynamics. What if the task in hand requires this? How does this contrast (if there is a contrast) with section 3.2?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Provides strong theoretical together with intuitive explanations.\nContrasts their two proposed methods mathematically and also experimentally.\nThe experimental results are excellent and definitely contributes to the field significantly.\nSupplementary material is comprehensive.", "weaknesses": "I think section 3.2 can be written more accessible.\n\nI believe Figure 1 is very important but can be made more explanatory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J0S9oYEz5i", "reviewer": "Reviewer_Wn4t", "summary": "This paper introduces Linear Oscillatory State-Space models (LinOSS), a novel approach to sequence modeling based on forced harmonic oscillators. The model comes in two variants: LinOSS-IM (implicit) and LinOSS-IMEX (implicit-explicit). The key innovation is using second-order ODEs with diagonal state matrices, offering stable dynamics while only requiring non-negative diagonal elements. The paper provides theoretical guarantees for stability and universal approximation, along with empirical validation showing significant improvements over state-of-the-art models like Mamba and LRU on long sequences.", "questions": "1. Could the model be extended to incorporate coupled oscillations while maintaining stability guarantees?\n\n2. What is the impact of the time step parameter Δt on model performance and stability?\n\n3. How does the choice between IM and IMEX variants affect training dynamics and convergence?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates strong theoretical foundations by providing rigorous mathematical analysis of stability conditions, proving universal approximation capability, and establishing clear connections to Hamiltonian systems and symplectic integration. Implementation-wise, it offers a remarkably simple parameterization requiring only non-negative diagonal elements, achieves efficient computation through parallel scans, and presents two complementary variants with different preservation properties. The empirical results are great, showing strong performance on diverse tasks.", "weaknesses": "1. Limited Analysis of Model Interpretability:\n\nWhile based on oscillatory dynamics, lacks discussion of learned frequencies\n\nNo analysis of how the model captures different timescales\n\n2. Experiment:\n\nNo ablation studies on the impact of different initialization schemes\n\nThe implementation details are unclear. For instance, how does it compare to Mamba or S5 in terms of speed, training time, FLOPs, and memory usage? Discussing these aspects could enhance its practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qHjbxVdLt", "reviewer": "Reviewer_w9Vn", "summary": "In this work, the authors propose a state-space model (SSM) architecture, Linear-Oscillatory SSM, derived from the discretization of second-order linear ODEs that models a network of forced harmonic oscillators. They develop two discretization schemes for the approach, study the stability properties of the parametrization and show universal approximation of LinOSS for approximating general continuous, causal input-output mapping. Through empirical evaluations, they demonstrate that their approach outperforms other Linear SSMs on time series classification, prediction and long-horizon forecasting.", "questions": "* L116-118. Since the oscillators are independent, can the proposed architecture model transient synchronization/desynchronization?\n* L250. `Assuming M is diagonalizable [...]`. If $M$ has real eigenvalues with algebraic multiplicity $ > 1$, would that make the system unstable? Admittedly, norm growth would be sub-exponential, so perhaps in practice it's fine.\n* L289 `constraint` $=>$ constrained", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The work is well motivated and the writing is clear and concise. The recurrence matrix afforded by the proposed approach has desirable stability properties and is less constrained compared to the typical SSM parametrizations. The expressivity is further supported by the theoretical analysis on universality of the proposed parametrization. The experimental results are thorough and demonstrate the efficacy of the proposed approach relative to baselines.", "weaknesses": "Given that the parametrization has been introduced in Rusch & Mishra (2021) and the universality results for the non-linear counterpart to this work have been shown in Lanthaler et al., 2024, the novelty of this work is somewhat limited. Still, I think that this is a useful contribution overall as it improves the ability of SSMs for learning long-range dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)