ICLR 2025

Published as a conference paper at ICLR 2025
OSCILLATORY STATE-SPACE MODELS
T. Konstantin Rusch
MIT
tkrusch@mit.edu
Daniela Rus
MIT
ABSTRACT
We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learn-
ing on long sequences. Inspired by cortical dynamics of biological neural net-
works, we base our proposed LinOSS model on a system of forced harmonic os-
cillators. A stable discretization, integrated over time using fast associative paral-
lel scans, yields the proposed state-space model. We prove that LinOSS produces
stable dynamics only requiring nonnegative diagonal state matrix. This is in stark
contrast to many previous state-space models relying heavily on restrictive param-
eterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can
approximate any continuous and causal operator mapping between time-varying
functions, to desired accuracy. In addition, we show that an implicit-explicit dis-
cretization of LinOSS perfectly conserves the symmetry of time reversibility of
the underlying dynamics. Together, these properties enable efficient modeling of
long-range interactions, while ensuring stable and accurate long-horizon forecast-
ing. Finally, our empirical results, spanning a wide range of time-series tasks from
mid-range to very long-range classification and regression, as well as long-horizon
forecasting, demonstrate that our proposed LinOSS model consistently outper-
forms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba
and LRU by nearly 2x on a sequence modeling task with sequences of length 50k.
Code: https://github.com/tk-rusch/linoss.
1
INTRODUCTION
State-space models (Gu et al., 2021; Hasani et al., 2022; Smith et al., 2023; Orvieto et al., 2023) have
recently emerged as a powerful tool for learning on long sequences. These models posses the state-
fullness and fast inference capabilities of Recurrent Neural Networks (RNNs) together with many of
the benefits of Transformers (Vaswani, 2017; ...