NeurIPS 2025

Bridging Expressivity and Scalability
with Adaptive Unitary SSMs
Arjun Karuvally
Salk Institute for Biological Studies
akaruvally@salk.edu
Franz Nowak
ETH Zürich
franz.nowak@inf.ethz.ch
T. Anderson Keller
The Kempner Institute for the Study of Natural
and Artificial Intelligence at Harvard University
t.anderson.keller@gmail.com
Carmen Amo Alonso
Computer Science Department
Stanford University
camoalon@stanford.edu
Terrence J. Sejnowski
Salk Institute for Biological Studies
terry@salk.edu
Hava T. Siegelmann
University of Massachusetts Amherst
hava@umass.edu
Abstract
Recent work has revealed that state space models (SSMs), while efficient for long-
sequence processing, are fundamentally limited in their ability to represent formal
languages—particularly due to time-invariant and real-valued recurrence structures.
In this work, we draw inspiration from adaptive and structured dynamics observed
in biological neural systems and introduce the Adaptive Unitary State Space Model
(AUSSM): a novel class of SSMs that leverages skew-symmetric, input-dependent
recurrence to achieve unitary evolution and high expressive power. Using algebraic
automata theory, we prove that AUSSM can perform modulo counting and simulate
solvable group automata at precision logarithmically bounded in the input length,
enabling SSMs to model a broad class of regular languages out of reach for other
SSM architectures. To overcome the practical inefficiencies of adaptive recurrence,
we develop a separable convolution formulation and a CUDA implementation that
enables scalable parallel training. Empirically, we show that AUSSM and its hybrid
variant—interleaved with Mamba—outperform prior SSMs on formal algorithmic
tasks such as parity and modular arithmetic, and achieve competent performance
on real-world long time-series classification benchmarks. Our results demonstrate
that adaptive unitary recurrence provides a powerful and efficient inductive bias
for both symbolic and continuous sequence modelin...