Title: The Expressive Limits of Diagonal SSMs for State-Tracking
Primary area: learning theory
Keywords: state-space model, SSM, LRNN, linear RNN, expressivity, complex, dynamical system, state-tracking, semigroup, group, automata, Krohn-Rhodes

Abstract: State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling tasks while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited.
In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) State-Space Models (SSMs) on sequential state-tracking tasks for abstract groups. It is easy to show that a single DCD SSM layer with a universal decoder can track any Abelian group at finite precision by decomposing it into a product of cyclic groups. We show that this is tight by proving that such a model cannot track any non-Abelian group at finite precision.
We further establish the expressivity of multi-layer DCD SSMs. We show that a $k$-layer DCD SSM tracks a group if and only if that group has a subnormal series of length at most $k$, with Abelian factor groups. Empirically, while multi-layer models are theoretically expressive enough for solvable non-Abelian groups, we find they often fail to learn such solutions in practice, highlighting a gap between expressivity and learnability.

TL;DR: This paper studies the expressivity of diagonal complex-valued state-space models. A single layer can track Abelian groups but not non-Abelian ones. Stacking layers allows tracking solvable automata but not non-solvable ones.