Background and Related Work
State Space Models. Our goal is to accurately describe the evolution of high-dimensional time-series data \bm{y}_{t}\in\mathbb{R}^{M} with the following state-space equations,

\bm{y}_{t} =\bm{D}\bm{x}_{t}+\bm{\epsilon}_{y_{t}}, \quad\bm{\epsilon}_{y_{t}}\sim\mathcal{N}(0,\bm{\Sigma}_{y}), \quad{\rm(Observations)} (1) \bm{x}_{t} =f_{t}(\bm{x}_{t-1})+\bm{\epsilon}_{x_{t}}, \quad\bm{\epsilon}_{x_{t}}\sim\mathcal{N}(0,\bm{\Sigma}_{x}), \quad{\rm(Dynamics)}

where \bm{x}_{t}\in\mathbb{R}^{N} is the latent state, f_{t}(\cdot) is the dynamics function, and \bm{D}\in\mathbb{R}^{M\times N} is a linear observation function. Our work focuses on the case when N<M , which compresses high-dimensional signals into a low-dimensional latent space. By choosing f_{t} to be a time-varying linear operator, we can approximate complex nonlinear dynamics with simple locally-linear components, balancing expressivity with mathematical interpretability. However, learning a time-varying linear operator from data can be challenging, and typically requires additional constraints on the underlying generative model to identify meaningful representations.

Switching Linear Dynamical System (SLDS). SLDS approximates nonlinear systems by introducing a discrete switching variable z_{t}=\{1,\dots,K\} into the time-varying linear dynamics equation,

\bm{x}_{t+1} =\bm{x}_{t}+\bm{F}_{z_{t}}\bm{x}_{t}+\bm{b}_{z_{t}}+\bm{\epsilon}_{x_{t}}.

At each time step t , the latent state \bm{x}_{t} evolves according to the z_{t} -th linear regime defined by \bm{F}_{z_{t}}\in\mathbb{R}^{N\times N} and \bm{b}_{z_{t}}\in\mathbb{R}^{N} while the switching variables evolve according to a Markov matrix. Inference is performed through a vEM algorithm, where the approximate posterior of the latent variables is estimated through coordinate ascent updates over tractable subgraphs. There are many extensions of SLDS, such as rSLDS [ 22 ] which modifies its generative behavior by informing the transitions of z_{t} with x_{t-1} . However, switching models are inherently limited when describing complex signals due to their discrete formulation. For instance, a switched representation is unable to learn that a dynamic regime may exhibit a range of variations. In neural systems, these variations may arise from random spiking processes [ 33 , 36 ] or systems with randomly distributed speeds [ 37 , 11 ] . In SLDS, each variation is learned as a separate discrete state, thus obscuring that the learned states are related. Furthermore, the switching formulation cannot adapt the learned system to unseen variations (i.e. different levels of random speeds). This can produce unstable inference behavior, where the switching state oscillates unpredictably or collapses to a single uninformative state.

Decomposed Linear Dynamical Systems (dLDS). dLDS [ 26 ] relaxes the discrete formulation by approximating nonlinear and nonstationary signals with a time-varying mixture of linear dynamical systems (LDS) defined by the following equations and constraints,

\bm{x}_{t+1}=\bm{x}_{t}+\bm{F}_{t}\bm{x}_{t}+\bm{\epsilon}_{x_{t}},\qquad\bm{F}_{t}=\textstyle\sum_{k=1}^{K}\bm{f}_{k}c_{t,k},\qquad\textrm{ s.t. {$\bm{c}_{t}$} is sparse}

Every transition \bm{F}_{t} is decomposed as a linear combination of sparse coefficients \bm{c}_{t}\in\mathbb{R}^{K} and a dictionary of K DOs \bm{f}_{k}\in\mathbb{R}^{N\times N} .
Figure 1 A shows the corresponding graphical model.
Inference of the latent variables is accomplished by solving the Basis Pursuit Denoising with Dynamic Filtering (BPDN-DF) [ 5 ] objective sequentially for all t and \lambda_{0},\lambda_{1},\lambda_{2}>0 ,

\begin{split}\bm{\widehat{x}}_{t},\bm{\widehat{c}}_{t}=\operatorname*{arg\,min}_{\bm{x}_{t},\bm{c}_{t}}&\|\bm{y}_{t}-\bm{D}\bm{x}_{t}\|_{2}^{2}+\lambda_{0}\|\bm{x}_{t}-\bm{\widehat{x}}_{t-1}-\bm{F}_{t}\bm{\widehat{x}}_{t-1}\|_{2}^{2}+\lambda_{1}\|\bm{c}_{t}\|_{1}+\lambda_{2}\|\bm{c}_{t}-\bm{\widehat{c}}_{t-1}\|_{2}^{2}.\end{split}

This produces a point estimate of \bm{x}_{t} and \bm{c}_{t} that matches the likelihood function resulting from Equations ( 1 ) and ( 2 ). In this objective, the dynamic coefficients are encouraged to be sparse through the \ell_{1} penalty and temporally smooth through the \ell_{2} penalty centered around the previous coefficient estimate. However, this approach is sensitive to noise because inference relies on propagating noisy point estimates of \widehat{\bm{c}}_{t-1} over time. As a result, BPDN-DF may accumulate errors that can lead to significantly different coefficient estimates on signals sampled from the same generative process. Furthermore, the lack of robustness to noise can degrade multi-step inference performance, causing the inferred system to quickly diverge from the true system. This suggests that the inferred latent variables only capture the local activity narrowly and are unable to accurately represent the dynamics beyond a single time-step.
Another drawback of dLDS arises from the dynamics model in equation ( 2 ) which implicitly assumes that the observed dynamics contain a single fixed point that revolves around the origin.
This limits dLDS’s ability to model systems that cannot be easily mean-centered such as those with multiple fixed points or nonstationary drifts.

Sparse Bayesian Learning with Dynamic Filtering. Sparsity is achieved in probabilistic models through hierarchical scale-mixture priors [ 9 , 2 , 4 ] .
To integrate dynamical information into probabilistic sparse signal inference, previous work [ 29 ] proposes the
Sparse Bayesian Learning with Dynamic Filtering (SBL-DF) framework where the following hierarchical model is defined,

p({\bf x}_{t},{\bf c}_{t},{\bf\gamma}_{t})=p({\bf x}_{t}|{\bf c}_{t})\prod_{k=1}^{K}p(c_{t,k}|\gamma_{t,k})p(\gamma_{t,k}|a_{t,k},b_{t,k})

Here, p({\bf x}_{t}|{\bf c}_{t})=\mathcal{N}({\bf\Phi c}_{t},\lambda_{t}{\bf I}) specifies the likelihood, where \bm{\Phi} is a measurement matrix. Sparsity is encouraged through the zero-mean Gaussian priors p(c_{t,k}|\gamma_{t,k})=\mathcal{N}(0,\gamma_{t,k}) independently placed on each element of the sparse vector. The variance parameters \gamma_{t,k} are defined by an inverse gamma hyperprior p(\gamma_{t,k}|\alpha_{t,k},\beta_{t,k})=\mathcal{IG}_{\gamma_{t,k}}(\alpha_{t,k},\beta_{t,k}) with shape parameters \alpha_{t,k} and \beta_{t,k} .
When marginalizing over \gamma_{t,k} ’s, we see that p(c_{t,k}|\alpha_{t,k},\beta_{t,k}) becomes a t-distribution known for its high kurtosis, which is essential for producing sparse solutions.
To propogate dynamics information, the past estimate \widehat{\bm{c}}_{t-1} informs the hyperprior parameters of the next estimate such that b_{t,k}/a_{t,k}=c_{t-1,k}^{2} .