SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyun Dong Lee
hdlee@stanford.edu
Joshua I. Glaser
Department of Neurology
j-glaser@northwestern.edu
Andrew Warrington
awarring@stanford.edu
An important problem in time-series analysis is modeling systems with time-varying dynamics. Probabilistic models with joint continuous and discrete latent states offer interpretable, efficient, and experimentally useful descriptions of such data. Commonly used models include autoregressive hidden Markov models (ARHMMs) and switching linear dynamical systems (SLDSs), each with its own advantages and disadvantages. ARHMMs permit exact inference and easy parameter estimation, but are parameter intensive when modeling long dependencies, and hence are prone to overfitting. In contrast, SLDSs can capture long-range dependencies in a parameter efficient way through Markovian latent dynamics, but present an intractable likelihood and a challenging parameter estimation task. In this paper, we propose switching autoregressive low-rank tensor (SALT) models, which retain the advantages of both approaches while ameliorating the weaknesses. SALT parameterizes the tensor of an ARHMM with a low-rank factorization to control the number of parameters and allow longer range dependencies without overfitting. We prove theoretical and discuss practical connections between SALT, linear dynamical systems, and SLDSs. We empirically demonstrate quantitative advantages of SALT models on a range of simulated and real prediction tasks, including behavioral and neural datasets. Furthermore, the learned low-rank tensor provides novel insights into temporal dependencies within each discrete state.
Many time series analysis problems involve jointly segmenting data and modeling the time-evolution of the system within each segment. For example, a common task in computational ethology [Datta et al., 2019] — the study of natural behavior — is segmenting videos of freely moving animals into states that represent distinct behaviors, while also quantifying the differences in dynamics between states [Wiltschko et al., 2015, Costacurta et al., 2022]. Similarly, discrete shifts in the dynamics of neural activity may reflect changes in underlying brain state [Saravani et al., 2019, Recanatesi et al., 2022]. Model-based segmentations are experimentally valuable, providing an unsupervised grouping of neural or behavioral states together with a model of the dynamics within each state.
One common probabilistic state space model for such analyses is the autoregressive hidden Markov model (ARHMM) [Ephraim et al., 1989]. For example, MoSeq [Wiltschko et al., 2015] uses ARHMMs for unsupervised behavioral analysis of freely moving animals. ARHMMs learn a set of linear autoregressive models, indexed by a discrete state, to predict the next observation as a function of previous observations. Inference in ARHMMs then reduces to inferring which AR
Figure 1: SALT imposes a low-rank constraint on the autoregressive tensor: (A) The probabilistic graphical model of an ARHMM. (B) An example multi-dimensional time series generated from an ARHMM. Background color indicates which discrete state (and hence autoregressive tensor) was selected at each time. (C) In SALT, each autoregressive dynamics tensor of an ARHMM is parameterized as a low-rank tensor.
process best explains the observed data at each timestep (in turn also providing the segmentation). The simplicity of ARHMMs allows for exact state inference via message passing, and closed-form updates for parameter estimation using expectation-maximization (EM). However, the ARHMM requires high order autoregressive dependencies to model long timescale dependencies, and its parameter complexity is quadratic in the data dimension, making it prone to overfitting.
Switching linear dynamical systems (SLDS) [Ghahramani and Hinton, 2000] ameliorate some of the drawbacks of the ARHMM by introducing a low-dimensional, continuous latent state. These models have been used widely throughout neuroscience [Saravani et al., 2019, Petreska et al., 2011, Linderman et al., 2019, Glaser et al., 2020, Nair et al., 2023]. Unlike the ARHMM, the SLDS can capture long timescale dependencies through the dynamics of the continuous latent state, while also being much more parameter efficient than ARHMMs. However, exact inference in SLDSs is intractable due to the exponential number of potential discrete state paths governing the time-evolution of the continuous latent variable. This intractability has led to many elaborate and specialized approximate inference techniques [Ghahramani and Hinton, 2000, Barber, 2006, Fox, 2009, Murphy and Russell, 2001, Linderman et al., 2017, Zoltowski et al., 2020]. Thus, the SLDS gains parameter efficiency at the expense of the computational tractability and statistical simplicity of the ARHMM.
We propose a new class of unsupervised probabilistic models that we call switching autoregressive low-rank tensor (SALT) models. Our novel insight is that when you marginalize over the latent states of a linear dynamical system, you obtain an autoregressive model with full history dependence. However, these autoregressive dependencies are not arbitrarily complex — they factor into a low-rank tensor that can be well-approximated with a finite-history model. We formalize this connection in Proposition 1. SALT models are constrained ARHMMs that leverage this insight. Rather than allowing for arbitrary autoregressive dependencies, SALT models are constrained to be low-rank. The low-rank property allows us to construct a low-dimensional continuous description of the data, jointly with the discrete segmentation provided by the switching states. Thus, SALT models inherit the experimentally useful representations and parsimonious parameter complexity of an SLDS, as well as the ease of inference and estimation of ARHMMs. We demonstrate the advantages of SALT models empirically using synthetic data as well as real neural and behavioral time series. Finally, in addition to improving predictive performance, we show how the low-rank nature of SALT models can offer new insights into complex systems, like biological neural networks.
This section introduces the notation used throughout the paper and describes preliminaries on low-rank tensor decomposition, vector autoregressive models, switching autoregressive models, linear dynamical systems, and switching linear dynamical systems.
Notation We follow the notation of Kolda and Bader [2009]. We use lowercase letters for scalar variables (e.g.  $a$ ), uppercase letters for scalar constants (e.g.  $A$ ), boldface lowercase letters for vectors (e.g.  $\mathbf{a}$ ), boldface uppercase letters for matrices (e.g.  $\mathbf{A}$ ), and boldface Euler script for tensors of
Source code is available at https://github.com/lindermanlab/salt.
order three or higher (e.g.  $\mathcal{A}$ ). We use  $\mathbf{A}_{i:},\mathbf{A}_{j:}$ , and  $\mathbf{A}_{:k}$  to denote the horizontal, lateral, and frontal slices respectively of a three-way tensor  $\mathcal{A}$ . Similarly, we use  $\mathbf{a}_i$ : and  $\mathbf{a}_{:j}$  to denote the  $i^{th}$  row and  $j^{th}$  column of a matrix  $\mathbf{A}$ .  $\mathbf{a} \circ \mathbf{b}$  represents the vector outer product between vectors  $\mathbf{a}$  and  $\mathbf{b}$ . The  $n$ -mode tensor-matrix (tensor-vector) product is represented as  $\mathcal{A} \times_n \mathbf{A} (\mathcal{A} \bar{\times}_n \mathbf{a})$ . We denote the vectorization of an  $n$ -way tensor  $\mathcal{G}$ , with dimensions  $D_{1:n}$ , as  $\operatorname{vec}(\mathcal{G})$ . This is performed by successively flattening the last dimensions of the tensor, and results in a vector of size equal to the product of the dimensions of the tensor. We denote the mode- $n$  matricization of a tensor  $\mathcal{G}$  as  $\mathcal{G}_{(n)}$ . This is defined as the stack of vectors resulting from vectorizing the matrix (or tensor) defined by each slice through the  $n^{\text{th}}$  dimension. This results in a matrix with leading dimension  $D_n$ , and second dimension equal to the product of the sizes of the other dimensions. We will denote a  $T$ -length time series of  $N$ -dimensional observed data as  $\mathbf{Y} \in \mathbb{R}^{N \times T}$ . Note that we will use the shorthand  $\mathbf{y}_t \in \mathbb{R}^N$  to denote the observation at time  $t$ , and  $y_{j,t} \in \mathbb{R}$  to denote the  $j^{\text{th}}$  element in the  $t^{\text{th}}$  observation. It will be clear from context which dimension is being indexed.
Tensor Decomposition For  $\mathcal{A} \in \mathbb{R}^{N_1 \times N_2 \times N_3}$ , the Tucker decomposition is defined as,
$$
\mathcal {A} = \sum_ {i = 1} ^ {D _ {1}} \sum_ {j = 1} ^ {D _ {2}} \sum_ {k = 1} ^ {D _ {3}} g _ {i j k} \mathbf {u}: i \circ \mathbf {v}: j \circ \mathbf {w}: k, \tag {1}
$$
where  $\mathbf{u}_{:i}$ ,  $\mathbf{v}_{:j}$ , and  $\mathbf{w}_{:k}$  are the columns of the factor matrices  $\mathbf{U} \in \mathbb{R}^{N_1 \times D_1}$ ,  $\mathbf{V} \in \mathbb{R}^{N_2 \times D_2}$ , and  $\mathbf{W} \in \mathbb{R}^{N_3 \times D_3}$ , respectively, and  $g_{ijk}$  are the entries in the core tensor  $\mathbf{G} \in \mathbb{R}^{D_1 \times D_2 \times D_3}$ .
The CANDECOMP/PARAFAC (CP) decomposition is a special case of the Tucker decomposition, with  $D_{1} = D_{2} = D_{3}$  and a diagonal core tensor  $\mathcal{G}$ .
Vector autoregressive models Let  $\mathbf{Y} \in \mathbb{R}^{N \times T}$  denote a multivariate time series with  $\mathbf{y}_t \in \mathbb{R}^N$  for all  $t$ . An order- $L$  vector autoregressive (VAR) model with Gaussian innovations is defined by,
$$
\mathbf {y} _ {t} \sim \mathcal {N} \left(\sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {L} \mathbf {a} _ {: j k} y _ {j, t - k} + \mathbf {b}, \mathbf {R}\right), \tag {2}
$$
where  $\mathcal{A} \in \mathbb{R}^{N \times N \times L}$  is the autoregressive tensor, whose frontal slice  $\mathbf{A}_{:l}$  is the dynamics matrix for lag  $l$ ,  $\mathbf{b} \in \mathbb{R}^N$  is the bias, and  $\mathbf{R} \in \mathbb{R}_{\geq 0}^{N \times N}$  is a positive semi-definite covariance matrix. The parameters  $\Theta = (\mathcal{A}, \mathbf{b}, \mathbf{R})$  can be estimated via ordinary least squares [Hamilton, 2020].
We note that, to our knowledge, there is no clear consensus on the best way to regularize the potentially large parameter space of vector autoregressive (hidden Markov) models; several possibilities exist, see, e.g., Melnyk and Banerjee [2016] or Ni and Sun [2005]. Many regularizers and priors are difficult to work with, and so are not widely used in practice. Beyond this, even well-regularized ARHMMs do not natively capture interpretable low-dimensional dynamics, as both SALT and SLDS models do (see Figure 3). These low-dimensional continuous representations are as experimentally useful as the discrete segmentation, and hence are a key desiderata for any method we consider.
Switching autoregressive models One limitation of VAR models is that they assume the time series is stationary; i.e. that one set of parameters holds for all time steps. Time-varying autoregressive models allow the autoregressive process to change at various time points. One such VAR model, referred to as a switching autoregressive model or autoregressive hidden Markov model (ARHMM), switches the parameters over time according to a discrete latent state [Ephraim et al., 1989]. Let  $z_{t} \in \{1, \dots, H\}$  denote the discrete state at time  $t$ , an ARHMM defines the following generative model,
$$
z _ {t} \sim \operatorname {C a t} \left(\boldsymbol {\pi} ^ {(z _ {t - 1})}\right), \quad \mathbf {y} _ {t} \sim \mathcal {N} \left(\sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {L} \mathbf {a} _ {: j k} ^ {(z _ {t})} y _ {j, t - k} + \mathbf {b} ^ {(z _ {t})}, \mathbf {R} ^ {(z _ {t})}\right), \tag {3}
$$
where  $\pmb{\pi}^{(h)}\in \{\pmb{\pi}^{(h)}\}_{h = 1}^{H}$  is the  $h$  -th row of the discrete state transition matrix.
A switching VAR model is simply a type of hidden Markov model, and as such it is easily fit via the expectation-maximization (EM) algorithm within the Baum-Welch algorithm. The M-step amounts to solving a weighted least squares problem.
Linear dynamical systems The number of parameters in a VAR model grows as  $\mathcal{O}(N^2 L)$ . For high-dimensional time series, this can quickly become intractable. Linear dynamical systems (LDS) [Murphy, 2012] offer an alternative means of modeling time series via a continuous latent state  $\mathbf{x}_t\in \mathbb{R}^S$ ,
$$
\mathbf {x} _ {t} \sim \mathcal {N} (\mathbf {A x} _ {t - 1} + \mathbf {b}, \mathbf {Q}), \quad \mathbf {y} _ {t} \sim \mathcal {N} (\mathbf {C x} _ {t} + \mathbf {d}, \mathbf {R}), \tag {4}
$$
where  $\mathbf{Q} \in \mathbb{R}_{\succeq 0}^{S \times S}$  and  $\mathbf{R} \in \mathbb{R}_{\succeq 0}^{N \times N}$ . Here, the latent states follow a first-order VAR model, and the observations are conditionally independent given the latent states. As we discuss in Section 3.3, marginalizing over the continuous latent states renders  $\mathbf{y}_t$  dependent on the preceding observations, just like in a high order VAR model.
Compared to the VAR model, however, the LDS has only  $\mathcal{O}(S^2 + NS + N^2)$  parameters if  $\mathbf{R}$  is a full covariance matrix. This further reduces to  $\mathcal{O}(S^2 + NS)$  if  $\mathbf{R}$  is diagonal. As a result, when  $S \ll N$ , the LDS has many fewer parameters than a VAR model. Thanks to the linear and Gaussian assumptions of the model, the parameters can be easily estimated via EM, using the Kalman smoother to compute the expected values of the latent states.
Switching linear dynamical systems A switching LDS combines the advantages of the low-dimensional continuous latent states of an LDS, with the advantages of discrete switching from an ARHMM. Let  $z_{t} \in \{1, \dots, H\}$  be a discrete latent state with Markovian dynamics (3), and let it determine some or all of the parameters of the LDS (e.g.  $\mathbf{A}$  would become  $\mathbf{A}^{(z_t)}$  in (4)). We note that SLDSs often use a single-subspace, where  $\mathbf{C}$ ,  $\mathbf{d}$  and  $\mathbf{R}$  are shared across states, reducing parameter complexity and simplifying the optimization.
Unfortunately, parameter estimation is considerably harder in SLDS models. The posterior distribution over all latent states,  $p(\mathbf{z}_{1:T}, \mathbf{x}_{1:T} \mid \mathbf{y}_{1:T}, \boldsymbol{\Theta})$ , where  $\boldsymbol{\Theta}$  denotes the parameters, is intractable [Lerner, 2003]. Instead, these models are fit via approximate inference methods like MCMC [Fox, 2009, Linderman et al., 2017], variational EM [Ghahramani and Hinton, 2000, Zoltowski et al., 2020], particle EM [Murphy and Russell, 2001, Doucet et al., 2001], or other approximations [Barber, 2006]. Selecting the appropriate fitting and inference methodologies is itself non-trivial hyperparameter. Furthermore, each method also brings additional estimation hyperparameters that need to be tuned prior to even fitting the generative model. We look to define a model that enjoys the benefits of SLDSs, but avoids the inference and estimation difficulties.
3 SALT: Switching Autoregressive Low-rank Tensor Models
Here we formally introduce SALT models. We begin by defining the generative model (also illustrated in Figure 1), and describing how inference and model fitting are performed. We conclude by drawing connections between SALT and SLDS models.
3.1 Generative Model
SALT factorizes each autoregressive tensor  $\mathcal{A}^{(h)}$  for  $h\in \{1,\dots ,H\}$  of an ARHMM as a product of low-rank factors. Given the current discrete state  $z_{t}$ , each observation  $\mathbf{y}_t\in \mathbb{R}^N$  is modeled as being normally distributed conditioned on  $L$  previous observations  $\mathbf{y}_{t - 1:t - L}$
$$
z _ {t} \sim \operatorname {C a t} \left(\boldsymbol {\pi} ^ {\left(z _ {t - 1}\right)}\right), \tag {5}
$$
$$
\mathbf {y} _ {t} \stackrel {\text {i . i . d .}} {\sim} \mathcal {N} \left(\sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {L} \mathbf {a} _ {\mathrm {S A L T},: j k} ^ {(z _ {t})} y _ {j, t - k} + \mathbf {b} ^ {(z _ {t})}, \boldsymbol {\Sigma} ^ {(z _ {t})}\right), \tag {6}
$$
$$
\mathcal {A} _ {\text {S A L T}} ^ {(z _ {t})} = \sum_ {i = 1} ^ {D _ {1}} \sum_ {j = 1} ^ {D _ {2}} \sum_ {k = 1} ^ {D _ {3}} g _ {i j k} ^ {(z _ {t})} \mathbf {u} _ {: i} ^ {(z _ {t})} \circ \mathbf {v} _ {: j} ^ {(z _ {t})} \circ \mathbf {w} _ {: k} ^ {(z _ {t})}, \tag {7}
$$
where  $\mathbf{u}_{:i}^{(z_t)}$ ,  $\mathbf{v}_{:j}^{(z_t)}$ , and  $\mathbf{w}_{:k}^{(z_t)}$  are the columns of the factor matrices  $\mathbf{U}^{(z_t)} \in \mathbb{R}^{N \times D_1}$ ,  $\mathbf{V}^{(z_t)} \in \mathbb{R}^{N \times D_2}$ , and  $\mathbf{W}^{(z_t)} \in \mathbb{R}^{L \times D_3}$ , respectively, and  $g_{ijk}^{(z_t)}$  are the entries in the core tensor  $\mathcal{G}^{(z_t)} \in \mathbb{R}^{D_1 \times D_2 \times D_3}$ . The vector  $\pmb{b}^{(z_t)} \in \mathbb{R}^N$  and positive definite matrix  $\pmb{\Sigma}^{(z_t)} \in \mathbb{R}_{\geq 0}^{N \times N}$  are the bias and covariance for state  $z_t$ . Without further restriction this decomposition is a Tucker decomposition [Kolda and Bader, 2009]. If  $D_1 = D_2 = D_3$  and  $\mathcal{G}_{z_t}$  is diagonal, it corresponds to a
Table 1: Comparison of number of parameters for the methods we consider. We exclude covariance matrix parameters, as the parameterization of the covariance matrix is independent of method. Throughout our experiments, we find  $S \approx D$ .
Model	Parameter Complexity	(Example from Section 5.4)
SLDS	O(NS + HS2)	2.8K
CP-SALT	O(H(ND + LD))	8.1K
Tucker-SALT	O(H(ND + LD + D3))	17.4K
Order-L ARHMM	O(HN2L)	145.2K
CP decomposition [Kolda and Bader, 2009]. We refer to ARHMM models with these factorizations as Tucker-SALT and CP-SALT respectively. Note that herein we will only consider models where  $D_{1} = D_{2} = D_{3} = D$ , where we refer to  $D$  as the "rank" of the SALT model (for both Tucker-SALT and CP-SALT). In practice, we find that models constrained in this way perform well, and so this constraint is imposed simply to reduce the search space of models and could easily be relaxed.
Table 1 shows the number of parameters for order-  $L$  ARHMMs, SLDSs, and SALT. Focusing on the lag dependence, the number of ARHMM parameters grows as  $\mathcal{O}(HN^2 L)$ , whereas SALT grows as only  $\mathcal{O}(HDL)$  with  $D \ll N$ . SALT can also make a simplifying single-subspace constraint, where certain emission parameters are shared across discrete states.
Low-dimensional Representation Note that SALT implicitly defines a low-dimensional continuous representation, analogous to the continuous latent variable in SLDS,
$$
\mathcal {P} = \sum_ {j = 1} ^ {D _ {2}} \sum_ {k = 1} ^ {D _ {3}} \mathbf {g} _ {: j k} ^ {(z _ {t})} \circ \mathbf {v} _ {: j} ^ {(z _ {t})} \circ \mathbf {w} _ {: k} ^ {(z _ {t})}, \tag {8}
$$
$$
\mathbf {x} _ {t} = \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {L} \mathbf {p}: j k y _ {j, t - k}. \tag {9}
$$
The low-dimensional  $\mathbf{x}_t\in \mathbb{R}^{D_1}$  vectors can be visualized, similar to the latent states in SLDS models, to further interrogate the learned dynamics, as we show in Figure 3. Note the vector  $\mathbf{x}_t\in \mathbb{R}^{D_1}$ , when multiplied by the output factors  $\mathbf{U}^{(z_t)}$ , is the mean of the next observation.
3.2 Model Fitting and Inference
Since SALT models are ARHMMs, we can apply the expectation-maximization (EM) algorithm to fit model parameters and perform state space inference. We direct the reader to Murphy [2012] for a detailed exposition of EM and include only the key points here.
The E-step solves for the distribution over latent variables given observed data and model parameters. For SALT, this is the distribution over  $z_{t}$ , denoted  $\omega_{t}^{(h)} = \mathbb{E}[z_{t} = h \mid \mathbf{y}_{1:T}, \boldsymbol{\theta}]$ . This can be computed exactly with the forward-backward algorithm, which is fast and stable. The marginal likelihood can be evaluated exactly by taking the product across  $t$  of expectations of (6) under  $\omega_{t}^{(h)}$ .
The M-step then updates the parameters of the model given the distribution over latent states. For SALT, the emission parameters are  $\pmb{\theta} = \{\mathbf{U}^{(h)},\mathbf{V}^{(h)},\mathbf{W}^{(h)},\mathbf{G}^{(h)},\mathbf{b}^{(h)},\boldsymbol{\Sigma}^{(h)},\boldsymbol{\pi}^{(h)}\}_{h = 1}^{H}$ . We use closed-form coordinate-wise updates to maximize the expected log likelihood evaluated in the E-step. Each factor update amounts to solving a weighted least squares problem. We include just one update step here for brevity, and provide all updates in full in Appendix A. Assuming here that  $\mathbf{b}^{(h)} = \mathbf{0}$  for simplicity, the update rule for the lag factors is as follows:
$$
\mathbf {w} ^ {(h) \star} = \left(\sum_ {t} \omega_ {t} ^ {(h)} \widetilde {\mathbf {X}} _ {t} ^ {(h) \top} \left(\boldsymbol {\Sigma} ^ {(h)}\right) ^ {- 1} \widetilde {\mathbf {X}} _ {t} ^ {(h)}\right) ^ {- 1} \left(\sum_ {t} \omega_ {t} ^ {(h)} \widetilde {\mathbf {X}} _ {t} ^ {(h) \top} \left(\boldsymbol {\Sigma} ^ {(h)}\right) ^ {- 1} \mathbf {y} _ {t}\right) \tag {10}
$$
where  $\widetilde{\mathbf{X}}_t^{(h)} = \mathbf{U}^{(h)}\mathcal{G}_{(1)}^{(h)}(\mathbf{V}^{(h)\top}\mathbf{y}_{t - 1:t - L}\otimes \mathbf{I}_{D_3})$  and  $\mathbf{w}^{(h)\star} = \mathrm{vec}(\mathbf{W}^{(h)})$ . Crucially, these coordinate wise updates are exact, and so we recover the fast and monotonic convergence of EM.
3.3 Connections Between SALT and Linear Dynamical Systems
SALT is not only an intuitive regularization for ARHMMs, it is grounded in a mathematical correspondence between autoregressive models and linear dynamical systems.
Proposition 1 (Low-Rank Tensor Autoregressions Approximate Stable Linear Dynamical Systems). Consider a stable linear time-invariant Gaussian dynamical system. We define the steady-state Kalman gain matrix as  $\mathbf{K} = \lim_{t\to \infty}\mathbf{K}_t$ , and  $\Gamma = \mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C})$ . The matrix  $\Gamma \in \mathbb{R}^{S\times S}$  has eigenvalues  $\lambda_1,\ldots ,\lambda_S$ . Let  $\lambda_{\mathrm{max}} = \max_s|\lambda_s|$ ; for a stable LDS,  $\lambda_{\mathrm{max}} < 1$  [Davis and Vinter, 1985]. Let  $n$  denote the number of real eigenvalues and  $m$  the number of complex conjugate pairs. Let  $\hat{\mathbf{y}}_t^{(\mathrm{LDS})} = \mathbb{E}[\mathbf{y}_t\mid \mathbf{y}_{1:t - 1}]$  denote the predictive mean under a steady-state LDS, and  $\hat{\mathbf{y}}_t^{(\mathrm{SALT})}$  the predictive mean under a SALT model. An order- $L$  Tucker-SALT model with rank  $n + 2m = S$ , or a CP-SALT model with rank  $n + 3m$ , can approximate the predictive mean of the steady-state LDS with error  $\| \hat{\mathbf{y}}_t^{(\mathrm{LDS})} - \hat{\mathbf{y}}_t^{(\mathrm{SALT})}\|_\infty = \mathcal{O}(\lambda_{\mathrm{max}}^L)$ .
Proof. We give a sketch of the proof here and a full proof in Appendix B. The analytic form of  $\mathbb{E}\left[\mathbf{y}_t\mid \mathbf{y}_{1:t - 1}\right]$  is a linear function of  $\mathbf{y}_{t - l}$  for  $l = 1,\dots ,\infty$  . For this sketch, consider the special case where  $\mathbf{b} = \mathbf{d} = \mathbf{0}$  . Then the coefficients of the linear function are  $\mathbf{C}\Gamma^l\mathbf{K}$  . As all eigenvalues of  $\Gamma$  have magnitude less than one, the coefficients decay exponentially in  $l$  . We can therefore upper bound the approximation error introduced by truncating the linear function to  $L$  terms to  $\mathcal{O}(\lambda_{\max}^{L})$  To complete the proof, we show that the truncated linear function can be represented exactly by a tensor regression with at most a specific rank. Thus, only truncated terms contribute to the error.
This proposition shows that the steady-state predictive distribution of a stable LDS can be approximated by a low-rank tensor autoregression, with a rank determined by the eigenspectrum of the LDS. We validate this proposition experimentally in Section 5.1. Note as well that the predictive distribution will converge to a fixed covariance, and hence can also be exactly represented by the covariance matrices  $\Sigma^{(h)}$  estimated in SALT models.