SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {\mathcal {A}} \times_ {2, 3} \mathbf {X} _ {t} = \mathbf {U} \mathcal {G} _ {(1)} \left(\mathbf {I} _ {D _ {2}} \otimes \mathbf {W} ^ {\top} \mathbf {X} _ {t} ^ {\top}\right) \operatorname {v e c} \left(\mathbf {V} ^ {\top}\right) (15) \\ = \mathbf {U} \mathcal {G} _ {(1)} \left(\mathbf {V} ^ {\top} \mathbf {X} _ {t} \otimes \mathbf {I} _ {D _ {3}}\right) \operatorname {v e c} (\mathbf {W}) (16) \\ = \left[ \mathbf {U} \otimes \operatorname {v e c} \left(\mathbf {V} ^ {\top} \mathbf {X} _ {t} \mathbf {W}\right) \right] \operatorname {v e c} (\boldsymbol {\mathcal {G}}). (17) \\ \end{array}
$$
We minimize the negative log likelihood by coordinate descent.
Optimizing the output factors Let
$$
\widetilde {\mathbf {x}} _ {t} = \mathcal {G} _ {(1)} \operatorname {v e c} \left(\mathbf {V} ^ {\top} \mathbf {X} _ {t} \mathbf {W}\right) \tag {18}
$$
for fixed  $\mathbf{V}$ ,  $\mathbf{W}$ , and  $\mathcal{G}$ . The NLL as a function of  $\mathbf{U}$  is,
$$
\mathcal {L} (\mathbf {U}) = \frac {1}{2} \sum_ {t} \left(\mathbf {y} _ {t} - \mathbf {U} \widetilde {\mathbf {x}} _ {t}\right) ^ {\top} \mathbf {Q} ^ {- 1} \left(\mathbf {y} _ {t} - \mathbf {U} \widetilde {\mathbf {x}} _ {t}\right). \tag {19}
$$
This is a standard least squares problem with solution
$$
\mathbf {U} ^ {\star} = \left(\sum_ {t} \mathbf {y} _ {t} \widetilde {\mathbf {x}} _ {t} ^ {\top}\right) \left(\sum_ {t} \widetilde {\mathbf {x}} _ {t} \widetilde {\mathbf {x}} _ {t} ^ {\top}\right) ^ {- 1}. \tag {20}
$$
Optimizing the core tensors Let  $\widetilde{\mathbf{X}}_t = \mathbf{U} \otimes \mathrm{vec}(\mathbf{V}^\top \mathbf{X}_t\mathbf{W}) \in \mathbb{R}^{N_1 \times D_1D_2D_3}$  denote the coefficient on  $\mathrm{vec}(\mathcal{G})$  in eq. (17). The NLL as a function of  $\mathbf{g} = \mathrm{vec}(\mathcal{G})$  is,
$$
\mathcal {L} (\mathbf {g}) = \frac {1}{2} \sum_ {t} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {g}\right) ^ {\top} \mathbf {Q} ^ {- 1} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {g}\right). \tag {21}
$$
The minimizer of this quadratic form is,
$$
\mathbf {g} ^ {\star} = \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \widetilde {\mathbf {X}} _ {t}\right) ^ {- 1} \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \mathbf {y} _ {t}\right) \tag {22}
$$
Figure 6: Depiction of the  $\times_{2,3}$  tensor operator we use [Kolda and Bader, 2009] This can be thought of as a generalization of matrix-vector products to tensor-matrix products.
Optimizing the input factors Let
$$
\widetilde {\mathbf {X}} _ {t} = \mathbf {U} \mathcal {G} _ {(1)} \left(\mathbf {I} _ {D _ {2}} \otimes \mathbf {W} ^ {\top} \mathbf {X} _ {t} ^ {\top}\right) \tag {23}
$$
for fixed  $\mathbf{U}$ ,  $\mathbf{W}$ , and  $\mathcal{G}$ . The NLL as a function of  $\mathbf{v} = \mathrm{vec}(\mathbf{V}^{\top})$  is,
$$
\mathcal {L} (\mathbf {v}) = \frac {1}{2} \sum_ {t} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {v}\right) ^ {\top} \mathbf {Q} ^ {- 1} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {v}\right). \tag {24}
$$
$$
\mathbf {v} ^ {\star} = \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \widetilde {\mathbf {X}} _ {t}\right) ^ {- 1} \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \mathbf {y} _ {t}\right) \tag {25}
$$
Optimizing the lag factors Let
$$
\widetilde {\mathbf {X}} _ {t} = \mathbf {U} \mathcal {G} _ {(1)} \left(\mathbf {V} ^ {\top} \mathbf {X} _ {t} \otimes \mathbf {I} _ {D _ {3}}\right) \tag {26}
$$
for fixed  $\mathbf{U}$ ,  $\mathbf{V}$ , and  $\mathcal{G}$ . The NLL as a function of  $\mathbf{w} = \mathrm{vec}(\mathbf{W})$  is,
$$
\mathcal {L} (\mathbf {w}) = \frac {1}{2} \sum_ {t} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {w}\right) ^ {\top} \mathbf {Q} ^ {- 1} \left(\mathbf {y} _ {t} - \widetilde {\mathbf {X}} _ {t} \mathbf {w}\right). \tag {27}
$$
$$
\mathbf {w} ^ {\star} = \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \widetilde {\mathbf {X}} _ {t}\right) ^ {- 1} \left(\sum_ {t} \widetilde {\mathbf {X}} _ {t} ^ {\top} \mathbf {Q} ^ {- 1} \mathbf {y} _ {t}\right) \tag {28}
$$
Multiple discrete states If we have discrete states  $z_{t} \in \{1, \dots, H\}$  and each state has its own parameters  $(\mathcal{G}^{(h)}, \mathbf{U}^{(h)}, \mathbf{V}^{(h)}, \mathbf{W}^{(h)}, \mathbf{Q}^{(h)})$ , then letting  $\omega_{t}^{(h)} = \mathbb{E}[z_{t} = h]$  denote the weights from the E-step, the summations in coordinate updates are weighted by  $\omega_{t}^{(h)}$ . For example, the coordinate update for the core tensors becomes,
$$
\mathbf {g} ^ {(h) \star} = \left(\sum_ {t} \omega_ {t} ^ {(h)} \widetilde {\mathbf {X}} _ {t} ^ {(h) \top} \mathbf {Q} ^ {(h) - 1} \widetilde {\mathbf {X}} _ {t} ^ {(h)}\right) ^ {- 1} \left(\sum_ {t} \omega_ {t} ^ {(h)} \widetilde {\mathbf {X}} _ {t} ^ {(h) \top} \mathbf {Q} ^ {(h) - 1} \mathbf {y} _ {t}\right) \tag {29}
$$
B SALT approximates a (Switching) Linear Dynamical System
We now re-state and provide a full proof for Proposition 1.
Proposition 1 (Low-Rank Tensor Autoregressions Approximate Stable Linear Dynamical Systems). Consider a stable linear time-invariant Gaussian dynamical system. We define the steady-state Kalman gain matrix as  $\mathbf{K} = \lim_{t\to \infty}\mathbf{K}_t$  and  $\Gamma = \mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C})$ . The matrix  $\Gamma \in \mathbb{R}^{S\times S}$  has eigenvalues  $\lambda_1,\ldots ,\lambda_S$ . Let  $\lambda_{\mathrm{max}} = \max_s|\lambda_s|$ ; for a stable LDS,  $\lambda_{\mathrm{max}} < 1$  [Davis and Vinter, 1985]. Let  $n$  denote the number of real eigenvalues and  $m$  the number of complex conjugate pairs. Let  $\hat{\mathbf{y}}_t^{(\mathrm{LDS})} = \mathbb{E}[\mathbf{y}_t\mid \mathbf{y}_{1:t - 1}]$  denote the predictive mean under a steady-state LDS, and  $\hat{\mathbf{y}}_t^{(\mathrm{SALT})}$  the predictive mean under a SALT model. An order- $L$  Tucker-SALT model with rank  $n + 2m = S$ , or a CP-SALT model with rank  $n + 3m$ , can approximate the predictive mean of the steady-state LDS with error  $\| \hat{\mathbf{y}}_t^{(\mathrm{LDS})} - \hat{\mathbf{y}}_t^{(\mathrm{SALT})}\|_{\infty} = \mathcal{O}(\lambda_{\mathrm{max}}^L)$ .
Proof. A stationary linear dynamical system (LDS) is defined as follows:
$$
\mathbf {x} _ {t} = \mathbf {A} \mathbf {x} _ {t - 1} + \mathbf {b} + \boldsymbol {\epsilon} _ {t} \tag {30}
$$
$$
\mathbf {y} _ {t} = \mathbf {C} \mathbf {x} _ {t} + \mathbf {d} + \boldsymbol {\delta} _ {t} \tag {31}
$$
where  $\mathbf{y}_t\in \mathbb{R}^N$  is the  $t$ -th observation,  $\mathbf{x}_t\in \mathbb{R}^S$  is the  $t$ -th hidden state,  $\epsilon_t\stackrel{\mathrm{i.i.d.}}{\sim}\mathcal{N}(\mathbf{0},\mathbf{Q})$ ,  $\delta_t\stackrel{\mathrm{i.i.d.}}{\sim}\mathcal{N}(\mathbf{0},\mathbf{R})$ , and  $\pmb{\theta} = (\mathbf{A},\mathbf{b},\mathbf{Q},\mathbf{C},\mathbf{d},\mathbf{R})$  are the parameters of the LDS.
Following the notation of Murphy [2012], the one-step-ahead posterior predictive distribution for the observations of the LDS defined above can be expressed as:
$$
p \left(\mathbf {y} _ {t} \mid \mathbf {y} _ {1: t - 1}\right) = \mathcal {N} \left(\mathbf {C} \boldsymbol {\mu} _ {t \mid t - 1} + \mathbf {d}, \mathbf {C} \boldsymbol {\Sigma} _ {t \mid t - 1} \mathbf {C} ^ {T} + \mathbf {R}\right) \tag {32}
$$
$$
\boldsymbol {\mu} _ {t \mid t - 1} = \mathbf {A} \boldsymbol {\mu} _ {t - 1} + \mathbf {b} \tag {33}
$$
$$
\boldsymbol {\mu} _ {t} = \boldsymbol {\mu} _ {t | t - 1} + \mathbf {K} _ {t} \mathbf {r} _ {t} \tag {34}
$$
$$
\boldsymbol {\Sigma} _ {t \mid t - 1} = \mathbf {A} \boldsymbol {\Sigma} _ {t - 1} \mathbf {A} ^ {T} + \mathbf {Q} \tag {35}
$$
$$
\boldsymbol {\Sigma} _ {t} = \left(\mathbf {I} - \mathbf {K} _ {t} \mathbf {C}\right) \boldsymbol {\Sigma} _ {t | t - 1} \tag {36}
$$
$$
p \left(\mathbf {x} _ {1}\right) = \mathcal {N} \left(\mathbf {x} _ {1} \mid \boldsymbol {\mu} _ {1 | 0}, \boldsymbol {\Sigma} _ {1 | 0}\right) \tag {37}
$$
$$
\mathbf {K} _ {t} = \left(\boldsymbol {\Sigma} _ {t | t - 1} ^ {- 1} + \mathbf {C} ^ {T} \mathbf {R} \mathbf {C}\right) ^ {- 1} \mathbf {C} ^ {T} \mathbf {R} ^ {- 1} \tag {38}
$$
$$
\mathbf {r} _ {t} = \mathbf {y} _ {t} - \mathbf {C} \boldsymbol {\mu} _ {t | t - 1} - \mathbf {d}. \tag {39}
$$
We can then expand the mean  $\mathbf{C}\pmb{\mu}_{t|t - 1} + \mathbf{d}$  as follows:
$$
\mathbf {C} \boldsymbol {\mu} _ {t \mid t - 1} + \mathbf {d} = \mathbf {C} \sum_ {l = 1} ^ {t - 1} \boldsymbol {\Gamma} _ {l} \mathbf {A} \mathbf {K} _ {t - l} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = 1} ^ {t - 1} \boldsymbol {\Gamma} _ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} _ {t - l} \mathbf {d}) + \mathbf {d} \tag {40}
$$
$$
\boldsymbol {\Gamma} _ {l} = \prod_ {i = 1} ^ {l - 1} \mathbf {A} (\mathbf {I} - \mathbf {K} _ {t - i} \mathbf {C}) \quad \text {f o r} \quad l \in \{2, 3, \dots \}, \tag {41}
$$
$$
\boldsymbol {\Gamma} _ {1} = \mathbf {I}. \tag {42}
$$
Theorem 3.3.3 of Davis and Vinter [1985] (reproduced with our notation below) states that for a stabilizable and detectable system, the  $\lim_{t\to \infty}\Sigma_{t|t - 1} = \Sigma$ , where  $\Sigma$  is the unique solution of the discrete algebraic Riccati equation
$$
\boldsymbol {\Sigma} = \mathbf {A} \boldsymbol {\Sigma} \mathbf {A} ^ {T} - \mathbf {A} \boldsymbol {\Sigma} \mathbf {C} ^ {T} \left(\mathbf {C} \boldsymbol {\Sigma} \mathbf {C} ^ {T} + \mathbf {R}\right) ^ {- 1} \mathbf {C} \boldsymbol {\Sigma} \mathbf {A} ^ {T} + \mathbf {Q}. \tag {43}
$$
As we are considering stable autonomous LDSs here, the system is stabilizable and detectable, as all unobservable states are themselves stable [Davis and Vinter, 1985, Katayama, 2005]
Theorem 3.3.3 (Reproduced from Davis and Vinter [1985], updated to our notation and context). The theorem has two parts.
(a) If the pair  $(\mathbf{A},\mathbf{C})$  is detectable then there exists at least one non-negative solution,  $\pmb{\Sigma}$ , to the discrete algebraic Riccati equation (43).
(b) If the pair  $(\mathbf{A},\mathbf{C})$  is stabilizable then this solution  $\pmb{\Sigma}$  is unique, and  $\pmb{\Sigma}_{t|t - 1}\rightarrow \pmb{\Sigma}$  as  $t\to \infty$  where  $\pmb{\Sigma}_{t|t - 1}$  is the sequence generated by (33)-(39) with arbitrary initial covariance  $\pmb{\Sigma}_0$ . Then, the matrix  $\Gamma = \mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C})$  is stable, where  $\mathbf{K}$  is the Kalman gain corresponding to  $\pmb{\Sigma}$ ; i.e.,
$$
\mathbf {K} = \left(\boldsymbol {\Sigma} ^ {- 1} + \mathbf {C} ^ {T} \mathbf {R} \mathbf {C}\right) ^ {- 1} \mathbf {C} ^ {T} \mathbf {R} ^ {- 1} \tag {44}
$$
Proof. See Davis and Vinter [1985]. Note that Davis and Vinter [1985] define the Kalman gain as AK.
The convergence of the Kalman gain also implies that each term in the sequence  $\Gamma_{l}$  converges to
$$
\boldsymbol {\Gamma} _ {l} = \prod_ {i = 1} ^ {l - 1} \mathbf {A} (\mathbf {I} - \mathbf {K C}) = \left(\mathbf {A} (\mathbf {I} - \mathbf {K C})\right) ^ {l - 1} = \boldsymbol {\Gamma} ^ {l - 1}, \tag {45}
$$
where, concretely, we define  $\mathbf{\Gamma} = \mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C})$ . We can therefore make the following substitution and approximation
$$
\begin{array}{l} \mathbf {C} \boldsymbol {\mu} _ {t \mid t - 1} + \mathbf {d} \stackrel {\lim  t \rightarrow \infty} {=} \mathbf {C} \sum_ {l = 1} ^ {t - 1} \boldsymbol {\Gamma} ^ {l} \mathbf {A} \mathbf {K} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = 1} ^ {t - 1} \boldsymbol {\Gamma} ^ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} \mathbf {d}) + \mathbf {d} (46) \\ = \mathbf {C} \sum_ {l = 1} ^ {L} \boldsymbol {\Gamma} ^ {l} \mathbf {A} \mathbf {K} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = 1} ^ {L} \boldsymbol {\Gamma} ^ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} \mathbf {d}) + \mathbf {d} + \sum_ {l = L + 1} ^ {\infty} \mathcal {F} \left(\boldsymbol {\Gamma} ^ {l}\right) (47) \\ \approx \mathbf {C} \sum_ {l = 1} ^ {L} \boldsymbol {\Gamma} ^ {l} \mathbf {A} \mathbf {K} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = 1} ^ {L} \boldsymbol {\Gamma} ^ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} \mathbf {d}) + \mathbf {d} (48) \\ \end{array}
$$
The approximation is introduced as a result of truncating the sequence to consider just the "first"  $L$  terms, and discarding the higher-order terms (indicated in blue). It is important to note that each term in (46) is the sum of a geometric sequence multiplied elementwise with  $\mathbf{y}_t$ .
There are two components we prove from here. First, we derive an element-wise bound on the error introduced by the truncation, and verify that under the conditions outlined that the bound decays monotonically in  $L$ . We then show that Tucker and CP decompositions can represent the truncated summations in (48), and derive the minimum rank required for this representation to be exact.
Bounding The Error Term We first rearrange the truncated terms in (46), where we define  $\mathbf{x}_l \triangleq \mathbf{A}\mathbf{K}\mathbf{y}_{t-l} + \mathbf{b} - \mathbf{A}\mathbf{K}\mathbf{d}$
$$
\begin{array}{l} \sum_ {l = L + 1} ^ {\infty} \mathcal {F} (\boldsymbol {\Gamma} ^ {l}) = \mathbf {C} \sum_ {l = L + 1} ^ {\infty} \boldsymbol {\Gamma} ^ {l} \mathbf {A} \mathbf {K} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = L + 1} ^ {\infty} \boldsymbol {\Gamma} ^ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} \mathbf {d}) + \mathbf {d}, (49) \\ = \sum_ {l = L + 1} ^ {\infty} \mathbf {C} \boldsymbol {\Gamma} ^ {l} \mathbf {x} _ {l}, (50) \\ = \sum_ {l = L + 1} ^ {\infty} \mathbf {C E} \boldsymbol {\Lambda} ^ {l - 1} \mathbf {E} ^ {- 1} \mathbf {x} _ {l}, (51) \\ = \sum_ {l = L + 1} ^ {\infty} \mathbf {P} \boldsymbol {\Lambda} ^ {l - 1} \mathbf {q} _ {l}, (52) \\ \end{array}
$$
where  $\mathbf{E}\pmb{\Lambda}\mathbf{E}^{-1}$  is the eigendecomposition of  $\Gamma$ ,  $\mathbf{P} \triangleq \mathbf{C}\mathbf{E}$ , and  $\mathbf{q}_l \triangleq \mathbf{E}^{-1}\mathbf{x}_l$ . We now consider the infinity-norm of the error, and apply the triangle and Cauchy-Schwartz inequalities. We can write the
bound on the as
$$
\begin{array}{l} \epsilon = \left| \left(\sum_ {l = L + 1} ^ {\infty} \mathcal {F} \left(\boldsymbol {\Gamma} ^ {l}\right)\right) _ {n} \right|, \quad \text {w h e r e} \quad n = \underset {k} {\arg \max } \left| \left(\sum_ {l = L + 1} ^ {\infty} \mathcal {F} \left(\boldsymbol {\Gamma} ^ {l}\right)\right) _ {k} \right| (53) \\ = \left| \sum_ {l = L + 1} ^ {\infty} \sum_ {s = 1} ^ {S} p _ {n s} \lambda_ {d} ^ {l - 1} q _ {l, s} \right|, (54) \\ \leq \sum_ {l = L + 1} ^ {\infty} \sum_ {s = 1} ^ {S} \left| p _ {n s} \right| \left| \lambda_ {s} ^ {l - 1} \right| \left| q _ {l, s} \right|. (55) \\ \end{array}
$$
Upper bounding the absolute magnitude of  $q_{l,s}$  by  $W$  provides a further upper bound, which we can then rearrange
$$
\begin{array}{l} \epsilon \leq W \sum_ {l = L + 1} ^ {\infty} \sum_ {s = 1} ^ {S} \left| p _ {n s} \right| \left| \lambda_ {s} ^ {l - 1} \right|, (56) \\ = W \sum_ {s = 1} ^ {S} \left| p _ {n s} \right| \sum_ {l = L + 1} ^ {\infty} \left| \lambda_ {s} ^ {l - 1} \right|. (57) \\ \end{array}
$$
The first two terms are constant, and hence the upper bound is determined by the sum of the of the  $l^{\mathrm{th}}$  power of the eigenvalues. We can again bound this sum by setting all eigenvalues equal to the magnitude of the eigenvalue with the maximum magnitude (spectral norm), denoted  $\lambda_{max}$ :
$$
\epsilon \leq W \sum_ {s = 1} ^ {S} \left| p _ {n s} \right| \sum_ {l = L + 1} ^ {\infty} \lambda_ {\max } ^ {l - 1}, \tag {58}
$$
where these second summation is not a function of  $s$ , and  $W \sum_{s=1}^{S} |p_{ns}|$  is constant. This summation is a truncated geometric sequence. Invoking Theorem 3.3.3 of Davis and Vinter [1985] again, the matrix  $\Gamma$  has only stable eigenvalues, and hence  $\lambda_{\max} < 1$ . Therefore the sequence sum will converge to
$$
\sum_ {l = L + 1} ^ {\infty} \lambda_ {\max } ^ {l - 1} = \frac {\lambda_ {\max } ^ {L}}{1 - \lambda_ {\max }}. \tag {59}
$$
Rearranging again, we see that the absolute error on the  $n^{\mathrm{th}}$  element of  $\mathbf{y}_t$  is therefore bounded according to a power of the spectral norm
$$
\begin{array}{l} \epsilon \leq W \sum_ {s = 1} ^ {S} \left| p _ {n s} \right| \frac {\lambda_ {\max } ^ {L}}{1 - \lambda_ {\max }}, (60) \\ = \mathcal {O} \left(\lambda_ {\max } ^ {L}\right). (61) \\ \end{array}
$$
More specifically, for a stable linear time-invariant dynamical system, and where  $\mathbf{q}$  — and hence  $\mathbf{y}$  — is bounded, then the bound on the error incurred reduces exponentially in the length of the window  $L$ . Furthermore, this error bound will reduce faster for systems with a lower spectral norm.
Diagonalizing the System We first transform  $\Gamma$  into real modal form, defined as  $\mathbf{E}\Lambda \mathbf{E}^{-1}$ , where  $\mathbf{E}$  and  $\Lambda$  are the eigenvectors and diagonal matrix of eigenvalues of  $\Gamma$ . Letting  $\Gamma$  have  $n$  real eigenvalues and  $m$  pairs of complex eigenvalues (i.e.,  $n + 2m = S$ ), we can express  $\mathbf{E}$ ,  $\Lambda$ , and  $\mathbf{E}^{-1}$  as:
$$
\mathbf {E} = \left[ \begin{array}{l l l l l l} \mathbf {a} _ {1} & \dots & \mathbf {a} _ {n} & \mathbf {b} _ {1} & \mathbf {c} _ {1} & \dots & \mathbf {b} _ {m} & \mathbf {c} _ {m} \end{array} \right] \tag {62}
$$
$$
\boldsymbol {\Lambda} = \left[ \begin{array}{c c c c c c c} \lambda_ {1} & & & & & \\ & \ddots & & & & \\ & & \lambda_ {n} & & & \\ & & & \sigma_ {1} & \omega_ {1} & & \\ & & & - \omega_ {1} & \sigma_ {1} & & \\ & & & & \ddots & & \\ & & & & & \sigma_ {m} & \omega_ {m} \\ & & & & & - \omega_ {m} & \sigma_ {m} \end{array} \right] \tag {63}
$$
$$
\mathbf {E} ^ {- 1} = \left[ \begin{array}{c} \mathbf {d} _ {1} ^ {T} \\ \vdots \\ \mathbf {d} _ {n} ^ {T} \\ \mathbf {e} _ {1} ^ {T} \\ \boldsymbol {f} _ {1} ^ {T} \\ \vdots \\ \mathbf {e} _ {m} ^ {T} \\ \boldsymbol {f} _ {m} ^ {T} \end{array} \right] \tag {64}
$$
where  $\mathbf{a}_1\ldots \mathbf{a}_n$  are the right eigenvectors corresponding to  $n$  real eigenvalues  $\lambda_1\ldots \lambda_n$ , and  $\mathbf{b}_i$  and  $\mathbf{c}_i$  are the real and imaginary parts of the eigenvector corresponding to the complex eigenvalue  $\sigma_i + j\omega_i$ . Note that
$$
\boldsymbol {\Gamma} ^ {l} = \left(\mathbf {A} (\mathbf {I} - \mathbf {K C})\right) ^ {l - 1} = \mathbf {E} \boldsymbol {\Lambda} ^ {l - 1} \mathbf {E} ^ {- 1} \tag {65}
$$
The  $l^{th}$  power of  $\Lambda, \Lambda^l$ , where  $l \geq 0$ , can be expressed as:
$$
\boldsymbol {\Lambda} ^ {l} = \left[ \begin{array}{c c c c c c c c} \lambda_ {1} ^ {l} & & & & & & \\ & \ddots & & & & & \\ & & \lambda_ {n} ^ {l} & & & & \\ & & & \sigma_ {1, l} & \omega_ {1, l} & & \\ & & & - \omega_ {1, l} & \sigma_ {1, l} & & \\ & & & & \ddots & & \\ & & & & & \sigma_ {m, l} & \omega_ {m, l} \\ & & & & & - \omega_ {m, l} & \sigma_ {m, l} \end{array} \right] \tag {66}
$$
where  $\sigma_{i,l} = \sigma_{i,l - 1}^2 -\omega_{i,l - 1}^2$ $\omega_{i,l} = 2\sigma_{i,l - 1}\omega_{i,l - 1}$  for  $l\geq 2$ $\sigma_{i,1} = \sigma_i,\omega_{i,1} = \omega_i,\sigma_{i,0} = 1$  , and  $\omega_{i,0} = 0$
Tucker Tensor Regression Let  $\mathcal{H} \in \mathbb{R}^{S \times S \times L}$  be a three-way tensor, whose  $l^{th}$  frontal slice  $\mathbf{H}_{:l} = \boldsymbol{\Lambda}^{l-1}$ . Let  $\mathcal{G} \in \mathbb{R}^{S \times S \times S}$  be a three-way tensor, whose entry  $g_{ijk} = \mathbb{1}_{i=j=k}$  for  $1 \leq k \leq n$ , and  $g_{ijk} = (-1)^{\mathbb{1}_{i+1=j=k+1}} \mathbb{1}_{(i=j=k) \vee (i-1=j-1=k) \vee (i=j+1=k+1) \vee (i+1=j=k+1)}$  for  $k \in \{n+1, n+3, \ldots, n+2m-1\}$ . Let  $\mathbf{W} \in \mathbb{R}^{L \times S}$  be a matrix, whose entry  $w_{lk} = \lambda_k^{l-1}$  for  $1 \leq k \leq n$ ,  $w_{lk} = \sigma_{k,l-1}$  for  $k \in \{n+1, n+3, \ldots, n+2m-1\}$ , and  $w_{lk} = -\omega_{k,l-1}$  for  $k \in \{n+2, n+4, \ldots, n+2m\}$ . We can then decompose  $\mathcal{H}$  into  $\mathcal{G} \in \mathbb{R}^{S \times S \times S}$  and  $\mathbf{W} \in \mathbb{R}^{L \times S}$  such that  $\mathcal{H} = \mathcal{G} \times_3 \mathbf{W}$  (Figure 7).
Figure 7: Decomposition of  $\mathcal{H}$  into  $\mathcal{G}$  and  $\mathbf{W}$  such that  $\mathcal{H} = \mathcal{G} \times_3 \mathbf{W}$ : Given an LDS whose  $\mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C})$  has  $n$  real eigenvalues and  $m$  pairs of complex eigenvalues, this decomposition illustrates how Tucker-SALT can approximate the LDS well with rank  $n + 2m$ .
With  $\mathbf{V} = (\mathbf{E}^{-1}\mathbf{A}\mathbf{K})^T$ ,  $\mathbf{U} = \mathbf{C}\mathbf{E}$ ,  $\mathbf{m} = \mathbf{C}\sum_{l=1}^{L}\Gamma^{l}(\mathbf{b} - \mathbf{A}\mathbf{K}\mathbf{d}) + \mathbf{d}$ , and  $\mathbf{X}_{t} = \mathbf{y}_{t-1:t-L}$ , we can rearrange the mean to:
$$
\begin{array}{l} \mathbf {C} \boldsymbol {\mu} _ {t \mid t - 1} + \mathbf {d} \approx \mathbf {C} \sum_ {l = 1} ^ {L} \mathbf {E} \boldsymbol {\Lambda} ^ {l - 1} \mathbf {E} ^ {- 1} \mathbf {A} \mathbf {K} \mathbf {y} _ {t - l} + \mathbf {C} \sum_ {l = 1} ^ {L} \boldsymbol {\Gamma} ^ {l} (\mathbf {b} - \mathbf {A} \mathbf {K} \mathbf {d}) + \mathbf {d} (67) \\ = \mathbf {U} \sum_ {l = 1} ^ {L} \mathbf {H} _ {: l} \mathbf {V} ^ {T} \mathbf {y} _ {t - l} + \mathbf {m} (68) \\ = \mathbf {U} \sum_ {l = 1} ^ {L} \left(\mathbf {G} \bar {\times} _ {3} \mathbf {w} _ {l}\right) \mathbf {V} ^ {T} \mathbf {y} _ {t - l} + \mathbf {m} (69) \\ = \mathbf {U} \sum_ {l = 1} ^ {L} \left(\left(\boldsymbol {\mathcal {G}} \times_ {2} \mathbf {V}\right) \bar {\times} _ {3} \mathbf {w} _ {l}\right) \mathbf {y} _ {t - l} + \mathbf {m} (70) \\ = \mathbf {U} \sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {S} \sum_ {k = 1} ^ {S} \mathbf {g} _ {: j k} \circ \mathbf {v} _ {: j} \left(w _ {l k} \mathbf {y} _ {t - l}\right) + \mathbf {m} (71) \\ = \mathbf {U} \sum_ {j = 1} ^ {S} \sum_ {k = 1} ^ {S} \mathbf {g} _ {: j k} \left(\mathbf {v} _ {: j} ^ {\top} \mathbf {X} _ {t} \mathbf {w} _ {: k}\right) + \mathbf {m} (72) \\ = \sum_ {i = 1} ^ {S} \sum_ {j = 1} ^ {S} \sum_ {k = 1} ^ {S} \mathbf {u} _ {: i} g _ {i j k} \left(\mathbf {v} _ {: j} ^ {\top} \mathbf {X} _ {t} \mathbf {w} _ {: k}\right) + \mathbf {m} (73) \\ = \left[ \sum_ {i = 1} ^ {n + 2 m} \sum_ {j = 1} ^ {n + 2 m} \sum_ {k = 1} ^ {n + 2 m} g _ {i j k} \mathbf {u} _ {: i} \circ \mathbf {v} _ {: j} \circ \mathbf {w} _ {: k} \right] \times_ {2, 3} \mathbf {X} _ {t} + \mathbf {m} (74) \\ \end{array}
$$
CP Tensor Regression By rearranging  $\mathbf{E},\boldsymbol{\Lambda}^l$  , and  $\mathbf{E}^{-1}$  into  $\mathbf{J},\mathbf{P}_l$  , and  $\mathbf{S}$  respectively as follows:
$$
\mathbf {J} = \left[ \begin{array}{l l l l l l l l} \mathbf {a} _ {1} & \dots & \mathbf {a} _ {n} & \mathbf {b} _ {1} + \mathbf {c} _ {1} & \mathbf {b} _ {1} & \mathbf {c} _ {1} & \dots & \mathbf {b} _ {m} + \mathbf {c} _ {m} & \mathbf {b} _ {m} & \mathbf {c} _ {m} \end{array} \right] \tag {75}
$$
$$
\mathbf {P} _ {l} = \left[ \begin{array}{c c c c c c c c} \lambda_ {1} ^ {l} & & & & & & \\ & \ddots & & & & & \\ & & \lambda_ {n} ^ {l} & & & & \\ & & & \sigma_ {1, l} & & & \\ & & & \alpha_ {1, l} & & & \\ & & & \beta_ {1, l} & & & \\ & & & & \ddots & & \\ & & & & & \sigma_ {m, l} & \\ & & & & & \alpha_ {m, l} & \\ & & & & & \beta_ {m, l} \end{array} \right] \tag {76}
$$
$$
\mathbf {S} = \left[ \begin{array}{c} \mathbf {d} _ {1} ^ {T} \\ \vdots \\ \mathbf {d} _ {n} ^ {T} \\ \mathbf {e} _ {1} ^ {T} + \mathbf {f} _ {1} ^ {T} \\ \mathbf {f} _ {1} ^ {T} \\ \mathbf {e} _ {1} ^ {T} \\ \vdots \\ \mathbf {e} _ {m} ^ {T} + \mathbf {f} _ {m} ^ {T} \\ \mathbf {f} _ {m} ^ {T} \\ \mathbf {e} _ {m} ^ {T} \end{array} \right] \tag {77}
$$
where  $\mathbf{J} \in \mathbb{R}^{S \times (n + 3m)}$ ,  $\mathbf{P}_l \in \mathbb{R}^{(n + 3m) \times (n + 3m)}$ ,  $\mathbf{S} \in \mathbb{R}^{(n + 3m) \times S}$ ,  $\alpha_{i,l} = \omega_{i,l} - \sigma_{i,l}$ , and  $\beta_{i,l} = -\omega_{i,l} - \sigma_{i,l}$ , we can diagonalize  $(\mathbf{A}(\mathbf{I} - \mathbf{K}\mathbf{C}))^l$  as  $\mathbf{J}\mathbf{P}_l\mathbf{S}$ .