Switching Autoregressive Low-rank Tensor Models
Authors: Hyun Dong Lee
Andrew Warrington
Joshua I Glaser
Scott Linderman
Venue: NeurIPS 2023 poster (2023)
Keywords: switching
autoregressive
low-rank tensor
time-series
probabilistic
neural
neuroscience
behavioral
arhmm
slds
TL;DR: We propose switching autoregressive low-rank tensor models, which retain the advantages of both ARHMMs and SLDSs, while ameliorating both of their weaknesses.

Abstract: An important problem in time-series analysis is modeling systems with time-varying dynamics.  Probabilistic models with joint continuous and discrete latent states offer interpretable, efficient, and experimentally useful descriptions of such data.  Commonly used models include autoregressive hidden Markov models (ARHMMs) and switching linear dynamical systems (SLDSs), each with its own advantages and disadvantages.  ARHMMs permit exact inference and easy parameter estimation, but are parameter intensive when modeling long dependencies, and hence are prone to overfitting.  In contrast, SLDSs can capture long-range dependencies in a parameter efficient way through Markovian latent dynamics, but present an intractable likelihood and a challenging parameter estimation task.  In this paper, we propose _switching autoregressive low-rank tensor_ SALT models, which retain the advantages of both approaches while ameliorating the weaknesses.  SALT parameterizes the tensor of an ARHMM with a low-rank factorization to control the number of parameters and allow longer range dependencies without overfitting.  We prove theoretical and discuss practical connections between SALT, linear dynamical systems, and SLDSs.  We empirically demonstrate quantitative advantages of SALT models on a range of simulated and real prediction tasks, including behavioral and neural datasets.  Furthermore, the learned low-rank tensor provides novel insights into temporal dependencies within each discrete state.

Switching Autoregressive Low-rank Tensor Models
Hyun Dong Lee
Andrew Warrington
Computer Science Department Stanford University
Department of Statisties
Stanford University
hdlee@stanford.co edu
awarringdstanford.com
Joshua I. Glaser
Scott W. Linderman
Department of Neurology
Department of Statistics
Northwestern University
Stanford University
j -gl whweernortweeeterr..cmmm edu
scott. ccot:.Lindermannsttaffooo edu
Abstract
An important problem in time-series analysis is modeling systems with time- varying dynamics. Probabilistic models with joint continuous and discrete latent states offer interpretable, efficient, and experimentally useful descriptions of such data. Commonly used models include autoregressive hidden Markov models (ARHMMs) and switching linear dynamical systems (SLDSs), each with its own advantages and disadvantages, ARHMMs permit exact inference and casy param- eter estimation, but parameter intensive when modeling long dependencies, and hence are prone to overfitting. In contrast, SLDSs can capture long-range dependencies in parameter efficient way through Markovian latent dynamics, but present an intractable likelihood and a challenging parameter estimation task. In this paper, we propose switching autoregressive low-rank tensor (SALT) models, which retain the advantages of both approaches while ameliorating the weaknesses. SALT parameterizes the tensor of ARHMM with low-rank factorization control the number of parameters and allow longer range dependencies without overfitting. We prove theoretical and discuss practical connections between SALT, linear dynamical systems, and SLDSs. We empirically demonstrate quantitative advantages of SALT models on range of simulated and real prediction tasks, including behavioral and neural datasets. Furthermore, the learned low-rank tensor provides novel insights into temporal dependencies within each discrete state.
Introduction
Many time series analysis problems involve jointly segmenting data and modeling the time-evolution of the system within each segment. For example, common task computational ethology [Datta et al., 2019] - the study of natural behavior - is segmenting videos of freely moving animals into states that represent distinct behaviors, while also quantifying the differences in dynamics between states [Wiltschko et al., 2015, Costacurta et al., 2022]. Similarly, discrete shifts in the dynamics of neural activity may reflect changes in underlying brain state [Saravani et al., 2019, Recanatesi et al., 2022]. Model-based segmentations experimentally valuable, providing unsupervised grouping of neural or behavioral states together with a model of the dynamics within each state.
One common probabilistic state space model for such analyses is the autoregressive hidden Markov model (ARHMM) [Ephraim et al., 1989]. For example, MoSeg [Wiltschko et al., 2015] uses ARHMMs for unsupervised behavioral analysis of freely moving animals. ARHMMs learn a set of linear autoregressive models, indexed by discrete state, to predict the next observation function of previous observations. Inference in ARHMMs then reduces to inferring which AR
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

B ==-
AR Tensor Tensor_A(e))
Core Tensor
Lag Factor w
Transition
Discrete states
Lag dim, L.
g g(et)
Matrix
Obs, dim, N
Input Factor
AR
(Ouiput)
V(Ra
Tensors
Obs. dim, N
Outpul Factor
50
100
155
200
(Input)
(()))
Observations
Time,
Y1
A
y
06
20) Mik
ye-net
Figure SALT imposes a low-rank constraint on the autoregressive tensor: (A) The probabilistic graphical model of an ARHMM. (B) An example multi-dimennsonal time series generated from an ARHMM. Background color indicates which discrete state (and hence autoregressive tensor) was selected at each time. (C) In SALT, each autoregressive dynamics tensor of an ARHMM is parameterized as a low-rank tensor.
process best explains the observed data at each timestep (in turn also providing the segmentation). The simplicity of ARHMMs allows for exact state inference via message passing, and closed-form updates for parameter estimation using expectation- expectation-maxiinzzaoon (EM). However, the ARHMM requires high order autoregressive dependencies to model long timescale dependencies, and its parameter complexity quadratic in the data dimension, making prone to overfitting.
Switching linear dynamical systems (SLDS) [Ghahramani and Hinton, 2000] ameliorate some of the drawbacks of the ARHMM by introducing low-dimensional, continuous latent state. These models have been used widely throughout neuroscience [Saravani et al., 2019, Petreska et al., 2011, Linderman et al., 2019, Glaser et al., 2020, Nair et al., 2023]. Unlike the ARHMM, the SLDS can capture long timescale dependencies through the dynamics of the continuous latent state, while also being much more parameter efficient than ARHMMs. However, exact inference in SLDSS is intractable due to the exponential number of potential discrete state paths governing the time-evolution of the continuous latent variable. This intractability has led to many elaborate and specialized approximate inference techniques [Ghahramani and Hinton, 2000, Barber, 2006, Fox, 2009, Murphy and Russell, 2001, Linderman et al., 2017, Zoltowski et al., 2020]. Thus, the SLDS gains parameter efficiency the expense of the computational tractability and statistical simplicity of the ARHMM.
We propose a new class of unsupervised probabilistic models that we call switching autoregressive low-rank tensor (SALT) models. Our novel insight is that when you marginalize over the latent states of linear dynamical system, you obtain an autoregressive model with full history dependence. However, these autoregressive dependencies are not arbitrarily complex- - they factor into low-rank tensor that can be -ppl--appo with a fiite-history model. We formalize this connection in Proposition 1. SALT models are constrained ARHMMS that leverage this insight. Rather than allowing for arbitrary autoregressive dependencies. SALT models are constrained to be low-rank. The low-rank property allows us construct a low-dimensional continuous description of the data, jointly with the discrete segmentation provided by the switching states. Thus, SALT models inherit the experimentally useful representations and parsimonious parameter complexity of an SLDS, as well as the ease of inference and estimation of ARHMMs. We demonstrate the advantages of SALT models empirically using synthetic data well real neural and behavioral time series. Finally, in addition to improving predictive performance, we show how the low-rank nature of SALT models offer new insights into complex systems, like biological neural networks.
2 Background
This section introduces the notation used throughout the paper and describes preliminaries on low- rank tensor decomposition, vector autoregressive models, switching autoregressive models, linear dynamical systems, and switching linear dynamical systems.
Notation We follow the notation of Kolda and Bader [2009]. We use lowercase letters for scalar variables (e.g, a), uppercase letters for scalar constants (c.g. A), boldface lowercase letters for vectors (c.g. a), boldface uppercase letters for matrices (e.g. A), and boldface Euler script for tensors of
Source code is available at https: http//ttti..mmm//mmmmmaaaaaaaaaa

---

order three or higher (e.g. A). We use As:, Ag:, and A to denote the horizontal, lateral, and frontal slices respectively of a three-way tensor A. Similarly, we use and j.j to denote the it row and jth column of matrix A. a b represents the vector outer product between vectors and b. The n-mode tensor-matris tensor-vecton product represented as X A (,),, We denote the vectorization of an n-way tensor S. with dimensions 1:n as vec (S). This is performed by successively flattening the last dimensions of the tensor, and results vector of size equal the product of the dimensions of the tensor. We denote the mode-n matricization of tensor 9 as S (n)) This is defined as the stack of vectors resulting from vectorizing the matrix (or tensor) defined by each slice through the n1 dimension. This results in matrix with leading dimension Dn. and second dimension equal to the product of the sizes the other dimensions. We will denote a T'llegthh time series of N N-dimensional observed data Y E IR RX Note that will use the shorthand y: R to denote the observation at time t, and Fj.t E R to denote the 1th element the 4th observation. will be clear from context which dimension is being indexed.
Tensor Decomposition For A E TR.N N the Tucker decomposition is defined as,
D2 D3 A A=
L2 j=
Bisk U. V. W.k.
(I)
RIXDI V E Na where u.s ViJ. and w are the columns of the factor matrices and W R NaN respectively, and 9ijk are the entries in the core tensor 9 E DDxxD2xD
The CANDECOMP/PARATAC (CP) decomposition is a special case of the Tucker decomposition, with D1 D2 D3 and diagonal core tensor S.
Vector autoregressive models Let Y RNxT denote a multivariate time series with y: RRN for all f. An order-T. vector autoregressive (VAR) model with Gaussian innovations is defined by,
N Σ
Y: ~~~ N
a.kkAya + b,, R
where A E R N is the autoregressive tensor, whose frontal slice A: is the dynamics matrix for lag I, b E RVV is the bias, and R R.N is positive semi-definite covariance matrix. The parameters θ (A,, b. R) be estimated via ordinary least squares [Hamilton, 2020]
We note that, to our knowledge, there no clear consensus on the best way to regularize the potentially large parameter space of vector autoregressive (hidden Markov) models; several possibilities exist, see, e.g., Melnyk and Banerjee [2016] or Ni and Sun [2005]. Many regularizers and priors are difficult widely used in practice. Beyond this, even well-regularized ARHMMS do not natively capture interpretable low-dimensional dynamics, as both SALT and SLDS models do (see Figure 3). These low-dimensional continuous representations are as experimentally useful as the discrete segmentation, and hence are a key desiderata for any method we consider.
work with, and
Switching autoregressive models One limitation of VAR models is that they assume the time series is stationary; i.e. that one set of parameters holds for all time steps. Time-varying autore- gressive models allow the autoregressive process to change at various time points. One such VAR model, referred as switching autoregressive model autoregressive hidden Markov model (ARHMM), switches the parameters over time according to discrete latent state [Ephraim et al., 1989]. Let 2, e (I...., H. denote the discrete state time t, ARHMM defines the following generative model,
L
Cat ( (
(2)) jkk
Uj + ((t)) R(*,)
Y: W
(3)
k=1
where T (h) E (Nt ( is the the h-th row of the discrete state transition matrix.
A switching VAR model is simply type of hidden Markov model, and as such it is easily fit via the expe expectation-mmxxiizaaioo (EM) algorithm within the Baum-Welch algorithm. The M-step amounts to solving weighted least squares problem.

---

Linear dynamical systems The number of parameters in a VAR model grows as L). For high-dimensional time series, this can quickly become intractable. Linear dynamical systems (LDS) [Murphy, 2012] offer an alternative means of modeling time series via continuous latent state X E R
X₁ (AAx-1 b, Q), Ye ~ NC(xx + R),
=0 where Q SxS and R E R Here, the latent states follow first-order VAR model, and the observations are conditionally independent given the latent states. As we discuss in Section 3.3, marginalizing over the continuous latent states renders y, dependent on the preceding observations, just like in high order VAR model.
Compared the VAR model, however, the LDS has only O + NS + N²) parameters if R full covariance matrix. This further reduces to O(S + NS) if R is diagonal. As a result, when S < N, the LDS has many fewer parameters than a VAR model. Thanks to the linear and Gaussian assumptions of the model, the parameters can be easily estimated via EM, using the Kalman smoother compute the expected values of the latent states.
Switching linear dynamical systems A switching LDS combines the advantages of the low- dimensional continuous latent states of an LDS, with the advantages of discrete switching from an ARHMM. Let E {1..... H) be discrete latent state with Markovian dynamics (3), and let it determine some all of the parameters of the LDS (e.g. A would become ( in (4)). We note that SLDSs often use a Single-subspace, where C, d and R. are shared across states, reducing parameter complexity and simplifying the optimization.
Unfortunately, parameter estimation is considerably harder in SLDS models. The posterior distri- bution over all latent states, p(z X1:T M1:T, )) where O denotes the parameters, is intractable [Lerner, 2003]. Instead, these models are fit via approximate inference methods like MCMC [Fox, 2009, Linderman et al., 2017]. variational EM [Ghahramani and Hinton, 2000, Zoltowski et al., 2020], particle EM [Murphy and Russell, 2001, Doucet et al., 2001]. or other approximations [Barber, 2006]. Selecting the appropriate fitting and inference methodologies is itself non-trivial hyperparameter. Furthermore, each method also brings additional estimation hyperparameters that need to be tuned prior to even fitting the generative model. We look to define a model that enjoys the benefits of SI.SSs, but avoids the inference and estimation difficulties.
3 SALT: Switching Autoregressive Low-rank Tensor Models
Here we formally introduce SALT models. We begin by defining the generative model (also illustrated in Figure 1), and describing how inference and model fitting are performed. We conclude by drawing connections between SALT and SLDS models.
3.1 Generative Model
SALT factorizes each autoregressive tensor A (h) for h E (1,..., H) } of an ARHMM as product of low-rank factors. Given the current discrete state 24, each observation y: R.N is modeled as being normally distributed conditioned on L previous observations 1:-1at
Cat ()))
(5)
1
lid y: ~ N
a
(3e) SALT SAALT:KkUjA.- bb(z), I(=)) ((+))
(6)
k=
D2
A
(x))
ΣΣΣ
Tisk u v((e) ww(().
(2)) (2))
SALT
(x)) where (2c) (ec) and w are the columns of the factor matrices U(a) E NxDi, (2t) v(x) E NxDx and ((#) E RLXDs respectively, and Pizk are the entries in the core ten- sor g(*x) E Pix are the bias and covariance for state 2,. Without further restriction this decomposition is Tucker decomposition [Kolda and Bader, 2009]. If D, D2 D3 and 9z, is diagonal, it corresponds to
Vij
The vector (2s) E R and positive definite matrix (+) E R N

---

Table Comparison of number of parameters for the methods we consider. We exclude covariance
matrix parameters, as the parameterization of the covariance matrix is independent of method. Throughout our experiments, we find S D.
Model
Parameter Complexity
(Example from Section 5.4)
SLDS
O(NS + H O(H(ND + LD))
2.8K
CP-SALT
8.1K
TucKer-SALT
(N + LD DD)) O(HN2L)
17.4K
Order-L ARHMM
145.2K
CP decomposition [Kolda and Bader, 2009]. We refer ARHMM models with these factorizations as Tucker-S ALT and CP-SALT respectively. Note that herein we will only consider models where D1 D2 = D3 D, where we refer D the "rank" of the SALT model (for both Tucker-SALT and CP-SALT). In practice, we find that models constrained in this way perform well, and so this constraint imposed simply reduce the search space of models and could easily be relaxed.
Table 1 shows the number of parameters for order- ARHMMs, SLDSs, and SALT. Focusing the lag dependence, the number ARHMM parameters grows as O(HN2L), whereas SALT grows as only (L) with D N. SALT can also make simplifying single-subspace constraint, where certain emission parameters are shared across discrete states.
Low-dimensional Representation Note that SALT implicitly deñines low-dimensional continu- ous representation, analogous the continuous latent variable in SLDS,
Dg
(=+)
(2t)
ow (2t) W
p
Bijk
(8)
j=
N
L
X = Σ
Pukkicck
(9)
The low-dimensional E R vectors can be visualized, similar to the latent states in SLDS models, to further interrogate the learned dynamics, as we show Figure 3. Note the vector x E PP when multiplied by the output factors U is the mean of the next observation.
3.2 Model Fitting and Inference
Since SALT models are ARHMMs, we can apply the expectation-maxxiizzni (EM) algorithm to fit model parameters and perform state space inference. We direct the reader to Murphy [2012] for detailed exposition of EM and include only the key points here.
The E-step solves for the distribution over latent variables given observed data and model parameters. For SALT, this is the distribution over 2,, denoted (h(h) E[z, h YI:7, O). This can be computed exactly with the forward-backwar algorithm, which is fast and stable. The marginal likelihood can
be evaluated exactly by taking the product across t of expectations of (6) under w((h)
The M-step then updates the parameters of the model given the distribution over latent states. For H (1)) Jh=1- We use
SALT, the emission parameters are 0 (A(A), (h)) (h)), ( (a), E(h), (1) closed-form coordinate-wise updates to maximize the expected log likelihood evaluated the E-step. Each factor update amounts to solving weighted least squares problem. We include just one update
step here for brevity, and provide all updates in full in Appendix A. Assuming here that (h)) = 0 for simplicity, the update rule for the lag factors is follows:
(M )
(h)+
(N) (N)T Wi
Σ
W ( (111)-1y
(b) (h)
(h)T
(10)
where {((h)
U (V - & ID.) and (h)+ vee (h)) Crucially, these coordi-
(1)
nate wise updates are exact, and so we recover the fast and monotonic convergence of EM.
5

---

3.3
Connections Between SALT and Linear Dynamical Systems
SALT is not only an intuitive regularization for ARHMMs, is grounded a mathematical corre- spondence between autoregressive models and linear dynamical systems.
Proposition (Low-Rand Tensor Autoregressions Approximate Stable Linear Dynamical Systems). Consider a stable linear time-invarianl Gaussian dynamical system. We define the steady-state Kalman gain matrix as K si K₁, and T A(I KC). The matrix E RSS has eigenvalues A1...., xg. Let Amax maX, [aal: for stable LDS, Xmax < 1 [Davis and Vinter, 1985]. Let n denote the number real eigenvalues and m the number of complex conjugate pairs. Let S (LLDS) Ely+ y1:t-1] denote the predictive mean under steady-state LDD, and y (SALT) the predictive mean under a SALT model. An order- I. Tucker-SAI.T model with rank n + 2m S, or CP-SALT model with rank n. + 3m, can approximate the predictive mean of the steady-state LDS with error ( Yi -/(AALT) 11 (Naaa)
Proof. We give sketch of the proof here and a full proof in Appendix B. The analytic form of E fy V1+-1] is a linear function of yt-r for 1.....,0, oc. For this sketch, consider the special case where b =d d 0. Then the coefficients of the linear function are CT K. As all eigenvalues of r have magnitude less than one, the coefficients decay exponentially in l. We can therefore upper bound the approximation error introduced by truncating the linear function to L terms to ((Arax)) To complete the proof, we show that the truncated linear function can be represented exactly by tensor regression with at most a specific rank. Thus, only truncated terms contribute to the error. D
This proposition shows that the steady predictive distribution of stable LDS can be approx- imated by a low-rank tensor autoregression, with a rank determined by the eigenspectrum of the LDS. We validate this proposition experimentally in Section 5.1. Note as well that the predictive distribution will converge to a fixed covariance, and hence can also be exactly represented by the covariance matrices (A) estimated in SALT models.
Connections with Switching Linear Dynamical Systems With this foundation, it is natural to hypothesize that switching low-rank tensor autoregression like SALT could approximate a switching LDS. There are two ways this intuition could fail: first, if the dynamics SLDS are unstable, then Proposition I would not hold; second, after a discrete state transition in SLDS, it may take some time before the dynamics reach stationarity. We empirically test how well SALT approximates an SLDS in Section 5 and find that, across variety of datasets, SALT obtains commensurate performance with considerably simpler inference and estimation algorithms.
a discrete state of an
4 Related Work
Low-rank tensor decompositions of time-invariant autoregressive models Similar to this work, Wang et al. [2021] also modeled the transition matrices as third-order tensor A E IR xNxL where the A. is the 1-th dynamics matrix. They then constrained the tensor to be low-rank via a Tucker decomposition, as defined in (1). However, unlike SALT, their model was time-invariant,did not have an ARHMM structure, or, make connections to the LDS and SLDS, as in Proposition 1.
Low-rank tensor decompositions of time-varying autoregressive models approaches have also been used to model viryina AR processes [Harris al., 2021, Zhang et al., 2021]. Harris et al. [2021] introduced TVART, which first splits the data into T contiguous fixed-length segments, each with its own AR-1 process. TVART can be thought of as defining a T x N x N ARHMM dynamics tensor and progressing through discrete states at fixed time points. This tensor is parameterized using the CP decomposition and optimized using an alternating least squares algorithm, with additional penalties such that the dynamics of adjacent windows are similar. By contrast, SALT automatically segments, rather than windows, the time-series into learned and re-usable discrete states.
Low-rank tensor-based
Zhang et al. [2021] constructed Bayesian model of higher-order AR matrices that can vary over time. First, H VAR dynamics tensors are specified, parameterized as third-order tensors with a rank- I CP decomposition. The dynamics at a given time are then defined as a weighted sum of the tensors, where the weights have a prior density specified by an Ising model. Finally, inference over the weights performed using MCMC. This method can be interpreted factorial ARHMM, hence offering substantial modeling flexibility, but sacrificing computational tractability when H is large.

---

Mean squared errox C
Mran squares ernor
log, lillliileoddd
-20.0
7e-2
Test Ing. reel-illlhodd
28.0
1m-6
-26.5
27.00
$
2e44
Grond tarh LUS
-280
CP-SALI cask t
-27.5
Ground LDS
CP-SALI
20-6
-9.0
ARRMM
TALT
10 11 .28.0
Tocher SALT
LDS
10 11
12
1e3
2:3
Sed
lel
Sel
-300
263
SeB
3ed
SALT rank, D
SALT rank, D
Timesttope of train data.
Timesteps of train data
Figure 2: SALT approximates LDS: Data simulated from an LDS for which = 1 and 3 (see Proposition 1). (A-B): Average mean squared error of the autoregressive tensor corresponding to the LDS simulation and the log-likelihood of test data, as function of SALT rank. According to Proposition 1, to model the LDS Tucker-$AL17 and CP-SALT require 7 and 10 ranks respectively (indicated by vertical dashed lines). Note the parameter error increases above the predicted threshold as a result of overfitting, (C-D): Mean squared error of the learned autoregressive tensor and log- likelihood of test data function of training data.
Low-rank tensor decompositions of neural networks Low-rank tensor decomposition methods have also been used make neural networks more parameter efficient. Novikov et al. [2015] used the tensor-train decomposition [Oseledets, 2011 1] on the dense weight matrices of the fully- cally-connected layers to reduce the number of parameters. Yu et [2017] and Qiu al. [2021] applied the tensor- train decomposition to the weight tensors for polynomial interactions between the hidden states of recurrent neural networks (RNNs) to efficiently capture high-order temporal dependencies. Unlike switching models with linear dynamics, recurrent neural networks have dynamics that are hard to interpret, their state estimates are not probabilistic, and they do not provide experimentally useful data segmentations.
Linear dynamical systems and low-rank linear recurrent neural networks Valente et al. [2022] recently examined the relationship between LDSs and low-rank linear RNNs. They provide the conditions under which low-rank linear RNNs can exactly model the first-order autoregressive distributions of LDSs, and derive the transformation convert between model classes under those conditions. This result has close parallels to Proposition 1. Under the conditions identified by Valente al. [2022], the approximation in Proposition becomes exact with just one lag term. However, when those conditions are not satisfied, we show that one still recovers an LDS approximation with a bounded error that decays exponentially in the number of lag terms.
Results
We now empirically validate SALT by first validating the theoretical claims made in Section 3, and then apply SALT to two synthetic examples to compare SALT to existing methods. We conclude by using SALT to analyze real mouse behavioral recordings and C. elegans neural recordings.
5.1 SALT Faithfully Approximates LDS
To test the theoretical result that SALT can closely approximate a linear dynamical system, we fit SALT models to data sampled from an LDS. The LDS has S = dimensional latent states with random rotational dynamics, where T has
1 real eigenvalue and 3 pairs of complex eigenvalues, and N = 20 observations with a random emission matrix.
For Figure 2, we trained CP-SALT and Tucker- with I. = 50 lags and varying ranks. We first analyzed how well SALT reconstructed the parameters of the autoregressive dynamies tensor. As predicted by Proposition 1. Figure 2A shows that the mean squared errors between the SALT tensor and the autoregressive tensor corresponding ranks of CP-SALT and Tucker-S S SALT are n + 3m = 10 and n + 2m = 7 respectively. We then computed log likelihoods on 5,000 timesteps of held-out test data (Figure 2B). Interestingly, the predictive performance of both CP-SALT and Sucker-SALT reach the likelihood of the ground truth LDS model with rank n + 2m = 7, suggesting that sometimes smaller tensors than suggested by Proposition 1 may still be able to provide good approximations to the data. We also show in Figures 2C and 2D that, as predicted, SALT models require much less data to fit than ARHMMS. We show extended empirical results and discussion on Proposition I in Appendix D.I.
the simulated LDS are the lowest when the
7

---

A
Ground truth observation
Ground truth traeecoory
B
Ground truth observation
Ground truth Lorenz attractor
-
SLDS fittered observstions
SLDS fittered trajectory
SLDS Eltered observations
SLDS fittered trajectory
2
SALT filtered obeervations
SALT filtered trajectory
SALT filtered observations
SALT flttered trajectory
100
200
D
400
Timestops
Timestope
Figure 3: SALT reconstructs simulated SLDS data and Lorenz attractor: (Top row) Observation generated from low-dimensional trajectory. (A) shows ten observations generated from recurrent "NASCAR" SLDS trajectory Linderman al. [2017]. J. (B) 20- 20-dimensional observations generated from Lorenz attractor (5 observed dimensions are shown). (Middle and bottom rows): filtered observations and inferred low-dimense -e-dimmeniooal trajectories from SLDS and SALT models. Colors indicate discrete state for ground truth (if available) and fitted models. SLDS and SALT find comparable filtered trajectories and observations. It is important to note that the latent spaces in both SLDS and SLT are only identifiable up to linear transformation. We therefore align the latent trajectories for ease of comparison. This latent structure is reliably found by both SALT and SLDS.
5.2 Synthetic Switching LDS Examples
Proposition quantifies the convergence properties of low-rank tensor regressions when approximat- ing stable LDSs. Next tested how well SALT approximate the more expressive switching LDSs. We first applied SALT to data generated from recurrent SLDS [Linderman et al., 2017]. where the two-dimensional ground truth latent trajectory resembles NASCAR track (Figure 3A). SALT accurately reconstructed the ground truth filtered trajectories and discrete state segmentation, and yielded very similar results to an SL.DS model. We also tested the ability of SALTT to model nonlinear dynamics- specifically, a Lorenz attractor which SLDSs are capable of modeling. Again, SALT accurately reconstructed ground truth latents and observations, and closely matched SLDS segmentations. These results suggest that SALT models provide good alternative to SLDS models. Finally, in Appendix D.3, we used SLDS-geenrann data to compare SALT and TVART [Harris et al., 2021], another tensor-based method for modeling autoregressive processes, and find that SALT more accurately reconstructed autoregressive dynamics tensors than TVART.
5.3 Modeling Mouse Behavior
Next we considered a video segmentation problem commonly faced in the field of computational neuroethology [Datta et al., 2019]. Wiltschko et al. [2015] collected videos of mice freely behaving circular open field. They projected the video data onto the top 10 principal components (Figure 4A) and used an ARHMM to segment the PCA time series into distinct behavioral states. Here, compared ARHMMS and CP-SALT with data from three mice. We used the first 35,949 timesteps of each recording. which were collected at 30Hz resolution. We used H 50 discrete states and fitted ARHMMs and CP-SALT models with varying lags and ranks.
The likelihood held-out validation set shows that the ARHMM overfitted quickly as the number of lags increased, while CP-SALT was more robust to overfitting (Figure 4B). We compared log- likelihoods of the best model (evaluated on the validation set) on separate held-out test set and found that CP-SALT consistently outperformed ARHMM across mice (Figure 4C).
We also investigated the quality of SALT segmentations of the behavioral data (Appendix E.3). We found that PCA trajectories upon transition into discrete SALT state were highly stereotyped, suggesting that SALT segments the data into consistent behavioral states. Furthermore, CP-SALT used fewer discrete states than the ARHMM, suggesting that the ARHMM may have oversegmented and that CP-SALT offers parsimonious description of the data.

---

A
Example Frame
B
C
Validation log. likelhhood
1.3
-14
Test log-likel Ing-likelhoool
-1.5
-0.4
-1.6-
ARHMM
Top 10 PCs
-1.7
CF-SAT 10nk=1
-0.5
Mouse
CF-SALT cank-11
1.8
CF-SALT rank=12
-0.6
Mouse
-1.
CF SALT cank=13
mank-1t
-2.0
30
51
52
331
31
35
36
37
38
39
00
ARHMM CP
time
[s]
Lags
Ing=5
ranke=18
lag=5
Figure 4: CP-SALT consistently outperforms ARHMM on mouse behavior videos and segments data into distinct behavioral syllables: (A) An example frame from the MoSeq dataset. The models were trained on the top 10 principal components of the video frames from three mice. (B) CP-SALT and ARHMM trained with different ranks and lags. Mean and standard deviation across five seeds evaluated on validation set are shown. CP-SALT parameterization prevents overfitting for larger lags, (C) Test log- licelioo,,,odd, averaged across 5 model fits, computed from the best ARHMM and CP-SALT hyperparameters in (B). CP-SALT outperforms ARHMM across all three mice.
5.4 Modeling C. elegans Neural Data
Finally, we analyzed neural recordings of an immobilized C. elegans worm from Kato et al. [2015]. SLDS have previously been used to capture the time-varying low-dimensional dynamics of the neural activity [Linderman et al., 2019, Glaser et al., 2020]. We compared SLDS, ARHMM, and CP-SALT with 18 minutes of neural traces (recorded at 3Hz; ~3200 timesteps) from one worm, in which 48 neurons were confidently identified. The dataset also contains 7 manually identified state labels based on the neural activity.
We used H = 7 discrete states and fitted SLDSs, ARHMMs, and CP-SALT with varying lags and ranks (or continuous latent dimensions for SLDSs). Following Linderman et al. [2019], we searched for sets of hyperparameters that achieve 90% explained variance on a held-out test dataset (see Appendix for more details). For ARHMMs and CP-SALT, we chose a larger lag (L equivalent to seconds) to examine the long-timescale correlations among the neurons.
We find that SALT can perform as well as SLDSS and ARHMMS in terms of held-out explained variance ratio (a metric used by previous work [Linderman al., 20191). As expected, we find that CP-SALT can achieve these results with far fewer parameters than ARHMMs, and with a parameter count closer to SLDS than ARHMM (as more continuous latent states were required in an SLDS to achieve 900 explained variance; see Appendix F). Figure 5A shows that SALT, SLDS and ARHMM produce similar segmentations to the given labels, evidenced by the confusion matrix having high entries on the leading diagonal (Figure 5B and Appendix F).
Figure 5C shows the one- mie-dimeniionl autorcgressive filters learned by CP-SALT, defined as (h) (b) (h) (h)
ED: ΣD Ebe Difk ugi Ugf w.k for neurons p and q. We see that neurons believed to be involved particular behavioral states have high weights in the filter (c.g.. SMDV during the "Ventral Turn" state and SMDD during the "Dorsal Turn" state [Linderman et al., 2019, Kato et al., 2015, Gray al., 2005, Kaplan et al., 2020, Yeon et al., 2018]). This highlights how switching autoregressive models can reveal dependent functional interactions between neurons (or observed states more generally). In Appendix F, we show the autoregressive filters learned by an ARHMM, an SLDS, and generalized linear model (GLM), method commonly used to model inter-neuronal interac- tions [Pillow et al., 2008]. Interestingly, the GLM does not find many strong functional interactions between neurons, likely because it is averaging over many unique discrete states. In addition to its advantages in parameter efficiency and estimation, SALT thus provides a novel method for finding changing functional interactions across neurons at multiple timescales.
Discussion
We introduce switching autoregressive low-rank tensor (SALT) models: novel model class that parameterizes the autoregressive tensors of an ARHMM with low-rank factorization. This constraint

---

A
B
Labels Traces
Given Labels
Rev. S.
Slow
600
Fwd
-400
of Timestops
VT
DT
-20
Rev.
Rev.
6517
210
400
600
500
1000
1200
1400
SALT Labels
Timesteps
Reverse Sustained
Slow
Forward
Ventral Turn
Dorsal Turn
Reverse I
Reverse 2
C
SALT State
(Ventral Turn)
SALT State (Dorsal Turn)
SALT Tensor Weights
1001
3MDV
94000
5NDOK
ABn
WER
RMED
SMDDW
X000
ABn
during:
Active
Ventral
SMIIVI
Turn
SMITIVR
AVTT
AVTR
SMIDX.
Dorsal
SNOOR
SNDOK
Turn
CP-SALT Lag (Rank-11, 11, Lag-9, equivalent seconds; from left right)
Figure 5: CP-SALT provides good segmentations C. elegans neural data, and inferred low- rank tensors give insights into temporal dependencies among neurons in each discrete state: (A) Example data with manually generated labels (Given), as well as segmentations generated by SALT, SLDS and ARHMM models. Learned states are colored based on the permutation of states that best matches given labels. All methods produce comparable segmentations, with high agreement with the given labels. (B) Confusion matrix of -AL.T-eeeeaaed labels. (C) One-dimensional autoregressive filters learned in two states by SALT (identified as ventral and dorsal turns). Colors indicate the neurons known mediate
area under curve (red is positive; blue is negative). The first four ventral turns, while the last two rows mediate dorsal turns [Kato et al., 2015, Gray et al., 2005, Yeon al., 2018]. These known behavior-tuned neurons generally have larger magnitude autoregressive filters. Interestingly, AVFL and AVER also have large filters for dorsal turns. These neurons do not have well-known function. However, they are associated with motor neurons, and so may simultaneously activate due to factors that co-occur with turning. This highlights how SALT may be used for proposing novel relationships systems.
allows SALT model time-series data with fewer parameters than ARHMMs and with simpler estimation procedures than SLDSs. We also make theoretical connections between low-rank tensor regressions and LDSs. We then demonstrate, with both synthetic and real datasets, that SALT offers both efficiency and interpretability. striking an advantageous balance between the ARHMM and SLDS. Moreover, SALT offers an enhanced ability to investigate the interactions across observations, such as neurons, across different timescales in data-efficient manner.
However, SALT is not without limitations. Foremost, SALT cannot readily handle missing obser- vations, or share information between multiple time series with variable observation dimensions. "Hierarchical SALT" is an interesting extension, where information is shared across time series, but the factors of individual time series are allowed to vary. Furthermore, SALT could be extended handle non-Gaussian data. For example, neural spike trains are often modeled with Poisson likelihoods instead of SALT's Gaussian noise model. In this case, the E-step would still be exact, but the M-step would no longer have closed-form coordinate updates. Despite these limitations, SALT offers simple, effective and complementary means of modeling and inference methodology for complex, time-varying dynamical systems.
Ethical Concerns
We note that there no new ethical concerns result of SALT.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by grants from the Simons Collaboration on the Global Brain (SCGB 697092), the NIH (U19NS113201, ROINS113119 ROINSI30787 and K99NS119787) the Sloan Foundation, and the Stanford Center for Human and Artificial Intelligence, We thank Liam Paninski and the anonymous reviewers for their constructive feedback the paper. We also thank the members of the Linderman Lab for their support and feedback throughout the project.
References
Sandeep Robert Datta, David J Anderson, Kristin Branson, Pietro Perona, and Andrew Leifer. Computational neuroethology: A call to action. Neuron, 104(1):11 2019.
Alexander B Wiltschko, Matthew J Johnson, Giuliano lurilli, Ralph E Peterson, Jesse M Katon, Stan Pashkovski, Victoria E Abraira, Ryan P Adams, and Sandeep Robert Datta. Mapping 2015.
sub-second structure in mouse behavior. Neuron,
Julia Costacurta, Lea Duncker, Blue Sheffer, Winthrop Gillis, Caleb Weinreb, Jeffrey Markowitz, Sandeep R Datta, Alex Williams, and Scott Linderman. Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs. Advances in Neural Information Processing Systems, 35:23838-238551 2022.
Aram Giahi Saravani, Kiefer J Forseth, Nitin Tandon, and Xaq Pitkow. Dynamic brain interactions during picture naming. eNeuro, 6(4), 2019.
Stefano Recanatesi, Ulises Pbrlira-Obliinovi, Masayoshi Murakami, Zachary Mainen, and Luca Mazzucato. Metastable attractors explain the variable timing of stable behavioral action sequences. 2022.
Neuron,
Yariv Ephraim, David Malah, and B-H Juang. On the application of hidden Markov models for enhancing noisy speech. IEEE Transactions Acoustics, Speech, and Signal Processing, 37((2): 1846-1856, 1989.
Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural Computation, 12(4):83 1-864, 2000.
Biljana Petreska, Byron M Yu, John Cunningham, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Maneesh Sahani. Dynamical segmentation of single trials from population neural data. Advances Neural Information Processing Systems, 24, 2011.
Scott Linderman, Annika Nichols, David Blei, Manuel Zimmer, and Liam Paninski. Hicrarchical recurrent state space models reveal discrete and continuous dynamics of neural activity in C. elegans. bioRxiv, page 621540, 2019.
Joshua Glaser, Matthew Whiteway, John P Cunningham, Liam Paninski, and Scott Linderman. Re- current switching dynamical systems models for multiple interacting neural populations. Advances in Neural Information Processing Systems, 33:14867-14878, 2020.
Aditya Nair, Tomomi Karigo, Bin Yang, Surya Ganguli, Mark J Schnitzer, Scott W Linderman, David Anderson, and Ann Kennedy. An approximate line attractor in the hypothalamus encodes an aggressive state. Cell, 178-111 2023.
David Barber. Expectation correction for smoothed inference in switching linear dynamical systems. Journal of Machine Learning Research, 7(11), 2006.
Emily Beth Fox. Bayesian nonparametric learning of complex dynamical phenomena. PhD thesis, Massachusetts Institute of Technology, 2009.
Kevin Murphy and Stuart Russell. Rao Blo-BBackwellised particle filtering for dynamic Bayesian networks. In Sequential Monte Carlo methods in practice, pages 499-515. Springer, 2001.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference recurrent switching linear dynamical systems. Artificial Intelligence and Statistics, pages 914-922. PMLR, 2017.

---

David Zoltowski, Jonathan Pillow, and Scott Linderman. A general recurrent state space framework for modeling neural dynamics during decision- International Conference on Machine Learning, pages 1660-1 1691. PMLR, 2020.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
James Douglas Hamilton. Time series analysis. Princeton University Press, 2020.
Igor Melnyk and Arindam Banerjee. Estimating structured vector autoregressive models. Maria Flo- rina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 830-839, New York, New York, USA, 20-22 Jun 2016. PMLR.
Shawn Ni and Dongchu Sun. Bayesian estimates for vector autoregressive models. Journal of Business & Economic Statistics, 23(1):105-117, 2005.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Uri Nahum Lerner. Hybrid Bayesian networks for reasoning about complex systems. PhD thesis, Stanford University, 2003.
Arnaud Doucet, Neil Gordon, and Vikram Krishnamurthy, Particle filters for state estimation of jump Markov linear systems. IEEE Transactions on signal processing. 49(3):613-624, 2001.
Mark H A Davis and Richard B Vinter. Stochastic modelling and control. Chapman and Hall London New York, 1985. ISBN 0412162008.
Di Wang. Yao Zheng. Heng Lian, and Guodong Li. High-dimensional vector autoregressive time series modeling via tensor decomposition. Journal of the American Statistical Association, pages 1-19, 2021.
Kameron Decker Harris, Aleksandr Aravkin, Rajesh Rao, and Bingni Wen Brunton. Time-varying autoregression with low-rank tensors. SIAM Journal on Applied Dynamical Systems, 20(4): 2335-2358, 2021.
Wei Zhang, Ivor Cribben, Sonia Petrone, and Michele Guindani. Bayesian time-varying tensor vector autoregressive models for dynamic effective connectivity. arXiv preprint arXiv:2106.14083, 2021.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in Neural Information Processing Systems, 28, 2015.
Ivan Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5): 2011.
Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using higher order tensor RNNs. arXiv preprint 171 2017.
Hejia Qiu, Chao Li, Ying Weng, Zhun Sun, Xingyu He, and Qibin Zhao. On the memory mechanism of tensor-power recurrent models. In International Conference Artificial Intelligence and Statistics, pages 3682-3690. PMLR, 2021.
Adrian Valente, Srdjan Ostojic, and Jonathan W Pillow. Probing the relationship between latent linear dynamical systems and low-rank recurrent neural network models. Neural Computation, 34(9): 1871-1892, 2022.
Saul Kato, Harris S Kaplan, Tina Schrödel, Susanne Skora, Theodore H Lindsay, Eviatar Yemini, Shawn Lockery, and Manuel Zimmer. Global brain dynamies embed the motor command sequence of Caenorhabditis elegans. Cell, (63(3):656-666) 2015.
Jesse M Gray, Joseph J Hill, and Cornelia Bargmann. A circuit for navigation in Caenorhabditis elegans. Proceedings of the National Academy of Sciences, 102(9):184 (199))318433111 8491,, 2005.
Harris S Kaplan, Oriana Salazar Thula, Niklas Khoss, and Manuel Zimmer. Nested neuronal dynamics orchestrate behavioral hierarchy across timescales. Neuron, 105(3):562-576, 2020. 2020.
12

---

Jihye Yeon, Jinmahn Kim, Do- Young Kim, Hyunmin Kim, Jungha Kim, Eun Jo Du, KyeongJin Kang. Hyun-Ho Lim, Daewon Moon, and Kyuhyung Kim. A sensory-motor neuron type mediates proprioceptive coordination of steering in C. elegans via two TRPC channels. PLoS biology, 16(6): e2004929, 2018
Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan Litke, EJ Chichilnisky, and Eero P Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454(7207);995-999 9999, 2008.
Tohru Katayama. Subspace Methods for System Identification, volume 1. Springer, 2005.
Martin Chaltie, John E Sulston, John G White, Eileen Southgate, Nicol Thomson, and Sydney Bren- ner. The neural circuit for touch sensitivity in Caenorhabditis elegans. Journal of Neuroscience, 964,, 1985.
Beverly J Piggott, Jie Liu, Zhaoyang Feng, Seth A Wescott, and XZ. Shawn Xu. The neural circuits and synaptic mechanisms underlying motor initiation in C. elegans. Cell, 147(4):922 233229933333 2011.
Amirreza Farnoosh, Bahar Azari, and Sarah Ostadabbas. Deep switching auto-regressive factorization: Application time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 7394-7403, 2021.
Andreas S Weigend and Neil A Gershenfeld. Time series prediction: Forecasting the future and understanding the past. Santa Fe Institute Studies in the Sciences of Complexity, 1994.
13

---

Supplementary Materials for: Switching Autoregressive Low-rank Tensor Models
Table of Contents
Appendix A: SALT Optimization via Tensor Regression.
Appendix B: SALT Approximates (Switching) Linear Dynamical System.
Appendix C: Single-subspace SALT.
Appendix D: Synthetic Data Experiments.
Appendix E: Modeling Mouse Behavior.
Appendix F: Modeling C. elegans Neural Data.
Appendix G: Deep Switching Autoregressive Model.
14

---

A SALT Optimization via Tensor Regression
Let Yr R be the f-th outputs and X, N be the t-th inputs. The regression weights are a tensor A E RRNIXXN, XN3, which model via Tucker decomposition,
D2 Ds A= ΣΣΣ j=
Bisk U. V.g o.k
(11)
where u, Vj. and We columns of the factor matrices U RNII VE R and W R Ds,Dss respectively, and Pijk entries in the core tensor 9 E DDDDDxDDs
We define j, to be tensor- iensor-matrix product over the jth and 1thh slices of the tensor. For ex- D2x ample, given whyee-way tensor A Pxx Ds and matrix X E DaxDs A X2 X
Ds Σラ Σ jighikk This operation is depicted in Figure 6.
Consider the linear model, y: NNA x2,3 X,, Q) where A X2,3 X, is defined using the Tucker decomposition of A as,
A x2 X, = (x)))e((X.)
(12)
= =US[n)(V)) & W'ee((X)
(13)
(14)
where A(1) E NNNNNNN and 9(1) E DDiDD2Ds are mode-1 matricizations the corresponding tensors. Note that these equations assume that matricization and vectorization are performed in row-major order, as in Python but opposite to what is typically used in Wikipedia articles.
Equation (14) can be written in multiple ways, and these equivalent forms will be useful for deriving the updates below. We have,
A x2,3 Xx = (ID, & W X. ]vec(VT) = US (V X, @ ID,) vec())) [U vec( V X.] vec(g).
(15)
(16)
(17)
We minimize the negative log likelihood by coordinate descent. Optimizing the output factors
Let
x veo veo VV X, W)
(18)
for fixed V, W, and 9. The NLL as function of U is,
((U) 2 Σ yy. UX,) UX:) (y:- UX.).
(19)
This is a standard least squares problem with solution
Exit] ) (IxEE) X
U
(20)
Optimizing the core tensors Let X, vec V X, W) E R DD denote the coefficient on vec(9) in eq. (17). The NLL as a function of g = vec(9) is,
((g) 1 -(x:-Xi8) X.g) QQ - -x.)).
(21)
The minimizer of this quadratic form is.
Q X )² ( XI xTTQ''y Q-yy)
g" -'= Σ x
(22)
15

---

Mode-1 (column) fibres a.jk
D3
Time
Yt A X2, 11-1--1
a. ja-k
Lag
Observation dimension,
11e-1:0-1
N D2
Σ
+
x
X
a Y/s-k
y.
Figure 6: Depiction of the 22, tensor operator we use [Kolda and Bader, 2009] This can be thought of as a generalization of -veroo products to tensor-matrix products.
Optimizing the input factors
Let
X, = (1 @ W ('x,))
(23)
for fixed U, W, and 9. The NLL as function of v vec(V is.
((v) = E(si QQ X(v).
(24)
The minimizer of this quadratic form is,
v Σ xx -
[xx 12 -
(25)
Optimizing the lag factors
Let
X, = (S ( X, & ID,)
(26)
for fixed U, V. and 9. The NLL function of vec(W) is,
1 1 -
[(w) The minimizer of this quadratic form is.
(27)
w' = Σ [XxQ-XX " y1
(28)
Multiple discrete states If we have discrete states 2 E {1, H and each state has its own parameters (g(h), U(N) v(h), wit), (()),, then letting w1 E/ 2 hi] denote the weights from the E-step, the summations in coordinate updates are weighted by w/ For example, the coordinate update for the core tensors becomes.
(h))
Σ wi
Q (h) - X
Σ
wi X (N) Q Q())-
(29)
16

---

B SALT approximates a (Switching) Linear Dynamical System
We now re-state and provide a full proof for Proposition
Proposition 1 (Low-Rank Tensor Autoregressions Approximate Stable Linear Dynamical Systems). Consider stable linear time-invariant Gaussian dynamical system. We define the steady-state Kalman gain matrix as K = lim Kg, and T A(I KC). The matrix I E S has eigenvalues .p..... Xg. Let Amax max, [Ag]: for stable LDS, Amax < [Davis and Vinter, 1985]. Let denote the number of real eigenvalues and m the number of complex conjugate pairs. Let y: (LDS) Eyly Y1a-1] denote the predictive mean under steady-state LDS, and y; (SALT) the predictive mean under a SALT model. An order-L Tucker-SALT model with rank n + 2m = S, or CP-SALI model with rank n + -m,, can approximate the predictive mean of the steady-state LDS with error ||8 (LD))
y: (SALT) 11 = -
Proof. A stationary linear dynamical system (LDS) is defined as follows:
X Ax b 6t
(30)
Yr = Cx2 + d d +
(31)
where Yt E R is the t-th observation, x, E R is the f-th hidden state, c₁ lid. N(O, Q), 6, Lid
N/O, R), and 0 (A, b, Q. C, d, R) are the parameters of the LDS.
Following the notation of Murphy [2012], the one-step-ahess posterior predictive distribution for the observations of the LDS defined above can be expressed as:
p(y:ly1a-1) = -N + + CE +R)
(32)
where
Mijo-1 = A +b
(33)
14 = Mij-1 + K,Ii
(34)
E = AΣ,1A A +Q
(35)
Σ, = (I K,C)Euj--1
(36)
p(x1) =N(xx 1110 Σ1jo)
(37)
K, = =(Eji-1 + C²RCC
(38)
y: CCu(e11 -d.
(39)
We can then expand the mean Cfp-1 + as follows:
Cp + d = C
-NAKK-N--IIO C
ri A
(40)
l=
1=1
where
H-1 I I A(I K.-C) for lE (2.
(41)
I
(42)
Theorem 3.3.3 of Davis and Vinter [1985] (reproduced with our notation below) states that for a stabilizable and detectable system, the lim - E, where Σ is the unique solution of the discrete algebraic Riccati equation
Σ AΣAT AECT (CECT + -CEA +
(43)
As we are considering stable autonomous LDSs here, the system is stabilizable and detectable, as all unobservable states themselves stable [Davis and Vinter, 1985, Katayama, 2005]
Theorem 3.3.3 (Reproduced from Davis and Vinter [1985], updated to our notation and context).
The theorem has two parts.
17

---

The convergence of the Kalman gain also implies that each term in the sequence T2 converges to (a) If the pair (A, C) is detectable then there exists at least one non-negative solution, Σ, to the discrete algebraic Riceati equation (43).
(b) If the pair (A, C) is stabilizable then this solution Σ is unique, and Emje-1 E aast- co, where Dijt11 the sequence generated by (33)-(39) with arbitrary initial covariance Eg. Then, the matrix r A(I KC) is stable, where K is the Kalman gain corresponding to Σ; i.e.
== CC'O R
(44)
Proof. See Davis and Vinter [1985]. Note that Davis and Vinter [1985] define the Kalman gain AK.
I A KC) (A (A(I- -C)
(45)
where, concretely, we define r = A(I - KC). We can therefore make the following substitution and approximation
Cpa--1
+d
lin 100
A'KYy + E
( (b AKd) +d
(46)
1
1 r' AKd) +d+
r A'KY - + +0
Σ F F(r))
(47)
1=1
1=L+1
C A (rbbAkk)) AKd) +d
(48)
The approximation is introduced as a result of truncating the sequence to consider just the "first" terms, and discarding the higher-order terms (indicated in blue). It is important note that each term in (46) is the sum of a geometric sequence multiplied elementwise with y..
There are two components we prove from here. First, we derive an element- wise bound on the error introduced by the truncation, and verify that under the conditions outlined that the bound decays monotonically in L.. We then show that Tucker and CP decompositions can represent the truncated summations in (48), and derive the minimum rank required for this representation be exact.
Bounding The Error Term We first rearrange the truncated terms in (46), where we define
X e AKy, + b AKd
Σ
r = C Σ
r'AKy,
+C Σ r'(b - ([b-AKd) +d,
(49)
1=++1
1=1+1
1=++1
cr'x1. X1:
(50)
1=
CEA 1-1 E- X1.
(51)
1=1-1
PA
1-1
q1.
(52)
1=1-1
where EAE-1 is the eigendecomposition r. P A CE, and qi - E-1x, We now consider the infinity-norm of the error, and apply the triangle and Cauchy-Schwartz incqualities. We can write the
18

---

bound on the as
Σ F (T')
20
(
Σ
20
where n = arg max
(53)
1=b+1
-=
8
Pustd g1.0
(54)
1=L+1
DC
Pnsl |
19,=l
(55)
Upper bounding the absolute magnitude of qts by W provides further upper bound, which then rearrange
S W
Pnsl 1
(56)
1=L+1
W
Pnsl
|
(57)
1=L+1
The first two terms are constant, and hence the upper bound is determined by the sum of the of the 4th power of the eigenvalues. We can again bound this sum by setting all eigenvalues equal the magnitude of the eigenvalue with the maximum magnitude (spectral norm), denoted Amaz:
CS W
PILS
X
(58)
I=L+1
where these second summation is not function of s, and W E |Pnel is constant. This summation a truncated geometric sequence. Invoking Theorem 3.3.3 of Davis and Vinter [1985] again, the matrix I has only stable eigenvalues, and hence Amax < 1. Therefore the sequence sum will converge to
AL
Σ 1++1
Amax
(59)
max
Amax
Rearranging again, we see that the absolute error on the th element of y: is therefore bounded according to a power of the spectral norm
X
L
S W
Pns
(60)
Amax
More specifically, for stable linear (AL max. bounded, then the bound on the error incurred reduces exponentially the length of the window Furthermore, this error bound will reduce faster for systems with lower spectral norm.
(61)
dynamical system, and where q and hence y
Diagonalizing the System We first transform I into real modal form, defined as EAE-1, where E and A are the eigenvectors and diagonal matrix of eigenvalues of T. Letting I have real eigenvalues and m pairs of complex eigenvalues (i.e., n + 2m = 5), we can express E, A, and E as:
E
aj
an bj C1 bm Cm
(22)
A1
A
01
W1
(63)
0
om Wm
Wm O
19

---

(64)
where a a, are the right eigenvectors corresponding to n real eigenvalues A.... An and b; and c2 the real and imaginary parts of the eigenvector corresponding to the complex eigenvalue
of + jw, Note that
I' (A(I- EA EA
(65)
The 1th power of A. A', where 2 can be expressed as:
X
X
A
11,1
W1
(66)
W1,1
01,
Om, Wmi
-mm, Om.
where di, = =o, wf, w1i-- Wiy = 20,,1-1451--1 for 1 2 2, 0i,1 = oi, Wi,1 Wi, T.u and
wi. = 0.
Tucker Tensor Regression Let K E 5X8XLL be three-way tensor, whose 1th frontal slice H.. 1-1 Let 8x5x8 be three-way tensor, whose entry Jijk li==j= for < k < n, for & E {n 1, + 3, 3,..., n + 2m 1}. Let W E L be a matrix, whose entry wt = -x 1-1 for
and 9itk 111---t-j-- ba
n, WTk = 2mm We can then decompose H into 9 E Sx and W E g such that H[ X3 W (Figure 7),
Thi-1 for (n+1,++3, ... n 2 and wis -wku-1 for & E (
20

---

D
D
Decompose
0m.
50mm
S
0m2
2
2m
11
1
S
H A-1
5
H
9
X3 W
Figure 7: Decomposition of H into 9 and W such that H S X3 W: Given an LDS whose A(I KC) has n real eigenvalues and pairs of complex eigenvalues, this decomposition illustrates how Tucker-SALa can approximate the LDS well with rank n + 2m.
With V AK- U= CE, m CE (b AKd) d, and X, Yt-1:t-L. we can rearrange the mean to:
- EA DA'-E-- B-
C r' AKd) +d
(67)
1=1
1=1
U
Σ H
(68)
V
U
V y
(69)
1=1
U
((S X2 +
(70)
1=1
U
gyjk V.y +
(71)
U
8.yk X.W:)) + m
(72)
8
U
X.w (.W...) +m
(73)
n+ n- nm2m n +m
U o V. o W.k
X2,3 X, m
(74)
CP Tensor Regression
By rearranging E, A and E-1 into Pt, and S respectively follows:
J = [aj b + b C₁ b + cm bm Cm
(75)
21

---

M
X/
P =
(76)
P1
mmd
amd
Bmd
df
s
(77)
c ffl
fT
where J (x((+))m) P. E R ((n+3m))((++m)) S E R(n+3m)XS 04,1 wi,1 oil and BL, -WI,1 oit diagonalize (A KC)) as JP,S Let V (SAK)",
CJ. m , AKd) + d. and X, Yt-1a-L- Let W (x+(()) be a matrix, whose element in the 1th row and 4th column is 1t-1,6k (i.e., the element in the htth row and tt column of Pí-1), and 9 [(++3m)(((+3m) ((++3m) x (3+33m) be a diagonal
3-way tensor, where Pijk -=j=k-- We then rearrange the mean to:
Cpp-1
d C
B- AKy, r' AKd) +
(78)
1=1
-c
JP -PK-
(79)
1=1
U
P PI yo-tm
(80)
1 2 3m n- 3m 3m 3m
3
Σ gizk U. o (.)(PI-1,NN0-1)+ + m
(81)
1=1
2-33m n 4m n433m
Σ
Pisk U. (V
(82)
n- n- n-3m n +m
Σ
Σ
Gifk Us o V.j o W.s
X2,3 X, + m
(83)
And so concludes the proof.
22

---

C Single-subspace SALT
Here we explicitly define the generative model of multi-subspaace and single-subspace Tucker-SALT and CP-SALT Single-subspace SALT is analogous to single-subspace SLDSs (also defined below), where certain emission parameters (e.g., C, d, and R) are shared across discrete states. This reduces the expressivity of the model, but also reduces the number of parameters in the model. Note that both variants of all models have the same structure on the transition dynamics of z;-
Multi-subspace SALT Note that the SALT model defined in (6) and (7) in the main text is multi-subspace SALT. We repeat the definition here for ease of comparison.
Do Da
Lid. ~ N
(2e)
(ze)
X2,3 - +b(a), g(())
Pigk
Vi o wit""
(84)
D1 = =2 D3 = and 9 is diagonal for CP-SALT.
Single-subspace TucKer-SALT In single-subspace methods, the output factors are shared across discrete states
Y1 idd N
U
m( +
(xx) (2d) Bikk vi owlnn (a)
X X2.3 12.300-1:0-1 + b. b.)))
(85)
1=1
where (3t) D..
Single-subspace CP-SALT Single-subspace CP-SALT requires extra tensor compared to TuL,er-SALT, as this tensor can no longer be absorbed in to the core tensor.
N Yt hid (u U' (()) p(a))
D2 D2
1) + b, (+t)
ΣΣ
Σ
Bijk
v
(2))
w (zx)
X2.3 11-11111))
j=1k=1
(86)
where U' E RNxD, p(a) E DD m (#)) E RDi, D1 = D2 D3 = D, and G is diago- nal.
Multi-subspace SLDS is much harder optimization problem, which found was often numerically unstable. We therefore do not consider multi-subspaac SLDS in these experiments, but include its definition here for completeness
Multi-subspace SLDS
X₁ N (A(x))x + b(a1), Q 0(z))
(87)
Ye > N (c d(e)) R((,))
(88)
Single-subspace SLDS
Single-subspace SLDS was used in all of our experiments, and is typically used practice [Petreska al., 2011, Linderman et al., 2017]
N (A (=x) x1-1 +b Q(x))
x, ~~W
(89)
ye ~ N + +d, R)..
(90)
23

---

D Synthetic Data Experiments
D.1 Extended Experiments for Proposition 1
In Section 5.1 we showed that Proposition 1 can accurately predict the required rank for CP- and Tucker-SALT models. We showed results for a single LDS for clarity. We now extend this analysis across multiple random LDS and SALT models. We randomly sampled LDSs with latent dimensions ranging from 4 to 10, and observation dimensions ranging from 9 to 20. For each LDS, we fit 5 randomly initialized CP-SALT and Tucker-SAL) models with L 50 lags. We varied the rank of our fit S SALT models according to the rank predicted by Proposition 1. Specifically, we computed the estimated number of ranks for given LDDS, denoted D" and then fit SALT models with {D - 2, D 1, D" D + 1, D + 2) ranks. According to Proposition 1, we would expect see the reconstruction error of the autoregressive tensor be minimized, and for prediction accuracy to saturate, at D D
To analyze these model fits, we first computed the average mean squared error of the autoregressive tensor corresponding to the LDS simulation, as function of SALT rank relative to the rank required by Proposition 1. We see, as predicted by Proposition 1, that error in the autoregressive tensor is nearly always minimized at D (Figure 8A). Tucker-SALT was always minimized D), Some CP-SALT fits have lower MSE at ranks other than predicted by Proposition 1. We believe this is due to local minima in the optimization. We next investigated the test log-like ligliielihood as a function of the relative rank (Figure 8B). Interestingly, the test likelikel shows that Tucker-S SALT strongly requires the correct number of ranks for accurate prediction, but CP-SALT can often perform well with fewer ranks than predicted (although still comparable number of ranks to Tucker-SALI As Figure 2, these analyses empirically confirm Proposition!
Rolative test log- lielliielihood
0.00
Relative MSE
-0.05
-0.10
-0.15
Tuckkr-SAII
CP-SALT
-0.20
-2
0 D
2
-2
-1
D D'
(4) Normalized MSE of autoregressive tensor.
(b) Normalized log- likelihood on held-out test set.
Figure& 8: Extended results examining Proposition 1. Results are shown for the ability of SALT to estimate ten randomly generated LDSs, using five SALT repeats for each LDS. MSEs (in panel A) and log-likelihoods (in panel B) are normalized by the mean MSE and mean log-likelihood of SALT models trained with D D*. D is the rank of the fit SALT model, and D* is the necessary rank predicted by Proposition 1.
D.2 Quantitative Performance: Synthetic Switching LDS Experiments
We include further results and analysis for the NASCAR and Lorenz attractor experiments presented in Section 5.2. We compare the marginal likelihood achieved by single-subspace SALT models of different sizes. We see that SALT outperforms ARHMMs, and can fit larger models (more lags) without overfitting (Figure 9). Note that the SLDS does not admit exact inference, and so we cannot readily compute the exact marginal likelihood for the SLDS.
D.3 TVART versus SALT in recovering the parameters of SLDSs
We compared SALT to TVART Harris et al. [2021], another tensor-based method for modeling autoregressive processes. We modified TVART (as briefly described in the original paper, Harris et al. [2021]) that it can handle AR(p) processes, opposed to only AR(1) processes. TVART is also
24

---

0.8
Tost log- likelibood
Tost log-Jiloo Pog-liililood
0.7
0.6
0.775
0.750
Torke-8AFI not
CPSAIT rokk
0.3
Tucke--ALT mank
CPSALT mak
0.725
TurkesSAfiT nark
CPSAIT rok
Turker-ALT mank
ARIMM
0.700
CP SALT makk
0.4
15
Lags
Lags
(a) NASCAR.
(b) Lorenz.
Figure 9: Quantitative performance of different SALT models and ARHMMS (averaged over different runs) the synthetic experiments presented in Section 5.2. The test-set log likelihood is shown as function of lags in the SALT model, for both (A) the NASCAR and (B) Lorenz synthetic datasets.
a probabilistic model (i.e., cannot compute log-likelihoods), and so we focus our comparison on how well these methods recover the parameters of ground-truth SLDS.
used the same SLDS that we used to generate the NASCAR used L = 7 CP-SALT and Tucker-SALT with ranks and 2, respectively, and computed the MSE between the ground truth tensor and SALT tensors. For TVART, we used and ranks 2 and to fit the model to the data. We then clustered the inferred dynamics parameters to assign discrete states. To get the TVART parameter estimation, we computed the mean of the dynamics parameters for each discrete state and computed the MSE against the ground truth tensor. The MSE results are as follows:
dataset Section 5.2. We then
7, bin size 10,
Table 2: Results comparing SALT and TVART Harris et al. [2021] on the NASCAR example,
Model
Rank
Tensor Reconstruction MSE (x10-3) Number of parameters
TVART
0.423
1.4K
TVART
3
0.488
2.0K
SALT
0.294
0.6K
CP-SALT
0.297
0.7K
Table 2 shows that SALT models recover the dynamics parameters of the ground truth SLDS more accurately. Furthermore, the dataset (as the number of parameters in TVART scales linearly with the number of windows). We also note that TVART cannot be applied to held-out data, and, without post-hoc analysis, does not readily have notion of re-usable dynamics or syllables.
that SALT models fewer parameters than TVART models for
D.4
The effect of the number switches on the recovery of the parameters of the autoregressive dynamic tensors
We asked how the frequency of discrete state switches affected SALT's ability to recover the autore- gressive tensors. We trained SALT, the ARHMM, all with from an SLDS with varying number of discrete state switches. The ground-trutt SLDS model had H! discrete states, N 20 observations and S
lags, and the SLDS data sampled
dimensional continuous latent states. The matrix A (1) (I- ( (()) of each discrete state of the ground-trute SLDS had 1 real eigenvalue and pairs of complex eigenvalues. We sampled 5 batches of T 15, 000 timesteps of data from the ground- fround-truth truth SLDS, with s, E {1, 10, 25, 75, 125) number of discrete state switches that were evenly spaced out across the data. We then computed the mean squared error (MSE) between SLDS tensors and the tensors reconstructed by SALT, the ARHMM, and the SLDS. (Figure 10). More precisely, we combined the 3rd order autoregressive tensors from each discrete state 4th order tensor, and calculated the MSE based on these 4th order tensors. As expected, the MSE
25

---

x10
Mean squared eπтоr
3.0
2.5
2.0
1.5
CP-SLT
1.0
TALT
0.5
ARHMM
SLDS
0.0
10
25
75
125
8n.
(Number of discrete state switchos)
Figure 10: The quality of SALT approximation of SLDSs decreases as the number of discrete state switches increases: The data comes from an SLDS with H = 2, N = 20, and S = 7. 15,000 timesteps were generated, with varying numbers evenly spaced discrete state switches (x-axis). The mean squared error of reconstructing the autoregressive tensors increased as function of the number discrete state switches. Note that we combined the 3rd order autoregressive tensors from each discrete state into a 4th order tensor, and calculated the MSE based on these 4th order tensors.
increased with the number of switches in the data, indicating that the quality of SALT approximation of SLDSs decreases the frequency of discrete state switches increases.
26

---

E
Modeling Mouse Behavior
We include further details for the mouse experiments in Scction 5.3.
E.1
Training Details
We used the first 35,949 timesteps of data from each of the three mice, which were collected at 30Hz resolution. We used H = 50 discrete states and fitted ARHMMs and CP-SALT models with varying lags and ranks. Similar to Wiltschko al. [2015], we imposed stickiness on the discrete state transition matrix via Dirichlet prior with concentration of 1.1 on non-diagonals and 6 10 on the diagonals. These prior hyperparameters were empirically chosen such that the durations of the inferred discrete states and the given labels were comparable. We trained each model 5 times with random initialization for each hyperparameter, using 100 iterations EM on a single NVIDIA Tesla P100 GPU.
E.2 Video Generation
Here we describe how the mouse behavioral videos were generated. We first determined the CP- SALT hyperparameters as those which led to the highest log-likelihood on the validation dataset. Then, using that CP-SALT model, we computed the most likely discrete states on the train and test data. Given discrete state h, we extracted slices of the data whose most likely discrete state was h. We padded the data by 30 frames (i.e. 1 second) both at the beginning and the end of each slice for the movie. A red dot appears on each mouse for the duration of discrete state h. We generated such videos for all 50 discrete states (as long as there existed at least one slice for each discrete state) on the train and test data. For given discrete state, the mice in each video behaved very similarly (e.g.. the mice in the video for state 18 "pause" when the red dots appear, and those in the video for state 32 "walk" forward), suggesting that CP-SALT is capable of segmenting the data into useful behavioral syllables. See "MoSeq_ salt videos train" and "MoSeg salt videos test" in the supplementary material for the videos generated from the train and test data, respectively. "salt crowd_ i.mp4" refers the crowd video for state i. We show the principal components for states 1, 2, 13, 32, 33, 47 in Figure 11.
E.3 Modeling Mouse Behavior: Additional Analyses
We also investigated whether SALT qualitatively led to good segmentation of the behavioral data into discrete states, shown in Figure 11. Figure 1 1 shows 30 second example snippet of the test data from one mouse colored by the discrete states inferred by CP-SALT. CP-SALT used fewer discrete states to model the data than the ARHMM (Figure 11B). Coupled with the finding that CP-SALT improves test-set likelihoods, this suggests that the ARHMM may have oversegmented the data and CP-SALT may be better able to capture the number of behavioral syllables. Figure 1 ICC shows average test data (with two standard deviations) for a short time window around the onset of discrete state (we also include mouse videos corresponding to that state in the supplementary materials). The shrinking gray area around the time of state onset, along with the similar behaviors of the mice in the video, suggests that CP-SALT is capable of segmenting the data into consistent behavioral syllables.
27

---

B
CP-SALT 1ank=13 lag-5
Mouse test data segmentation
Top 10 PCs
Mouse 1 train data Camulative mumher
15000
10000
CP-SALT 1ack-13 lag-5
ARHMM lag=5 uniform:
total tumber frames
30
31
32
33
34
35
36
37
38
40
0-
Time
0
5
10
30 36 40 45 50 State iudex [ordered]
C
State 32 (3rd most used state)
State (4th most used state)
State 13 (8th most used state)
Average lest. data
Average test data
Average test data
Top 10 PCs
State onset
State 47 (15th most used state) State 1 (16th most used state) State 2 (26th most used state) Average test data
Average test data
Average test data
Top 10 PCs
-..0
-0.5
11.0
0.5
1.0
-..
.0.5
D.I
0.5
1.0
1.5
20
-1.0
-0.5
0.11
0.5
At
A
[8]
At
Figure 11: CP-SALT leads to qualitatively good segmentation of the mouse behavioral data into distinet syllables.: (A) 30 seconds of test data (Mouse 1) with the discrete states inferred by CP-SALT as the background color. (B) For one mouse, the cumulative number of frames that are captured by each discrete state, where the discrete states are ordered according how frequently they occur. (C) The average test data, with two standard deviations, for six states of CP-S ALT, aligned to the time of state onset. The shrinkage of the gray region around the state onset tells that CP-SALT segments the test data consistently.
28

---

Modeling C. elegans Neural Data
We include further details and results for the elegans example presented in Section 5.4. This example highlights how SALT can be used to gain scientific insight in to the system.
F.1 Training Details
We used ~3200 timesteps of data (recorded at 3Hz) from one worm, for which 48 neurons were confidently identified. The data manually segmented in seven labels (reverse sustained, slow, forward, ventral turn, dorsal turn, reversal (type 1) and reversal (type 2). We therefore used H = 7 discrete states in all models (apart from the GLM). After testing multiple lag values, we selected interactions and
L = 9 for all models. as these longer lags allow us to examine Longer-timeseed produced better segmentations across models, with only small reduction in variance explained. We trained each model 5 times with KMeans initialization, using 100 iterations of EM on single NVIDIA Tesla V100 GPU. Models that achieved 90% explained variance on held-out test set were then selected and analyzed (similar to Linderman et al. [2019]).
F.2 Additional Quantitative Results
Figure 12 shows additional results for training different models. In Figure 12A we see that models with larger ranks (or latent dimension) achieve higher explained variance. Interestingly, longer lags can lead to a slight reduction in the explained variance, likely due to overfitting. This effect less pronounced in the more constrained single-subspace SALT, but, these models achieve lower explained variance ratios throughout. Longer lag models allow us to inspect longer-timesee dependencies, and more experimentally insightful. Figure 12B shows the confusion matrix for discrete states between learned models and the given labels. The segmentations were similar across all models that achieved 90% explained variance.
F.3 Additional Autoregressive Filters
Figures 13 and 14 show extended versions of the autoregressive filters included in Section 5.4. Figure 13 shows the filters learned for ventral and dorsal turns (for which panel was included in Figure
0.950
Explained variance ratio
0.925
Held-out test data
0.000
0.875
0.850
0.825
CP-SALT L.:.
MS CP-SALT L-3 MS CP-SALT L=5
0.800
SS CI-SALT L
0.775
SS CPSSLT L-0
SS SLDS
0.750
18
SALT rank or SLDS continuous latent state dimension
B
Number of T'meestops
REVSUS
Given Labels
SLOW
FWD
VT
000
DT
REV1
REV2
2 6
1 3
2 4 ARHMM Labels
SALT Labels
SLDS Labels
Figure 12: SALT and SLDS perform comparably on held-out data: (A): Explained variance on held-out sequence. Single-subspace (SS) SALT and SLDS perform comparably. Multi-subspace (MS) SALT achieves a higher explained variance with fewer ranks. Multi-subspace SLDS was numerically unstable. (B): Confusion matrices between given labels and predicted labels. All methods produce similar quality segmentations.
29

---

5), while Figure 14 shows the filters for forward and backward locomotion. Note that the GLM does not have multiple discrete states, and hence the same filters are used across states. We see for ARHMM and SALT that known-behavion tuned neurons have higher magnitude filters (determined by area under curve), whereas the SLDS and GL.M do not recover such strong state-specific tuning. Since the learned SLDS did not have stable within-state dynamics, the autoregressive filters could not be computed using Equation (48). We thus show CA for lag I, where C+ denotes the Moore-Penrose pseudoinverse of C. as proxy for the autoregressive filters of discrete state h of the SLDS. Note that this is a post-hoc method and does not capture the true dependencies in the observation space.
We see that SALT consistently assigns high autoregressive weight to neurons known to be involved in certain behaviors (see Figures 13 and 14). In contrast, the ARHMM identifies these relationships less reliably, and the estimate of the SLDS autoregressive filters identifies few strong relationships. As the GLM only have one "state", the autoregressive filters are averaged across state, and few strong relationships are found. This highlights how the low-rank and switching properties of SALT can be leveraged to glean insight into the system.
30

---

A
SALT State (Ventral Turn
SALT State (Dorsal Tum)
Active
during:
4056
100
100
SALT Tensor Weights
Ventral
Turn
59000
Dorsal
SUEDE
TTurn
B
SLDS State (Ventral Tum)
SLDS State (Dorsal Tum
SLDS Tensor Weights
RIV1
ARHMM Tensor Weights C
ARHMM State 4 (Ventral Turn)
ARHMM State 5 (Dorsal Turn)
001L
111/1 15EVL 22000/
1251
GLM Tensor Weights D
SUDER
Lag-9, equivalent to 3 seconds; from left to right
Figure 13: Autoregressive tensors learned by different models (Ventral and Dorsal Turns): (A-C) iie-dimensional autoregressive filters learned two states by SALT, SLDS, ARHMM (identified as ventral and dorsal turns), and (D) by a GLM. RIV and SMDV are known to mediate ventral turns, while SMDD mediate dorsal turns [Kato et al., 2015, Gray al., 2005, Yeon al., 2018].
31

---

A
SALT State (Forward)
SALT State 7 (Reverse)
Active during:
SALI SALT Tensor Weights
1970
20138
100.
Forward
AVAL
Reverse
SLDS State (Forward)
SLDS State 7 (Reverse)
000
SLDS SLDS Tensor Weights
A036
10331
Axtrt
88
00991
180
1000. -88
10/32
ARHMM T'nnsor Weights C
ARHMM State (Forward)
ARHMM State 7 (Reverse)
1010
10113
10/3
D
GLM Tensor Weights
Lag 9, equivalent to 3 seconds; from left to right
Figure 14: Autoregressive tensors learned by different models (Forward Locomotion and Reversal): (A-C) One-dinensione autoregressive filters learned in two states by SALT, SLDS, ARHMM (identified forward and reverse). and (D) by a GLM. AVB and RIB are known mediate forward locomotion, while AVA and AVE are involved in initiating reversals [Kato et al., 2015, Gray al., 2005, Chalfie al., 1985, Piggott al., 1].
32

---

G Deep Switching Autoregressive Model
Here we compare SALT to deep switching auto-regressive factorization (DSARF) models [Farnoosh et al., 2021]. DSARF models construct deep generative model using set of low-rank factors, which are then weighted according to inferred discrete states at each timestep. This combination then defines the time evolution of autoregressive process, which in turn defines the distribution over the observed variables. As such, non-linear function approximators can be used to parameterize many of the link functions, gaining expressivity, but retaining the parameter efficiency and interpretability of conventional methods. Variational inference is used to learn the parameters and perform inference. Unlike SALT, DSARF handle missing data, as autoregressive processes defined only in the latent space.
We compare SALT against DSARF example drawn from Farnoosh et al. [2021], proposed for studying switching systems by Ghahramani and Hinton [2000] (see also Weigend and Gershenfeld [1994]). This example studies patient believed have sleep apnea, typified by periods where normal rhythmic breathing ceases, resulting in periods low or zero respiratory rates. The data are a one- dimensional time series of measure of chest volume, such that oscillations in the data correspond to rhythmic breathing (see Figure 15). Periods of constant volume correspond to apnea bouts. Disjoint thousand length sequences are used for train and test sets, such that y train E R 1x and y ytest E R 1110000
We apply DSARF described in Farnoosh al. [2021] and SALT. The SALT model used uses = discrete states, = ranks and L = 10 lags, and with a single latent subspace. We use L2 weight penalty of 10-4. As per the mouse experiments, we add Dirichlet stickiness prior with parameters ry = 10-2 and λ 10 (See Table 3). Inference results on the test set are shown in Figure 15. SALT hyperparameterss were selected through manual tuning. SALT achieves normalized next-step prediction RMSE of 22.57%, vs 23.86% achieved by DSARF (described by [Farnoosh al., 2021] as "short-term prediction") ") This result highlights that SALT is competitive with "deep" methods that provide the desired discrete-continuuus representation.
13000
15000
DEANZ
10000
10000
SALT
SALT
DSARF
DSARP
200
100
1000
1225
250
325
3000
100
Titeeteo
Timestep
Figure 15: Comparison DSARF and SALT-CP on the apnea example as presented in Farnoosh et al. [2021] Shown are filtering reconstructions of the observed trace and binary discrete label. SALT achieves a normalized RMSE of 22.57% vs 23.86% achieved by DSARE. Right panel is zoomed in version of the left panel. We that reconstructions and scores achieved by both models are comparable.
33

---

H Experiment Configurations
In this section we provide extended details for the experiments presented in the main text, including the hyperparameters selected and the hyperparameter tuning process.
H.1 List of Hyperparameters
Table 3 outlines the key hyperparameters we consider when comparing and selecting models. Details of the values for each of these hyperparameters are then specified in the sections afterwards.
Table 3:
Outline of the key hyperparameters in the three main models families we consider: SALT, SLDS and ARHMM.
Hyperparameter
Description
Applicable to models
Permissible Values
Tensor factorization
Factorization structure autoregressive tensor
SALT, ARHMM
(CP, Tucker, None)
Subspace type
Whether the output factors are shared across discrete states.
SALT, SLDS
(Single, Multi]
Dimension of the tensor.
D, Tensor ranks
SALT
E21
Assumed throughout that D D; D2 D3.
H. Number of discrete states
Number of discrete states in switching models.
SALT. SLDS, ARHMM
Σ21
L. Number of lags
Number of previous observations in autoregressive models.
SALT, ARHMM
Σ21
S, Latent space dimension
Dimension of continuous latent state.
SLDS
Zz1
L2 penalty applied lag parameters longer lags.
Temporal L2 penalty
The penalty strength is defined as a g where / is the lag.
SALT, ARHMM
E 12
B E Rg1
Stickiness
Dirichlet prior that can be added to penalize discrete state switches.
SALT, SLDS, ARHMM SALT, SLDS, ARHMM
Y
Rx E R20
Global L2 penalty
12 penalty applied across all parameters. (See note below)
R Rz0
Unless otherwise specified, we performed grid search over range of values within the permissible set. In certain circumstances, the hyperparameter was selected to match known properties of the data, e.g. we used H for the NASCAR data, because we know there are four underlying states in the data.
During pilot experiments we experimented with global L2 penalty applied to all parameters in the model. We found that varying this parameter did not affect the key outcomes of each experiment. We therefore set the L2 penalty strength to the same value for all the models for all the experiments, with strength 0.0001, unless otherwise specified.
Experiment Specific Hyperparameters Here specify experiment--peccffe details.
H.2
H.2.1
Section 5.1: SALT Faithfully Approximates LDS
For given LDS with latent dimension size of and initialized with random rotation matrix, computed the estimated number of CP-SALT and Tucker-S ranks, denoted D = n + 3m and 2m, respectively, where is the number of real eigenvalues and m the number of complex conjugate pairs of I defined in Proposition 1. We then fitted CP-S ALT models with (min(S,De Dip) 2, ... max S, D'p) + 2} and SALK models with {min(S, Duckerr) 2.... (S, D ) + 2}. We chose L = 50, as this was the horizon at which the autoregressive parameters were approximately zero. The temporal L2 penalty was 1.0.
Ducker
H.2.2 Section 5.2: Synthetic Switching LDS Examples
Synthetic NASCAR dataset All models used the true number of discrete latent states, H 4. We fitted single-subspace CP-SALT and Tucker-S
Bocker--ALLe SALT models with ranks D E 2, For both SALT models and ARHMMs, we used L E {1, 5, 6, 7, 9, 10, 15. 20}. A temporal L2 penalty of 1.0 was used for CP-SALT and Tucker-SALT models. Single-subspace SLDSs were fitted with the true underlying latent dimension, S
Synthetic Lorenz attractor dataset All models used the approximate number of discrete latent states the data, H (following Linderman et al. [2017]). We fitted single-subspace CP-SALT and Tucker-SALI models with ranks D E {1, 3, 4). For both SALT models and ARHMMs, we used {1, 5, 10, 15, 20). A temporal 1.2 penalty of 2.0 was used for CP-SALT and Tucker-SALT models. Sungle-subspace SLDSs were fitted with the true underlying latent dimension, S $=1
34

---

H.2.3 Section 5.3: Modeling Mouse Behavior
We fitted multi--ubsspace CP-SALT models with ranks D (10. 11, 12, 13, 14).
For both CP-SALT
models and ARHMMs, we used E {1, 3, 4, 5, 6, 7,8, 9, 10, 15. 20} and the number of discrete states was We use a temporal L2 penalty of 1.0. Similar to Wiltschko et al. [2015], we imposed stickiness on the discrete state transition matrix of both SALT models and ARHMMS via Dirichlet prior. For discrete state h, the concentration parameters v E 1 of the Dirichlet prior is 7 for Us & h, and
H 50 (51 behavioral states explained 95% of videos Wiltschko et al. [2015]).
+ K. for vn- For this experiment, " was set to 1.1 and K to 6 10", which were empirically chosen such that the duration of the inferred discrete states and the given labels were comparable.
H.2.4 Section 5.4: Modeling C. elegans Neural Data
Following Linderman et al. [2019], we empirically searched for sets of hyperparameters that achieve ~90% explained variance on a held-out test dataset. We fitted both single and multi multi-subssaace CP-SALT models with ranks D f8, 9, 10, 11, 12, 13, 14, 15. 16, 17. 18). Similarly, SLDSs were fitted with the same range of latent dimension size. For both CP-SALT models and ARHMMs, we used L (1,3, 6, 6, and the number of discrete states
set to H unique manual labels. After testing multiple lag values, we selected = 9 for SALT models and ARHMMs, these longer lags allow to examine longer-timescale interactions and produced better segmentation across models, with only a small reduction in variance explained. For ARHMMs, we set the global L2 penalty strength to 20.0 with the temporal L2 penalty set to 1.0. For SLDSs, we set the global L2 penalty strength to 100.0. For SALT models, we set the global L2 penalty strength to 0.2 and the temporal I.2 penalty to 1.1. We additionally imposed smoothness penalty to the weights of the lag factors of SALT models by adding 7/ FW to the NLL for optimizing the lag factors, where 7 is the smoothness penalty strength and F is the first difference matrix. We set 0.8 for S SALT models.
which is the number of
35


Reviews:
[{"review_id": "jzR9ZOsrzz", "reviewer": "Reviewer_BTJL", "summary": "The paper proposes Switching Autoregressive Low-Rank Tensor (SALT) models, a variant of an Autoregressive Hidden Markov Model (ARHMM) in which the model’s temporal dynamics are captured by a low-rank tensor approximation, thus combining the parameter efficiency of Switching Linear Dynamical Systems (SLDS) with the simple learning and inference techniques available for ARHMMs. The paper further shows that a SALT model can approximate a stable LDS model to a degree that depends only on the eigenspectrum of the LDS model and the order of the SALT model.\n", "questions": "Minor comments:\n\n- It would improve the accessibility of the paper if it included graphical models of (S)LDS and (S)VAR, similar to Figure 1, so that the advantages and disadvantages of the different model classes can be analyzed in terms of their (marginalized) conditional independence assumptions.\n\n- I wish the presentation had pointed out the strong connections of the Tucker decomposition to (higher-order) PCA/SVD. In contrast to the Tucker decomposition, this is something most readers are familiar with and would have helped with the intuition of an orthogonal/unitary factor approximation.\n\nQuestions:\n\n- Section 5.1: how were the random matrices sampled?\n- Figure 2(A): why does a rank *higher* than 7 (Tucker-SALT) and 10 (CP-SALT) lead to *worse* approximations. Should the MSE not be strictly decreasing?", "limitations": "- The paper briefly mentions possible extensions but does not include an explicit discussion of limitations.\n\n- The paper does not discuss any ethical concerns related to the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper makes two main contributions: (1) a parametrization of ARHMM dynamics with a low-rank Tucker decomposition; and (2) a theoretical analysis of the resulting model’s approximation error relative to a stable LDS. The latter is an interesting theoretical insight and includes a rigorous proof in Appendix B.\n\n- The background section (Section 2) provides a useful recap of (switching) autoregressive models and (switching) linear dynamical systems that motivates the need for parameter-efficient architectures with tractable learning and inference algorithms.\n\n- The experiments (Section 5) confirm that SALT models can learn (S)LDS dynamics and require less training data than their ARHMM parent due their parameter-efficient representation. The experiments on real-world data demonstrate that SALT models can learn semantically meaningful filters / state representations and outperform ARHMMs in terms of test log-likelihood.", "weaknesses": "- As discussed in Section 4, structural decompositions of temporal dynamics, including low-rank approximations, have been explored in a variety of different contexts. This includes the representation of the autoregressive tensor as a Tucker decomposition and, while I appreciate the differences in the graphical structure compared to these models and the theoretical insights provided in Proposition 1, the technical contribution of this paper is not particularly strong.\n\n- Although the core ideas of this paper are relatively simple, the confusing tensor notation makes the paper unnecessarily hard to follow. Uncommon operations like the $n$-mode tensor-matrix product or the $n$-mode tensor matricization must be properly defined, ideally including visual illustrations. Having to figure out which high-dimensional slices are being multiplied is a burden on the reader and diverts the focus from the underlying ideas. Where possible, I would recommend to express the model dynamics in summation notation instead of tensor operations; it would greatly improve readability.\n\n- Ultimately, the representation of the autoregressive tensor with a Tucker decomposition is a structural assumption that modulates the spectrum between flexible models prone to overfitting and robust models prone to large bias. Since SALT models cannot be more expressive than a generic ARHMM, I view them primarily as a form of implicit regularization and, as such, would have liked to see a comparison with other regularization techniques (e.g., a Bayesian treatment with strong priors).\n\n- The experimental validation compares the proposed method to two traditional time-series models (ARHMM and SLDS) but does not include any state-of-the-art baselines (e.g., based on deep variants of autoregressive or state-space models, Gaussian processes, Transformer architectures, normalizing flows, etc.). Even if the proposed method is more related to ARHMM or SLDS, it is expected that the evaluation takes other time-series architectures into account, including more competitive and more recent developments. The data is relatively simple as well, with two toy datasets consisting of (S)LDS simulations and a small number of real-world sequences (3 mice, 1 worm).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer BTJL\n\nWe thank the reviewer for taking the time to read our submission and for their detailed and insightful feedback.  The strengths you outline really neatly encapsulated our objectives, and so that was great to hear!  We will now provide some more detailed feedback beyond the General Response provided above.\n\n## R.E. Weaknesses:\n\n**1.  Technical Contributions:**\n\n>[...]  While I appreciate the differences in the graphical structure compared to these models and the theoretical insights provided in Proposition 1, the technical contribution of this paper is not particularly strong.\n\nWhile the technical contributions of our work may not be a paradigm shift, we are confident it will be useful to many practitioners who regularly use these types of models.  SALT sits at the intersection of several widely-used methods, leveraging efficient components and capturing the benefits of each model to yield a model that offers a unique insight into the system.  We also release our fast JAX code.  We also highlight, as raised in General Response B, that inference in SLDSs is not a “solved” problem and often requires complex, hard-to-tune inference schemes – see recent publications such as Zoltowski et al [2020, ICML] and Berger et al [2022, NeurIPS].  Therefore, achieving a parameter complexity comparable with SLDS, while retaining fast and exact inference, and bringing additional benefits, is a valuable contribution to the field that will be of particular interest to the NeurIPS readership.\n\n**2.  Improving Notation**\n\n>Although the core ideas of this paper are relatively simple, the confusing tensor notation makes the paper unnecessarily hard to follow. [...]\nSee General Response A.  We have simplified and clarified the notation where possible, and added diagrammatic explanations to the supplement.  Thank you for this feedback.  \n\n**3.  SALT as implicit regularization:**\n\n>Ultimately, the representation of the autoregressive tensor with a Tucker decomposition is a structural assumption that modulates the spectrum between flexible models prone to overfitting and robust models prone to large bias. [...]\n\nAll models trained used L2 regularization, including the ARHMM (we have added details to the supplement).  Further to our response in General Response B, we note that there is no clear consensus on the correct way to regularize the parameters of vector autoregressive hidden Markov models – especially ones with low-rank factors.  Therefore, SALT provides a natural and intuitive way to control the expressivity of the model family.  Nonetheless, exploring more powerful Bayesian regularization or hierarchical components to SALT are important directions for future research.  \n\n**4.  Empirical evaluation:**\n\n>The experimental validation compares the proposed method to two traditional time-series models (ARHMM and SLDS) but does not include any state-of-the-art baselines (e.g., based on deep variants of autoregressive or state-space models, Gaussian processes, Transformer architectures, normalizing flows, etc.). [...]\n\nSee General Response B.  We stress, our objective with this work was not to create a state-of-the-art regression model.  Instead, SALT models provide an “interpolation” between the widely used SLDS and ARHMM.  These models provide a very experimentally valuable hybrid continuous-discrete representation, and are used regularly by practitioners owing to their simplicity, efficiency and utility.  As such, SALT is a valuable model that allows practitioners to retain the benefits of these models, while flexibly ameliorating their individual weaknesses.  \n\nThe tasks we examined were taken from the literature, but, if the reviewer has suggestions for additional experiments then we are more than happy to add them ahead of a camera-ready version.  Since receiving the reviews, we have compared SALT to DSARF (a “deep” switching model, as suggested by 6MVR) on the apnea example from that paper.  SALT outperforms comparably than DSARF in terms of reconstruction accuracy and segmentation quality.  These results are included in the additional PDF. \n\n## Minor Comments:\n\n1.  **Visualization of model families**:  We have added extra graphical models for each model to the supplement, and discussed the implications in the main text.  Thank you for the suggestions.\n\n2.  **Links to SVD**:  This is a great connection, we have added discussion of the links to SVD to the background, and also re-highlighted them when we discuss the factors later on.\n\n## Questions:\n\n1.  **How were random matrices sampled**:  Matrices were sampled as random rotational matrices.\n\n2.  **Why can increasing rank reduce performance**:  Great observation – the answer is actually overfitting to the training data.  Here we are plotting the error on held-out test data.  The degradation also exists for the log-likelihood but is less visible.  We have added clarification on this.  We stress that this overfitting is caught by cross-validation. \n\n## Limitations:\n\n1.  **Inclusion of limitations**:  See General Response A.  We have added discussion of limitations including model mismatch and missing observations. \n\n2.  **Inclusion of ethical concerns**:  We have added clarification that we see no ethical concerns specifically related to SALT.  \n\n\n**Thank you again for your response.**  If we have successfully allayed your concerns, we ask if you would consider upgrading your score.  Of course, if you have further questions, we are happy to answer them! \n\n--- The SALT authors.  \n"}, {"review_id": "XSBRgDhZo9", "reviewer": "Reviewer_sUaQ", "summary": "This paper proposes a new time-series model called Switching Autoregressive Low-rank Tensor Model (SALT) that combines the advantages of autoregressive hidden Markov models and switching linear dynamical systems while addressing their weaknesses. SALT allows for longer range dependencies without overfitting and has been proven to be effective in various prediction tasks. The paper also explains the low-rank factorization used in SALT parameterization and provides experimental results demonstrating the effectiveness of SALT in various real-world applications.", "questions": "1. How generalizable are the results of this paper to other types of time-series data and applications?\n\n2. How robust are the results of this paper to variations in the experimental setup, such as the choice of evaluation metrics or hyperparameters?\n\n3. How does the complexity of SALT compare to other time-series models, and what are the implications of this for its practical use?\n\n4. How does the choice of low-rank tensor regression in SALT affect its performance, and how might other regression methods be more appropriate for certain types of data?\n\n5. What are the potential ethical implications of using SALT or other time-series models for analyzing sensitive or personal data, and how can these be addressed?", "limitations": "1. Limited scope: The paper focuses on a specific type of time-series data and does not explore the applicability of SALT to other types of time-series data. This may limit the generalizability of the model to other domains.\n\n2. Limited sample size: While the paper provides empirical evidence of the effectiveness of SALT, the sample size of the experiments is relatively small. This may limit the generalizability of the results to larger datasets.\n\n3. Limited evaluation metrics: The paper primarily evaluates the effectiveness of SALT using prediction accuracy metrics. While these are important metrics, they may not capture all aspects of the model's performance, such as its ability to capture complex temporal dependencies.\n\n4. Limited discussion of hyperparameters: The paper does not provide a detailed discussion of the hyperparameters used in the experiments. This may limit the understanding of how sensitive the model's performance is to the choice of hyperparameters.\n\n5. Limited discussion of computational complexity: While the paper briefly mentions the computational complexity of SALT, it does not provide a detailed analysis of the model's computational requirements. This may limit the understanding of the practical implications of using SALT in real-world scenarios with large datasets.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty: The paper proposes a new time-series model, SALT, that combines the advantages of two existing models while addressing their weaknesses. This is a novel approach that has not been explored before.\n\n2. Clarity: The paper is well-written and easy to understand, even for readers who are not experts in the field. The authors provide clear explanations of the model and its components, as well as the experimental setup and results.\n\n3. Empirical evaluation: The paper provides empirical evidence of the effectiveness of SALT in various real-world applications, including neural and behavioral time series. The authors compare SALT to other commonly used time-series models and demonstrate its superior performance.\n\n4. Reproducibility: The authors provide code and data to facilitate reproducibility of their experiments. This is important for other researchers who want to build on this work or apply SALT to their own datasets.\n\n5. Generalizability: The authors demonstrate the effectiveness of SALT in various real-world applications, suggesting that it is a generalizable model that can be applied to a wide range of time-series data.", "weaknesses": "1. Lack of theoretical analysis: While the paper provides empirical evidence of the effectiveness of SALT, it does not provide a detailed theoretical analysis of the model. This may limit the understanding of the model's properties and limitations.\n\n2. Limited comparison to state-of-the-art models: While the paper compares SALT to other commonly used time-series models, it does not compare it to the most recent state-of-the-art models. This may limit the understanding of how SALT compares to the best-performing models in the field.\n\n3. Limited discussion of limitations: The paper does not provide a detailed discussion of the limitations of SALT. This may limit the understanding of the situations in which SALT may not be the best choice for modeling time-series data.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for their detailed and insightful feedback.  The five strength points were particularly heartening.  We now provide more detailed feedback.\n\n## RE Weaknesses\n\n**1. Theoretical analysis:**\n\n>Lack of theoretical analysis: While the paper provides empirical evidence of the effectiveness of SALT, [...]\n\nWe make a novel and broadly applicable link between low-rank tensor regressions and linear dynamical systems in Proposition 1, linking the core components of SALT and SLDS models.  Furthermore, SALT models are still fundamentally ARHMMs, allowing us to directly use existing results and methods to further understand and control the behavior of SALT models, eg, enforcing sparsity [Shah et al, 2015, NeurIPS] and non-negativity of the factors [Shashua and Hazan, 2005, ICML].  \n\n**2. Choice of baselines:**\n\n>Limited comparison to state-of-the-art models: While the paper compares SALT [...]\n\nSee General Response B.  We compare SALT to SLDS, ARHMMs and TVART (in the supplement).  As raised by 6MVR, we also compared to DSARF on an example from the original paper, and find that SALT performs comparably if not better than this deep method.  \n\nWe note that the relevant comparisons are models that yield a hybrid continuous-discrete representation.  Many common methods are therefore not directly applicable.  If the reviewer has suggestions of models with this structure, we are happy to include a comparison ahead of any camera ready version.  \n\n**3. Discussion of Limitations:**\n\n>Limited discussion of limitations: The paper does not provide a detailed discussion of the limitations of SALT. [...]\n\nSee General Response (A).  We have included thorough discussion of SALTs limitations. \n\n## RE Questions\n\n**1. Generalizability of SALT:**  \n\n>How generalizable are the results of this paper to other types of time-series data and applications?\n\nWe believe the analysis allowed by SALT is fully generalizable, particularly when the interdependencies between observations are insightful to model and a mixed discrete-continuous representation is sought.  The ease of tuning and fitting SALT models make it an excellent option for exploratory data analysis.  Beyond the two time-series applications in the paper, we added a third on Sleep Apnea data experiment from DSARF (as pointed to by 6MVR, see additional PDF), further demonstrating the generalizability of SALT.\n\n**2. Robustness to hyperparameters and metrics:**  \n\n>How robust are the results of this paper to variations in the experimental setup, such as the choice of evaluation metrics or hyperparameters?\n\nSee General Response.  SALT models are fairly robust to the choice of hyperparameters (within reason).  There are also fewer hyperparameters, as there is no learning rate or decay schedule required, no tuning of variational/interactive/approximate inference algorithms, and EM guarantees convergence to a (local) maximum so there is no need for tuning early stopping.  \n\nWith regard to metrics, we considered multiple different metrics for each model.  We found that the use of confusion matrices, segmentations and marginal log-likelihoods (where applicable) yielded comparable results across methods and applications.  \n\n**3. Complexity of SALT:**\n\n>How does the complexity of SALT compare to other time-series models, and what are the implications of this for its practical use?\n\nThe complexity of SALT lies between the ARHMM and the SLDS.  The ARHMM is difficult — often intractable —  to fit because of overparameterization.  The SLDS is tricky to fit because it requires iterative variational inference methods.  SALT admits fast computation of an exact marginal likelihood, allowing fair comparison **between** different model classes.   In contrast, SLDS estimates a bound and is dependent on approximate or variational inference schemes.  \n\nSALT is also easier to implement and more extensible than SLDS.  Each new SLDS model variant requires deriving and tuning a new variational inference scheme.  New SALT models can be derived by simply updating the update equations (falling back to coordinate gradient descent).  As in Weakness 1, SALT can therefore directly leverage the rich literature in tensor factorization methods.\n\n**4. Role of rank in optimization:** \n\n>How does the choice of low-rank tensor regression in SALT affect its performance [...]\n\nThe rank trades off expressivity with overfitting, acting as an implicit regularizer, in contrast to other models that require additional regularizers and hyperparameters to prevent overfitting.  As noted in General Response B, there is no consensus on the best way to regularize an ARHMM, so SALT actually provides an intuitive regularization.  \n\n**5. Ethical concerns:**\n\n>What are the potential ethical implications of using SALT [...]\n\nWe do not believe there are any new ethical implications from SALT. \n\n\n## RE Limitations\n\n1. **Type of evaluation**: See Questions 1.\n\n2. **Scope of evaluation**: We tested SALT on a number of standard benchmarks common to this domain.  We have also tested SALT on the apnea dataset used in DSARF (noted by Reviewer 6MVR).  SALT outperforms DSARF in both reconstruction accuracy and segmentation quality.\n\n3. **Temporal dependencies**: Quantifying temporal dependencies is tricky.  We explored this in the C. elegans example, using a known neural function as a proxy for capturing time dependencies.  We have added some clarification on this to the results section.\n\n4. **Hyperparameters**: See General Response / Questions 2.\n\n5. **Computational complexity**: See General Response and additional PDF.  SALT models are broadly the same speed or faster to train and apply than ARHMMs and SLDSs.\n\n**Thank you again for your response.**  If we have successfully allayed your concerns, we ask if you would consider upgrading your score.  Of course, if you have further questions, we are happy to answer them! \n\n--- SALT authors.  \n"}, {"review_id": "QsaKSFjlhP", "reviewer": "Reviewer_6MVR", "summary": "The paper introduces a new model for time-series called SALT (switching autoreg. low-rank tensor). The goal with SALT is to offer a \"best of both worlds\" alternative to AR-HMMs and switching linear dynamical systems (SLDS).\n\nSALT's relative advantages are:\n* enjoys closed-form parameter estimation (unlike SLDS)\n* enjoys lower parameter count, meaning less likely to overfit for long-range models on small datasets (unlike AR-HMM)\n\n|                  | AR-HMM      | SALT        | SLDS                  |\n|------------------|-------------|-------------|-----------------------|\n| param estimation | closed-form EM | closed-form EM | need MCMC/variational |\n| parameter count  | O(N^2)      | O(N)        | O(N)                  |\n| hyperparameters  | H, L        | D, H, L     | D, L                  |\n\nwhere\n\nD = rank / number of latent state dims,\nH = num hidden states,\nL = autoregressive lag\n\nThe key idea behind SALT is to essentially force the autoregressive coefficient matrix of an AR-HMM to take a specific factorization structure (See Eq 6-7). \n\nIn Sec 3.2, a closed-form EM estimation algorithm is derived.\n\nIn Sec 3.3, Prop 1 presents a formal error analysis of how well a SALT model can approximate a *stable* LDS\n\nExperiments in Sec 4 cover\n\n* 5.1: analysis of Prop 1 ideas on a simulated dataset\n* 5.2: evidence that SALT might be as expressive as SLDS on synthetic data\n* 5.3/5.4: real experiments on mouse behavior and worm neurons\n", "questions": "\n1. How can we use Prop. 1 to decide which D value is optimal? Caption of Fig 2 implies we can use Prop. 1 to pick/rank D for SALT given a known LDS model. But doesn't Prop 1 as stated just provide a statement on error when SALT's D is set to one specific value D = n + 2m?\n\n2. Seems like SALT has more hyperparameters (H, L, and D) to select than either AR-HMM or SLDS (each requires 2 of the 3). Are there good rules of thumb to avoid the expense of 3-dimensional grid search?\n\n3. Can you report typical runtimes on the large datasets you study? Not just for training, but also hyperparameter selection as well.\n\n", "limitations": "Sec. 6 is missing entirely a proper paragraph discussing limitations of the work at the end of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n+ elegant proposal for factorizing coefficients of AR-HMM\n+ parameter estimation via EM allows all steps to be closed-form\n+ thoughtful experiments on synthetic and real data", "weaknesses": "\n### Why prefer SALT over SLDS? What real computational advantage?\n\nIn Sec 5.4, on a big real dataset it is argued that SALT \"can perform as well as\" SLDS. While this is all fine, is there a strong advantage that makes a practioner studying this data prefer SALT?\n\nSeems like ultimately, parameter count SALT is larger than SLDS (Table 1).\n\nClosed-form parameter estimation is nice of course, but if the approximate methods for SLDS worked well enough on this dataset, when should a practitioner think that SALT is preferrable, if ever?\n\n\n### Missing comparison to AR-HMM with L2-regularized coefs\n\nThe paper's story suggests that AR-HMM overfitting is the major reason a new model is needed. However, a natural way to prevent overfitting (that preserves closed-form EM parameter estimation) is to provide L2-regularization of (some) AR coefficients. As best as I can tell, the results shown emphasize maximum likelihood estimation, without regularization.\n\nWhile SALT is elegant, why should a practitioner invest in SALT and its more complex parameterization over a well-studied way to prevent overfitting?\n\n### Hyperparameter settings need to be clarified\n\n**Update after response**: The author response resolved this concern. Text below from original maintained for posterity.\n\nIn my understanding, SALT needs the user to select D, H, and L as hyperparameters. In the main paper, I found it difficult to understand which settings were used in several cases and how those were determined.\n\nMy concern is that their SALT model could be using substantially higher D/H/L values than alternatives. There needs to be a transparent way these hyperparameters are selected on each dataset.\n\nI'd suggest in Fig 3 caption, reporting N/D/H/L for both racetrack and Lorenz data. Similarly with Fig 4 for mouse data and Fig 5 for worm neural activity.\n\n### Missing some related work\n\n**Update after response**: The author response resolved this concern. Text below from original maintained for posterity.\n\nI'd suggest the authors consider a conceptual comparison to deep switching auto-regressive factorization (DSARF), published by Farnoosh et al at AAAI '21.\n\nLike SALT, DSARF produces discrete segmentations and continuous latent trajectories for a time-series. SALT's factorization is conceptually distinct and has closed-form estimation, while DSARF requires non-conjugate inference via VI with Monte Carlo approximations of gradients. However, DSARF could be more flexible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 6MVR:\n\nWe thank the reviewer for taking the time to read our submissions and for their detailed and insightful feedback.  We especially appreciated the description of our method as “elegant”!\n\nThe main theme of your review seems to be centred on the experimental utility of SALT – in terms of comparative performance compared and hyperparameter tuning.   We provided some feedback in our General Response, and provide more detailed responses here.   \n\n## R.E. Weaknesses:\n\n**1.  Why prefer SALT over SLDS? What real computational advantage?**\n\n>In Sec 5.4, on a big real dataset it is argued that SALT \"can perform as well as\" SLDS. While this is all fine, is there a strong advantage that makes a practitioner studying this data prefer SALT?  [...]\n\nA key advantage of SALT over SLDS is that exact inference (posterior over state and evaluation of the log marginal likelihood) in SALT models is straightforward; whereas SLDS can only evaluate a bound on the likelihood and a variational posterior over the latent state (or resort to MCMC).  This makes model development more straightforward and model comparison more accurate.  Inference in SALT models is notably faster than (even approximate) inference in SLDS models.  Finally, the autoregressive dependency structure parameterised by SALT allows more direct insight into the system, compared to the somewhat abstract latent states recovered by SLDS.  The factors can be individually inspected to garner further insight into the system.  \n\n*We stress, however, that SALT analysis is a good complement to conventional SLDS analysis, providing different, but equally useful, insights into the system.*\n\n**2. Missing comparison to AR-HMM with L2-regularized coefs:**\n\n>The paper's story suggests that AR-HMM overfitting is the major reason a new model is needed. [...]\n\nWe refer the reviewer to General Response B for general discussion of the practical utility of SALT.  Beyond this, however, SALT recovers a low-dimensional continuous description of the data as well, defined as the vector value computed prior to multiplication with the output factors.  The example traces of this low-dimensional variable are shown in Figure 3.  We have highlighted this benefit over ARHMMs in the updated text.  \n\nAs noted in General Response B, there is no consensus on the best or most principled way to regularize the tensor of an ARHMM, and our experiences in previous work was one of the motivations behind SALT!  For fair comparisons across all models, we opted for the simplest approach and used L2 regularization.  We have added further discussion of this point to the main text.  \n\n\n**3. Hyperparameter settings need to be clarified:**\n\n\n>In my understanding, SALT needs the user to select D, H, and L as hyperparameters.  [...]\n\nThe parameter complexities of SALT and SLDS were comparable across most of the models we considered.  Note that for the C. elegans experiment, we used longer lags than was strictly necessary to model the data (to examine the longer time dependencies in the data).  Models with commensurate numbers of parameters performed similarly.  We will add further clarification of this to the text.\n\n**4. Missing some related work:**\n\n>I'd suggest the authors consider a conceptual comparison to DSARF [...]\n\nThank you for the very relevant reference!  We have added qualitative comparison between SALT and DSARF to the revised manuscript.  We have also compared SALT to DSARF on the apnea example used by Farnoosh et al [2021].  This application is of particular interest because of the clinical relevance of the discrete states (respiration vs no respiration, see additional PDF).  We find that SALT performs comparably or slightly better than DSARF, achieving a normalized error of 22.57% vs DSARFs 23.86%, and producing qualitatively better discrete segmentations.  We will include these results and further discussion in a camera-ready version.  \n\n## R.E. Questions:\n\n**1. Selecting D using Prop. 1:**\n\n>How can we use Prop. 1 to decide which D value is optimal? [...]\n\nWe explored using Proposition 1 to set D in Supplementary Figure 7, where we are able to predict the optimal rank for a SALT model when fitting to an LDS.  Proposition 1 further states that the accuracy (and consequently the marginal likelihood) is only dependent on the number of lags above a certain rank, a result which we verify in the additional PDF.  This result, in part, motivated the use of low-rank approximations. \n\n**2.  Hyperparameter fitting:**\n\n>Seems like SALT has more hyperparameters (H, L, and D) to select than either AR-HMM or SLDS [...]\n\nSee General Response.  In general, we find SALT models faster and more robust to fit than SLDS models, and so even a three-way grid search takes a comparable amount of “user” and computational effort to the two-way SLDS search (see additional PDF).  In our experience, setting good hyperparameter ranges is straightforward with domain knowledge (eg. length of time dependencies).   We also believe that the simpler inference in SALT constitutes a “gain”, whereas the inference pipeline used in SLDS is a hidden hyperparameter which dramatically impacts performance. \n\n**3.  Runtimes:**\n\n>Can you report typical runtimes on the large datasets you study? [...]\n\nSee General Response.  We have added typical runtimes for execution and training, and estimates for the time for hyperparameter tuning.  SALT models retain the runtime cost of ARHMMs, but the resistance to overfitting of SLDS, overall making hyperparameter selection easier.  \n\n## R.E. Limitations:\n\n**1. Discussion of limitations**:  See General Response.  We have added a more detailed limitations section touching on/ameliorating many of the points you raise.  \n\n**Thank you again for your response.**  We ask if the reviewer would consider upgrading their score if we have successfully allayed your concerns.  Of course, if you have further questions, we are happy to answer them! \n\n--- The SALT authors.  "}, {"review_id": "SQlN4K39VB", "reviewer": "Reviewer_4hZ1", "summary": "This paper introduces a new unsupervised probabilistic model for time series analysis: the Switching Autoregressive Low-rank Tensor (SALT). It combines a Low Rank Tensor parametrization of autoregressive (AR) models with switching Dynamics. The two main contributions are:\n\n1)  The SALT model itself. Although the tensor parametrization is not new, they succesfully augment it with switching dynamics and derive an EM-based learning and inference algorithm with closed form updates.  \n\n2) A theoretical connection between Linear Dynamical Systems (LDS) and Low-Rank Tensor Autoregression. \n\nThe model is evaluated on simulated, behavioral and neural datasets.", "questions": "1) Would it be possible to concatenate the 3-way autoregressive tensors $\\mathcal{A}^{h}$ in a 4-way low rank tensor $\\mathcal{A}$ of size $N \\times N \\times L \\times h$ ? This might yield an even more compact description of the data.\n\n2) I understood Proposition 1 was a novel contribution from the authors. If it's indeed the case, it's worth mentioning it (more) explicitly in the introduction. \n\n3) How stable are the CP-SALT discovered factors from one initialization to the next ? It could be interesting to look at the factors directly to characterize dynamics, but I think the discovered dynamics could be roughly preserved despite different $\\mathcal{A}^{h}$ ? ", "limitations": "1) The proposed method is based on Gaussian noise models, which is acknowledged by the author. In its current form, it cannot, for example, efficiently handle image data without pre-processing steps. \n\n2) The link between SALT and SLDS is \"only\" demonstrated for non switching dynamics ($h=1$ ).", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written despite the need for numerous notations. The author fluently expose their model and relate it to existing approaches.\n\nThe link between LDS and Low-Rank Tensor AR models is important and novel (to my understanding). Not only it helps grasping the relationship between those two models, but it also provides a bound linking the dynamics spectrum, the tensor rank and the approximation error one makes when using the tensor AR model. Among other things, I believe it can guide the specification of the tensor rank, which is a non trivial operation. \n\nSALT builds upon existing approaches, and although it is not more expressive than sLDS, inference and learning use closed form updates which is very convincing from a practical perspective. Importantly, it retains the possibility to analyze a low dimensional representation of the observed time series.\n\nBoth simulated and real world experiments are convincing and the code is provided. ", "weaknesses": "1) Title 3.3 is confusing. I agree that the theoretical connection discovered by the author is significant, but it does not concern SALT and sLDS. The link is between LDS and Low Rank Tensor AR models.\n\n2) I think the graphical of Figure 1 is wrong. Why isn't there any arrow between $z_t$ and $z_{t+1}$ despite $z_{t+1} \\sim \\text{Cat}(\\pi^{(z_t)})$ ?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 4hZ1\n\nWe thank the reviewer for the time to review our submission; for their detailed and insightful review; and for highlighting the contributions and the clarity of our submission.\n\n## R.E. Weaknesses:\n\n**1. Section title:**\n\n>Title 3.3 is confusing. I agree that the theoretical connection discovered by the author is significant, but it does not concern SALT and sLDS. The link is between LDS and Low Rank Tensor AR models.\n\nWe have removed switching from the section title. \n\n**2. Graphical model:**\n>I think the graphical of Figure 1 is wrong. Why isn't there any arrow between z_t and z_t+1?\n\nGreat catch – we also noticed this the day after submission…  Yes, there should be a link between z’s.  We have corrected the figure.\n\n## R.E. Questions:\n\n**1.  Concatenating into 4-way tensor:**\n\n>Would it be possible to concatenate the 3-way autoregressive tensors A^h in a 4-way low rank tensor A of size N x N x L x H? This might yield an even more compact description of the data.\n\nConcatenating across states is certainly possible.  However, it isn’t immediately clear how to “mine” additional understanding from the representation.  \n\n**2.  Highlight novelty of Proposition 1:**\n\n>I understood Proposition 1 was a novel contribution from the authors. If it's indeed the case, it's worth mentioning it (more) explicitly in the introduction.\n\nThank you, yes, Proposition 1 is novel.  We have highlighted this more clearly in the introduction.\n\n**3.  Stability of factors:**\n\n>How stable are the CP-SALT discovered factors from one initialization to the next ? It could be interesting to look at the factors directly to characterize dynamics, but I think the discovered dynamics could be roughly preserved despite different A^h?\n\nFor the NASCAR and Lorenz experiments, the factors are very stable (up to permutations and rotations) between runs.  For the worm and mice experiments, they are slightly more variable.  We have added a brief discussion of this, explicitly highlighting the equivalence classes and how these can be examined/identified.  \n\n## R.E. Limitations:\n\n1.  **Handling alternative modalities:** See General Response.  SALT could be easily applied to handle, for instance, image embeddings generated by a deep autoencoder.  This is a very interesting direction for future research!  \n\n2.  **Links between SALT and SLDS around switches:**  We attempted to analyze both SALT and SLDS around switches.  However, the analysis rapidly became intractable.  Practically speaking however, we only require that SALT models produce “similar” switching behavior.  Further understanding the theoretical difference between the two methods is an interesting and challenging opportunity for follow-up work.  \n\n\n**Thank you again for your response.**  If you have further questions, we are happy to answer them! \n\n--- The SALT authors.  \n"}]

Author rebuttal:
Thank you to all four reviewers for taking the time to read our submission and provide insightful and constructive feedback. We presented Switching Autoregressive Low-Rank Tensor (SALT) Models, which combine the benefits of ARHMMs and SLDS models (such as parameter efficiency, fast exact inference and interpretability), while ameliorating the drawbacks of each method individually. We compared SALT to similar models across a range of problems.  

Here we respond to two themes that were touched on by multiple reviews: clarity and experimental utility. We then respond to individual reviews in more detail below each review.  

## (A) Clarity

Reviewers 4hZ1 and sUaQ commended the clarity of our submission. However, there were comments on the complexity of the notation, the explanation of hyperparameter tuning, and the discussion of limitations.  

**R.E. Notation:** We generally followed the tensor notation of Kolda and Bader [17]. However, we agree that tensor notation can sometimes be a lot to process. We have simplified the notation and used summation operators in several places. We have also added explicit definitions of tensor operations (with diagrams!).  

**R.E. Hyperparameter tuning:** We conducted extensive hyperparameter tuning for all models using grid searches, and have included extensive details. Generally, SALT models are fairly robust to hyperparameter settings and across seeds. ARHMMs are very robust across seeds for good hyperparameters, but more are more susceptible to bad hyperparameter settings (resulting in very poor and more variable performance). The best SALT models use similar dimensions commensurate to the best SLDS models. We include exact dimensions in the additional PDF.  

We also note that the number of hyperparameters in SALT is actually notably lower than many modern “deep” or kernel-based methods. Closed form updates alleviate the need for tuning learning rates; there are only three architectural dimensions that need tuning (compared to many layers widths, activations etc for deep methods); and exact inference removes any hyperparameters required by approximate or variational inference methodologies. Therefore, we assert that SALT actually has *fewer* parameters to tune than comparable models.  

**R.E. Limitations:** We have added extensive discussion of the limitations of SALT, including:
- Model mismatch: SALT is a simpler model family compared to (eg.) DSARF. This simplicity accelerates optimization and reduces overfitting, but may increase the bias in predictions if the SALT model used is poorly tuned. 
- Missing observations: SALT cannot natively handle missing observations. We considered using linear interpolation to bootstrap SALT, but believe that a more principled method for handling missing observations should be possible because of the nature of SALT models.  
- Bayesian treatment: It is not currently possible to share information between time series with different dimensional observations.  “Hierarchical SALT” is a possible extension to tackle this.

**R.E. Ethics:** We have added a short discussion outlining that there are no new ethical concerns as a result of SALT.  

## (B) Experimental Utility

Several reviewers commented on the baselines we compared to. We note that a hybrid continuous-discrete representation of the data is experimentally valuable. For instance, MoSeq [Wiltschko et al, 2015] uses ARHMMs to segment mouse behavior into discrete labels and a continuous state. The requirement precludes many common models (transformers, GPs, RKNs etc) which lack a discrete component. Our objective was not to create a state-of-the-art regression model, but rather explore the space between two widely used models that offer this description. 

We highlight that inference in SLDSs is not a “solved” problem and an active area of research. Numerous inference approaches exist, often requiring approximate or hard-to-tune inference schemes. Recent examples include Laplace-EM [Zoltowski et al, 2020, ICML] and linear programming [Berger et al, 2022, NeurIPS]. Developing new SLDS model variants therefore requires a deep understanding of the associated inference techniques. In contrast, new SALT models can be derived by simply updating the update equations (which can always fall back to coordinate gradient descent if a closed-form update cannot be derived).  SALT can therefore tap easily into the rich existing literature on tensor regressions, such as enforcing sparsity [Shah et al, 2015, NeurIPS] or non-negativity [Shashua and Hazan, 2005, ICML], in a way that SLDS cannot.    

Similarly, to our knowledge, there is no clear consensus on the best way to regularize vector autoregressive (hidden Markov) models; several possibilities exist, see, eg, Melnyk and Banerjee [2016, ICML] or Ni and Sun [2005, ASA]. Many regularizers and priors are difficult to work with, and are not widely used in practice. Beyond this, even well-regularized ARHMMs do not natively capture interpretable low-dimensional dynamics, as both SALT and SLDS models do (see Figure 3). These low-dimensional dynamics are experimentally as useful as the discrete segmentation.  

As also highlighted above, SALT has fewer hyperparameters than other methods, when methods are considered in their entirety. Therefore, SALT models combine the benefits and ameliorate the weaknesses of both ARHMMs and SLDSs, and provide an accessible, extensible, interpretable and performant alternative that can be easily tuned and deployed by practitioners.  

Finally, we have also compared SALT to DSARF (suggested by 6MVR) on the apnea task included in Farnoosh et al [2021] and have added these results and discussion. SALT matches or exceeds the performance of DSARF in terms of the NRMSE% on held-out test data and the quality of the segmentation (see attached PDF). 

**Again, thank you for taking the time to review our paper**. If there are any further questions, please do not hesitate to ask! 

--- The SALT authors

Meta-review: This paper describes an extension to switched linear dynamical systems where rather than a 1-step AR model, the switched system is an N-tap model. The method, named SALT, switches a low-rank tensor representing the system. The method is leverages low CP- or Tucker-rank representations of the AR-P model, and provides efficient fitting procedures as well as comparisons to toy data (similar to prior work in SLDS), mouse behavior data, and C. elegans data.

Overall the reviewers were generally positive about the contributions. Reviewer BTJL however brings up the point that the general model is fairly incremental to past work in ARHMMs, and questioned if the modeling and supporting experiments support the method as a valuable contribution to the conference. The reviewer does make good points, especially that the main benefit in some of the experiments seems to primarily be in increased modeling efficiency rather than new insights. The authors responded by noting that often modeling choices, such as the ability to compare Tucker and CP with other ARHMM assumptions, as well as the release of efficient code for the community do serve as a contribution even if the model is not completely new. Given the discussion and especially the support of the other reviewers, I recommend this work for publication.
Decision: Accept (poster)