SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section we provide extended details for the experiments presented in the main text, including the hyperparameters selected and the hyperparameter tuning process.
H.1 List of Hyperparameters
Table 3 outlines the key hyperparameters we consider when comparing and selecting models. Details of the values for each of these hyperparameters are then specified in the sections afterwards.
Table 3: Outline of the key hyperparameters in the three main models families we consider: SALT, SLDS and ARHMM.
Hyperparameter	Description	Applicable to models	Permissible Values
Tensor factorization	Factorization structure of autoregressive tensor	SALT, ARHMM	{CP, Tucker, None}
Subspace type	Whether the output factors are shared across discrete states.	SALT, SLDS	{Single, Multi}
D, Tensor ranks	Dimension of the core tensor.	SALT	Z≥1
Assumed throughout that D1=D2=D3.
H, Number of discrete states	Number of discrete states in switching models.	SALT, SLDS, ARHMM	Z≥1
L, Number of lags	Number of previous observations in autoregressive models.	SALT, ARHMM	Z≥1
S, Latent space dimension	Dimension of continuous latent state.	SLDS	Z≥1
Temporal L2 penalty	L2 penalty applied lag parameters at longer lags.	SALT, ARHMM	α ∈ R≥0, β ∈ R≥1
The penalty strength is defined as α · βl-1where l is the lag.
Stickiness	Dirichlet prior that can be added to penalize discrete state switches.	SALT, SLDS, ARHMM	γ ∈ R>0, κ ∈ R≥0
Global L2 penalty	L2 penalty applied across all parameters. (See note below)	SALT, SLDS, ARHMM	R≥0
Unless otherwise specified, we performed a grid search over a range of values within the permissible set. In certain circumstances, the hyperparameter was selected to match known properties of the data, e.g. we used  $H = 4$  for the NASCAR data, because we know there are four underlying states in the data.
During pilot experiments we experimented with a global L2 penalty applied to all parameters in the model. We found that varying this parameter did not affect the key outcomes of each experiment. We therefore set the L2 penalty strength to the same value for all the models for all the experiments, with strength 0.0001, unless otherwise specified.
H.2 Experiment Specific Hyperparameters
Here we specify experiment-specific details.
H.2.1 Section 5.1: SALT Faithfully Approximates LDS
For a given LDS with latent dimension size of  $S$  and initialized with a random rotation matrix, we computed the estimated number of CP-SALT and Tucker-SALT ranks, denoted  $D_{\mathrm{CP}}^{*} = n + 3m$  and  $D_{\mathrm{Tucker}}^{*} = n + 2m$ , respectively, where  $n$  is the number of real eigenvalues and  $m$  is the number of complex conjugate pairs of  $\Gamma$  defined in Proposition 1. We then fitted CP-SALT models with  $\{\min(S, D_{\mathrm{CP}}^{*}) - 2, \ldots, \max(S, D_{\mathrm{CP}}^{*}) + 2\}$  and Tucker-SALT models with  $\{\min(S, D_{\mathrm{Tucker}}^{*}) - 2, \ldots, \max(S, D_{\mathrm{Tucker}}^{*}) + 2\}$ . We chose  $L = 50$ , as this was the horizon at which the autoregressive parameters were approximately zero. The temporal L2 penalty was set to 1.0.
H.2.2 Section 5.2: Synthetic Switching LDS Examples
Synthetic NASCAR dataset All models used the true number of discrete latent states,  $H = 4$ . We fitted single-subspace CP-SALT and Tucker-SALT models with ranks  $D \in \{1,2,3,4\}$ . For both SALT models and ARHMMs, we used  $L \in \{1,5,6,7,8,9,10,15,20\}$ . A temporal L2 penalty of 1.0 was used for CP-SALT and Tucker-SALT models. Single-subspace SLDSs were fitted with the true underlying latent dimension,  $S = 2$ .
Synthetic Lorenz attractor dataset All models used the approximate number of discrete latent states in the data,  $H = 2$  (following Linderman et al. [2017]). We fitted single-subspace CP-SALT and Tucker-SALT models with ranks  $D \in \{1,2,3,4\}$ . For both SALT models and ARHMMs, we used  $L \in \{1,5,10,15,20\}$ . A temporal L2 penalty of 2.0 was used for CP-SALT and Tucker-SALT models. Single-subspace SLDSs were fitted with the true underlying latent dimension,  $S = 3$ .
H.2.3 Section 5.3: Modeling Mouse Behavior
We fitted multi-subspace CP-SALT models with ranks  $D \in \{10, 11, 12, 13, 14\}$ . For both CP-SALT models and ARHMMs, we used  $L \in \{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20\}$  and the number of discrete states was set to  $H = 50$  (51 behavioral states explained  $95\%$  of videos Wiltschko et al. [2015]). We use a temporal L2 penalty of 1.0. Similar to Wiltschko et al. [2015], we imposed stickiness on the discrete state transition matrix of both SALT models and ARHMMs via a Dirichlet prior. For discrete state  $h$ , the concentration parameters  $\nu \in \mathbb{R}_{>0}^{H}$  of the Dirichlet prior is  $\gamma$  for  $\nu_{i}$ ,  $i \neq h$ , and  $\gamma + \kappa$  for  $\nu_{h}$ . For this experiment,  $\gamma$  was set to 1.1 and  $\kappa$  to  $6 \times 10^{4}$ , which were empirically chosen such that the duration of the inferred discrete states and the given labels were comparable.
H.2.4 Section 5.4: Modeling C. elegans Neural Data
Following Linderman et al. [2019], we empirically searched for sets of hyperparameters that achieve  $\sim 90\%$  explained variance on a held-out test dataset. We fitted both single and multi-subspace CP-SALT models with ranks  $D \in \{8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18\}$ . Similarly, SLDSs were fitted with the same range of latent dimension size. For both CP-SALT models and ARHMMs, we used  $L \in \{1, 3, 6, 9\}$  and the number of discrete states was set to  $H = 7$ , which is the number of unique manual labels. After testing multiple lag values, we selected  $L = 9$  for SALT models and ARHMMs, as these longer lags allow us to examine longer-timescale interactions and produced better segmentation across models, with only a small reduction in variance explained. For ARHMMs, we set the global L2 penalty strength to 20.0 with the temporal L2 penalty set to 1.0. For SLDSs, we set the global L2 penalty strength to 100.0. For SALT models, we set the global L2 penalty strength to 0.2 and the temporal L2 penalty to 1.1. We additionally imposed a smoothness penalty to the weights of the lag factors of SALT models by adding  $\eta ||\mathbf{FW}||_2^2$  to the NLL for optimizing the lag factors, where  $\eta$  is the smoothness penalty strength and  $\mathbf{F}$  is the first difference matrix. We set  $\eta = 0.8$  for SALT models.