Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs
Authors: Julia C Costacurta
Lea Duncker
Blue Sheffer
Winthrop Gillis
Caleb Weinreb
Jeffrey Evan Markowitz
Sandeep R. Datta
Alex H Williams
Scott Linderman
Venue: NeurIPS 2022 Accept (2022)
Keywords: naturalistic behavior
clustering
markov models
time series

Abstract: A core goal in systems neuroscience and neuroethology is to understand how neural circuits generate naturalistic behavior. One foundational idea is that complex naturalistic behavior may be composed of sequences of stereotyped behavioral syllables, which combine to generate rich sequences of actions. To investigate this, a common approach is to use autoregressive hidden Markov models (ARHMMs) to segment video into discrete behavioral syllables. While these approaches have been successful in extracting syllables that are interpretable, they fail to account for other forms of behavioral variability, such as differences in speed, which may be better described as continuous in nature. To overcome these limitations, we introduce a class of warped ARHMMs (WARHMM). As is the case in the ARHMM, behavior is modeled as a mixture of autoregressive dynamics. However, the dynamics under each discrete latent state (i.e. each behavioral syllable) are additionally modulated by a continuous latent ``warping variable.'' We present two versions of warped ARHMM in which the warping variable affects the dynamics of each syllable either linearly or nonlinearly. Using depth-camera recordings of freely moving mice, we demonstrate that the failure of ARHMMs to account for continuous behavioral variability results in duplicate cluster assignments. WARHMM achieves similar performance to the standard ARHMM while using fewer behavioral syllables. Further analysis of behavioral measurements in mice demonstrates that WARHMM identifies structure relating to response vigor. 

Distinguishing discrete and continuous behavioral variability using warped autoregressive HMMs
Julia C. Costacurta
Lea Duneker
Stanford University
Stanford University
j costac@@tanfford.. edu
1duncker@stanford.co
Blue Sheffer
Winthrop Gillis Harvard Medical School
Caleb Weinreb Harvard Medical School
Stanford University
Jeffrey E. Markowitz
Sandeep R. Datta
Georgia Institute of Technology. Emory University
Harvard Medical School
Alex H. Williams New York University, Flatiron Institute alex. h. wwwwwilllae wwwwwiammssyuu.ee edu edu
Scott W. Linderman
Stanford University
scott www.::iindemmmntstee.mm waw.ord.dddd..o.. edu
Abstract
A core goal in systems neuroscience and neurocthology is to understand how neural circuits generate naturalistic behavior. One foundational idea is that complex naturalistic behavior may be composed of sequences of stereotyped behavioral syllables, which combine to generate rich sequences of actions. To investigate this, a common approach is to use autoregressive hidden Markov models (ARHMMs) to segment video into discrete behavioral syllables. While these approaches have been successful in extracting syllables that are interpretable, they fail account for other forms of behavioral variability, such as differences speed, which may be better described continuous in nature. To overcome these limitations, we introduce a class warped ARHMMs (WARHMM). As is the case in the ARHMM, behavior is modeled as mixture of autoregressive dynamics. However, the dynamics under each discrete latent state (i.c. each behavioral syllable) are additionally modulated by continuous latent "warping variable." We present two versions of warped ARHMM in which the warping variable affects the dynamics of each syllable either linearly nonlinearly. Using depth-camera recordings of freely moving mice, we demonstrate that the failure of ARHMMs to account for continuous behavioral variability results in duplicate cluster assignments. WARHMM achieves similar performance to the standard ARHMM while using fewer behavioral syllables. Further analysis of behavioral measurements in mice demonstrates that WARHMM identifies structure relating to response vigor.
Introduction
fundamental question in systems neuroscience is how neural activity generates complex behavior [1- 3]. Specifically, a key goal is to understand how changes in neural activity determine which actions selected executed essential to study ethologically relevant, naturalistic behavior. A common way of studying naturalistic behavior is observe animals as they freely explore an environment [4-6]. Such unconstrained,
moment-by-monen
basis. To make progress towards this goal, is
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A.
Frame
B.
Segmented Time Series C.
AR Dynamics
D. Syllables
Top 10 PCs
PC2
time [s]
5
PC
Figure 1: A. Single frame of depth video of a freely behaving mouse. B. The frames are projected onto their top ten principal components (black lines) and then segmented into discrete syllables (colors) with an autoregressive HMM (ARHMM). C. Each syllable is defined by autoregressive (AR) dynamics, which can be visualized as a flow field in PCA space. Here, the scatter plot shows the distribution of frames projected onto the top two PCs, and the red-to-blue trajectories correspond to multiple instances of one discrete state. D. Empirically, syllables correspond interpretable behaviors like investigating, rearing, and falling from rear, as illustrated here.
spontancous behavior offers a rich setting for studying behavioral variability. However, the hours of raw video data required to capture naturalistic behavior in detail are high-dimensional and difficult to use for follow-up analyses. Thus, a key goal in behavioral neuroscience research is to develop data-analysis strategies that can extract interpretable lower-dimensional summaries of behavior [4, 5, 7, 7-10]. Ultimately, such descriptions will facilitate relating complex naturalistic behaviors to the underlying neural activity patterns that generate them [3, 1 11] This will provide insight into how and why behavior may differ across environmental contexts [2, 3, 11, 12], under pharmacological manipulations [6] or across health and disease [13, 14).
One hypothesis is that the brain generates complex behaviors by concatenating a series of simpler, stereotyped actions [4, 5, 11, 15, 16]. Just like syllables form the building blocks of spoken language, behavioral syllables may be composed to perform complex sequences of behavior. A large focus of previous work has been to discover such behavioral syllables in an unsupervised manner, thereby obtaining low-dimensional description of high-dimensional behavior [4, 5, 12, 16-19].
Autoregressive Hidden Markov Models (ARHMMs) are well-suited to this task [5, 6]. For example, Wiltschko et al. [5] used depth video to capture the posture of freely moving mice (Fig. IA). In this work, authors projected the video frames onto the top D principal components (PCs) to obtain a D-dimensional time series of behavior. The authors then used an ARHMM to segment the behavioral time series into discrete syllables (Fig. IB). Each discrete syllable corresponds to vector autoregressive dynamics in PC space, which can be conceptualized as a vector field (Fig. IC). Each instance of a syllable corresponds to a short, stereotyped trajectory following this vector field (red-to-bluc trajectories in Fig. IC). Empirically, these often correspond to stereotyped patterns of movement like rearing, darting, or grooming (Fig. ID).
However, current modeling approaches have focused on clustering discrete behavioral syllables while failing to account for other, continuous forms of behavioral variability. For example, the same type of behavior (e.g. dart) could be performed more or less vigorously. While these two actions might be identified by a human observer as the same behavior under different speeds (fast vs. slow darts), current models lack the ability to allow for such structured continuous variability within states, and thus might assign these actions to entirely separate behavioral states. As a result, current approaches often ever-segment video data, by allocating distinct clusters to the same movement type. The data might be better described by a model which incorporates a continuous spectrum of structured variability within a specific behavior.
We extend the ARHMM by incorporating a latent warping variable. With this Warped Autoregressive Hidden Markov Model (WARHMM), we are able to capture continuous variability within a discrete syllable. Thus, we are able to disentangle variability due to (discrete) movement type from other forms of (continuous) variability, such as movement speed. We consider two types of warping: warping which captures changes in the speed of evolution of the autoregressive dynamies, and Gaussian process warping, which allows for nonlinear changes in dynamics. We develop an

---

efficient inference algorithm for the WARHMM and show that it can correctly identify the underlying states and parameters simulated data. Then, using behavioral measurements of freely behaving mice, we demonstrate that accounting for continuous sources of behavioral variability can resolve issues of over-segmentation commonly observed using previous approaches. Furthermore, using behavioral data of mice treated with either saline solution or amphetamine, we demonstrate that the WARHMM identifies differences the distribution of latent warping variables across both groups of mice. This result reflects potential modulations of movement vigor and speed distributions due to the pharmacological intervention.
2
Background
We first review related work and then present a brief description of two classes of Hidden Markov Models that have been used for time series segmentation and behavioral modeling. The key features of each model form the basis for our warped extension to the ARHMM model, which represents the main contribution of our work.
Related work in unsupervised behavioral segmentation. We build upon rich body of work in the unsupervised behavioral segmentation space. Our model is inspired by the ARHMM approach to unsupervised behavioral segmentation (MaSeq) proposed by Wiltschko et al. [5], which is described in the introduction. Berman et al. [4] addressed similar task fruit fly video using MotionMapper, which identifies discrete behavioral states as peaks in non-linear, two-dimensional embedding of postural speetrograms. Hsu and Yttri [17] have proposed B-SOiD, which uses a random forest to classify non-linear postural feature embedding clusters into multiple behavioral classes. Harris et al. [18] fit an autoregressive linear model to timewwindowed postural features using low-rank tensor decomposition, and interpret clusters in the fit model parameters as discrete behavioral states. Luxem et al. [19] identify behavioral states by clustering the latent vectors produced by training a variational autoencoder input from markerless pose estimation [9]. For
thorough treatment of work in unsupervised behavioral quantification, see McCullough and Goodhill [20] for a recent review, The majority of these related approaches have focused on segmenting behavioral video based movement type or on tracking animal pose. The model we introduce section 3 extends on this work by explicitly taking other forms of behavioral variability, such movement speed, into account.
Autoregressive Hidden Markov Models. An ARHMM (Fig. 2A, top) consists of a discrete latent state variable 2, {1, ..., K} and observations x, E RRD. Transitions between the discrete state values over time are governed by a transition matrix P. where the entry PA, indicates the probability of advancing from state 24 = k to state 2++1 = k'. Given the discrete state value, the observations modeled to evolve according linear dynamics, where dynamics matrix Ag, and bias term b determine the mapping from x; to X;+1 in the presence of Gaussian noise with covariance matrix Q.,- Thus, the model represents an extension of classic Gaussian mixture models to time series with autoregressive dynamics. The ARHMM model can be summarized as
2+11 24 = Cat( PR.:)
KE (L.....K))
Xx+1 Xx, रτ = k N' (x: + A&X, + bb. Q+)
x; ER
In the context of behavioral modeling [5], the discrete latent state reflects the behavioral syllable (such as a dart, leftward turn, rear, etc.) and determines which dynamics are used describe the temporal evolution of the observed posture
Factorial Hidden Markoy Models. In Factorial Hidden Markov Models (FHMMs) [21], multiple discrete hidden state variables influence the distribution of observed states. The FHMM model can be summarized as
2+41 2 ks ~ A--Cat(P
(3)
Cat(Pl
KE f1,....,
K()), 1= ..M
(2)
{3) )
(40 p(xe (2, (
R
FHMMs are useful in cases when aspects of data can be distributed across multiple states. For example, to fit data that varies according to M binary variables, standard HMM would need 2M discrete states, while FHMM would only need M binary states [21].
3

---

A.
ARHMM
B.
Ax - -1)
Ax(T =0) 0)
Ax(T = +1)
discrete states
21
#2
Time- Warped ARHMM
PCA
embedding
I1
22
23
Warped ARHMM
warping variables
72
T3
GP-Warped ARHMM
discrete states
11
22
23
PCA embedding
T1
D3
Figure 2: A. Probabilistic graphical models for the ARHMM and the Warped ARHMM. The warping variables and discrete states (i.e. syllables) together determine the AR dynamics. B. We consider two types of warping: warping where dynamics are sped up or slowed down by the warping variable, and GP warping, where the dynamics vary smoothly but nonlinearly with the warping variable.
In the context of behavior, adding additional latent variables is appealing way provide structured variability within discrete syllables. Assuming this variability included in a reasonable way, an FHMM has the potential to describe dataset with fewer discrete latent states than an HMM.
3 Warped Autoregressive Hidden Markov Models
ARHMMS have been successful in clustering video measurements of behavior into discrete sets of behavioral syllables. However, in practice they are prone to over-segmentatiinn, where behaviors that appear similar to a human expert are split into distinct clusters. We hypothesize that oversegmentation arises because the ARHMM conflates discrete sources of behavioral variability the expression of distinct behavioral syllables - with continuous sources of variability that cannot be captured by linear autoregressive dynamics with Gaussian noise.
To address this limitation, we develop Warped Autoregressive Hidden Markov Models (WARHMMs). WARHMMs extend the ARHMM with additional latent variable, like in Factorial HMM. In addition to the discrete state variable 2t. WARHMM includes a latent warping state тą E [-1, 1] at each time step (Fig. 2A, bottom). The warping state modulates the dynamics associated with each state 2t. While 2n can model rapid switches in dynamics, Tr can account for additional variability in the dynamics associated with given latent state.
The general form of this model class can be summarized as follows, where Au (Tr), by (T.), and Qu(T.) are functions of Tz:
2+41 Zt ki Cat PR.))
E (1,..,, K}
T++1 Tt ~ p(F(+1|T))
T T [-1,]]
(3)
X++1 X6, क k, Tt NN(x + As(T)) X + bx(r), Qx))))
x; E R
Below, consider two possibilities for how exactly the warping variable modulates the dynamics.
3.1 Time-Warped ARHMM: linear modulation of autoregressive dynamics
Our first specific instance of WARHMM has direct motivation in terms of time-warping, and thus refer to it as lime-warped ARHMM (T- (T-WARHMM), Specifically, we aim capture continuous changes in how quickly trajectories move through observation space. consider the change in X++1 due to the dynamics within given state 2 k and let 7 be log
current state Ax; step-size parameter, we can write
C-7 Ax; Axx; + bj Q
4 ~~011
(4)
X x, CT + ba) + Q
(5)

---

When r = 0 the step size C" is always one, so equation (5) is equivalent to the classic ARHMM. For nonzero т, however, the warping variable modulates how far a single update can move the state, akin to a time-constant in ordinary differential equations. The constant C' determines the maximum multiplicative factor by which dynamics can be scaled; in our experiments we set = 2. In terms of the functional mapping, T-WARHMM corresponds to the following mapping between the warping state Tr and the parameters determining the autoregressive dynamics of the observed states X;:
(T) Asu (T,( C'.
(T. C Q=:
(6)
Fig. 2B (top) shows how an example syllable's dynamies are sped up or slowed down by changing т. As increases, the trajectories traverse a greater distance in the same number of time steps. In the context of behavioral modeling, the time-warping variable may capture modulations in movement speed. As we will see in section 5, the inferred warping variables correlate with other intuitive notions of vigor, like centroid velocity in darting syllables, while also offering a quantification of vigor in stationary syllables such as grooming.
3.2 Gaussian Process- Wireess-WAAHMMM Nonlinear modulation of autoregressive dynamics
The model formulation for T-WARHMM interpetable and intuitive, but also makes strong para- metric assumptions about how continuous variability affects the dynamics of behavior. As a point of comparison, we consider a more flexible model, in which the influence of T. via Ag, (T;) and b., (T:) is specified nonparametrically in terms of Gaussian processes. This Gaussian Process WARHMM (GP-ARRMM) moves beyond using Tr to describe speed or vigor, as 7; could have effects beyond modulation of time-constants of the dynamics. Here Ti can modulate entries of the dynamics matrix according to a smooth nonparametric function. In particular, the dynamics are modeled as follows:
ARtz (T) GP(O, (o(T,
j ,..., D; k
K
(T)(T) (o
j j= 1. D; k 1. K
(T(T) Qk:
h = 1, 1.....,.. K.
where (T))(T denotes the ji, entry of the matrix (T) and (T) is the i-th entry of the vector (T), each modeled as a Gaussian Process with mean zero and covariance function Ko(T, r'). Thus, each coordinate varies smoothly and independently as a function of r, a priori. The prior mean Akij by = 0 corresponds to a pause with the state x, remaining stationary in expectation. In our experiments, we choose a squared exponential kernel with kernel hyperparameters 0 = (p,o).
Kø(T, 7') exp {
20 ( - -1)
(8)
Fig. 2B (bottom) shows how an example syllable's dynamics are modulated by changing r a GP-WA U.-WARHMM. The central dynamics (A₃(T 0)) are the same in the T-WARHMM above, but now T can have nonlinear effects on the dynamics that do more than simply slow them down or speed them up. This extra flexibility could be useful for capturing more general types of continuous variability within discrete syllables, but it may also come at the cost of interpretability.
3.3 Inference and Learning
Though we have presented T, E [-1,1] as continuous latent variable, in practice we find it sufficient to discretize Tr over a fine grid of evenly spaced points, since it is a one-dimensional, bounded random variable. Once discretized, we model the warping variable dynamics as a Markov process with a banded transition matrix to encourage small, local changes over time. Similarly, the GP prior in eq. (7) reduces to a multivariate normal after discretization. Full details are provided in the Supplementary Material.
As in standard ARHMMs, we estimate the states and parameters with the Expectation -axiattioo--aiiizzatiii algorithm (EM) [22]. We perform exact inference over the discrete syllables and warping variables using the forward-backward memory. For both the time-warped and GP-warped ARHMMs, the parameter updates have closed- form solutions. For large datasets, we used stochastic EM [23] to speed convergence. For the GP- WARMM, we learned the kernel hyperparameters through optimization of the variational lower bound. Again, complete details are in the Supplementary Material.
algorithm, which runs in ( + J2)) time and uses O JJ)
5

---

A. 2" 0.75
2 1.0
2"
1.25 C
2=1
2
0.4
true
IN
Aa
for increasing
0044
E.
true
B.
tw
33
-2
gp
2
2" 1.25
time
time
Figure 3: A. Illustration of the dynamics of the simulated example. Two discrete latent states represent bottom) and counterclockwise (= 1, top) rotation, each shown as vector fields. The speed of the rotation is modulated via different values of т, where (T 2T Ar. B. Illustration of one instance of the 2-dimensional state trajectory (t), (t)) the generative dynamics are modulated by both state switches in z (discrete variability, blue versus pink) and changes in rotation angle via (continuous variability, shading of color bar). C. The generative values for 2 matrices (T) = 2 A: (top row) together with the learned matrices for T-WARHMM (tw, middle) and GP. WARHMM (gp, bottom). T-WARHMM is able to learn the variations in the autoregressive dynamics. GP-TWARHMA
a clockwise
learns the overall rotation structure of this example, but the increased flexibility of the GP model makes it harder for it to disentangle discrete and continuous contributions to changes in (T). D.-E. The true and inferred latent paths (posterior mode) for ž (D) and T (E) under each model.
Synthetic data validation
We begin by generating two-dimensional synthetic data x, R22 with K = 2 discrete states. The autoregressive dynamics under each discrete state are chosen to be a clockwise and counterelockwise rotation. We modulated the dynamics with a time-warped ARHMM, setting A&(T) 2 AA. as illustrated in Fig. 3A. For this simple experiment, we limited the true time-warping variables to values evenly spaced on [-1, 1]. snippet of simulated data and the corresponding time constants and states is shown in Fig. 3B. We fit GP-WARHMM and T- T-WARHMM with K = states and 5 warping variables data simulated from this model, and additionally compare results in terms of log-likelihood performance to classic ARHMM with K 2, 4, 10 discrete states.
The results from this experiment are summarized in Fig. 3C-E and Table 1. Panel shows the true and learned values of entries of the autore- gressive dynamics matrix A,(T) as both z and vary. We see that both the T-WARHMM and GP- WARHMM learn dynamics that reflect the overall rotational structure of the matrix. However, only T-WARHMM is able to recover the clockwise and counterclockwise variation with and correct speed modulation with T.
Model
Train LL
Test LL
T-WARHMM
0.035
0.066
GP- WARHMM
-0.018
0.024
2-state ARHMM
-0.192
-0.155
4-state ARHMM
-0.082
-0.030
10-state ARHMM
0.007
0.037
True Model
0.048
0.077
Comparison of test log likelihoods on simulated data across true and fitted mod- els. Higher test LL overall is due to random sampling train/test data.
Table 1:
The increased flexibility of GG--WRRHMM. allows it to achieve test log likelihood which outperforms the K = 2, 4 state ARHMMs (Table 1), without correctly disentangling discrete and continuous contributions to the observed data variability, Fig. 3D and E show the true state changes in and т., together with the mode of the inferred posterior distributions under each model. T-WARHMM is able to recover the correct segmentation of the data, while GP-WA---- WARHMM's increased flexibility allows it to fit the data, but in a less interpretable way.

---

A.
B.
ARHMM
-1.25
Test log likelihood
-1.50
syllables identified
5
-1.75
0
-2.00
T-WARHMM
ARHMM
15
-2.25
T-WARHMM
-2.50
GP-WARHMM
5
10
20
30
40
50
60
Behavioral syllables
0
dart groom rear
Figure A. Comparison of logeliikl on held-out test data function of the number of discrete latent states after training the different model classes on the MoSeq dataset. B. For an ARHMM and T-WARHMM with similar test log- likeliihodods total number of syllables that are grouped into the same behavior class by expert neuroethologists. The ARHMM creates new syllables to model qualitatively similar behaviors.
5 Modeling depth video of freely behaving mice
For the remainder of the paper, we are interested in extracting discrete and continuous structure in behavior from posture data x, extracted from depth-imaging recordings of freely behaving mice. To do this, we reanalyze data from Wiltschko al. [5], which represents the original application of the ARHMM to clustering mouse behavior. In the context of this dataset, the ARHMM approach is often also referred to as MoSeq, and we thus refer to the dataset as the MoSeq dataset.
In the MoSeq dataset, the observations are taken to be the first 10 principal components of depth camera video data of mice exploring an open field. The dataset consists of 20-minute depth camera recordings of 24 mice. In preprocessing. the videos are cropped and centered around the mouse centroid, and then filtered remove recording artifacts. Finally, the preprocessed video projected onto the top principal components to obtain a 10-dimensional time series.
5.1 Comparison of model performances on the MoSeq dataset
To provide a direct comparison of performance between the ARHMM. T-WARHMM. and GP- WARHMM, we trained each model using 50 epochs of stochastic EM. The ARHMM is equivalent to setting the number of v-vllues in T-WARHMM J T-WARHMM and GP-WARHMM each had / = 31 evenly spaced values of 7 on the interval [-1, [-1,1] 1].
Fig. 4A shows the log-likelihood of each model held-out test data for range of K values. For given value K, both T-WARHMM and GP-WARMMM outperform the classic ARHMM in terms of their generalization performance log-likelihoods for an ARHMM with K = 40 syllables, a T-WARHMM with K = 25 syllables, and GP-WARRMMM with K 10 syllables. This further illustrates that the ARHMM has to account for continuous variability by creating a larger number of discrete states, while the factorial structure the WARHMMs enables syllables be merged when the difference in their dynamics can be explained by changes in 7. The GP- HI-WWARHMM outperforms the T-WARHMM in terms of test fog-likelihood. which can be attributed to the fact that the GP-WARMMM is more flexible than the T-WARHMM and can capture additional structure in the data. However, the GP-WARHMM accomplishes this at the cost of interpretability. The T- WARHMM warping constant physiologically motivated and, as we will show in the later results, has direct correlations to centroid velocity and other notions of vigor. While the GP- MA--WAHMMM warping variable is allowed to vary the syllable whichever way will increase data log-liken likelihoooo the T-WARHMM is restricted to modulating syllables in way that is biologically interpretable. believe that this interpretability of T-WARHMM makes more useful for bchavioral analysis purposes and thus focus on the utility of T-WARHMM in the remainder.
unseen data. However, we achieve similar test

---

rear
groom
dart
3
3
8
Centroid velocity
6
2
4
2
0
0
0
0.5
2 0.5
0.5
Vigor (21)
Figure 5: Centroid velocity (pixels/ms) vs. inferred vigor (2") for three representative states of the TW-ARHMM, showing a clear positive correlation between the variables.
5.2 Interpreting the time-warping variable
Which syllables are "warped'?? To begin our analysis of the T-WARHMM results, we a model with K = 20 discrete states and J = 31 time constants. As shown in Fig. 4A, this model attains similar test likeliiil an ARHMM with K 35 discrete states. We then determine which types of syllables are over-represented as multiple distinct discrete states in the ARHMM by analyzing the syllable labels from each model. After generating sample videos of behavior from both models, we asked expert neuroethologists and MoSeq users to label the syllables produced. We then gathered the labels under three general behaviors: darts, grooms/pauses, and rears. The results are shown in Fig. 4B. Both models have similar numbers of darting states, while the numbers of grooms and rears are reduced T-WARHMM, with T-WARHMM halving the number of rear states. While T-WARHMM does not completely resolve the oversegmentation issue, it is able to perform well as the ARHMM with fewer syllables. More details and video examples of behaviors identified by T-WARHMM are provided in the Supplementary Material.
Connections to centroid velocity. From our formulation of the time warping variable, it is expected that the time warping variable and centroid velocity of the mouse would be highly correlated. In Fig. 5 we plot measured centroid velocity (in pixels/ms) vs. inferred vigor (2") for four representative states. We a strong relationship between centroid velocity and vigor for the darting state, where we would expect such a relationship to occur. The other states also show this relationship. but less consistently. In these states, the warping variable may be accounting for additional forms of timing- relate- variability that are not well accounted for by centroid speed. These analyses validate that the time warping parameter is able extract speed-related information also contained in centroid velocity for behavioral syllables such as darting motions.
5.3 Identifying drug-induced changes in behavior
Finally, we analyze video recordings of two groups of mice treated with either saline solution or amphetamine. As shown in Fig. 6A, the distribution of centroid speed varies slightly between the groups, with the amphetamine--reeted mice performing actions at higher speeds more frequently than the saline mice. We fit T-WARHMM with J 31 time constants and K 20 syllables to both sets of data. Fig. 6B shows the the inferred vigor (2") distributions from two representative syllables, demonstrating that there is clear rightward shift between amphetamine and saline treated mice. This indicates that amphetamine mice use faster time warping variables more frequently than saline mice. The difference between means of the index distributions a significant difference (p < 0.05, independent t-test) between the means of the r distributions for all except one of the inferred syllables. Thus, T-WARHMM is able to detect and dynamically track drug-induced differences in behavior across both groups of mice.
shown in Fig. 6C. Our model shows
6 Conclusion
We have introduced an approach for unsupervised behavioral modeling which allows for the iden- tification of continuous variability within discrete behavioral syllables. Our work builds prior state-of-the-art technique, which uses a discrete hidden state in combination with autoregressive linear dynamics to cluster behavior. In our proposed model extension (WARHMM), underlying the

---

A.
B.
distribution of centroid speed
State: Rear
State: Dart
proportion of time in cach vigor
0.5
saline amphetamine
0.10
0.4
0.05
Density
0.3
0.00
0.2
0.5
2
Vigor (2*)
C.
0.1
22.5
iinex
0.0
20.0
10
20
3D centroid speed (pixels)
17.5
5
Syllable
Figure A. The distribution of centroid speed across mice treated with saline (blue) or amphetamines (pink). B. Histograms showing the vigor distributions across the two classes of mice treated with saline (blue) or amphetamines (pink) for two example syllables (discrete states). The dashed lines indicate the mean of distributions, showing clear increase in mean vigor in the amphetamine treated group. C. Mean index for each of the 20 discrete states the model for both the saline (blue) and amphetamine (pink) treated mice. The inferred indices in the amphetamine treated group are larger (higher vigor, faster movement speed) for all but one syllable.
behavioral measurement at each time point both discrete state variable and a warping variable. Both of these latent variables modulate the dynamics of the syllable, either through discrete switches (via the discrete state variable) or through more continuous forms of modulation (via the warping variable). We found that WARHMM achieved similar performance in terms of log-likelihood on held test data while utilizing fewer discrete latent states than the ARHMM.
Our analyses on mouse behavioral data focused on an interpretable subclass of warped ARHMM, where the warping variable interacts linearly with the autoregressive dynamics of the observed data (the T-WARHMMI), Notably, the warping variable in this simple model can be viewed as implementing local rescaling of time. Similar time warping models have been fruitfully applied to trial-structured neural data [24, 25], and our work illustrates how these ideas can be extended to naturalistic time series data that lack repeated data illustrates that continuous variability in behavior may be induced through pharamacological interventions and can be extracted directly from video recordings.
Our work mouse behavioral
Limitations. For simplicity, discretized on fine grid instead of explicitly performing in- ference over the continuous variable. This was done in favor of computational efficiency and case of inference. While we believe the fine grid we've 'e used for offers good approximation of continuous function, future work could capture continuous variability by including truly continuous posterior, e.g. via Gaussian Process [24]. Furthermore, our model assumes that the behavioral PCs evolve according to linear dynamics, and though the GP-WARHMM allows these dynamics to be modulated nonlinear ways, the basic assumption could be relaxed to more general nonlinear dynamics in future work.
Unsupervised deep learning approaches exist address similar problems in the behavioral segmen- tation space, e.g. B-SOiD [17]. Another approach could involve including speed as an input and clustering features for these methods. However, since clusters are defined based on differences in input features, we predict that this approach would lead to further segmentation of syllables. In contrast, the motivation behind WARHMM is to collapse syllables together by allowing for structured variation within a single syllable.
Future work. This work establishes multiple directions for future research. The first involves further extending the ARHMM model by adding in additional forms of structured, interpretable variability, What aspects of behavior, in addition to speed, may vary within syllables? the context of mouse behavior, variables such turning radius depth-camera pixel height may provide additional

---

continuous axes along which behaviors can be modulated. Another future direction lies in using T-WARHMM analyze neural correlates behavior. Similar behavioral segmentation approaches have been used to simultaneously analyze the evolution of behavioral and neural recordings [8, 26- 28]. future research area of particular interest lies understanding how the dorsolateral striatum encodes action selection and vigor [13, 29-32]. Neural activity in this midbrain nucleus is known to be disrupted in motor diseases like Parkinson's ' and Huntington's 's [33, 34]. The T-WARHMM approach allows to simultancously extract descriptions of movement type (how frequently actions are selected) and movement speed (how vigorously actions are performed) from behavioral measurements. We expect that the ability to disentangle the contributions of these two factors could provide an important basis for better understanding how variations in neural activity determine changes in action selection and vigor. Finally, the models we have proposed here could be applied more widely to analyze other types of behavioral data.
Acknowledgments and Disclosure of Funding
We would like to thank Andy Warrington, Laura Driscoll, Libby Zhang, Jimmy Smith, Michael Salerno. Mohammed Osman, Sherry Lin, Akshay Jaggi. and our reviewers for their feedback and suggestions.
This work is supported by the Simons Collaboration on the Global Brain (SCGB 697092) and NIH BRAIN Initiative Grant 1U19NS1 13201-01. JCC is funded by the NSF GRFP and Stanford Graduate Fellowship. SRD is supported by NIH grants 119NS113201, RF1AG073625, ROINSI 14020, the Brain Research Foundation, and the Simons Collaboration by a Career Award at the Scientific Interface (CASI) from the Burroughs Wellcome Fund. WG is supported NIH grant F31NS1 13385. SWL is additionally supported by the Sloan Foundation.
the Global Brain. JM supported
10

---

References
[1] John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin, Malcolm A MacIver, and David Poeppel. Neuroscience needs behavior: correcting reductionist bias. Neuron, 93(3):880-45 2017.
[2] Sandeep Robert Datta, David Anderson, Kristin Branson, Pietro Perona, and Andrew Leifer. Computa- tional neurocthology: A call to action. Neuron, 104(1):11-22 24, 2021.
[3] Ann Kennedy. The what, how, and why of naturalistic behavior. Current Opinion in Neurobiology, 74, 2022.
[4] Gordon J Berman, Daniel M Choi, William Bialek, and Joshua W Shaevitz. Mapping the stereotyped behaviour of freely moving fruit flies. Journal of The Royal Society Interface, 11(99):20140777 2014.
[5] Alexander B Wiltschko, Matthew J Johnson, Giuliano Iurilli, Ralph E Peterson, Jesse M Katon, Stan L Pashkovski, Victoria E Abraira. Ryan P Adams, and Sandeep Robert Datta. Mapping sub-second structure in mouse behavior. Neuron, 88(6): 121-1135, 2015.
[6] Alexander Wiltschko, Tatsuya Tsukahara, Ayman Zeine, Rockwell Anyoha, Winthrop F Gillis, Jeffrey Markowitz, Ralph E Peterson, Jesse Katon, Matthew J Johnson, and Sandeep Robert Datta. Revealing the structure of pharmacohchavioral space through motion sequencing. Nature neuroscience, 23(11):
1433-1443, 2020.
[7] Talmo D Pereira, Diego E Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-11 Wang, Mala Murthy, and Joshua W Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods, 16(1):
117-125, 2019.
[8] Eleanor Batty. Matthew Whiteway, Shreya Saxena. Dan Biderman, Taiga Abe, Simon Musall, Winthrop Gillis, Jeffrey Markowitz, Anne Churchland, John P Cunningham, et al. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. Advances in Neural Information Processing Systems, 32, 2019.
[9] Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Wey- gandt Mathis, and Matthias Bethge. Deeplabeut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroxcience, 128111281-, 2018
[10] Jacob M Graving, Daniel Chae, Hemal Naik, Liang Li, Benjamin Koger. Blair Costelloe, and lain Couzin. Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife. 8:e47994, 2019.
[11] André EX Brown and Benjamin de Bivort. Ethology as a physical science. Nature Physics, 14:653 657,, 2018.
[12] Matthew R Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, E Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathanicl J Miska, Jean-Paul Nocl, al. Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders. PLoS computational biology, 17():: e1009439, 2021.
[13] Yael Niv, Nathaniel D Daw, Daphna Joel, and Peter Dayan. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmucology. 191(3):507-52) 2007.
[14] Pietro Mazzoni, Anna Hristova, and John W Krakauer. Why don't we move faster? parkinson's 's disease, movement vigor, and implicit motivation. Journal of neuroscience, 27(27):7105-111 2007.
[15] Nikolaas Tinbergen. The study instinct. Clarendon Press, 1951.
[16] Greg J Stephens, Lestie C Osborne, and William Bialck. Searching for simplicity in the analysis of neurons and behavior. Proceedings of the National Academy of Sciences, 108(Supplement 5):15665-15711 2011
[17] Alexander Hsu and Eric A Yttri. B-soid, open-source unsupervised algorithm for identification and fast prediction behaviors. Nature Communications, 12, 2021.
[18] Kameron Decker Harris, Aleksandr Aravkin, Rajesh Rao, and Bingni Wen Brunton. fime-varying autoregression with low-rank tensors. SIAM Journal on Applied Dynamical Systems, 20(4). 2021.
[19] Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stefan Remy, and Pavol Bauer. Iden- tifying behavioral structure from deep variational embeddings of animal motion. bioRxiv, 2022. doi: 10.1101/2020.05.1 14.095433.

---

[20] Michael И McCullough and Geoffrey J Goodhill. Unsupervised quantification of naturalistic animal behaviors for gaining insight into the brain. Current Opinion in Neurobiology, 70:89-100, 2021.
[21] Zoubin Ghahramani and Michael Jordan. Factorial hidden markov models. Advances in Neural Information Processing Systems, 8, 1995.
[22] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
[23] Olivier Cappé and Fric Moulines. On-line expectation-maaxiizziiin algorithm for latent data models. Journal the Royal Statistical Society: Series B (Statistical Methodology). 71(3):593-613, jun 2009. doi:
10.11 10.1111//114679910 000.98888820000000 x.0..006...x.
[24] Lea Duncker and Maneesh Sahani. Temporal alignment and latent gaussian process factor inference in population spike trains. S. Bengio. H. Wallach, Larochelle, K. Grauman, N. Cesa- Besa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[25] Alex Н Williams, Ben Poole, Niru Maheswaramathan, Ashesh K Dhawale, Tucker Fisher, Christopher D Wilson, David H Brann, Eric M Trautmann, Stephen Ryu, Roman Shusterman, Dmitry Rinberg, Bence P Ölveczky, Krishna V Shenoy, and Surya Ganguli. Discovering precise temporal pattems in Large-Scale Scale neural recordings through robust and interpretable time warping. Neuron, 105(2):246 259,c8, January 2020.
[26] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis, Learnable latent embeddings for joint behavioral and neural analysis, 2022.
[27] Joshua Vogelstein, Youngser Park, Tomoko Ohyama, Rex Kerr, James Truman, Carey E Preibe, and Marta Zlatic. Discovery of brainwide neural-behaving maps via multiscale unsupervised structure learning. Science, 344, 2014.
[28] Ryan A. York, Arnaldo Carreira-Rosario, Lisa M. Giocomo, and Thomas R. Clandinin. Flexible analysis of animal behavior via time-resolved manifold embedding, bioRxiv, 2021. doi: 10. 10.1101/2020.09.03 30....33040000
[29] Babita Panigrahi, Kathleen A. Martin, Yi Li, Austin R. Graves, Alison Vollmer, Lars Olson, Brett D. Mensh, Alla Y. Karpova, and Joshua T. Dudman. Dopamine is required for the neural representation and control of movement vigor. Cell, 1418)14411-130, 2015. ISSN 0092-8674, doi: https://doi.org/10.. 1016/ 1..00..011.08.0144.
[30] Joshua T Dudman and John W Krakauer. The basal ganglia: from motor commands the control of vigor. Current Opinion Neurobiology, 37:158-166, 2016. ISSN 0959-43888. doi: https://doi.org// 10.b...conb. 00.6002.005. Neurobiology of cognitive behavior.
[31] Allison E Hamilos, Giulia Spedicato, Ye Hong, Fangmiao Sun, Yulong Li, and John A Assad. Slowly evolving dopaminergic activity modulates the moment-to-momeet prohability of reward-related self-timed movements. el.ife, 10:e62583, dec 2021. ISSN 2050-084X. doi: (0.7554/cLiife.65555
[32] Jaeeon Lee, Wengang Wang, and Bernardo Sabatini. Anatomically segregated basal ganglia pathways allow parallel behavioral modulation. Nature Neuroscience, 23:1388-1398, 2020.
[33] Julie Lotharius and Patrik Brundin. Pathogenesis of parkinson's 's disease: dopamine, vesicles and a- synuclein. Nature Reviews Neuroscience, 3:932-942, 2002.
[34] Lactitia Schwab, Shaady Garas, Janelle Drouin-Ouellet, Sarah Mason, Simon Stout, and Roger Barker. Dopamine and huntington's 's disease. Expert Review of Neurothraapeuties, 15(4):445458, 2015. doi: 10.1115614777777 10055383. PMID: 25773746.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The limitations section is
contained in the discussion, on page 9.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Discussed in Supplemental Material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Discussed in Supplemental Material
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Discussed in Supplemental Material
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] Error bars are reported Fig. 5, and p- p-values are reported in section 5.3. Due to computational strain we did not report error bars for Fig. 4A.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Discussed in Supplemental Material
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our work uses an existing dataset from [5].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "vs2AiRV5ysw", "reviewer": "Reviewer_w5pw", "summary": "This paper develops a useful extension to the unsupervised behavioral syllable segmentation model space (specifically ARHMM), with an additional latent variable for time warping (handled as in a factorial HMM fashion). Time warping is achieved by having a log step-size parameter that allows the behavior to speed up or slow down dynamically. The model is validated in synthetic data, the applied to analyze mouse behavior data. The results demonstrate that the time-warping method enables clustering together same behavior type but with different speed/rigor (e.g., fast vs. slow runs), which were often identified as distinct behavioral syllables in previous models. The paper also proposed a more flexible model, GP-warping, but the results indicate that GP warping is less interpretable than the more straightforward time-warping.\n", "questions": "i) Step sizes: With the time-warping model, do you observe/anticipate any overshooting issues when the step sizes are too large? The dynamical matrices shown in the examples are simple and smooth rotations, but in reality the dynamics can have more structure with varying scale and complexity. In practice, would there be any signatures in the results that tell you when the step size is too large for the specific dynamics?\n\nii) Figure 3: I think I got most of the messages that the authors wanted to convey here, but it required going back and forth multiple times between text and captions and some educated guess. Some color legends and labels would be useful. Some things I still have not understood: Why are there four colors in each box in Fig 3C, and what are these colors? Also, what are x1 and x2 in Fig 3B; are they just two realizations from the same latent states or is something different? With improved figure annotations, my clarity rating for this paper could go up to 4.\n\niii) Description of GP model results on synthetic data: I am generally happy with how clearly and transparently the paper is written, but I feel that the GP-warped model results in the synthetic data section is a little sugar coated... For example, L195: \"... GP-WARHMM's increased flexibility allows it to fit the data, but in a less interpretable way\". From Fig 3 and Table 1, I would say that the GP model is simply not fitting the data very well, at least as it is. (Please feel free to correct me though; I might have mis-read Fig 3.) Of course you don't need to be as blunt, and I understand that it is constructive to put things on a positive context. But as I mentioned in the strengths/weaknesses above, the consideration of GP model can be valuable even if the results are not as good, and I think the authors can be more straightforward.\n", "limitations": "The authors have addressed the limitations of their methods and assumptions. They also discussed the potential societal impact of their work in the supplementary material.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a nice paper and certainly an addition to the community assets. The presented method will be readily useful for more interpretable analysis of natural behavior. The paper followed good practice: well defined problems, reasonable choice of models, validation on synthetic data, and finally application to real data. The paper is very clear and well written.\n\nI also appreciate that the paper considered two models (time-warping and GP-warping) as examples of interpretable vs. flexible models, and was clear about their assumptions and performances. Although the GP-warping model performance was not as good on synthetic data, which might count as a weakness per se, I think it was still valuable to share this part of the research and would like to credit the authors for doing so. The juxtaposition of two models provided a better context to the choice of time-warping model.\n\nA weakness of the paper might be that, from the technical viewpoint, the method is a relatively small extension of the existing models. However, it is an extension that brings value both practically (more interpretable data analysis with less syllables) and conceptually (allowing the timescale variability can be important in behavior analysis).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "t7apALMNOig", "reviewer": "Reviewer_N8jX", "summary": "This paper provides an extension to autoregressive Hidden Markov Models (ARHMM), in which the state transitions are modulated by a temporal component that controls the rate of state change. The paper provides two variants: in one of them, the step size is a parametric (time-wrapped), and the other one is based on Gaussian Processes. The model is applied to two datasets, one synthetic and one real-world dataset (MoSeq). The results on the synthetic dataset show that the model generally can infer the hidden state and also the temporal dynamic. The results on the real dataset show that the vanilla ARHMM can reach a similar performance (in terms of test likelihood) as the proposed model but requires a higher number of hidden states, which might hurt the interpretability of the model.\n", "questions": "C is chosen to be 2, and \\tau is within [-1, 1], which limits how much states can change. Whare the basis for this assumption?\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation for the model is sound, and it addressed an interesting problem with ARHMM.\n- The proposed models are simple but motivated by the target problem.\n- The presentation of the paper is good, and the paper is easy to follow.\n\nWeaknesses:\n- The method is not compared to any state-of-the-art action recognition/segmentation techniques (such as deep-learning approaches).\n- The choice of hyper-parameters is rather unclear. It is mentioned in the supplementary section that a range value for \\sigma was tried, which didn't affect the performance much. Are these performances on the train or test set? I'd expect the parameters to be tuned on a separate validation set.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nGN2Y0uYzlC", "reviewer": "Reviewer_4Fyi", "summary": "This is an application + algorithm paper. Application-wise, it focuses on the summarization of videos of freely moving mice for the purpose of neuroscience research. For this purpose, a new HMM model is proposed, whose state space is such that a discrete state variable captures discrete behaviors (sometimes referred to as movement primitives) and a continuous variable captures an element of warping of the dynamics within discrete states.\nMore precisely, the paper focuses on modeling of continuous-valued multivariate time-series observations (such as a video stream that has been reduced to a time-series of principal components). It starts from Auto Regressive Hidden Markov Model (ARHMM) models, in which within every discrete state z_i, (spanning some variable length of time), the observations within the state are modeled by a linear-gaussian model (i.e. linear dynamics with gaussian noise) whose parameters (A, b and Q) are state dependent. The paper then suggests a novel extension of the ARHMM model in which an additional continuous variable, tau (discretized for sake of computation) is added in 2 possible ways, both of which allow the linear-gaussian model for z_i to be \"warped:. In the version given more attention within the paper, the dynamics are simply sped up or down by 2^tau (tau in [-1 - 1]). The other warping method involves a gaussian process which, while fitting realt data better, is discarded from further analysis due to lack of interpretability.\nThe method is applied to (pre-processed) videos (IR grayscale + depth) of mice \"freely exploring\" within their lab environment. The paper shows how the proposed Time-Warped-AHRMM (T-WARHMM) can achieve log-likelihood values that are better than ARHMM for the same number of discrete hidden states (or equivalent to an ARHMM with more hidden states). This is explained by the ARHMM requiring several states to represent the same dynamic at different speeds whereas the WARHMM can do so with the addition of the warping hidden variable. The estimated warping variable, can be interpreted as a measure of \"rigorousness\" and thus conveys biologically-relevant meaning.\n", "questions": "* In equation 7,  the entries of A and b are modeled as GP(0, K(tau, tau`)). Does 0 refer to a zero mean? aren't these equations missing a base A and b which is only modified by a zero-mean GP?\n\n* The results in figure 3 and table 1 (for a synthetic time series) could be extended to include the regular ARHMM (which serves as a baseline in this study. I'd be interested to see how well that model captures the discrete state (perhaps as a function of the number of available model \"syllables”), as was done in the \"real-data\" experiment.\n", "limitations": "I see no  limitations within the context of behavioral analysis nor any potential negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the modeling sound and so is the interpretation of the experimental results. The benefits of such modeling for automating the annotation of behavioral videos of mice with the purpose of relating their behavior (as annotated by the model) to experiment conditions is clear. The model and its inference method is relatively clear. So are the experimental results.\n\nThe results are limited to a single non-public real dataset. It is therefore hard to assess whether the new method is a good fit to the particular application (analysis of behavior with discrete types of actions that can be warped in time). An analysis of a larger set of datasets (or one which is comprehensive and publicly available) would have been more convincing.\n\nThe MoSeq dataset seems to lack ground-truth annotations of the behaviors. I would have liked to see an analysis where a ground-truth annotation can be compared against the HMM so that a measure can be calculated which captures how well the HMM states capture behavior that the researchers care about. For example, I assume that MoSeq or some other dataset exists where (expert) human annotations are available, segmenting the video into human-recognisable behaviors such as those mentioned in the paper (dart, groom, rear and, I presume, \"unspecified\"). I would then like to see **how well HMM states fit within human annotated states** (as opposed to the results in the paper were states where assigned to behaviors by experts) and whether WARHMMs are better under _that_ sort of performance measure (which would be more relevant to behavioral scientists).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept