Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops a useful extension to the unsupervised behavioral syllable segmentation model space (specifically ARHMM), with an additional latent variable for time warping (handled as in a factorial HMM fashion). Time warping is achieved by having a log step-size parameter that allows the behavior to speed up or slow down dynamically. The model is validated in synthetic data, the applied to analyze mouse behavior data. The results demonstrate that the time-warping method enables clustering together same behavior type but with different speed/rigor (e.g., fast vs. slow runs), which were often identified as distinct behavioral syllables in previous models. The paper also proposed a more flexible model, GP-warping, but the results indicate that GP warping is less interpretable than the more straightforward time-warping.

strengths_and_weaknesses: This is a nice paper and certainly an addition to the community assets. The presented method will be readily useful for more interpretable analysis of natural behavior. The paper followed good practice: well defined problems, reasonable choice of models, validation on synthetic data, and finally application to real data. The paper is very clear and well written.

I also appreciate that the paper considered two models (time-warping and GP-warping) as examples of interpretable vs. flexible models, and was clear about their assumptions and performances. Although the GP-warping model performance was not as good on synthetic data, which might count as a weakness per se, I think it was still valuable to share this part of the research and would like to credit the authors for doing so. The juxtaposition of two models provided a better context to the choice of time-warping model.

A weakness of the paper might be that, from the technical viewpoint, the method is a relatively small extension of the existing models. However, it is an extension that brings value both practically (more interpretable data analysis with less syllables) and conceptually (allowing the timescale variability can be important in behavior analysis).

questions: i) Step sizes: With the time-warping model, do you observe/anticipate any overshooting issues when the step sizes are too large? The dynamical matrices shown in the examples are simple and smooth rotations, but in reality the dynamics can have more structure with varying scale and complexity. In practice, would there be any signatures in the results that tell you when the step size is too large for the specific dynamics?

ii) Figure 3: I think I got most of the messages that the authors wanted to convey here, but it required going back and forth multiple times between text and captions and some educated guess. Some color legends and labels would be useful. Some things I still have not understood: Why are there four colors in each box in Fig 3C, and what are these colors? Also, what are x1 and x2 in Fig 3B; are they just two realizations from the same latent states or is something different? With improved figure annotations, my clarity rating for this paper could go up to 4.

iii) Description of GP model results on synthetic data: I am generally happy with how clearly and transparently the paper is written, but I feel that the GP-warped model results in the synthetic data section is a little sugar coated... For example, L195: "... GP-WARHMM's increased flexibility allows it to fit the data, but in a less interpretable way". From Fig 3 and Table 1, I would say that the GP model is simply not fitting the data very well, at least as it is. (Please feel free to correct me though; I might have mis-read Fig 3.) Of course you don't need to be as blunt, and I understand that it is constructive to put things on a positive context. But as I mentioned in the strengths/weaknesses above, the consideration of GP model can be valuable even if the results are not as good, and I think the authors can be more straightforward.

limitations: The authors have addressed the limitations of their methods and assumptions. They also discussed the potential societal impact of their work in the supplementary material.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides an extension to autoregressive Hidden Markov Models (ARHMM), in which the state transitions are modulated by a temporal component that controls the rate of state change. The paper provides two variants: in one of them, the step size is a parametric (time-wrapped), and the other one is based on Gaussian Processes. The model is applied to two datasets, one synthetic and one real-world dataset (MoSeq). The results on the synthetic dataset show that the model generally can infer the hidden state and also the temporal dynamic. The results on the real dataset show that the vanilla ARHMM can reach a similar performance (in terms of test likelihood) as the proposed model but requires a higher number of hidden states, which might hurt the interpretability of the model.

strengths_and_weaknesses: Strengths:
- The motivation for the model is sound, and it addressed an interesting problem with ARHMM.
- The proposed models are simple but motivated by the target problem.
- The presentation of the paper is good, and the paper is easy to follow.

Weaknesses:
- The method is not compared to any state-of-the-art action recognition/segmentation techniques (such as deep-learning approaches).
- The choice of hyper-parameters is rather unclear. It is mentioned in the supplementary section that a range value for \sigma was tried, which didn't affect the performance much. Are these performances on the train or test set? I'd expect the parameters to be tuned on a separate validation set.

questions: C is chosen to be 2, and \tau is within [-1, 1], which limits how much states can change. Whare the basis for this assumption?

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is an application + algorithm paper. Application-wise, it focuses on the summarization of videos of freely moving mice for the purpose of neuroscience research. For this purpose, a new HMM model is proposed, whose state space is such that a discrete state variable captures discrete behaviors (sometimes referred to as movement primitives) and a continuous variable captures an element of warping of the dynamics within discrete states.
More precisely, the paper focuses on modeling of continuous-valued multivariate time-series observations (such as a video stream that has been reduced to a time-series of principal components). It starts from Auto Regressive Hidden Markov Model (ARHMM) models, in which within every discrete state z_i, (spanning some variable length of time), the observations within the state are modeled by a linear-gaussian model (i.e. linear dynamics with gaussian noise) whose parameters (A, b and Q) are state dependent. The paper then suggests a novel extension of the ARHMM model in which an additional continuous variable, tau (discretized for sake of computation) is added in 2 possible ways, both of which allow the linear-gaussian model for z_i to be "warped:. In the version given more attention within the paper, the dynamics are simply sped up or down by 2^tau (tau in [-1 - 1]). The other warping method involves a gaussian process which, while fitting realt data better, is discarded from further analysis due to lack of interpretability.
The method is applied to (pre-processed) videos (IR grayscale + depth) of mice "freely exploring" within their lab environment. The paper shows how the proposed Time-Warped-AHRMM (T-WARHMM) can achieve log-likelihood values that are better than ARHMM for the same number of discrete hidden states (or equivalent to an ARHMM with more hidden states). This is explained by the ARHMM requiring several states to represent the same dynamic at different speeds whereas the WARHMM can do so with the addition of the warping hidden variable. The estimated warping variable, can be interpreted as a measure of "rigorousness" and thus conveys biologically-relevant meaning.

strengths_and_weaknesses: I find the modeling sound and so is the interpretation of the experimental results. The benefits of such modeling for automating the annotation of behavioral videos of mice with the purpose of relating their behavior (as annotated by the model) to experiment conditions is clear. The model and its inference method is relatively clear. So are the experimental results.

The results are limited to a single non-public real dataset. It is therefore hard to assess whether the new method is a good fit to the particular application (analysis of behavior with discrete types of actions that can be warped in time). An analysis of a larger set of datasets (or one which is comprehensive and publicly available) would have been more convincing.

The MoSeq dataset seems to lack ground-truth annotations of the behaviors. I would have liked to see an analysis where a ground-truth annotation can be compared against the HMM so that a measure can be calculated which captures how well the HMM states capture behavior that the researchers care about. For example, I assume that MoSeq or some other dataset exists where (expert) human annotations are available, segmenting the video into human-recognisable behaviors such as those mentioned in the paper (dart, groom, rear and, I presume, "unspecified"). I would then like to see **how well HMM states fit within human annotated states** (as opposed to the results in the paper were states where assigned to behaviors by experts) and whether WARHMMs are better under _that_ sort of performance measure (which would be more relevant to behavioral scientists).

questions: * In equation 7,  the entries of A and b are modeled as GP(0, K(tau, tau`)). Does 0 refer to a zero mean? aren't these equations missing a base A and b which is only modified by a zero-mean GP?

* The results in figure 3 and table 1 (for a synthetic time series) could be extended to include the regular ARHMM (which serves as a baseline in this study. I'd be interested to see how well that model captures the discrete state (perhaps as a function of the number of available model "syllables”), as was done in the "real-data" experiment.

limitations: I see no  limitations within the context of behavioral analysis nor any potential negative societal impact of this work.
review_title: null

Meta-review: metareview: This paper introduces warped auto-regressive HMMs, where both discrete state transitions z_{t+1} | z_t and continuous observations x_{t+1} | x_t, z_t, depend on the previous time step, and where the linear dynamics for observation transitions are either time-warped (i.e., the linear transition matrix and bias are multiplied by a time-dependent latent step size) or obtained by Gaussian Process regression with latent prototypes. The method is evaluated and applied to modeling “syllables” of behaviour in free-moving mice in neuroscience experiments directly from depth camera recordings.

Reviewers praised the clarity and structure of the paper (w5pw, N8jX, 4Fyi), the motivation for the paper (N8jX, 4Fyi), the honesty in reporting results on slightly worse GP warped ARHMM (w5pw)

The reviewers noted that the contribution was incremental (w5pw) and did not compare to any modern deep learning baseline (N8jX) - for the latter, the authors partially addressed this point by responding that existing work did not handle tracking and segmentation without supervised training. Reviewers noted that evaluation was limited to a proprietary dataset (it is actually publicly accessible) without human ground-truth annotations (4Fyi) and wished more evaluations had been done - the authors replied that results were reviewed by experimentalists. They also had some questions regarding figures that were properly addressed (w5pw).

Reviewers agree on high scores (6, 7, 7) and therefore I would recommend this paper for acceptance.

Sincerely,
Area Chair
confidence: Certain
Decision: Accept