AI-driven decoding of naturalistic behaviors enables tailored detection of depressive-like behavior in mice

Domain: biology
Published: 2025-12-30
Editorial outcome: major_revision

Abstract
Major depressive disorder (MDD) is an etiologically diverse psychiatric disease with heterogeneous manifestations, making it difficult to diagnose with conventional assessment standards. In addition, the obvious incompatibility of the standard survey-based tests for human MDD and the behavioral assays for depressive-like phenotypes in mice makes clear the requirement for a non-invasive method for quantifying the expression of depressive-like state in naturalistic contexts. Here, we introduce a self-supervised machine learning platform, CLOSER (Contrastive Learning-based Observer-free analysis of Spontaneous behavior for Ethogram Representation), to monitor the spontaneous behavior in a depressive disease model with enhanced precision, reliability, and efficiency. This framework incorporates 3D pose skeleton data and kinematic features in a unique data augmentation strategy to characterize semantic behavioral syllables with a high-quality feature space. Using CLOSER, we uncovered distinct motion profiles in chronically stressed mice across both sexes and different disease stages. Furthermore, we quantified the drug-specific recovery of psychomotor symptoms, highlighting CLOSER’s discriminative power for identifying drug efficacy. In offering an artificial intelligence (AI)-driven decoding of exploratory behaviors, CLOSER proposes the standardization of depressive-like phenotype assessment in mouse models, thereby bridging preclinical and clinical diagnostics for psychiatric drug discovery.

Introduction
Major depressive disorder (MDD) is a stress-induced disorder associated with persistent feelings of sadness and loss of interest. MDD has diverse causes, including stress, and is associated with an elevated risk of developing a host of other severe medical disorders. Despite its prevalence across the world, the heterogeneity of MDD symptoms and their severity in patients of different ages and genders constrains the therapeutic effectiveness of pharmacological interventions1,2. Recently, researchers have tried to discriminate genetic and pathophysiological depression subtypes, while little attempt has been made to classify depressive episodes according to the motion parameters, which are the symbolic symptoms of psychiatric diseases3–6. We aimed, therefore, to develop a diagnostic module encompassing the heterogeneity of depression for use in symptom-based antidepressant drug discovery.

Traditional behavioral screening tests in rodents (e.g., the tail suspension test (TST), forced swim test (FST), and sucrose preference test) have been used to evaluate depressive-like symptoms that are affected by stress exposure and antidepressant treatment. Unfortunately, they show low construct validity7, making them incapable of reflecting the complex psychiatric symptoms and prognoses of MDD. The highly controlled environments and oversimplified conditions used to elicit non-spontaneous behaviors in these tests have limited the development of the most common antidepressants, resulting in high false positive rates for novel antidepressant discovery8,9. Moreover, the over-reliance of the tests on manual scoring introduces subjectivity and inter-observer variability, further limiting the accuracy and reproducibility of depressive disorders models10.

In human mental diseases, the motion-associated symptoms, or psychomotor activity, have been assessed through behavioral observation of limb movements, trunk posture, voice, facial expression, and eye movements5, in parallel with identifications of circuital and functional connectivity between the amygdala and motor cortex11–14. For rodents, the machine learning approaches (both supervised and unsupervised learning) have started to distinguish the psychiatric state of mice with distinct motion features, decomposing behavioral structures induced by genetic manipulation or antipsychotics15–17. In general, the supervised method trains the machine to provide a theory-driven classification for the reproduction of human annotation, while the performance of such models is strongly limited because the supervised classifier can never outperform human perception18. The alternative approach is a data-driven method that mathematically organizes behavior syllables to preserve the natural principle of a motion sequence19. The major advantage of the unsupervised approach is the ability to uncover hidden behavioral patterns without human intervention. A popular unsupervised algorithm, contrastive learning, is a self-supervised model that learns behavioral representations to capture relevant features in the data20–22, by contrasting pairs of augmented views from the same sample against unrelated negative samples. During the training process of contrastive learning, data augmentation compels the machine to generate discriminative representations such that similar data pairs are located near one another in the embedding space. Importantly, contrastive learning semantically strengthens unsupervised approaches by modulating the combination of appropriate augmentations to capture invariance in behavioral data.

Therefore, we present Contrastive Learning-based Observer-free analysis of Spontaneous behavior for Ethogram Representation (CLOSER), a platform for scalable measurement of exploratory behaviors in depressive disorder model mice, employing 3D skeleton coordinates obtained from the precise motion recognition framework, AVATAR23. Trained on both 3D action skeleton coordinates and kinematic features, CLOSER surpasses the quality of embedding structures produced by current unsupervised pose estimation algorithms and shows improved predictive performance for manually annotated behavior datasets. Using CLOSER, we identified motion profiles for spontaneous movements in depressive mouse models that are affected by sex and disease severity. This discovery permits the computational evaluation of depressive behaviors with consideration for symptom variance between distinct cohorts. Finally, our study demonstrates that CLOSER accomplished a pivotal improvement in the identification of distinctive signatures for monoaminergic and non-monoaminergic antidepressants in naturalistic behavior, facilitating the interpretation of behavioral data required in future preclinical screens for new psychiatric drugs.

Results
Self-supervised learning of mouse exploratory behaviors using 3D action skeleton representations
To assess mouse behavioral structure in a quantitative manner, we constructed a contrastive learning module leveraging a distillation that interchanges information learned from joint and motion modalities to acquire both allocentric (global) and egocentric (local) action skeletal knowledge from recorded video in AVATAR studio (Fig. 1a). The module was trained with a motion dataset established by shifting a 10-frame-long sliding window through the recorded videos one frame (or about 33 msec) at a time. Since our model learns the characteristic invariances of mouse behavior by capturing semantic similarities from two augmented views (positive pairs), we conducted an ablation study for eight different combinations of augmentation strategies to optimize the feature space of the behavior sequence (Supplementary Table 1). In the estimation of model prediction accuracy, the algorithm incorporating size scaling, joint angle distortion, motion crop, and x- or y-axis mask presented the best performance on all evaluation metrics for the classification of manually labeled mouse behavior segments. We observed that reported augmentation sets optimized for human 3D skeleton representations produced relatively low accuracy (Supplementary Table 1, column a), suggesting human and mouse behaviors exhibit distinct semantic invariance.

Next, to evaluate the geometric variance of latent spaces across multiple augmentation strategies, we measured Pearson correlation coefficients for representations of 13 manually labeled behaviors (Supplementary Table 2) in an unseen human dataset to compare within- (or intra-) and between (or inter)-cluster relationships (Supplementary Fig. 1). A coefficient of +1 or −1 denotes perfect positive or negative linear association, respectively, while 0 indicates no linear relationship. We found that augmentation schemes with superior classification accuracy also exhibited stronger intra-cluster correlation than inter-cluster correlation. Importantly, our mouse-optimized augmentation yielded enhanced embedding for grooming behaviors (clusters 6 and 7) compared with augmentation combinations lacking channel masking (strategy 3) or scaling (strategy 2), highlighting the necessity of a complete augmentation strategy for mouse behavior latent structures.

To determine and categorize behavioral syllables, which are minimal motion units involving ethological context, we leveraged K-means clustering in 128 dimensions to partition behavior segments into 50 clusters. Then, a cluster label for each frame was determined by majority voting on candidates assigned from respective sliding windows at the first learning step (Fig. 1a). Given the continuous nature of mouse behavior, a behavioral syllable was denoted as a consecutive frame with the same cluster label. For validation of temporal segmentation, we monitored whether the dissection of a postural sequence captured the temporal dynamics of behavior shifts from one motion to another by utilizing a keypoint change-score, which highlighted the robust shift within the 3D action skeleton. Observing the transition of behavior motifs longer than 10 frames (or about 330 msec) with z-scored keypoint change scores, we found that the start frame (0% frame fraction) recorded the maximal change-score compared to 50% frame fractions in either direction, which matched the change-point of keypoint-MoSeq, the existing unsupervised motion analysis frameworks, and manually annotated segments (Fig. 2a). The duration of the defined clusters (17.69±24.58 frame, 590±819 ms) was slightly shorter than both the manually defined behavior syllables (28.78±25.83 frame, 959±861 ms) and segments created by keypoint-MoSeq (27.93±34.95 frame, 931±1165 ms) but longer than those created by Variational Animal Motion Embedding (VAME) (7.75±12.52 frame, 258±417 ms) (Fig. 2b), while also satisfying the length requirement for general sub-second behavior motifs (<500 ms)18.

To estimate whether the models could follow the criteria of human annotation, we evaluated the prediction power of five individual methods on the discrimination of labeled behaviors through k-nearest neighbor (k-NN) probing, a reliable indicator of downstream task performance24 (Fig. 2c). In benchmarking, we compared VAME trained for 20 epochs (to match CLOSER) against the default 500-epoch setting and employed the default setting, achieving optimal performance, although the improvement over 20 epochs was marginal (Supplementary Fig. 2). Moreover, since keypoint-MoSeq employs an HMM that learns transition probabilities between states, behavioral syllables IDs, and emission probabilities from a given 13-dimensional Principal Component Analysis (PCA) projection of motion sequences, we leveraged 270-dimension representation with 13-dimension PCA and Uniform Manifold Approximation and Projection (UMAP) embedding for 10-frame length behavioral segments, instead of keypoint-MoSeq. In behavioral classification, CLOSER provided the highest balanced (66%) and top-1 accuracy (55%), accompanied by consistently strong precision, recall, and F1 score. For further intrinsic evaluation of embedding spaces, we computed the Pearson correlation, which quantifies the strength and direction of the linear relationship between two behavioral segments. Using this method, we evaluated intra- and inter-cluster correlation coefficients for manually annotated behaviors within the latent spaces of multiple embedding strategies (Supplementary Fig. 3). These results demonstrated that CLOSER consistently yielded higher intra-cluster correlations than inter-cluster correlations, signifying superior embedding quality compared to all other algorithms. Among alternative methods, UMAP achieved the second-highest quality of embedding structure, whereas VAME produced relatively high correlations in both intra- and inter-cluster, indicating limitations in discriminative ability.

We then applied the dimensionality reduction algorithm, UMAP, to visualize the structure of feature space for CLOSER (Fig. 2d). This result implied that 128-dimensional behavior representations are well-organized even in two-dimensional (2D) space, with explicit compartmentalization of embedding space for individual behavioral clusters. Next, to determine if the resulting clusters comprised behavior syllables with uniform ethological contexts, we examined the embedding structures and kinematic features of the clusters with strong annotation as representative compositional behaviors, such as walking, turning, rearing up, and rearing down (Fig. 2e). Notably, behavioral clusters classified in the same groups closely distributed with unique cluster areas. When we extracted the moved distance, head-body angle, and nose height from the 3D pose skeleton, each cluster class showed motion hallmarks appropriate for its corresponding behavior label (Fig. 2f–i). These model qualifications with typical postural phenotypes demonstrate that CLOSER produces a fine characterization of sub-second motion segments containing meaningful behavioral sequences, which is a fundamental building block of the mouse ethogram.

To determine whether the latent space of CLOSER learned to rely on pose dynamics rather than confounding variables, we assessed mouse-size robustness in CLOSER and benchmarking pipelines, the former having received explicit scaling augmentation. Eighty male and female mice were binned into five weight groups, from 18 g upward in 2-g steps, and we extracted embedding vectors in non-overlapping 100-frame strides for all mice. These embedding vectors were used as inputs to a Random Forest classifier trained with five-fold validation to predict weight group. CLOSER yielded the lowest accuracy, precision, recall, and F1 scores among all algorithms (Supplementary Fig. 4), indicating that weight information is the hardest to extract from its latent space and confirming that the scaling augmentation effectively enforces size invariance in mouse behavioral segmentation.

Chronic unpredictable stress (CUS) induces typical depressive-like behaviors in mice in a sex-dependent manner
While chronic stress is known to modify the behavior of mice in a sex-dependent manner, most studies have included only a partial examination of depressive-like phenotypes, requiring an array of behavior tests to compare depressive behaviors in males and females. To address this issue, we exposed male and female mice to 4 weeks of CUS to generate an etiologically relevant depressive disorder model, followed by an evaluation of depressive-like behaviors. In male CUS mice, we observed a significant increase in immobility at TST and FST (Fig. 3a, b, left) and decrease in sucrose preference (Fig. 3c, left), as well as a decrease in body weight (Fig. 3d). Although female CUS mice showed reduced sucrose preference (Fig. 3c, right) and a notable loss of body weight (Fig. 3d), we detected immobility increase only with the FST, not the TST (Fig. 3a, b, right). The reduced appearance of stress-related behavior in females during TST sessions may be attributed to hormonal influences associated with estrous cycle fluctuations, which were not measured in this experiment. When we evaluated motor activity, we did not see any critical changes as a result of stress (Fig. 3e, f).

The CLOSER pipeline tracks the sex-specific behavior of CUS model mice in an open field
Rather than contribute to the current oversimplified model of depressive-like symptoms, we traced sophisticated chronic stress-induced behaviors of freely moving mice. To detect variations in the spontaneous behavior of individual mice, we used the AVATAR system to record mice at the beginning (0 weeks) and end (4 weeks) of the CUS paradigm, followed by normalization of behaviors at four weeks with the initial behavior frequency. Initially, we performed a Linear Discriminant Analysis (LDA) to transform the syllable frequency and transition summaries of five-minute intervals into a low-dimensional subspace (Fig. 3g), which enhances the expressiveness of stress-induced ethograms to address the sexual disparity. As LDA searches the axes showing the best classification performance, the larger distances between samples from different classes indicate the higher disparity in the original data25. The result revealed a divergence in the summaries of syllable frequency between stressed male and female mice. In contrast, syllable transition information converged from distinct embeddings, implying a discrete pattern of alteration in syllable usage and bigram. Using these behavioral structures, we measured the diagnostic accuracy of a random forest classifier trained by syllable frequency or inter-syllable transitions (Fig. 3h). In both males and females, CLOSER-identified syllables and sequences enabled the classifier to predict depressive disorder, with the sequence-trained decoder performing greater and suggesting larger changes in the temporal dynamics than syllable usage with the CUS model. Thus, our behavior analysis framework can extract fundamental motion features from exploratory behaviors affected by stress exposure.

To further evaluate the efficiency of CLOSER in the detection of depressive-like behavior, we applied the same analysis on male and female groups using the syllable combinations identified by VAME and keypoint-MoSeq. Classification accuracy indicated that CLOSER outperformed VAME and keypoint-MoSeq in the male mice, achieving superior predictive performance for both classifiers trained using syllable frequency of transition (Supplementary Fig. 5a, b). In contrast, for non-stressed and stressed females, the random forest classifier leveraging syllable frequency from keypoint-MoSeq demonstrated greater performance compared to other behavior recognition pipelines (Supplementary Fig. 5c); nevertheless, the syllable transition calculated by CLOSER consistently yielded the highest accuracy (Supplementary Fig. 5d). Notably, behavioral deconstruction in males offered higher classification rates than in females for all segmentation methods, aligning with the observed sex-dimorphic alterations of syllable frequency and transition (Fig. 3i, l).

Before interpreting the stress-induced changes in behavioral structure, we annotated the clusters with one or two stereotyped components while considering the discrete property of behavior. In the hierarchical structure of defined behavior syllables, clusters assigned analogous labels were located adjacent to one another in the hierarchy table (Supplementary Fig. 6), for example, the “wall rear up” clusters (clusters 49, 8, and 34), revealing the strong proximity of relevant behavior segments in the CLOSER embedding space. With this knowledge, we next explored the heterogenic alteration in behavior that emerged from chronic stress exposure. When evaluating syllable frequency in CUS mice, males and females showed statistical changes in the usage of twelve and six syllables, respectively (Fig. 3i), consistent with the greater predictive performance of the frequency-based classifier for males (Fig. 3h, frequency). Although there was a single overlap (cluster 41) in significant clusters, the trends in the major alterations were maintained across the sexes, with minor sex-dimorphic syllables (clusters 30, 24, and 6) showing sexual dimorphism in the depressive mouse model. In further kinematic profiling, chronic stress selectively reduced the occurrence of motifs with elevated body angles and sniff-related behaviors with large nose shifts relative to body shifts (Supplementary Fig. 7a), while higher nose heights were observed in stress-elevated syllables (Supplementary Fig. 7b). In female mice, stress-elevated syllables displayed greater body angles, in agreement with manual annotation, turning behaviors (Supplementary Fig. 7c, d).

Once we identified discriminant-relevant syllables, we wondered whether depressive phenotypes in naturalistic behavior had a direct correlation with conventional depressive-like behaviors. Hence, we used a linear regression model to predict the immobility time or sucrose preference using discriminant-relevant syllables and calculated a coefficient of determination (R2 score) to evaluate the accuracy of predictive value compared to actual data (Supplementary Fig. 8). In contrast, the structure of spontaneous behavior showed only a weak linear correlation with TST, FST, and SPT in females, implying a sexually dimorphic relationship between naturalistic behaviors and depressive-like behaviors (Supplementary Fig. 8d, f). In addition, syllable frequency was more closely related to stress-related behaviors in TST and FST (Supplementary Fig. 8a, b, d, e) than anhedonia-like behavior in SPT (Supplementary Fig. 8c, f) in both males and females, as an analogous result has been observed in the previous study utilizing social defeat stress mice for a depressive disorder model26.

We then dissected the total recording into five-minute time bins and looked for the temporal windows in which those syllables showed statistically significant changes (Fig. 3j, k). Interestingly, while only one significant stress effect (cluster 23) was found in the initial time bin, all other frequency alterations (clusters 25, 2, 23, and 18) were observed primarily in the third and fourth time bins. Given that initial cluster frequencies were similar in chronically stressed and non-stressed mice, we concluded that chronic stress disturbs the frequency of behavioral motifs in a semi-naturalistic environment only after sufficient exposure. Moreover, to investigate the “grammar” of the behavioral outcomes of stress exposure, we visualized the cluster transition matrix in a circular diagram depicting significantly adjusted conversions probability from one cluster type to another (Fig. 3l). Even though the transition summaries revealed a robust transition shift in CUS-treated male and female mice compared to unstressed controls, there were minimal differences between stressed male and female mice, implying that chronic stress induces greater changes in behavioral grammar than does sex. Together, these results demonstrate that the CLOSER pipeline identifies sex-specific modifications in naturalistic behavior structure elicited by chronic stress.

Model-specific alterations in behavioral syntax and architecture in depressive-like behavior mouse models
To explore the characteristics of spontaneous behavior across depressive-like behavior models, we first established stress paradigms using distinct pathological triggers. We employed two depressive-like behavior models: an immune system-based model through a single intraperitoneal (i.p.) injection of lipopolysaccharide (LPS); and a steroid hormone-based model using corticosterone administration for three weeks. Given prior reports suggesting that 1-week corticosterone does not elicit depressive-like phenotypes27, an additional group receiving a 1-week corticosterone injection was included to estimate the validity of the model. In the LPS-induced depressive-like phenotype model, mice exhibited hallmark stress-induced behaviors, as evidenced by increased immobility in the FST and reduced sucrose preference (Supplementary Fig. 9a). In the corticosterone paradigm, stress-related behaviors in TST and SPT emerged exclusively after 3 weeks of administration, with no significant alterations in the 1-week group (Supplementary Fig. 9b), in line with existing literature27.

In the first step of behavioral analysis, we evaluated syllable frequency and transition dynamics employing LDA to interrogate the broader architecture of spontaneous behaviors across these models (Supplementary Fig. 9c). While non-stressed groups and saline-treated groups, regardless of models, clustered together, chronically stressed and LPS-treated groups diverged markedly from controls in both syllable usage and bigram transitions. Interestingly, while both stress- and inflammation-induced models displayed distinct separations, their behavioral trajectories diverged in opposing directions. In contrast, corticosterone-treated mice were closely positioned with controls, suggesting comparatively modest modifications in spontaneous behavior patterns. Then, we implemented a random forest classifier to quantify the discriminability among stress models (Supplementary Fig. 9d). Consistent with LDA findings, confusion matrices indicated that CUS and LPS models were readily separable, whereas corticosterone-induced models were less distinguishable from controls.

We next applied an analysis of individual syllable contributions to unveil model-specific alterations. LPS treatment elicited marked changes in the frequency of 16 syllables, whereas corticosterone treatment affected only three, all of which were also altered in the LPS-treated model (Supplementary Fig. 9e). A subset analysis using the Venn diagram illustrated the degree of overlap across the different mouse models (Supplementary Fig. 9f). Of note, while six syllables were commonly altered between stress- and inflammation-induced models, the directionality of syllable changes was predominantly opposite between two models, aside from one shared cluster (cluster 2). These data suggest that frequency alterations are limited to a subset of syllables and exhibit distinct patterns depending on the etiology of depressive-like behavior. These results may also account for the divergent trajectories of depressive-like behavior models observed in the LDA (Supplementary Fig. 9c). Beyond the frequency alterations, we observed pronounced disruptions in grammatical structure, particularly in the LPS model (Supplementary Fig. 9g), and few conserved transition shifts were identified across depressive-like behavior models (Supplementary Fig. 9h). These findings imply that the source of stress (stress, inflammation) can be inferred based on distinctive patterns of spontaneous behavior identified in the CLOSER platform, extending beyond conventional dichotomous assessments of depressive-like behaviors.

Anomaly detection revealed no irregular motion in the stress mouse model
A key challenge in the psychomotor analysis of depressive disorder is the identification of abnormal behavioral syllables that only appear after exposure to chronic stress. To determine whether the CUS paradigm induces abnormal behaviors, we performed a regularity detection method with global and local pose representations to scan skeleton trajectories for anomalies. In this analysis, we utilized a recurrent neural network to calculate the frame-based anomaly scores to quantify the predictability of behavior sequences in an experimental group. We examined behavior datasets from male and female depressed mice, using the A53T Parkinson’s disease model28 as a positive control group with clear motor dysfunction. Consistent with our expectations, there was a clear segregation between the loss distributions of normal (µ=3.844, σ=0.789) and A53T mice (µ=5.412, σ=0.635) (Supplementary Fig. 10a), with discrete distribution patterns of behavior segments in embedding spaces for global and local components built with latent space-based anomaly detection29 (Supplementary Fig. 10b, c). In contrast, the computational losses and embedding structures of chronically stressed mice were similar for both male (control: µ=3.834, σ=0.946/CUS: µ=3.793, σ=0.877) and female groups (control: µ=3.661, σ=0.932/CUS: µ=3.679, σ=0.825) (Supplementary Fig. 10d–i), indicating that most of the behavior patterns of CUS mice can be generalized as behavior sequences from normal mice. We, therefore, concluded that chronic exposure to stressors modifies the structure of free exploratory mouse behaviors rather than causing irregular behaviors.

CLOSER uncovers gradual changes in the structure of mouse exploratory behavior depending on the duration of exposure to chronic stress
Since the discrimination of stages in the appearance of depressive symptoms would provide an additional regulatory node in the clinic, we asked whether CLOSER can discriminate time-dependent behavioral phenotypes in exploratory movements generated by chronic stress. We obtained 20-minute behavioral sequences of stressed and non-stressed male mice from 3D motion videos sampled every week of the four-week CUS protocol, along with weekly assessments of depressive-like behaviors. This analysis excluded sucrose preference since its outcome is sensitive to adaptation with repetitive uptake (Supplementary Fig. 11a–d). In the fourth week, we observed clear stress-induced effects in the TST, FST, SPT, and body weight measurements of CUS mice compared to control mice. Additional tests for locomotion identified a critical change in time spent in the arena’s center for the control group (Supplementary Fig. 11e, g), with a significant reduction in distance traveled regardless of the presence of chronic stimuli, suggesting a decline in exploratory instinct in familiar environments (Supplementary Fig. 11f, h). These results with conventional assays were sufficient to confirm the expression of depressive-like behaviors, but they failed to show a direct correlation with specific timepoints in the CUS paradigm. In other words, this type of behavioral analysis was insufficient for tracking the development of depressive-like behaviors.

To resolve this limitation, we devised a method to capture the characteristics of the behavioral repertoire of stressed mice following various durations of exposure to chronic stress. As previously described, we employed the random forest algorithm to estimate the predictive accuracy of the classifier for time points (1–4 weeks after), training it with syllable usage or with transition probability calculated from behavior sequences recorded in the AVATAR studio. The resulting confusion matrix indicated that the classifier discriminated between the experimental conditions (CTRL vs. CUS) and captured the manifestation of behavioral change from the beginning of depressive disorder, as represented by the isolation of prediction errors within the same conditions (Fig. 4a, b, left). Likewise, we were able to diagnose experimental conditions and exposure periods through LDA1 and LDA2 axes, respectively, which were derived from information on syllable frequency (Fig. 4c) and transitions (Fig. 4d). Then, we performed an additional investigation of syllables differentially expressed throughout the various stages of stress paradigm (Fig. 4e), which indicated decisive modification in syllable usage over the course of psychiatric pathogenesis.

Notably, we observed a peak in classification performance and the largest fold change at three weeks of stress exposure (Fig. 4e), marking it as the point of the largest deviation from the normal exploration pattern. To overcome the black-box nature of the machine learning framework, we augmented the explainability of the classifier model by harnessing SHAP analysis to identify clusters that most strongly impact the model’s decision-making process (Fig. 4f, g). The SHAP summary plot characterized the behavior signature for 3-week CUS, showing increases in “stand” (cluster 28) and “sedate (high nose)” (cluster 8) and decreases in “rotate” (cluster 17), “back groom” (cluster 25), and “sniff” (cluster 23) (Fig. 4g). Conversely, the minimal changes in cluster frequency we observed at the final week of the CUS protocol indicated that context adaptation by means of repetitious (in our case, five) experiences reduced the behavioral disparity caused by chronic stress, considering the larger variance in syllable usage between depressed and control mice exposed to the AVATAR studio only twice (Fig. 3i). The examination of diagnosis performance for depressive disorder furthers supports this observation (Fig. 4h). Further analysis of the transition map revealed that three-week exposure to chronic stressors altered the transition matrix from its original structure (Fig. 4i, left). Familiarity with the open field environment strongly affected syllable bigrams in a single experimental group (Fig. 4i, right), suggesting a large contribution of context familiarity to the transition matrix of spontaneous behavior.

CLOSER can quantify the pharmacological effects of monoaminergic and non-monoaminergic antidepressants on psychomotor symptoms
To develop targeted therapeutic strategies for depressive disorders, we examined whether CLOSER captures antidepressant-induced reversal of chronic stress-related behavioral modifications, assessing distinct behavioral phenotypes in freely moving mice. We administered traditional monoaminergic antidepressants (fluoxetine and imipramine) or emerging non-monoaminergic candidates (agmatine, scopolamine, and rapastinel) to observe the distinct action of different drug groups (Fig. 5a). In chronically stressed male mice, most antidepressants significantly reduced immobility durations in FST, while imipramine alone enhanced sucrose preference rather than influencing immobility (Supplementary Fig. 13a, b). Additionally, rapastinel treatment significantly elevated sucrose preference, whereas other drugs did not induce significant improvements in either the TST or SPT. Thus, despite the high predictive validity of FST in male mice, traditional behavioral assays generally failed to differentiate individual or grouped antidepressant actions, with the exception of imipramine and rapastinel.

As the initial step of our ML-guided behavioral evaluation, we implemented a machine learning-guided behavioral evaluation to discern subtle alterations within exploratory behaviors. A random forest classifier, informed by syllable frequencies and transitions, yielded sensitivities of 40.35% and specificities of 88.00% (Fig. 5b). Incorporating transition information enhanced performance, raising sensitivity to 50.07% and specificity to 90.01%. We examined the classification errors, which revealed strong confusion within drug groups; however, inter-group and vehicle-associated misclassifications remained moderate (Supplementary Fig. 14).

Further analysis of syllable frequency subsets was performed to explore on- and off-target effects of antidepressants (Fig. 5c). Chronic stress significantly altered the usage patterns of four behavior clusters (31, 12, 49, 28). Notably, cluster 31 was consistently restored by all tested drugs, while others displayed selective restoration profiles (Fig. 5d). Imipramine and agmatine demonstrated complete recovery across stress-relevant clusters, whereas other treatments elicited partial effects. Off-target behavioral alterations were more pronounced with non-monoaminergic compounds, displaying considerable overlap within this drug class, unlike the monoaminergic group, which lacked common off-target syllables (Fig. 5c, column 12). Eight additional clusters appeared at the intersection of two non-monoaminergic drugs, emphasizing high group similarity. Utilizing SHAP analysis for deeper behavioral profiling, we identified syllables that critically influence antidepressant classification, indicative markers reflecting antidepressant heterogeneity (Fig. 5e). Different variants of “rear up” behaviors emerged prominently as discriminative features. Specifically, increased occurrences of clusters 48, 47, and 4 and decreased cluster 3 activity characterized positive predictions for non-monoaminergic antidepressants, while inverse patterns distinguished monoaminergic treatments (Supplementary Fig. 16a). Within groups, fluctuations in cluster 19 contributed to precise drug identification. These findings underline that the CLOSER pipeline effectively leverages profound behavioral compositions of mice to distinguish psychiatric drug actions, offering potential for personalized therapeutic recommendations grounded in behavioral phenotypes.

In grammatical structure, we revealed partial normalization of stress-induced behavioral transitions following antidepressant treatment, though aberrant transitions persisted (Fig. 5f). Fluoxetine and imipramine demonstrated modest efficacy in restoring syllable bigram structures compared to the recovery in syllable frequency, accompanied by considerable off-target transition modifications (Fig. 5f, monoaminergic). Conversely, non-monoaminergic antidepressants presented comparatively lower off-target transition alterations (Fig. 5f, non-monoaminergic).

For the drug administration without stress exposure (Drug Only), the syllable frequency and transitions of spontaneous behavior were strongly discriminated from those of vehicle-treated male mice (Supplementary Fig. 17a). Consistent with observations in the stressed group, fluoxetine induced more changes in syllable frequency compared to imipramine (Supplementary Fig. 17b, c). In SHAP-based fingerprint identification, clusters 19 (“stand”) and 48 (“wall rear up”) overlapped with the top five impactful features of the stressed group. Instead, other syllables emerged as unique fingerprints of antidepressants, such as cluster 37 (“sniff and pause”) for fluoxetine (Supplementary Fig. 16b). This observation underscores the importance of pathological context in assessing the efficacy of psychiatric medications and their side effects via the monitoring of psychomotor behavior. Therefore, we have demonstrated that CLOSER can decode the hidden patterns of psychomotor symptoms into interpretable information for quantifying the pharmacological effects of antidepressants.

Sex-dependent behavioral fingerprints in naturalistic behavior for monoaminergic antidepressants
To elucidate sex-dependent antidepressant responsiveness and its impact on naturalistic behavior, we evaluated female mice with the same experimental paradigm previously applied to male mice using monoaminergic antidepressants. Unlike the male CUS model, female mice exhibited antidepressant efficacy exclusively in SPT and not in FST, highlighting substantial limitations in the predictive validity of traditional assays (Supplementary Fig. 13d, e).

To test if the behavioral structure identified by CLOSER could be applied to reliable discrimination of stress and antidepressant administration states within the female cohort (Supplementary Fig. 18a). Analysis based on syllable usage yielded a sensitivity of 47.22% and specificity of 82.41%, whereas classifiers leveraging transition profiles demonstrated enhanced performance, achieving sensitivity and specificity values of 61.11% and 87.04%, respectively, showing an analogous trend observed in males. Subsequent SHAP analyses elucidated antidepressant behavioral fingerprints, revealing significant sex-dependent differences (Supplementary Fig. 18b). In contrast to the rearing behaviors pivotal in males, female fingerprints predominantly featured head movements and sniffing behaviors, with cluster 22 exhibiting decreased frequencies following monoaminergic antidepressant treatment, while clusters 14 and 29 differentiated fluoxetine from imipramine (Supplementary Fig. 16c). Although cluster 4 displayed analogous antidepressant responsiveness across sexes, disparities in other syllable fingerprints underscored sex-specific differences in responses of naturalistic behavior to depressive disorders and their pharmacological interventions.

Next, we investigated drug-induced on- and off-target effects within naturalistic behavior syllables (Supplementary Fig. 18c). Chronic stress significantly altered five behavioral clusters; fluoxetine and imipramine successfully reversed three to baseline states (Supplementary Fig. 18d). Notably, off-target effects were substantially pronounced in females, with fluoxetine altering eleven syllables and imipramine influencing nineteen. Cross-sex comparisons revealed minimal overlap between male and female responses to fluoxetine (two common syllables) and none for imipramine, underscoring sexual dimorphism in behavioral responses due to divergent baseline behavioral structures post-stress induction. Moreover, fluoxetine consistently exhibited more off-target effects than imipramine across both sexes.

Finally, analysis of syllable bigram transitions revealed marked differences between female and male responses to antidepressants. Fluoxetine robustly normalized stress-induced transition modifications toward non-stressed states, whereas imipramine treatment predominantly induced off-target transitions (Supplementary Fig. 18e). In conclusion, while off-target effects on syllable usage were more evident in females, off-target transitions within syllable sequences were dominant in males, further emphasizing substantial sex-dependent variability in antidepressant pharmacodynamics.

Methods
Animals
All experimental procedures were performed in accordance with the guidelines of and approved by the Institutional Animal Care and Use Committee at KAIST (approval number No. KA2024-007-v4). Wild-type C57BL/6J male and female mice were used in the behavioral studies. The mice were group-housed for the first week at 7 weeks of age and then transferred at 8 weeks of age to separate cages under a 12-h light-dark cycle (light on from 8:00 to 20:00) at room temperature (22°C) and 40% humidity to allow them to accommodate to the housing facility. We used mice from at least three litters, and mice were randomly allocated into groups to balance potential genetic or developmental variance. Food and water were provided ad libitum.

Mouse models for depressive-like behavior
Chronic unpredictable stress (CUS) model. Wild-type C57BL/6 male or female mice (8-week-old) were subjected to nine different stressors: (i) 24 h of food and water deprivation, (ii) 8 h of cage tilting (45°), (iii) 24 h of light cycle reversal, (iv) 5 min of swimming in cold water (13±1°C), (v) 2 min of tail pinching 1 cm from the end of the tail, (vi) 8 h of moist bedding, (vii) 30 min of cold exposure (4°C), (viii) 12 h of sleep deprivation with a cylindrical pedestal (2.5 cm diameter and 3 cm height) placed on the cage floor filled with 1 cm-deep tap water to prevent the sleep during dark phase, and (ix) 1 h of physical restraint. These stressors were randomly scheduled over four weeks, with one or two stressors performed daily in a random combination to magnify unpredictability. During the test week (week 4), we omitted the cold swimming and tail pinching to prevent repetitive exposure and habituation effects. In periodic assessments of depressive-like behaviors, behavioral tests (TST and FST) were conducted at the beginning of each week, while forced swimming and tail-pinch stressors were excluded from the CUS paradigm to avoid confounding effects. Control mice were individually housed for four weeks with short daily handling in the housing room. We measured the mouse body weight each week at a fixed time during depressive-like behavior induction. Experimental batches and the number of mice per batch are listed in the Supplementary Table 4.

Lipopolysaccharide-induced mouse model. LPS (serotype O55:B5, Sigma-Aldrich, Cat#: L2880) was freshly prepared on the test day by dissolving in sterile 0.9% NaCl solution (normal saline). The solution was administered to 8-week-old male C57BL/6 mice with an intraperitoneal injection (i.p.) at a volume of 0.1 ml per mouse. The dose of LPS (0.83 mg/kg) was selected based on the meta-analysis for experimental studies on inflammation-associated depressive-like behavior, which is known to elicit the full spectrum of acute sickness behaviors39, including physiological and behavioral changes relevant to depressive-like phenotypes40. Control animals received an equivalent volume of sterile saline.

Corticosterone-induced mouse model. Eight-week-old male C57BL/6 mice were randomly assigned to three groups and received the following administration protocol: (i) corticosterone (20 mg/kg, Sigma-Aldrich, Cat#: D27840) for consecutive 21 days, (ii) vehicle (normal saline containing 0.1% dimethyl sulfoxide and 0.1% Tween-80) for consecutive 21 days, or (iii) vehicle for 14 days followed by corticosterone for 7 days. All injections were delivered subcutaneously once daily at a volume of 5 ml kg−1 body weight27,41.

Behavioral studies
AVATAR studio recording. All mice were handled in the behavior room for 10 min per day for 3 consecutive days and habituated to the room for an hour before the behavioral experiments. AVATAR studio consists of an opaque hemispheric chamber with five recording cameras (Full-HD CMOS, 1200×1200 pixels, 30 frames per second) and an LED light device (20 lux) at the bottom23. An open field apparatus, 5-mm-thick transparent acrylic panels (20 cm×20 cm×30 cm), is located at the center of the AVATAR studio. The mice were placed in the center of the chamber and allowed to freely explore the apparatus for 20 min. For the investigation of individual variance, we conducted recordings a day before the induction of a depressive-like state model. To estimate the effects of CUS on exploratory behaviors, the mice were recorded a day after exposure to the last stressor. From the video, the 3D coordination of nine key body points (nose, head, anus, tail tip, torso, left and right hind limb, left and right forelimb) was calculated from multiple 2D images following a computer-vision 3D calibration and reconstruction algorithm, which calculates the intersection of the ray from the optical center of the camera through anatomical landmarks for a given camera arrangement42. The obtained 3D coordinate parameters were processed to generate joint connections for the formation of the action skeleton, a vectorial representation of body configuration in 3D space, on both allocentric and egocentric coordinate systems. The accuracy of AVATARnet for keypoint prediction was evaluated using three standard metrics: Area of Overlap (IoU), Average Precision (AP), and Root Mean Squared Error (RMSE). AVATARnet achieved IoU consistently above 75% across nine anatomical key points and mAP values around 90% after sufficient network training. The positional error, measured as MSE, ranged from 7 to 15 pixels, equivalent to 1.4–4.5 mm in physical space, and the average RMSE recorded 10.19 pixels, corresponding to approximately 2–3 mm23.

Sucrose preference test. For habituation, we provided mice with two bottles, one containing 1% sucrose (Sigma, Cat# S7903) solution and the other containing regular water alone, for 48 h before the sucrose preference test (SPT). We changed the left-right position of the bottles every 24 h in the adaptation phase to avoid any location preference. In the formal experiment, mice were allowed to freely access the 1% sucrose and water bottles for 24 h with food withholding. The positions of the bottles were switched after 12 h of testing to exclude the potential effects of side preference. After the test, we moved the mice to the home cage and measured the total intake of each fluid. Sucrose preference was calculated as follows: sucrose preference (%) = 100 × sucrose consumption (g) / (sucrose + water consumption) (g). A minimal layer of sawdust was added to each cage to detect any bottle leakage.

Tail suspension test. Mice were suspended from a stick by their tails 50 cm above the ground for six minutes. The sticky tape was used to attach the stick to a spot 1 cm from the end of the tail41. We recorded video from the side and measured the immobility duration during 6 min of suspension using a video tracking system (Smart 3.0). Mice were considered immobile when hanging motionless, swinging passively, or remaining completely motionless.

Forced swimming test. The forced swimming test (FST) was conducted to assess the behavioral effect of stress exposure, at least four hours after the TST to minimize any potential effect of physical discomfort caused by tail suspension. A clear glass cylinder (20 cm height, 12 cm diameter) was filled to a depth of 15 cm with water (24±1°C) to prevent any physical contact of the tail or hind limbs with the bottom of the container. Mice were placed in a water-filled cylinder for six minutes. Video was recorded from the side with a web camera recording system and monitored using a video tracking system (Smart 3.0). We only quantified the duration of immobility during the final four minutes in an observer-blinded manner, as most mice move around at the beginning of the FST43. We defined immobility as the status in which mice floated (motionless) or merely moved their paws to maintain balance in the water.

Time in the center and moving distance. We extracted the time spent in the arena center and the distance traveled from the AVATAR studio recording videos to measure the locomotive function of mice. In 20-min sessions, we measured the time mice stayed in the center square of the open field (10×10 cm) and the total distance traveled by tracking the x- and y-axis coordinates of the body center.

Pharmacological treatment of animals
Monoaminergic antidepressants. Male or female 10-week-old C57BL/6J mice were used to investigate the effects of monoaminergic antidepressants on free exploratory behaviors. All drugs were dissolved in sterile 0.9% NaCl solution (normal saline) and injected intraperitoneally for 14 consecutive days. For both fluoxetine hydrochloride (Abcam, Cat#: ab120077) and imipramine hydrochloride (Abcam, Cat#: ab141046), 15 mg kg−1 was dissolved in 200 µl of vehicle. Control mice were only injected with the vehicle (normal saline). For all behavioral experiments, mice were tested 24 h after the final injection.

Non-monoaminergic antidepressants. Non-monoaminergic antidepressants were administered to 12-week-old male C57BL/6J mice exposed to chronic stress for four weeks. Agmatine sulfate salt (50 mg kg−1, Sigma-Aldrich, Cat#: A7127)44, scopolamine hydrobromide (0.5 mg kg−1, Tocris Bioscience, Cat#: 1414)45,46, and Rapastinel (or GLYX-13) (10 mg kg−1, Tocris Bioscience, Cat#: 3406)47 were dissolved in 0.9% saline and injected intraperitoneally at a volume of 5 ml kg−1 body weight, 24 h before behavioral experiments.

Manual label assignment and cluster annotation
We established a mouse behavior database for the evaluation of our machine learning framework. Thirteen different behavior labels were assigned to each frame of a 20-min video of a freely moving mouse. Coarse behavior phenotypes were defined according to the Mouse Ethogram database taxonomy (www.mousebehavior.org). We further concretized these coarse labels with fine-grained labels to provide more detailed discrimination of the spontaneous motions of well-known behavior sequences (Supplementary Table 2). After this process, each annotated behavior segment contained information about the start/end frame number and a single fine-grained label. To annotate the unsupervised learning-driven clusters with behavior components, experts explored twenty corresponding video clips for each cluster and determined if the behavior sequences in the video clip could be associated with fine-grained labels. The behavior labels that appeared in the video clip were scored and averaged as part of a numerical evaluation of cluster characteristics. We then summarized the features of the behavioral phenotypes (a maximum of two) with more subtle descriptions.

CLOSER Framework
Data augmentation. Contrastive learning requires an appropriate data augmentation strategy to maximize the performance of the machine learning model. We introduced four augmentation methods: “Scale”, “ChangeStartPoint”, “ChangeAngle”, and “ChannelMask”. “Scale” is accomplished by random scaling of the original mouse 3D skeleton vector size by a factor of 0.5 to 1.5. For the given motion sequence fi = {xi, yi, zi}t, the augmented motion sequence was defined as Scale(fi) = α{xi, yi, zi}t, while 0.5 ≤ α ≤ 1.5. Scale enables the model to learn the irrelevance of mouse size in its behavior. “ChangeStartPoint” simulates the temporal heterogeneity of behavior sequences that originates in variations in the duration of the motion segments. To accomplish this, we randomly cropped 60% of the original motion sequence and then performed a linear interpolation to the original length according to this relationship: ChangeStartPoint fi 1...T = Interpolate fi s::e, 1 ≤ s, e ≤ T, e-s+1 = 0.6 × T. We employed “ChangeAngle” to distort the angle between the skeleton vectors of a chosen joint and its parent skeleton vectors with a maximum distortion of 1.75 radians. Three joints were selected randomly. The angles between the skeleton vectors v involving the chosen joints and their parent skeleton vectors v were calculated using the following equation: θ = acos (v1·v2 / |v1||v2|); θnew = θ + Δ, -30° ≤ Δ ≤ 30°. For the investigation of new cartesian coordinates X, we built a linear system as follows: Concat(n, v1, v2) X = Concat(P v1 + |v2| |v1| cos θnew, |v2| |v1| sin θnew) with n = v1 × v2 and d = n × joint, where joint denotes the cartesian coordinates of the chosen joints. p1 and p2 refer to the parent joints of the chosen joints. Similarly, we rotated the skeleton vectors with the chosen joints orthogonal to the plane containing v1 and v2 by at most 0.875 radians. Through this augmentation, the model learned the angular difference in key joints. “ChannelMask” is a random mask of the x- or y-axis with a 25% probability of implementing independence of behavior contents on a specific axis. For example, if the y-axis is masked in a motion sequence fi = {xi, yi, zi}t, ChannelMask(fi) = {xi, 0, zi}t. To validate augmentation combinations through ablation analysis, we introduced existing augmentation strategies widely used in human action recognition. For the first augmentation combination20, we adopted two spatial skeleton augmentations (“Shear” and “JointJittering”) and a single temporal skeleton augmentation (“Temporal Crop-resize”). The “Shear” transformation gave variations in viewpoint and distance to the camera with pose conservation. In more detail, for the segment X ∈ Rbatchsize×numjoints×joint size, the modified action representation Shear(X) = X * [[1, r01, r02], [r10, 1, r12], [r20, r21, 1]]. “JointJittering” inserts a perturbation in joint connections through their regular movement of random joints to teach the model invariance to noise in joint positions. To achieve this, we multiplexed three randomly selected joints j, with a jitter matrix as JointJitter(X) = X[:, j, :] * [[r00, r01, r02], [r10, r11, r12], [r20, r21, r22]]. All entries in the shear and jitter matrices are randomly drawn from a uniform distribution [-1, 1]. Since “Temporal Crop-resize” adjusts a similar distortion to the “ChangeStartPoint” method, described above, we utilized our augmentation module without modification. In a second augmentation strategy, we integrated augmentation methods proposed for human depression detection48: “Shear”, “Rotate”, “GaussianNoise”, “JointMask”, and “ChannelMask”. In “Rotate”, we randomly rotated a 3D action skeleton with respect to the x-, y-, and z-axes with random variables α, β, γ ∈ Rbatchsize×1×jointsize×jointsize. “GaussianNoise” adds appropriate Gaussian noise to the action sequence. We applied the Gaussian noise matrix ε ∈ Rbatchsize×1×numjoints×jointsize, whose entries are taken from a Gaussian distribution with mean of 0 and a standard deviation of 1, to the action segment X ∈ Rbatchsize×inputlength×numjoints×jointsize as GaussianNoise(X) = X + ε. “JointMask” generates positive pairs by concealing three random joints by X[:, :, j, :] = 0. For self-supervised representation learning, we employed Cross-Modal Mutual Distillation (ST)49, following the structure of Spatio-Temporal Graph Convolutional Network (ST-GCN) building encoders50. A given full-length motion sequence obtained from AVATAR studio was divided into segments with a single motion segment unit of 10, introducing a 1-frame stride to provide generalization in input action sequences. The dataset consisted of 120 label-free videos of control and depressive-like behavior model mice for 4 weeks, and each video contains a whole 36,000 frame motion sequence of freely moving mice. In the CLOSER framework, training, validation, and test datasets are generated with a ratio of 60:20:20 while randomly sorting motion sequences. We used encoders composed of 9 layers of the spatio-temporal graph convolution operators; 3 layers of 32 output channels, 3 layers of 64 output channels, and 3 layers of 128 output channels stacked in order. This implies that the encoder is designed to map the input space of R10×27 to an embedding space of R128. In the CMD modality design, we implemented two approaches, joint movement and motion change. Joint modality represents Cartesian coordinates of key points from a “fix pose” motion sequences, and motion modality defines a difference in Cartesian coordinates of key points between adjacent frames from “pose” motion sequences. For the projection head module, linear layers of input dimension I ∈ R128 and output dimension O ∈ R128 without any activation function. The model was trained to optimize InfoNCE (Info Noise-Contrastive Estimation) loss51 and Kullback–Leibler (KL) divergence. For InfoNCE loss LSCL and CMD loss LCMD, the total loss function L = LjointSCL + LmotionSCL + LCMD was calculated. For the size of a queue-based memory bank N and temperature hyper-parameter τ, InfoNCE loss for joint and motion modality is defined as LSCL = -log [exp(zTq zk / τ) / (exp(zTq zk / τ) + ΣNi=1 exp(zTq zmi / τ))]. The knowledge distillation was achieved by minimizing Kullback–Leibler (KL) divergence of similarity distributions LCMD = KL(p(zjoint, τt) || p(zmotion, τs)) + KL(p(zmotion, τt) || p(zjoint, τs)), where p(z, τ) = Σ exp(zT ni / τ) / Σ exp(zT ni / τ), i=1, 2, ..., K. All hyperparameters used to optimize the CLOSER framework are listed in Supplementary Table 3.

Temporalsegmentation.Forconsiderationoftemporaldynamicsinbehaviorsequences,wemodifiedthemajorityvotealgorithmtosplitsequentialframes into semantic motion segments.An action representation z was computed by zi = ST-GCNjoint(segmenti) for i-th segment segmenti and the projection head-free query encoder ST-GCNjoint for joint modality.In dynamic temporal segmentation, we define the set Si as Si = {segmentj | i-T+1 ≤ j ≤ i+T-1} where T is the segment length, which is set to 10. We labeled each segment in Si with k-means clustering in the embedding space, then assigned the most frequent frame-level label. Label(frame i) = argmax s∈Si Label(ST-GCNjoint(s)). The consecutive frames with identical labels were paired to represent a behavioral unit. In addition, we employed a keypoint change score function52 for further estimation of the ability of the framework to capture the robust shift in the action skeleton.

Linear regression
The linear correlation of conventional depressive-like behaviors and spontaneous behavior was driven by fitting a linear regression model given variable Xi and observations yi, the linear regression model calculates coefficients βi and residual β0 by minimization of the following equation. L = 1/2 Σ (yi - ^yi)2, where yi = β0 + βiXi. Then, we estimated the coefficient of determination, the R2 score, for the numerical expression of predictability for the dependent variable from discriminant-relevant syllables.

Anomaly detection
Model configuration. To estimate the behavioral abnormalities of the disease model mice, we employed a Message-Passing Encoder-Decoder Recurrent Neural Network (MPED-RNN) model53. At each time point t, we decomposed the 3D pose representation into a global component and a local component fi = (xi, yi, zi)t. The global component represents a motion of the body center, implying a change in the absolute position of the 3D pose representation. It is modeled as the combination of the coordinates of the body center and the body’s bounding cuboid. Hence, we defined the global components as fgt = (xg, yg, zg, w, l, h). Each feature of the global component was calculated with the following equations: xg = (max xi + min xi) / 2; yg = (max yi + min yi) / 2; zg = (max zi + min zi) / 2; w = max xi - min xi; l = max yi - min yi; h = max zi - min zi. We defined the local component as a relative motion of key joints with respect to the body center. It was calculated from the global component as follows: xl,i = (xi - xg) / w; yl,i = (yi - yg) / l; zl,i = (zi - zg) / h. We introduced encoder Gated Recurrent Units (GRU) modules and decoder GRU modules for both the global and local components to implement MPED-RNN. Each module sends messages to its counterpart and receives messages for the calculation of the latter state. The messages exchanged in the module with current state h were computed as specified below, where σ denotes the activation function, W represents the weight, and b is the bias term: messagel→g = σ(Wl→g hl + bl→g); messageg→l = σ(Wg→l hg + bg→l). Then, to generate training segments, we used a utilized sliding-window strategy as follows: segment i = {ft} t=bi::ei for bi = s×i, ei = s×i+T, where the sliding stride s=10 and the segment length T=60. In model training, we measured reconstruction and prediction loss using the loss function suggested by Morais, Romero, et al.: L = 1/T Σ ||f*t - f*t||2 + 1/P Σ ||f*t - f*t||2, where P represents prediction length, which was set to 30, and * denotes either l or g. The training and validation datasets were split into an 80:20 ratio. To calculate the frame anomaly score, we used the following equation: anomaly score(frame i) = Σ segment∈Si fT l,gL * (segment), where Si = {segmentj | i-T+1 ≤ j ≤ i+T-1} denotes the set of segments containing frame i. For simplicity, we only considered frames belonging to T segments and analyzed the distribution of frame-wise anomaly scores for the evaluation of abnormality in the disease group compared to its control group.

Visualization. To visualize the embedding spaces of the global and local components, we leveraged an unsupervised deep anomaly detection approach called LatentOut, which was developed by Fabrizio Angiulli et al.29. With the mean μ and standard deviation for an anomaly score distribution σ, we normalized anomaly scores with the following equation: anomaly normalized score(frame i) = (anomaly score(frame i) - μ) / σ. Then, we projected the results for the embedding structure of global and local components using PCA onto a 3D coordinate system with the z-axis defined as z(frame i) = Σ frame∈Nk (frame i) anomaly score normalized(frame), where Nk(:) is the k-nearest-neighbor using default parameters (k=5). In the k-nearest neighbor module, distance was defined as Euclidean distance between projected cartesian coordinates and the normalized anomaly score.

Hierarchical clustering
To estimate the overall clustering structure of the CLOSER framework, we performed a hierarchical recategorization of the embedding. An average embedding zc for behavioral cluster c was measured according to zc = Σ zi / N, where zi is the embedding information of the ith segment in cluster c, and N is the number of segments in cluster c. We leveraged the Scikit-learn (sklearn) module for agglomerative clustering to group clustered objects based on similarity using complete linkage.

Dimensionality reduction
Principal component analysis (PCA). To extract the features of high-dimensional behavioral sequences using variance-maximizing projection54, we employed the Scikit-learn implementation of PCA with 13 components, with default parameters: (i) whiten=False, which determines the original variance in scalar values, (ii) svd solver=0auto0, which solves based on the input dimensions and the chosen number of components.

Uniform manifold approximation and projection (UMAP). We introduced UMAP to perform nonlinear dimensionality reduction, with the preservation of the topological structure of behavioral data within a low-dimensional embedding55. We leveraged the UMAP implementation in 13-dimension, with default settings: (i) n neighbors=15, (ii) min dist=0.1, and (iii) metric=0euclidean0 in pairwise distance measurements.

Benchmarking
To estimate the quality of the embedding precisely, we compared it to that of existing motion analysis frameworks, VAME and keypoint-MoSeq. Variational animal motion embedding (VAME). For the VAME, we modified the input structure to accept 3D behavioral segments as input through the extension of the number of features, from 12 to 27, and the embedding dimension, from 30-dimension to 108-dimension. In model training and inference, we adjusted the frame lengths of reconstruction and prediction methods to ten and five frames, respectively. All 3D pose trajectories were concatenated across animals and conditions and then windowed into 10-frame segments. Each segment was normalized and fed into the VAME encoder in batches of 4096 segments. From each batch, the VAME encoder produced a 4096×108 array of latent means (μ), followed by the concatenation of all batches to reconstruct a full embedding matrix. As the variational autoencoder architecture of the VAME consists of an encoder and a decoder, each using GRUs, we trained the model with the given parameters for reconstruction, prediction, and KL divergence, which shows the maximal performance in behavioral classification. For the i-th segment segment, we generated the action representation zi = VAMEencoder(segmenti). We applied both a Hidden Markov Model and k-means clustering after majority voting to assign frame-wise cluster labels.

Keypoint-MoSeq. For the keypoint-MoSeq, we configured the model to align with its key joints and connections, maintaining consistency with the variational autoencoder structure. The 3D pose trajectories were loaded and directly subjected to PCA, while omitting the noise-calibration step, which had been applied in AVATAR calibration. We then initialized and fit an autoregressive hidden Markov model on the PCA projections, followed by fitting the MoSeq model. Calculated inferred state sequences were exported to obtain frame-wise behavioral syllable labels for downstream analysis.

Investigation of representative behavior phenotypes
A behavior segmentation system requires the ability to cluster motion sequences into discrete modules of action for further ethological interpretation. Hence, we inspected exemplary behavior phenotypes (i.e., walking, turning, rearing, and falling) from automatically categorized motion segments to identify the semantic implementation of the framework. For behavior segments longer than 10 frames, we equally divided the segments into 10 sections and averaged each metric over a single section to summarize the behavioral features on an identical time scale. The set of frames Fn involved in cluster n was defined as Fn = {Fi = (xi, yi, zi) | j ∈ Joint, s ≤ i ≤ e} with start frame s and end frame e for key joints Joint. The i-th behavior segment in “walking” clusters was evaluated with moving distance d where dist(:,:) denotes the Euclidean distance between two body points from consecutive time sections t and t-1. di = 1/N Σ dist(xj, yj, zj, xj-1, yj-1, zj-1), N = e-s+1, s+(i-1)N+1 ≤ t ≤ s+iN. To measure the degree of “turning”, we measured the angle ai = 1/N Σ angle(v1, v2), N = e-s+1, s+(i-1)N+1 ≤ t ≤ s+iN between the anus-body center vector, v1, and the body center head vector, v2, where angle(v1, v2) = acos(v1·v2 / |v1||v2|). For the qualification of behavior segments in “rearing” and “falling”, we measured the height of the nose as follows: hi = 1/N Σ znose,t, N = e-s+1, s+(i-1)N+1 ≤ t ≤ s+iN.

Normalization
We normalized the frequency and transition probabilities for the behavioral clusters using the initial behavior structure while considering individual variance in the exploratory behavior paradigm throughout the subjects. We implemented fold change of a cluster frequency Freq and transition probability p for the k-th segment xk: FreqNorm(c, week, mouse) = Σ j f(xk | xk ∈ week m ∩ mouse n, Label(xk) = c) / Σ l m n j f(xk | xk ∈ week m ∩ mouse n, Label(xk) = c), pNorm(ci → cj, week, mouse) = Σ k f(xk-1, xk | xk-1, xk ∈ week m ∩ mouse n, Label(xk-1) = ci, Label(xk) = cj) / Σ k f(xk-1, xk | xk-1, xk ∈ week m ∩ mouse n, Label(xk-1) = ci, Label(xk) = cj).

Linear discriminant analysis (LDA)
To calculate the class-separability through a dimensionality reduction, we projected the training configuration of Linear Discriminant Analysis (in sklearn) results into the plane (nx, ny)projected = LDA(ST-GCNjoint(xi)) for labeled data x = {t, t, y=c}.

Pearson correlation coefficient
We calculated the Pearson Correlation Coefficient (PCC) by dividing the covariance of the variables by the product of their standard deviations, yielding a value between -1 and 1, as follows: rxy = E[(X-μX)(Y-μY)] / σX σY, where X and Y are latent vectors, μX and μY are respective means, and σX and σY are standard deviations.

k-nearest neighbor classifier
For quality evaluation of latent embedding, we employed a non-parametric, k-Nearest Neighbor (k-NN) classifier that predicts the label of an input instance by identification of the k closest labeled neighbors within the feature space and assignment of label using majority voting. We utilized the KNeighborsClassifier from the Scikit-learn library with default parameters: five neighbors (Nneighbors = 5) with uniform voting weights, using the Minkowski distance (p=2) corresponding to the Euclidean metric. The classifier was configured to automatically select an optimal search algorithm (algorithm=0auto0) with a leaf size of 30, without additional metric parameters.

Random forest classifier
To verify the behavioral correlation between free movement and depressive-like behaviors, we obtained the confusion matrix for control and depressive-like behavior model mice utilizing the Random Forest Classifier (from sklearn) with default settings. The Random Forest Classifier was trained with five-fold validation. To prepare the dataset, we partitioned 20-min sample recording videos into 5-min sections and calculated fold changes for cluster frequencies and transition probabilities as described in “Normalization”. To validate the size invariance of CLOSER and other benchmarking algorithms, eighty male and female mice were divided into five weight categories, spanning 18 g to 28 g in 2-g increments. The Random Forest classifier was trained to predict the weight categories from the embedding vectors extracted in non-overlapping 100-frame strides from behavioral sequences. We trained the classifier with 5-fold cross-validation in the default setting and evaluated the performance using precision, recall, and F1 score.

SHapley additive exPlanations (SHAP) analysis
To investigate the influence of behavioral cluster frequencies on classifier decisions, we applied a SHAP analysis56 using the scikit-learn library. Initially, we trained a Random Forest Classifier without employing k-fold validation followed by TreeSHAP57, which defines the value function through conditional expectation to extract SHAP values. Given the feature set N, a subset S, the classifier function f, and the input x, fS(x) denotes the classifier output restricted to S, which can be further described as the conditional expectation fS(x) = E[f(x) | xS]. The SHAP value for the i-th feature was then calculated as follows: ϕi = Σ S⊆N\{i} |S|!(|N|-|S|-1)! / |N|! [fS∪{i}(x) - fS(x)]. Leveraging the additivity of the SHAP value, average absolute SHAP values for overall feature importance were calculated and depicted as cumulative bar graphs, while SHAP values for individual groups were represented using beeswarm plots.

Transition probability
The transition probability from behavioral cluster ci to behavioral cluster cj = w was calculated from the transition rate from the behavioral cluster of the k-th segment xk: pi→j = Σ k f(xk-1, xk | Label(xk-1) = ci, Label(xk) = cj) / Σ k f(xk-1 | Label(xk-1) = ci). To compute the change in transition probability for a behavioral cluster, we estimated the difference of each transition probability and visualized leveraging the network analysis framework NetworkX. Only transitions whose alteration was significant (p<0.05) were displayed in transition graphs.

Statistics
Statistical analysis and graphs were produced in Python (3.10.12) and GraphPad Prism 10. Time-series datasets were analyzed by two-way ANOVA with Geisser-Greenhouse corrections. Post hoc analysis for multiple comparisons leveraged the Benjamini–Hochberg test to control the false discovery rate, which we fixed at 0.05. For the estimation of significance in groups of more than two, we estimated one-way ANOVAs with a 95% confidence interval and Dunnett’s post hoc analysis for multiple comparisons to the control group. Two-group comparisons, including those for syllable usage and syllable subset analysis, were performed using two-tailed unpaired t-tests (parametric) if the group passed the Anderson–Darling (AD) normality test. Otherwise, we used a Mann–Whitney test (non-parametric) to deal with a non-Gaussian distribution. Classification performance of k-NN and random forest classifiers was summarized with average values and standard deviation of precision, recall, and F1 scores. Box graphs show the 25th–75th percentiles for the whole dataset with a line indicating the median. Bar graphs and timelines present means ± standard error of the mean. Asterisks denote levels of statistical significance as follows: *p<0.05, **p<0.01, ***p<0.001, and ****p<0.001. More detailed descriptions are summarized in Supplementary Data 1, with statistical significance marked with yellow blocks.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The manuscript describes detailed quantification of spontaneous behavior in a semi-naturalistic environment using high-powered hardware and software. The authors then attempt to apply this method to advance research into mouse models of chronic stress-induced behavioral changes relevant to the study of depression and anti-depressant effects. Whilst the approach is potentially interesting, there are major issues with the manuscript.\n1. The authors are correct to, in the Introduction, be highly critical of the readout tests of forced swim test and tail suspension test, given their lack of face validity. However, they then proceed to use these very same tests to compare the behavioral effects of chronic stress in standardized tests with that in their locomotion paradigm. Furthermore, they interpret the effects of stress in the standardized tests in the highly subjective way of mice really being in a state of \"despair\" or \"helplessness\". These are antidepressant screening tests and do not provide information relevant to the major symptoms of depression. Examples of translational behavioral paradigms relevant to major depression symptoms would be reward motivation and Pavlovian fear condiitoning.\n2. The manner in which the authors describe the outputs of their algorithm renders it impossible to comprehend what actual behavioral changes they observed in locomotion in a semi-naturalisic environment and how this might be relevant to major symptoms of depression. In principle it would be very interesting to know what locomotor changes are induced by chronic stress, but the data need to be presented in a way that is understandable and using measures of overall activity, general body posture etc.", "reviewer_2": "Here, Oh and colleagues describe a self-supervised approach for segmenting three-dimensional keypoint trajectories into discrete behaviors that they call CLOSER. Specifically, CLOSER leverages self-supervised learning – a technique increasingly being used for action segmentation in human data – to learn an embedding of 3D kinematic data from spontaneously behaving mice. The learning objective is constructed to learn an embedding space where points representing similar behaviors are close together and dissimilar behaviors are far apart, while also remaining invariant to “nuisance” features such as body size. Points in the embedding space are clustered to form a segmentation of 3D kinematic data into discrete behaviors. They benchmark CLOSER against the currents state-of-the-art, Keypoint-MoSeq (KP-MoSeq) and Variational Animal Motion Embedding (VAME), and find that their technique has better correspondence with manual labels. Finally, the authors demonstrate that CLOSER can be used to dissect the behavioral consequences of inducing depression in a chronic stress model.\nOverall, I found that the authors have an intriguing idea that is likely to be well-received by the Nature Communications readership. However, I found some major technical issues that must be addressed before this can be considered for publication. I hope most of this can be remedied by some analytic controls without any additional experiments.\nMajor problems that must be addressed before the manuscript can be considered for publication:\n1. Contrastive learning is used to learn an embedding space of 3D kinematic data that is purportedly sensitive to changes in behavior and insensitive to other factor such as mouse size. This is a reasonable approach in principle; however, unless I missed it, I found no validation that their method achieves the stated objective.\n1a. Specifically, the authors do not provide evidence that CLOSER’s embedding strategy is actually invariant to size while remaining sensitive to the differences between kinematically distinct behaviors. Here, the authors need to provide direct quantification that their embedding space, as they say, works better than PCA, autoencoders, t-SNE, etc (line 380). This cannot be asserted without direct quantification. A simple path would be to use their manual labels and quantify the average distance between all pairs of points in different manually identified behavioral states and within behavioral states but across different subjects, especially across subjects with different weights and sizes.\n1b. Second, it would be extremely useful to show that, if their embedding shows better qualities that alternatives, how important the specifics of their augmentation strategy is. The authors use prediction accuracy of manually identified labels to validate their particular combination of augmentations. This should be complemented with analyzing the geometry of their embedding (see point above), and repeating this analysis across different combinations of augmentation functions.\n1c. Third, I was a little puzzled by some of the augmentations. If I understand correctly, Scale simply scales x, y, z coordinates by the same factor. Can the authors justify why this is a reasonable thing to do? Can keypoint coordinates across different mouse sizes truly be reduced to a single scaling factor across x, y, and z?\n2. The performance comparison with VAME and KP-MoSeq, which is central to the manuscript, is confusing.\n2a. If I understand correctly, the authors specifically tailored their augmentation strategy (Sup. Table 1) to maximize prediction accuracy on the same dataset that is used for the performance comparison in Figure 2. If this is the case, then the authors need to come up with two completely separate datasets – one for tuning hyperparameters, and another for performance evaluation. Either way, any hyperparameters in VAME and KP-MoSeq must also be tuned in the same fashion; that is, if the authors are going to tune hyperparameters of their own algorithm to maximize performance, then this must be done with all other algorithms to foster a fair comparison.\n2b. Can the authors elaborate on the embedding space that was used for KP-MoSeq? KP-MoSeq is not really an embedding algorithm, it simply uses PCA of keypoint trajectories as the input space for a probabilistic time series model. Did the authors attempt to make predictions using an MLP using the standard input for KP-MoSeq? If so, the authors need to make it clear that they’re not comparing their algorithm to KP-MoSeq, but rather to the input of KP-MoSeq. If I am misunderstanding what they did, and they in fact used the output of the model, how were the model hyperparameters tuned?\n2c. The comparison with VAME seems more apples to apples than the comparison with KP-MoSeq, which is good. That said, the performance difference between the two algorithms looks fairly subtle (Fig. 2c), and as mentioned above, the hyperparameters of CLOSER appear to have been tuned for maximum performance. Were the hyperparameters of VAME tuned at all? If not, this must be done in the same way hyperparameters were tuned for CLOSER. Additionally, given that the performance difference between CLOSER and VAME is small, the authors should provide uncertainty estimates, using different cross-validation folds or multiple restarts of each model with different random seeds.\n3. Another relevant detail here is that AVATAR uses a different form of keypoint tracking. Rather than using a standard markerless keypoint tracker, my understanding is that AVATAR learns the centroid of pixels associated with a given body part. If so, can the authors elaborate on the noise characteristics of this type of tracking relative to the output of, e.g., DeepLabCut, SLEAP, or LightningPose? I could not find these details in either this manuscript or the preprint describing AVATAR. Does the apparent performance benefit of CLOSER still apply when using standard keypoint trackers rather than AVATARnet?\n4. If the authors are going to compare KP-MoSeq and VAME with CLOSER, they should also show that CLOSER provides clear benefits in the analysis of the chronic stress model (e.g. the decoding in Fig. 3h).\nMinor issues:\n1. This may just be my misunderstanding. Line 611 of the methods describes the training/test/validation split. Since the authors are analyzing data in a 10 frame window with 1 frame stride, can the same frame appear in both the train and test data? If so, the analysis needs to be repeated while accounting for any potential data leakage.\n2. The writing is a bit hard to follow in places. For instance, Line 376-379 I am not quite sure what the authors mean by the “Neuroethological nature” of their “psychomotor analysis”.\n3. Could the authors provide a more detailed explanation of their dynamic temporal segmentation? My understanding is that each frame is given the most frequent label assigned to that frame across all windows that contain it; however, this is a bit hard to discern from figure 1.\n4. Time complexity is mentioned as a major benefit of their segmentation algorithm relative to HMMs. However, can the authors provide a simple comparison of performance between the two? Curious how they compare against the manual labels.\n4a. A related point about time complexity. Can the authors remark on the compute time for each step of the complete CLOSER pipeline? It would be useful to show this relative to VAME and KP-MoSeq as well.", "reviewer_3": "In this manuscript, Oh and colleagues present the self-supervised learning platform CLOSER, which is aimed to assess parameters of spontaneous behavior in mice (3D allocentric and egocentric skeleton plus kinematic features) and link them with altered depression scores in healthy controls vs UCMS (unpredictable chronic mild stress) model in both sexes as well as predictive validity-associated alterations induced by antidepressant drugs (the SSRI fluoxetine and the tricyclic imipramine). The authors claim that CLOSER results are of enhanced translational value compared to standard assays (forced swim, tail suspension test sucrose preference test).\nComparable efforts to assess psychomotor activity in mice focusing on variations (i) across strains (healthy and disease models) or (ii) variations induced by drugs have been published recently (e.g. Sheppard et al 2022 and Wiltschko et al 2020). Here, the authors provide interesting longetudinal data where CLOSER measures the development of a „depression-phenotype“ across 4 weeks of UCMS as well as the ability of CLOSER to measure distinct properties (as well as off-target effects) of antidepressant treatment which could be potentially interesting in shaping antidepressant treatment to a specific depression subtype as well as potentially detecting different disease stages in depression. This is a highly interesting manuscript, however I think the are several weaknesses that lower the value, which there are:\n- the authors did only test one depression model (UCMS). They do not show that CLOSER is working in other models (e.g. learned helplessness, social defeat stress, chronic CORT or genetic models like BDNF Val66Met or SERT polymorphisms).\n- they claim CLOSER can show sex-specific effects but then only include male animals when testing predicitive validity with the 2 antidepressant drugs\n- they claim that CLOSER has better predictive valdity than classical tests for depression but then only test drug classes that actually perform well in the classical tests too (monoaminergic drugs)\n- Also „rodents“ would imply that they tested more than one species which they did not (they only tested mice, it is untested if CLOSER would work in rats).\n- The extended figures were not provided to me so I could not review them in detail. However, statistical summaries were present and allowed some conclusions. Due to the excellent quality of the rest of the data presented I would not expect major mistakes in this graphs but please provide them in the next round.\nI recommed a major revision and suggest that the authors (i) add another (genetic not stress-based) depression model to strenghten their results or (ii) add a non-monoaminergic antidepressant (e.g. ketamine) to show that CLOSER could performing better in screenings, where the classical depression tests fail.\nAlso I strongy advice to remove overreaching statements e.g. that CLOSER has enhanced translational value – this statement is only based on the presence of psychomotor symptoms (slowed speech, decreased movement, and impaired cognitive function) in human patients. There are no data in this manuscript allowing a conclusion regarding the translational value of CLOSER. Psychomotor retardation scoring in humans is more complex, limb-trunk movement is only as small part of it. The translational value of the elegant, but also simplified, scoring of CLOSER has still to be elucidated. So please phrase this with a little bit more caution.\nAlso: Manifestations of psychomotor symptoms are common in patients with melancholic depression and those with psychotic features – in other words psychomotor symptoms are not a common feature in depression. There are various forms of depression, including atypical, seasonal, melancholic, psychotic, catatonic, double depression (comorbid dysthymia) and anxious/agitated forms of depression. This limitation should be mentioned!\nThere are several minor issues which I will summarize according to chapter/lines:\nIntroduction\nGeneral: I‘m missing a section describing how psychomotor retardation is scored in humans (MARS, SRSS etc) and how the authors take into account that for example in MARS five major body categories including eyes, face, voice, limbs, and trunk are used, while CLOSER tries to do the same with only limbs and trunks in mice.\nLine 49-52: Reference 3-5 are underscoring the efforts to discriminate depression subtypes, none of them supports the claim that „motion parameters … are among the most symbolic symptoms of psychiatric diseases“ please add a Reference there.\nLine 56-57: The forced swim test has high predictive validity for monoaminergic antidepressant drugs (SSRIs, SNRIs as well as MAO-I and tricyclic actually reduce the immobiltity in the FST), but not non-monoamine compounds (e.g. ketamine, neuropeptides). I would formulate that a little bit more precise (or less harsh) especially as you are not testing CLOSER with non-monoamine compounds\nLine 76: please add one sentence to explain the concept of contrastive learning\nResults:\nLine 127-129: something wrong here; the written numbers do not match the graph Fig 2b?? please correct\nLine 167-169: as this was no real Open field test (20x20cm arena was to small), Lux illumination?? I suggest to remove the time in the center graph (Fig 3e) as conclusions regardings anxiety cannot be drawn in this setting; Furhtermore, please split the distance moved into 5min segments as you did with the clusters (Fig 3j-k); the difference in the clusters could be driven by alterations in locomotion, please show if this is the case\nLine 209 / Suppl Fig 2d-f: estrous cycle analysis is missing here, that could explain the low correlational values, might mention that in the text\nLine 318-333: or this means that the antidepressants didnt work in all animals. did you check for individual variety within your dataset? According to your classical behavioral results you have some „non-responders“ to the UCMS protocol what about the antidepressants (should be in the extended dataset that wasnt provided so i couldnt check). It would be great if CLOSER could predict non-responders!\nMethods:\n- Please add the License number of the animal experimental license approving the experiments\n- please add that both sexes were used and the total n number, estrous cycle measurement for females?\n- point out that drug tests were performed in male animal only (this is a pity as here there would have been the chance to show a potential advantage of CLOSER as the classical tests dont work well in females!)\n- please define your room temperatur in °C\n- how many batches of recordings? are these litter mates?\n- was the analysis repeated with a seperated experiment to varify that the clusters identfied by CLOSER can be reproduced in a different set of animals?\n- the very small open field (20 x 20cm) likely results in your high center values (approx. 35-50% center time: in this small arena the exploration of the walls perpendicular to the wall direction will most probably result in detection of the animal in the center;)\n- OF: center plus wall area are nearly all of the arena, wall area usually defined as +8cm distance from wall, center defined as 10x10cm at a complete size of 20x20 the apparatus is much too small for analyzing center time! Please remove all statements regarding anxiety\nSee guidelines of Mouse Behavior phenotypes from https://mousebehavior.org/ethogram-comparison/\n- First there are only 2 recordings (prior and post UCMS) then there is 1 recording per week for Figure 4, please clarify the details (n mice, sex, age, how you dealt with repeated FST/TST etc)\nUCMS Protocol:\n- 13 degree Celsius water for 5min?! that is a very strong stressor - shallow water exposure is what is used normally and at 24-26 degrees! (e.g. JOVE Burskin & Doron 2018), TST can be part of UCMS but problematic here because its also used as readout, food/water deprivation unusualin UCMS, cold exposure too, sleep deprivation as well. Very strong stressors for an UCMS!\nI wouldnt get this protocol through the ethics committee in my country (in Europe). Please consider that for follow up experiments.\n- sucrose preference – why additional food deprivation? normally not necessary, just take the data from your ‘habituation phase’? does this indicate that you did not get nice results there and increased the drive via food deprivation?; please at least state, if the bottles were switched in the “real test” after 12h to exclude side preference influences? sawdust? does this mean they animals didn’t have proper bedding depth (that’s a stressor that will influence your results)\n- performing tail suspension test and forced swim test on the same day might have affected the FST data, you would for example expect some pain in the tail after the TST that can affect swimming performance;\n- in addition: the UCMS protocol already contains something very similiar to the FST and this will affect your FST results too, especially if the 2 are temporally close (how did you deal with this in the depression progression protocol where you did FST and TST every week?)\nDiscussion\nline 409-410: overstatement only one depression model was used\nline 416: you are stating that this algorithm is used in humans? where is the reference for that?\nline 419 – overstatment, where did you show a translational relevance of CLOSER? there are no human data in this manuscript\nLimitations need to be formulated more precisely (no other depression models, only 2 drugs, mice only etc)\nFigures\nGeneral: please make sure that n numbers can be found in the figure legends e.g. missing in Fig5\nFigure 3 legend:\nLine 1017: typo\nLine 1019: I think you mean decreased, same in 1020; or better gained less weight\nLine 1044-1045: this doesnt match what you state in the beginning? please correct and explain why there are sometimes more and sometimes less animals in the behavioral tests (e.g. Female TST 22 vs 21, FST 24 vs 24, SPT 24 vs 24), are these different experimental batches?\nSupplements:\nminor:\n- line 60: delete SPT statistics in figure legends (they are there 2x)\n- Suppl Figure 4e-h, clarify what’s the CTRL in e-f) in comparison to the data of the control in g-h) I guess these are separate experiments but please clarify it\nstats table\n- typo – Cluter instead of Cluster nearly everywhere\n- Fig 4 stats: why are there significant values for Cluster 6, 16 etc but they are not shown in the graph? even if only the results not shown in the graph are summarized here (and i would have expected otherwise), cluster 7 is on both??? please check this there is something wrong\nStatistics\ngeneral: do you test for normal distribution before performing ttest? if yes please add to the Statistics paragraph.", "reviewer_4": "The manuscript presents CLOSER, a self-supervised contrastive learning framework designed to analyze spontaneous rodent behavior in a chronic mild stress model of depression. This approach represents a step forward in behavior quantification by integrating 3D skeleton tracking and unsupervised learning. CLOSER provides a tool for advancing preclinical psychiatric research, particularly in identifying nuanced depressive-like behaviors and tracking responses to antidepressant treatments in a more ecologically valid setting.\nKey contributions of the manuscript include the following:\nFirst, CLOSER enhances behavior quantification by integrating 3D pose skeleton data, which captures detailed motion patterns and adds granularity to behavioral segmentation. This approach enables a more refined quantification of behavior, which is especially beneficial for nuanced analyses in psychiatric and behavioral neuroscience research.\nSecond, the framework leverages a self-supervised contrastive learning approach, organizing behavioral representations effectively without reliance on labeled data. While contrastive lost is common in other areas of ML it was not applied to animal behavior quantification yet.\nThird, CLOSER applies its analytical power to the Chronic Mild Stress (CUMS) model, identifying behavior changes associated with chronic stress in rodents and revealing sex-specific and time-dependent behavior patterns. These findings offer insights into the differential impact of chronic stress on behavior, and the model’s application to pharmacological studies demonstrates its potential for preclinical drug screening.\nFinally, the model shows potential for creating standardized, reproducible behavioral assays, with the ability to quantify behavioral changes linked to both depressive-like phenotypes and antidepressant responses. This represents a promising advancement for translational research and reproducibility in animal models of psychiatric disorders.\nCLOSER’s implementation builds upon the MoCo (Momentum Contrast) framework by Meta, incorporating several similar components for contrastive learning, such as a dual encoder system, momentum-based updates, and a projection head. The model architecture mirrors MoCo’s query-key structure, with two encoders to process different behavioral data modalities, stabilized by a momentum mechanism that supports consistent embeddings over time.\nThe originality in CLOSER lies in its domain-specific adaptations, such as using 3D skeletal and kinematic data and tailoring augmentations to the behavioral nuances of rodents. However, the modifications are incremental rather than fundamentally new: they focus on adapting MoCo’s principles to the nuances of behavior analysis rather than introducing novel learning techniques or architectural elements.\nDespite these contributions, there are several points that raise questions and could benefit from further clarification or refinement.\nMajor points:\nFirst, while the reported accuracy increase over VAME is an improvement, it remains marginal. The absence of error bars or confidence intervals for the reported benchmarks raises questions about the robustness and generalizability of these findings, especially considering potential dataset-specific effects or stochastic variations. Including error bars, confidence intervals, or statistical tests would support the accuracy claims and enhance clarity regarding model generalizability across different datasets.\nSecond, the use of 3D skeleton data indeed enhances behavioral segmentation, yet this could likely be achieved by combining DANNCE or similar 3D tracking tools with VAME. This integration raises questions regarding the necessity of developing an entirely new model to incorporate 3D capabilities. Comparing CLOSER’s performance with a 3D-augmented VAME (i.e. running DANNCE and adding the 3D data as input to VAME) would clarify whether CLOSER’s use of contrastive learning advantages over an adaptation of existing frameworks, or if the improvement stems only by the fact that more data (3D coordinates) are fed to the model.\nThird, while the findings on chronic stress-induced behavior changes are compelling, it remains unclear if these insights are due to the unique capabilities of CLOSER or simply the result of analyzing a new benchmark model. Including baseline results with existing models for the same chronic stress data would help establish the unique value of CLOSER’s findings in this context.\nFinally, the paper does not provide a detailed comparison of the findings related to the CUMS model with existing literature to establish their novelty in the neuroscience field. Without such a comparison, it's challenging to determine whether these behavioral patterns represent new discoveries in neuroscience or if they align with previously reported observations. Have there been perhaps some new substantial findings using the new method that have been not known so far?\nIn terms of reproducibility, while the code is available, the absence of example datasets limits the ability of other researchers to replicate the findings or to test the model on new datasets. Providing a sample dataset or detailed data specifications would allow others to reproduce the results for reviewers AND readers and evaluate CLOSER’s effectiveness across diverse datasets.\nMinor points:\nThe manuscript presents confusion matrices for classifier accuracy, but the interpretation of these matrices is limited without details on error patterns. This limitation restricts insights into potential misclassifications. Adding evaluation metrics like precision, recall, and F1 scores, along with error bars, could illustrate model reliability and provide a more comprehensive understanding of classification performance.\nThe use of LDA to visualize antidepressant effectiveness could be strengthened by a rationale for its selection over other dimensionality reduction techniques. Including a comparison with alternatives like t-SNE or UMAP would offer a broader perspective on clustering patterns and lend further support to the use of LDA.\nThe robustness and interpretability of SHAP summary plots for antidepressant efficacy would benefit from both global and individual feature importance, helping to identify consistent features across different conditions. Presenting individual SHAP values for specific animals or conditions could clarify which features are universally important or context-dependent in the assessment of antidepressant efficacy.\nThe interpretability of the results for drug classifications based on confusion matrices is unclear. Supplementing these matrices with precision-recall metrics or analyzing specific types of errors (e.g., false positives vs. false negatives) could offer insights into the model’s strengths and weaknesses in classifying stress conditions and drug responses. Clarifying why certain drugs or conditions are more challenging to classify would make the results more actionable.\nThe syllable usage and transition analyses would benefit from statistical validation to support claims of generalizability. Adding statistical tests, such as permutation tests, and validating the results on additional datasets would reinforce the claims of generalizability.\nThe reliance on balanced accuracy as the sole metric for classifier accuracy may not fully capture the model’s performance in distinguishing depressive symptoms. Including metrics like sensitivity and specificity would offer a comprehensive evaluation and ensure that the classifier’s performance reflects accuracy across various symptom categories.\nDetailed points (citations from the draft):\nIn the estimation of model prediction accuracy, the algorithm incorporating Scale, ChangeStartPoint, ChangeAngle, and ChannelMask presented the best performance for both Top-1 (73%) and balanced accuracy (70%) in the classification of manually labeled mouse behavior segments\nComment: This is absolutely unclear, how should the reader know what the algorithm incorporating Scale, etc, means?\nTo determine and categorize behavioral syllables, which are minimal motion units involving ethological context, we leveraged K-means clustering in 128 dimensions to partition behavior segments into 50 clusters .\nComment: How were this numbers picked?\nGiven the continuous nature of mouse behavior, a behavioral syllable was denoted as a consecutive frame with the same cluster label .\nComment: No minimum length was picked or no smoothing or filtering applied?\nCluster labels were determined by majority voting on a maximum of ten cluster candidates collected at the first learning step (Fig. 1a).\nComment: Unclear, how were the ten cluster candidates collected?\nObserving the transition of behavior motifs longer than 10 frames with z-scored keypoint change-scores, we found that the start frame\nComment: It would be much more scientific to report seconds / miliseconds then number of frames, so that the reader can compare this value directly to other publications without reading up which frame rate you were using.\n... and segments created by keypoint-MoSeq (27.93 ± 34.95 frame) but longer than those created by Variational Animal Motion Embedding (VAME) (7.75 ± 12.52 frame) ...\nComment: which configurations have you used both packages?\nIn this analysis, CLOSER provided a higher balanced accuracy (73%) than VAME (68%) or keypoint-Moseq (37%).\nComment: Could you provide this as +/- STD over multiple videos to see the degree of error in generalization?\nInitially, we performed an LDA projection for syllable frequency and transition summaries of five-minute intervals to address the sexual disparity in stress-induced ethograms (Fig. 3g).\nComment: So far the LDA and tranistion summaries have not been introduced, hence ist hard to follow here.\nIn both males and females, CLOSER-identified syllables and sequences enabled the classifier to predict depressive disorder,\nComment: To make sure, has this been tested on unseen data?\nIn the hierarchical structure of defined behavior syllables, clusters assigned analogous labels were located adjacent to one another in the hierarchy table\nComment: How come there are so many labels related to „sniff“? It seems to me that these are labels that have been randomly assined during labeling but not a concise dictionary of behaviors defined a priori (ideally comparable to different publications).\n...with discrete distribution patterns of behavior segments in embedding spaces for global and local components built with LatentOut ..\nComment: What is LatentOut? Please don’t use code functions in an article, that’s really hard for the reader to interpret and make sense out, if the code is not known of not of interest.\nThis finding indicated that the antidepressants only partially alleviated the depressive symptoms in spontaneous behavior\nComments: Its OK to say „indicated“, but making some judgments based on LDA embeddings is rather naive.\nWe addressed this problem by applying a self-supervised contrastive learning algorithm with augmentation modules that force the machine to embrace the underlying behavioral structure while ignoring irrelevant features\nComment: How do you define “irrelevant features”?\nThis leads to the development of transferable and generalizable ML agents\nComment: How is you codebase a ML agent?\nIndeed, the algorithm generated high-quality embedding for naturalistic behavior compared to other recent unsupervised toolboxes, with clear recognition of novel motion patterns imperceptible on visual inspection .\nComment: How can you say that if you score was marginably better then for competition? This needs to be re-assesed with a larger scope and at least an additional dataset in comparison.\nFinal Verdict\nWhile CLOSER is a carefully implemented application of contrastive learning for analyzing rodent behavior, its contributions appear primarily incremental. The model heavily relies on the established MoCo framework, with adaptations that are domain-specific but do not introduce fundamentally new methodologies or architectural innovations. These modifications—such as using 3D skeletal data and behavior-specific augmentations—tailor MoCo’s principles to the nuances of rodent behavior analysis but lack technical novelty.\nFurthermore, the manuscript does not clearly establish whether its primary contribution lies in the methodological adaptation or in producing new insights within the neuroscience of depressive behavior. Although it reports findings of sex-specific and time-dependent behavior changes in a chronic stress model, the novelty of these results for neuroscience is not convincingly discussed (hopefully this will be confirmed with reviewers with more understanding of the particular field). Without detailed comparisons with existing literature, it is challenging to assess whether these behavioral observations offer new understanding or align with previously known patterns.\nFor these reasons, the paper would benefit from a clearer positioning of its main contribution—whether as a novel behavioral quantification tool or a source of new biological insights—and from substantial additional justification for the technical and scientific novelty of its findings. The the authors should aim to clearly establish unique methodological contributions that distinguish CLOSER from existing models like MoCo, provide detailed comparative benchmarks to demonstrate added value over simple adaptations of MoCo, and strengthen generalizability through robust statistical validation."}, "author_response": "Response: We appreciate the reviewer for the critical feedback and hope to have addressed all the reviewer’s concerns in our responses below.\nComment 1. The authors are correct to, in the Introduction, be highly critical of the readout tests of the forced swim test and tail suspension test, given their lack of face validity. However, they then proceed to use these very same tests to compare the behavioral effects of chronic stress in standardized tests with that in their locomotion paradigm. Furthermore, they interpret the effects of stress in the standardized tests in a highly subjective way of mice really being in a state of \"despair\" or \"helplessness\". These are antidepressant screening tests and do not provide information relevant to the major symptoms of depression. Examples of translational behavioral paradigms relevant to major depression symptoms would be reward motivation and Pavlovian fear conditioning.\nResponse:\nWe thank the reviewer for raising this important issue about the interpretation of conventional behavioral tests (FST, TST, and SPT), and fully agree that these tests have significant limitations in construct validity, yet they remain essential for validation of the depression models in mice. Although our manuscript primarily aims to demonstrate the utility of motion profiles from naturalistic behavior analysis for depression diagnosis, initial validation of depression models using these conventional tests was a necessary foundational step.\nHowever, we acknowledge that our earlier descriptions might have overly linked the conventional behavioral tests to our behavioral analysis, potentially undermining the novelty and application of our new methodological framework. Thus, we have revised our manuscript to remove excessive comparisons between conventional antidepressant screening tests and naturalistic behavior patterns in our new analysis of monoaminergic and non-monoaminergic antidepressants (Figure 5). The current manuscript describes antidepressant efficacy measured through conventional tests more cautiously as follows (Results, line 402):\n“[…] We administered traditional monoaminergic antidepressants (fluoxetine and imipramine) or emerging non-monoaminergic candidates (agmatine, scopolamine, and rapastinel) to observe the distinct action of different drug groups. In chronically stressed male mice, most antidepressants significantly reduced immobility durations in FST, while imipramine alone enhanced sucrose preference rather than influencing immobility (Supplementary Fig. 9a, b). Additionally, rapastinel treatment significantly elevated sucrose preference, whereas other drugs did not induce significant improvements in either the TST or SPT. Thus, despite the high predictive validity of FST in male mice, traditional behavioral assays generally failed to differentiate individual or grouped antidepressant actions, with the exception of imipramine and rapastinel.”\nMoreover, while FST and TST are widely used to represent behavioral despair1, we agree that these tests alone could not capture learned helplessness. Thus, we have removed the term ‘learned helplessness’ from our manuscript and replaced any use of the phrase ‘depressive symptom’ with the more precise term ‘depressive-like behavior’ or ‘depressive-like phenotypes’ to avoid overstating the clinical implications of our findings:\n Line 193: “In male CUS mice, we observed a significant increase in behavioral despair (TST, FST) (Fig. 3a, b, left) and anhedonia (SPT) (Fig. 3c, left), as well as a decrease in body weight (Fig. 3d).”\n Line 262: “In addition, syllable frequency was more closely related to depressive-like behaviors reflecting behavioral despair (Supplementary Fig. 6a, b, d, and e) than anhedonia (Supplementary Fig. 6c, f) in both males and females, as an analogous result has been observed in the previous study utilizing social defeat stress mice for depressive disorder model.”\n Line 361: “These results with conventional assays were sufficient to confirm the expression of depressive-like behaviors, but they failed to show a direct correlation with specific time points in the CUS paradigm.”\nComment 2. The manner in which the authors describe the outputs of their algorithm renders it impossible to comprehend what actual behavioral changes they observed in locomotion in a semi-naturalistic environment and how this might be relevant to major symptoms of depression. In principle, it would be very interesting to know what locomotor changes are induced by chronic stress, but the data needs to be presented in a way that is understandable and uses measures of overall activity, general body posture, etc.\nResponse:\nWe fully agree with the reviewer that a comprehensive analysis of kinematics for syllables altered by chronic stress could provide potential readers a clear interpretation of behavioral characteristics. To address the concern, we performed a time-series kinematic analysis of depression-related syllables by measuring four motion metrics: nose height, body angle, body shift, and nose shift. We then grouped syllables that significantly increased or decreased following chronic stress and extracted representative 3D action skeleton sequences to visualize body posture and dynamics. We incorporated the analysis result in the new Supplementary Figure 5.\nSupplementary Fig. 5 ǀ Kinematic profiling of depression-specific behavioral syllables in male and female mice. Kinematic features and representative postural sequences are shown for behavioral motifs that changed by chronic stress exposure in male (a, b) and female (c, d) depression model mice. Behavioral syllables decreased in stressed male mice were grouped in threes following a hierarchical order (a). Line plots illustrate four kinematic measures (nose height, body angle, body shift distance, and nose shift distance), which are expressed as a fraction of duration for each behavioral segment. 3D action skeleton depicts the representative syllable dynamics (head: read, upper torso: coral, lower torso: orange, tail: yellow, four paws: green, blue, magenta, and purple)\nIn male mice, we observed a reduction in motifs with elevated body angles and sniff-related motifs, which show larger nose shifts than body shifts. Conversely, syllables that increased under stress featured higher nose heights. In females, stress-elevated syllables showed comparatively greater body angles, corresponding to turning behaviors in manual annotations. These findings are now described in the manuscript (Results, line 249).\n“[…] In further kinematic profiling, chronic stress selectively reduced the occurrence of motifs with elevated body angles and sniff-related behaviors with large nose shifts relative to body shifts (Supplementary Fig. 5a), while higher nose heights were observed in stress-elevated syllables (Supplementary Fig. 5b). In female mice, stress-elevated syllables displayed greater body angles, in agreement with manual annotation, turning behaviors.”\nReviewer #2\nReviewer #2 (Remarks to the Author):\nHere, Oh and colleagues describe a self-supervised approach for segmenting three-dimensional keypoint trajectories into discrete behaviors that they call CLOSER. Specifically, CLOSER leverages self-supervised learning – a technique increasingly being used for action segmentation in human data – to learn an embedding of 3D kinematic data from spontaneously behaving mice. The learning objective is constructed to learn an embedding space where points representing similar behaviors are close together and dissimilar behaviors are far apart, while also remaining invariant to “nuisance” features such as body size. Points in the embedding space are clustered to form a segmentation of 3D kinematic data into discrete behaviors. They benchmark CLOSER against the currents state-of-the-art, Keypoint-MoSeq (KP-MoSeq) and Variational Animal Motion Embedding (VAME), and find that their technique has better correspondence with manual labels. Finally, the authors demonstrate that CLOSER can be used to dissect the behavioral consequences of inducing depression in a chronic stress model.\nOverall, I found that the authors have an intriguing idea that is likely to be well-received by the Nature Communications readership. However, I found some major technical issues that must be addressed before this can be considered for publication. I hope most of this can be remedied by some analytic controls without any additional experiments.\nResponse: We appreciate the reviewer for their helpful and constructive comments. Please see below for our detailed responses.\nMajor problems that must be addressed before the manuscript can be considered for publication:\nMajor comments:\nComment 1. Contrastive learning is used to learn an embedding space of 3D kinematic data that is purportedly sensitive to changes in behavior and insensitive to other factors such as mouse size. This is a reasonable approach in principle; however, unless I missed it, I found no validation that their method achieves the stated objective.\nResponse:\nThe reviewer raises a relevant point that was not fully clear in the previous manuscript. We agree with the reviewer that the characterization of invariances in our module is mandatory to fully validate the contrastive learning training process. In this regard, we performed additional computational analysis and addressed all comments in detail below.\nComment 1a. Specifically, the authors do not provide evidence that CLOSER’s embedding strategy is actually invariant to size while remaining sensitive to the differences between kinematically distinct behaviors. Here, the authors need to provide direct quantification that their embedding space, as they say, works better than PCA, autoencoders, t-SNE, etc (line 380). This cannot be asserted without direct quantification. A simple path would be to use their manual labels and quantify the average distance between all pairs of points in different manually identified behavioral states and within behavioral states but across different subjects, especially across subjects with different weights and sizes.\nResponse:\nWe thank the reviewer for their insightful comments on the evaluation of the embedding structure. In addition to the extrinsic evaluation presented earlier, we performed a comprehensive intrinsic analysis to estimate the quality of the embedding space. We employed the Pearson correlation coefficient to quantitatively assess intra- and inter-cluster correlations across multiple embedding methods, including CLOSER, VAME, and other pose representations (PCA and UMAP).\nTo investigate the invariance to factors, in our case, mouse size, individuality, and sex, we calculated the correlation coefficients across distinct mouse subjects and sexes. The underlying rationale was that if a variable is learned as invariant, the intra-cluster correlations from different animals should be significantly higher than the inter-cluster correlations. Indeed, our analysis revealed that CLOSER achieves high intra-cluster correlation and low inter-cluster correlation for these variables, confirming that it captures behavioral features independent of size, sex, and individuality. A similar analysis of VAME demonstrated a clear separation of correlations, whereas the 3D pose representation (including PCA and UMAP) showed minimal difference or opposite trend, indicating poorer cluster distinction. These additional results are now included in the Results section (line 157) of the revised manuscript and the new Supplementary Figure 2.\n“[…] For further intrinsic evaluation of embedding spaces, we calculated the Pearson correlation, which quantifies the strength and direction of the linear relationship between two behavioral segments. Using this method, we evaluated whether our augmentation strategy induced invariance within the embedding space, such that syllable definitions depended solely on motion sequences. To achieve this, we selected two distinct syllables from mice differing in weight, individuality, or sex, and computed the intra- and inter-cluster correlation coefficients (Supplementary Fig. 2). In analysis, CLOSER exhibited high intra-cluster correlations alongside minimal inter-cluster correlations across all tested conditions, implying that the latent structure is independent to these three variables. Likewise, VAME showed strong intra-cluster correlation and negative inter-cluster correlations, whereas embeddings derived directly from 3D pose representations (including PCA and UMAP) displayed low intra-cluster correlation or greater variance, reflecting poor cluster separation.”\nFurthermore, to evaluate the reproducibility of our embeddings on unseen human-labeled data, we compared intra- and inter-cluster correlations across two independent mouse labeling datasets. Consistently, we observed higher intra-cluster correlations across most of the clusters (except for cluster 2, right turn, and cluster 5, freezing with four paws), which emphasizes that CLOSER effectively learns invariant and discriminative behavioral features.\nComment 1b. Second, it would be extremely useful to show that, if their embedding shows better qualities than alternatives, how important the specifics of their augmentation strategy are. The authors use the prediction accuracy of manually identified labels to validate their particular combination of augmentations. This should be complemented by analyzing the geometry of their embedding (see point above) and repeating this analysis across different combinations of augmentation functions.\nResponse:\nWe thank the reviewer for the constructive suggestion on an estimation of the augmentation strategies. While we have listed the accuracies of behavioral prediction for augmentation combinations, we have now incorporated the geometric analysis of embedding space into a revised manuscript to address this point more clearly. In detail, we extended our intrinsic evaluation across multiple augmentation combinations using an unseen human-labeled dataset with 13 manually defined clusters to strengthen the following points:\n1) Impact of augmentations on embedding geometry: For each augmentation combination, we computed intra- and inter-cluster Pearson correlations. The selected augmentation strategy produced the highest gap between intra- and inter-cluster correlation (Supplementary Figure 1). Compared to other moderate augmentation methods, our combination enhanced the embedding quality of grooming behaviors.\n2) Correlation between geometry and task performance: We observed a relationship that augmentation schemes with high clustering consistency also showed high classification metrics.\n3) Justification of the augmentation strategy through additional performance validation: We measured precision, recall, and F1 scores on the human-labeled test set and proved that our augmentation strategy consistently outperformed other augmentations (Supplementary Table 1).\nThese new analyses confirm that our mouse-specific augmentation combination is critical for learning invariant, discriminative behavioral embeddings. We have now addressed this point in the Results section (line 117) as follows:\n“Next, to evaluate the geometric variance of latent spaces across multiple augmentation strategies, we measured Pearson correlation coefficients for representations of 13 manually labeled behaviors (Supplementary Table 2) in an unseen human dataset to compare within- (or intra-) and between (or inter)-cluster relationships. A coefficient of +1 or -1 denotes perfect positive or negative linear association, respectively, while 0 indicates no linear relationship. We found that augmentation schemes with superior classification accuracy also exhibited stronger intra-cluster correlation than inter-cluster correlation. Importantly, the mouse-optimized augmentation yielded enhanced embedding for grooming behaviors (clusters 6 and 7) compared with augmentation combinations lacking channel masking (strategy 3) or scaling (strategy 2), highlighting the necessity of a complete augmentation strategy for mouse behavior latent structures.”\nComment 1c. Third, I was a little puzzled by some of the augmentations. If I understand correctly, Scale simply scales x, y, and z coordinates by the same factor. Can the authors justify why this is a reasonable thing to do? Can keypoint coordinates across different mouse sizes truly be reduced to a single scaling factor across x, y, and z?\nResponse:\nThe reviewer raised an important point about scaling augmentation in contrastive learning. We understand that the multiplication of a single scaling factor on 3D keypoint coordinates seems unelaborated and inadequate for the augmentation method. First of all, scaling and resize augmentation is a well-known augmentation method for enforcing size invariance in contrastive learning, and these methods have been employed in the action recognition field (Fida Mohammad Thoker et al., 2021) (Jingjing Yang et al., 2022) (Jiahang Zhang et al., 2023).\nNotably, as a contrastive learning framework learns an invariance between two augmented views from the same data, the model learns that action skeletons with different scaling factors represent the same behavior by minimizing the distances between augmented views in the feature space. Truly, it is a nature of contrastive learning that the encoder of the model is invariant to linear scaling of the mouse size, as stated in Ting Chen et al. (2020) and Kaiming He et al. (2020).\nHowever, as the reviewer pointed out, there should be additional validation for whether scaling 3D coordinates by the same factor reduces or enlarges the size of a mouse maintaining pose information. To clarify this confusion, we extracted a random segment and visualized it in 3D space after applying ‘Scale’, ‘ChangeAngle’, and ‘ChangeStartPoint’ augmentations. The original and scaled 3D action skeletons implied that multiplying a single scaling factor is sufficient for size scaling without pose distortion.\nComment 2. The performance comparison with VAME and KP-MoSeq, which is central to the manuscript, is confusing.\nComment 2a. If I understand correctly, the authors specifically tailored their augmentation strategy (Sup. Table 1) to maximize prediction accuracy on the same dataset that is used for the performance comparison in Figure 2. If this is the case, then the authors need to come up with two completely separate datasets – one for tuning hyperparameters and another for performance evaluation. Either way, any hyperparameters in VAME and KP-MoSeq must also be tuned in the same fashion; that is, if the authors are going to tune hyperparameters of their own algorithm to maximize performance, then this must be done with all other algorithms to foster a fair comparison.\nResponse:\nWe fully agree with the reviewer that the performance comparison for the CLOSER, VAME, and KP-MoSeq should be analyzed in various hyperparameter conditions to evaluate the maximal performance of other methods. Therefore, in the revision process, we carefully examined the model performance by adjusting four parts of the measurement of behavior prediction accuracy: 1) employment of the embedding distance-based classifier, 2) introduction of an independent test dataset, 3) subtle hyperparameter tuning in VAME, and 4) specification of hyperparameters in CLOSER.\nIn detail:\n1) We discarded an action module using a simple multi-layer perceptron (MLP) decoder. Commonly, self-supervised learning is operated in pre-training models using encoder architectures, then fine-tuned for specific downstream tasks after the addition of decoders. Accordingly, we adopted an exemplary algorithm in the BERT framework to investigate whether the amount and quality of information in embeddings are sufficient to be decoded into action modules using a simple multi-layer perceptron (MLP), and trained a two-layer MLP with a hidden dimension size of 100 and a RELU activation function for action module prediction. However, we found that the structure of the embedding space cannot be accurately estimated in the presence of a decoder if K-means clustering or Hidden Markov Model (HMM) are performed directly on the embedding structure. Fortunately, the k-nearest neighbor (k-NN) probing exhibits a high efficiency without any distribution assumption and is a reliable indicator of downstream task performance2. Indeed, many studies have employed the k-NN classifier to assess the quality of the latent space3,4. Hence, we revised our validation, adopting the k-NN classifier in embedding space qualification throughout the study.\n2) To ensure the fairness of the comparison step, we generated an independent test dataset that was never seen during the training process. Similar to the original data production, we assigned one of thirteen behavior labels in Supplementary Table 2 to each video frame for a freely moving mouse.\n3) Regarding the hyperparameter tuning in the VAME framework, we listed the default and optimized hyperparameters of VAME in the table below. It must be highlighted that we modified the VAME structure to develop an optimal environment comparable to our framework. The VAME structure, which originally managed 2D skeletal input, was adjusted to receive a 3D action skeleton while complying with the ratio of hidden dimension. Also, we reshaped the 30-frame reconstruction / 15-frame prediction training format into a 10-frame reconstruction / 5-frame prediction to compare with other methods without bias. In addition to former optimizations, we further modulated the learning rate and the number of maximum epochs in the revised manuscript, allowing us to monitor the effect of hyperparameter manipulation. The classification performance of VAME trained with two learning rates (5e-4 or 1e-3) and two maximum epochs (500 or 20) was tested.\nHyperparameters of VAME Default Optimized\nInput dimension 12 27\nEmbedding dimension 30 108\nFrame lengths for reconstruction 30 10\nFrame lengths for prediction 15 5\nInput normalization Yes Yes\nOptimizer Adam Adam\nBatch size 256 256\nLearning rate 5e-4 5e-4 or 1e-3\nMax epochs (patience step) 500 (50) 500 (50) or 20\nThe details of hyperparameters are now added to the Methods section (line 923) as follows:\n“Variational Animal Motion Embedding (VAME). For the VAME, we modified the input structure to accept 3D behavioral segments as input through the extension of the number of features, from 12 to 27, and the embedding dimension, from 30-dimension to 108-dimension. In model training and inference, we adjusted the frame lengths of reconstruction and prediction methods to ten and five frames, respectively. As the variational autoencoder architecture of the VAME consists of an encoder and a decoder, each using Gated Recurrent Units (GRUs), we trained the model with the given parameters for reconstruction, prediction, and KL divergence, which shows the maximal performance in behavioral classification. […]”\n4) To clarify the hyperparameter information for the CLOSER pipeline, we categorized any hyperparameters used on our platform. Specifically, the momentum of a key encoder and temperature for the InfoNCE loss and distillation calculations originate from the MoCo framework, which we utilized in the CLOSER. The maximum angle for the ‘ChangeAngle’ augmentation module was determined after a comparison of linear evaluation tests for various degrees. The corresponding table is now included as Supplementary Table 3.\nAlong with the modification of the benchmarking embedding of Keypoint-MoSeq as described in Comment 2b, we re-analyzed the performance of the behavioral classification across different methods. As a result, our framework exhibited the highest balanced (66%) and top-1 (55%) accuracy. Moreover, additional metrics revealed that CLOSER consistently showed strong precision, recall, and F1 scores. We have revised Figure 2C and the corresponding Results section (line 155) to reflect the updated performance metrics as follows:\n“[…] In this analysis, CLOSER provided the highest balanced (66%) and top-1 accuracy (55%), accompanied by consistently strong precision, recall, and F1 score. […]”\nFig. 2 ǀ The CLOSER framework recognized temporal and behavioral characteristics for classifying behavior segments into semantic syllables. […] c, Normalized confusion matrix (top) and associated performance metrics (bottom) showing the accuracy of the k-nearest neighbor classifier based on the latent spaces of different embedding methods. The classifiers were trained on top of the embedding spaces with an unseen manual annotation dataset to predict the behavior labels for given segments. Accumulated performance over the fivefold cross-validation is shown. The classification rate is represented in grayscale, with white for a classification rate of 1 and black for a classification rate of 0. Box plots show the 25th-75th percentiles of the precision, recall, and F1 score for methods. […]\nComment 2b. Can the authors elaborate on the embedding space that was used for KP-MoSeq? KP-MoSeq is not really an embedding algorithm, it simply uses PCA of keypoint trajectories as the input space for a probabilistic time series model. Did the authors attempt to make predictions using an MLP using the standard input for KP-MoSeq? If so, the authors need to make it clear that they’re not comparing their algorithm to KP-MoSeq, but rather to the input of KP-MoSeq. If I am misunderstanding what they did, and they in fact used the output of the model, how were the model hyperparameters tuned?\nResponse:\nThe reviewer raises a valid concern that keypoint-MoSeq uses the PCA projection of keypoint trajectories only as an input for its sequential analysis. Indeed, keypoint-MoSeq employs an HMM that learns transition probabilities between states, behavioral syllables IDs, and emission probabilities from a given PCA projection of motion sequences. In the previous analysis, we employed linear evaluation to assess the quality of behavior representations, excluding the segmentation step. Based on this framework, we initially believed that it was reasonable to use the 13-dimensional PCA projection of individual frames (i.e., the input to the HMM) as the behavior representation of keypoint-MoSeq in performance estimation. To account for sequential context, we concatenated 10 consecutive frames of these PCA projections to create 10-frame-length behavioral segments. However, we recognize that evaluating the input of keypoint-MoSeq alone does not fully capture the motion recognition efficiency of the entire tool. To address this limitation, we now provide an alternative comparison utilizing dimensionality reduction methods (PCA and UMAP) on raw data, comprising 270-dimensional vectors representing the 3D coordinates of keypoints over 10 consecutive frames, instead of the keypoint-MoSeq input. We have included the result of the performance comparison with raw pose data (270 dimensions), PCA (13 dimensions, identical to Keypoint-MoSeq input dimension), and UMAP (13 dimensions), and addressed the detailed description, in conjunction with our response to Comment 2b. Moreover, we have revised the Results section (line 150) to clarify the employment of the PCA and UMAP embedding as follows:\n“[…] Since keypoint-MoSeq employs an HMM that learns transition probabilities between states, behavioral syllables IDs, and emission probabilities from a given 13-dimensional Principal Component Analysis (PCA) projection of motion sequences, we leveraged 270-dimension representation with 13-dimension PCA and Uniform Manifold Approximation and Projection (UMAP) embedding for 10-frame length behavioral segments, instead of keypoint-MoSeq. […]”\nComment 2c. The comparison with VAME seems more apples to apples than the comparison with KP-MoSeq, which is good. That said, the performance difference between the two algorithms looks fairly subtle (Fig. 2c), and as mentioned above, the hyperparameters of CLOSER appear to have been tuned for maximum performance. Were the hyperparameters of VAME tuned at all? If not, this must be done in the same way hyperparameters were tuned for CLOSER. Additionally, given that the performance difference between CLOSER and VAME is small, the authors should provide uncertainty estimates using different cross-validation folds or multiple restarts of each model with different random seeds.\nResponse:\nWe believe that, to some extent, the answers concerning comment 2a are applicable to this note. As previously discussed in the response, the performance of VAME in manual labeling prediction was calculated by a effective k-NN evaluation method in various hyperparameter conditions, which allows subtle tuning of the VAME framework in our environment. This analysis revealed that the classification performance of VAME was the highest with the original hyperparameter combinations. Notably, the CLOSER framework outperformed the VAME and other dimensionality reduction algorithms (e.g., PCA and UMAP), highlighting that the feature space of CLOSER is well fitted to human labeling consensus. We also incorporated statistics for the 5-fold cross-validation process under the classification performance matrices (Figure 2c) to clarify the performance estimation step.\nComment 3. Another relevant detail here is that AVATAR uses a different form of keypoint tracking. Rather than using a standard markerless keypoint tracker, my understanding is that AVATAR learns the centroid of pixels associated with a given body part. If so, can the authors elaborate on the noise characteristics of this type of tracking relative to the output of, e.g., DeepLabCut, SLEAP, or LightningPose? I could not find these details in either this manuscript or the preprint describing AVATAR. Does the apparent performance benefit of CLOSER still apply when using standard keypoint trackers rather than AVATARnet?\nResponse:\nWe thank the reviewer for this important point. While the description of the characteristics and advantages of the recording system is beyond the scope of our manuscript, we are glad to introduce the AVATAR network.\nAVATAR’s approach differs from standard keypoint-based methods in how the underlying data is labeled and how the final body-part locations are computed. Whereas methods like DeepLabCut, SLEAP, or LightningPose typically rely on precisely annotated keypoints—individual points placed by human annotators at the exact location of a joint—AVATAR leverages an object detection paradigm. Instead of labeling a single point, the annotators define a region corresponding to each body part (e.g., a bounding box around the hand, foot, or head). From this region, AVATAR extracts the centroid of the associated pixels to determine the body-part position.\nThis difference in approach yields distinct noise characteristics:\n1. Labeling Consistency\nFor point-based systems, human annotators must pinpoint the exact joint location. Even with careful guidelines, there can be subtle inconsistencies—one labeler might place the marker slightly off-center or on a different anatomical landmark than another. This can introduce labeling noise that propagates into the trained model and its predictions. By contrast, with AVATAR’s region-based labeling, it is generally easier and more natural for annotators to consistently mark the entire area of a body part. This leads to more uniform and reliable training data, potentially reducing the variability introduced by subjective judgments about a single “correct” point.\n2. Aggregation over Pixels\nKeypoint-based methods provide a single coordinate per joint, making them sensitive to small deviations in that coordinate. AVATAR’s centroid estimation effectively averages over a group of pixels associated with a body part. As a result, small variations at the edges of the region or slight shifts in pose translate into more stable centroid estimates. This averaging effect can smooth out noise, reducing frame-to-frame jitter in tracked positions compared to methods that hinge on a single pixel-level determination.\n3. Robustness to Annotation Errors\nIn a keypoint labeling scenario, a single misplacement by a few pixels can cause a noticeable change in the predicted location. In the region-based approach of AVATAR, the centroid calculation is more robust in dealing with such minor discrepancies. A slightly over- or undersized bounding box still likely captures the true location of the body part fairly accurately once the centroid is computed.\nIn summary, while keypoint-based models may face more noise from subtle, hard-to-control labeling differences at a single coordinate, AVATAR’s object detection and centroid-based approach can mitigate such issues by relying on more consistent and easily defined training labels. This approach, in turn, can produce more stable positional estimates and potentially lower the overall noise in the final tracked outputs.\nSince CLOSER extracts the behavioral syllables from detected keypoints, we anticipate that the reviewer’s concern regarding the compatibility of CLOSER with alternative pose estimation systems could be resolved by the high sensitivity and specificity demonstrated by the AVATAR pipeline. Additionally, supplementary videos provided in the preprint (https://www.biorxiv.org/content/10.1101/2021.12.31.474634v1.supplementary-material) further illustrate the strong pose tracking accuracy of the AVATAR pipeline.\nComment 4. If the authors are going to compare KP-MoSeq and VAME with CLOSER, they should also show that CLOSER provides clear benefits in the analysis of the chronic stress model (e.g. the decoding in Fig. 3h).\nResponse:\nWe appreciate this excellent suggestion, which was also raised by another reviewer. We share the same interest and had anticipated that analyzing performance in the diagnosis of mouse depressive disorder using three different motion recognition frameworks could underscore the necessity of our tool. However, the GPU-RAM demand of VAME exceeded our computational resources due to the HMM processing on the massive dataset. Fortunately, Miller, Stephanie R., et al. recently introduced a k-means algorithm for latent space clustering, which has been shown to improve the classification accuracy of the genetic Alzheimer’s disease model (Miller, Stephanie R., et al., 2024). Building on this advancement, we used the keypoint-MoSeq and k-means version of VAME segmentation to explore the effectiveness of alternative segmentation methods in the detection of subtle behavioral changes in stressed mice.\nFollowing the analogous approach to the examination of diagnostic accuracy shown in Figure 3h, we investigated the predictive performance of individual pipelines for depressive disorder in mice. To demonstrate this, we partitioned the motion sequences of non-stressed and stressed mice into 5-minute segments for both males and females. Behavioral syllables and summaries were then extracted employing alternative methods, which were subsequently used to train random forest classifiers for diagnosis. We have now incorporated these results in the main manuscript (Results, line 223) with a new supplementary figure 3. The relevant section and the figure are added below:\n“To further evaluate the efficiency of CLOSER in depression diagnosis, we applied the same analysis on male and female groups using the syllable combinations identified by VAME and keypoint-MoSeq. Classification accuracy indicated that CLOSER outperformed VAME and keypoint-MoSeq in the male mice, achieving superior predictive performance for both classifiers trained using syllable frequency of transition (Supplementary Fig. 3a, b). In contrast, for non-stressed and stressed females, the random forest classifier leveraging syllable frequency from keypoint-MoSeq demonstrated greater performance compared to other behavior recognition pipelines (Supplementary Fig. 3c); nevertheless, the syllable transition calculated by CLOSER consistently yielded the highest accuracy (Supplementary Fig. 3d). Notably, behavioral deconstruction in males offered higher classification rates than in females for all segmentation methods, aligning with the observed sex-dimorphic alterations of syllable frequency and transition (Fig. 3i, l).”\nSupplementary Fig. 3 ǀ Diagnostic accuracy of behavioral parsing pipelines in a mouse model of depressive disorder. (a, b) Confusion matrices illustrating the classification accuracy of CLOSER (left), VAME (middle), and keypoint-MoSeq (right) pipelines in male mice. The matrices compare the diagnostic performance based on syllable frequency (a) and syllable transition (b), where brighter colors along the diagonal represent a better classification rate. The CLOSER pipeline consistently outperforms VAME and keypoint-MoSeq in the male group. Box plots indicate 25th-75th percentiles of the precision, recall, and F1 score for each analytic pipeline, based on a 5-fold validation step of the random forest classifier, and all individual outcomes are displayed as data points. (c, d) Confusion matrices and box plots describing classification performance and statistics for non-stressed (CTRL) and stressed (CUS) female mice, using classifiers trained with frequency (c) and transition (d).\nMinor comments\nComment 1. This may just be my misunderstanding. Line 611 of the methods describes the training/test/validation split. Since the authors are analyzing data in a 10-frame window with 1 frame stride, can the same frame appear in both the training and test data? If so, the analysis needs to be repeated while accounting for any potential data leakage.\nResponse:\nWe agree with the reviewer that the same frame can appear in both the training and test datasets, as we randomly sorted the motion sequences into training, validation, and test datasets. For more details, we split those datasets to check validation loss for early stopping and to compare the loss distribution between the training and unseen datasets. However, we are confident that there is no direct information leakage from the training dataset to the test dataset for two reasons.\nFirst, our module receives data in the form of a motion segment with a 10-frame length and, therefore, recognizes the motion in a sequence rather than a single pose (or frame). Even if the frame is repeatedly shown in different motion segments, each motion segment is distinct data, and a single frame could represent different contexts. Moreover, we mainly evaluated the quality of embedding through Linear Discriminant Analysis on the feature space instead of the test loss. Through performance evaluation of models with separate datasets, as the reviewer suggested, the data leakage problem could be solved (reviewer 2, comment 2a).\nComment 2. The writing is a bit hard to follow in places. For instance, Lines 376-379, I am not quite sure what the authors mean by the “Neuroethological nature” of their “psychomotor analysis”.\nResponse:\nWe apologize for the confusion in describing the advantages of our framework. We originally intended to inform you that the linkage between depressive disorder and psychomotor symptoms could strengthen the face validity of our diagnostic system. To clarify our statement, the text in the Discussion section (line 507) was changed as follows:\n“Importantly, the neurological and molecular linkages between emotion and motor activity allow it to produce high face validity for depressive symptoms appearing in spontaneous movement.”\nComment 3. Could the authors provide a more detailed explanation of their dynamic temporal segmentation? My understanding is that each frame is given the most frequent label assigned to that frame across all windows that contain it; however, this is a bit hard to discern from Figure 1.\nResponse:\nWe thank the reviewer for this important comment. The scheme of dynamic temporal segmentation has now been revised in Figure 1a and the main text (line 130) as follows:\n“[…] Then, a cluster label for each frame was determined by majority voting on candidates assigned from respective sliding windows at the first learning step (Fig. 1a). Given the continuous nature of mouse behavior, a behavioral syllable was denoted as a consecutive frame with the same cluster label.”\nComment 4. Time complexity is mentioned as a major benefit of their segmentation algorithm relative to HMMs. However, can the authors provide a simple comparison of performance between the two? Curious how they compare against the manual labels.\nResponse:\nWe thank the reviewer for pointing us to the estimation of time complexity in segmentation algorithms, k-means clustering, and HMM. We wish to emphasize that we have mentioned the Big-O complexity in the second paragraph of the Discussion section (line 521). As the exact execution times for HMM and k-means clustering are diverse depending on the running environment, we would like to indicate the theoretical time complexity for input size in the manuscript.\nFurthermore, we extended the Results (line 133) and Methods (line 829) sections in the main manuscript to include the detailed methodology for evaluating the quality of temporal segmentation. We employed a ‘Keypoint change-score’ from Weinreb, Caleb, et al. (2024), which estimates the sudden shifts in skeletal pose straightforwardly (Figure 2a). The keypoint change score was calculated with the following equation using egocentric keypoint coordinates transformed with a Gaussian kernel, where 𝑦 are the keypoint coordinates 𝑦 after Gaussian smoothing.\n𝑦ℎ𝑦𝑦𝑦𝑦 𝑦𝑦𝑦𝑦𝑦 (𝑦)=𝑦 𝑦𝑦𝑦𝑦𝑦 (|𝑦 −𝑦 |)\n𝑦 𝑦−1\nThe Result section now reads:\n“[…] For validation of temporal segmentation, we monitored whether the dissection of a postural sequence captured the temporal dynamics of behavior shifts from one motion to another by utilizing a keypoint change-score, which highlighted the robust shift within the 3D action skeleton.”\nThe Methods section for temporal segmentation now reads:\n“[…] In addition, we employed a keypoint change score function for further estimation of the ability of the framework to capture the robust shift in the action skeleton.”\nComment 4a. A related point about time complexity. Can the authors remark on the compute time for each step of the complete CLOSER pipeline? It would be useful to show this relative to VAME and KP-MoSeq as well.\nResponse:\nWe thank the reviewer for bringing this to our attention. Regarding this point, we generated an additional table below for the execution times of individual motion recognition pipelines from training to segmentation. However, since the computational environment and detailed running conditions are distinct from the original frameworks, we decided not to include the table in the supplementary data.\nRuntime\nFramework Step\n(minutes)\nData preparation (train, validation, and test set) 10\nEncoder training (per an epoch) 10\nEncoder\ntraining\nEncoder validation (per an epoch) 2\nEncoder test 2\nCLOSER\nEmbedding extraction 2\nSegmentation Calculation of behavioral syllable hierarchy 5\nDynamic temporal segmentation 1\nTotal runtime (Runtime for pre-trained model) 300 (30)\nData preparation (train, validation, and test set) 10\nEncoder\nTraining Variational Autoencoder (VAE) (per an epoch) 2\ntraining\nValidation of VAE (per an epoch) 1\nVAME\nEmbedding extraction 2\nSegmentation of single motion sequence\nSegmentation 2\n(HMM) (per an iteration)\nSegmentation of single motion sequence 1\n(K-means) (per an iteration)\nTotal runtime (HMM) 1800\nTotal runtime (K-means) (Runtime for pre-trained model) 100 (30)\nData preparation 2\nInitialization of HMM 2\nKeypoint-\nMoSeq\nHMM fitting 20\nTotal runtime 30\nReviewer #2 (Remarks on code availability):\nI was only able to perform a cursory review of the code. A README is available. However, I do not have the author's raw data in the format required for the code, so I was unable to run it.\nResponse:\nWe sincerely apologize for omitting the sample datasets, including mouse 3D coordinates used in our study. We have now uploaded the whole raw dataset at the provided Zenodo link to guarantee the utility of the CLOSER.\nReviewer #3\nReviewer #3 (Remarks to the Author):\nIn this manuscript, Oh and colleagues present the self-supervised learning platform CLOSER, which is aimed to assess parameters of spontaneous behavior in mice (3D allocentric and egocentric skeleton plus kinematic features) and link them with altered depression scores in healthy controls vs UCMS (unpredictable chronic mild stress) model in both sexes as well as predictive validity-associated alterations induced by antidepressant drugs (the SSRI fluoxetine and the tricyclic imipramine). The authors claim that CLOSER results are of enhanced translational value compared to standard assays (forced swim, tail suspension test, and sucrose preference test).\nComparable efforts to assess psychomotor activity in mice focusing on variations (i) across strains (healthy and disease models) or (ii) variations induced by drugs have been published recently (e.g., Sheppard et al. 2022 and Wiltschko et al. 2020). Here, the authors provide interesting longitudinal data where CLOSER measures the development of a “depression-phenotype” across 4 weeks of UCMS as well as the ability of CLOSER to measure distinct properties (as well as off-target effects) of antidepressant treatment which could be potentially interesting in shaping antidepressant treatment to a specific depression subtype as well as potentially detecting different disease stages in depression. This is a highly interesting manuscript; however, I think there are several weaknesses that lower the value, which are:\nResponse:\nWe appreciate the critical feedback and hope to have addressed all concerns in our responses to the reviewer.\nMajor comments.\nComment 1. The authors only test one depression model (UCMS). They do not show that CLOSER is working in other models (e.g., learned helplessness, social defeat stress, chronic CORT, or genetic models like BDNF Val66Met or SERT polymorphisms).\nResponse:\nWe agree with the reviewer that further application of CLOSER-based motion analysis on different depression mouse models strengthens the power of our framework in depression diagnosis. As the reviewer suggested in the latter comment, we primarily discussed the introduction of the genetic-based depression model exemplified by the Brain-Derived Neurotrophic Factor (BDNF) Val66Met mouse line. BDNF, a neurotrophic factor playing a significant role in the survival and development of neurons, has been suggested to be related to major depressive disorder as single-nucleotide polymorphism (SNP) in the gene correlates with mood disorders. Mechanistically, the methionine substitution in BDNF leads to defective activity-dependent BDNF secretion in neural dendrites, which results in trafficking abnormality and a following increase in anxiety-related behaviors (Chen, Zhe-Yu, et al., Science, 2006). Indeed, BDNF mediates the synaptogenic and antidepressant effect of ketamine in mice (Liu, Rong-Jian, et al. Biological psychiatry, 2012). Therefore, there is no doubt that BDNF has a major role in MDD (Hashimoto, Kenji., Psychiatry and Clinical Neurosciences, 2010).\nHowever, we would like to acknowledge several limitations in the genetic model, which may contribute to its low adoption in depressive disorder research. First, the allele frequency of the Val66Met polymorphism varies across ethnicities and sub-population groups, leading to population-specific heterogeneity. In addition, the inconsistent findings regarding the influence of polymorphisms on mood disorders, as well as widespread effects on cognition or brain morphology, suggest that both Val66Met knock-in and BDNF conditional knockout mice are more suitable for monitoring the role of BDNF in depressive disorders. (Notaras, Michael, R. Hill, and Maarten van den Buuse. Molecular Psychiatry, 2015). Most importantly, the importation and breeding of the new genetic depression mouse line (BDNF polymorphism or conditional knockout mouse) is expected to take more than six months. Given the significant time required for the establishment and stabilization of this mouse line, we decided to substitute the genetic depression model with immune system-based and hormone-based models.\nWe now present the comparison of behavioral structures of stress-, immune system-, and hormone-induced depression models utilizing the CLOSER platform. For the immune system-based depression model, we chose the lipopolysaccharide-induced neuroinflammation model for the onset of depressive-like behavior through acute activation of the peripheral innate immune system. In addition, we employed a mouse model of depression induced by corticosterone injection to study the effect of systemic elevation of the stress hormone. In the analysis of characteristics of spontaneous behavior across depression models, we focused on (1) the distinct and overlapping alterations in exploratory behaviors in depression mouse models and (2) the power of the CLOSER platform in the discrimination of the etiology of depression models. The results of our investigation are addressed in the Results section (line 283) and new Supplementary Figure 7:\nModel-specific alterations in behavioral Syntax and architecture in depression mouse models\nTo explore the characteristics of spontaneous behavior across depression models, we first established depression paradigms using distinct pathological triggers. We employed two depression models: an immune system-based model through a single intraperitoneal (i.p.) injection of lipopolysaccharide (LPS); and a steroid hormone-based model using corticosterone administration for three weeks. Given prior reports suggesting that 1-week corticosterone does not elicit depressive phenotypes27, an additional group receiving a 1-week corticosterone injection was included to estimate the validity of the model. In the LPS-induced depression model, mice exhibited hallmark depressive-like behaviors, as evidenced by increased immobility in the FST and reduced sucrose preference (Supplementary Fig. 7a). In the corticosterone paradigm, depressive-like behaviors in TST and SPT emerged exclusively after 3 weeks of administration, with no significant alterations in the 1-week group (Supplementary Fig. 7b), in line with existing literature27.\nIn the first step of behavioral analysis, we evaluated syllable frequency and transition dynamics employing LDA to interrogate the broader architecture of spontaneous behaviors across these models (Supplementary Fig. 7c). While non-stressed groups and saline-treated groups, regardless of models, clustered together, chronically stressed and LPS-treated groups diverged markedly from controls in both syllable usage and bigram transitions. Interestingly, while both stress- and inflammation-induced models displayed distinct separations, their behavioral trajectories diverged in opposing directions. In contrast, corticosterone-treated mice were closely positioned with controls, suggesting comparatively modest modifications in spontaneous behavior patterns. Then, we implemented a random forest classifier to quantify the discriminability among depression models (Supplementary Fig. 7d). Consistent with LDA findings, confusion matrices indicated that CUS and LPS models were readily separable, whereas corticosterone-induced models were less distinguishable from controls.\nWe next applied an analysis of individual syllable contributions to unveil model-specific alterations. LPS treatment elicited marked changes in the frequency of 16 syllables, whereas corticosterone treatment affected only three, all of which were also altered in the LPS-treated model (Supplementary Fig. 7e). A subset analysis using the Venn diagram illustrated the degree of overlap across the different depression mouse models (Supplementary Fig. 7f). Of note, while six syllables were commonly altered between stress- and inflammation-induced models, the directionality of syllable changes was predominantly opposite between two models, aside from one shared cluster (cluster 2). These data suggest that frequency alterations are limited to a subset of syllables and exhibit distinct patterns depending on the etiology of depression. These results may also account for the divergent trajectories of depression models observed in the LDA (Supplementary Fig. 7c). Beyond the frequency alterations, we observed pronounced disruptions in grammatical structure, particularly in the LPS model (Supplementary Fig. 7g), and few conserved transition shifts were identified across depression models (Supplementary Fig. 7h). These findings imply that the source of depression (stress, inflammation) can be inferred based on distinctive patterns of spontaneous behavior identified in the CLOSER platform, extending beyond conventional dichotomous assessments of depressive-like behaviors.\nSupplementary Fig. 7 ǀ Distinct and overlapping alterations in spontaneous behavioral patterns in mouse models of stress, inflammation, and corticosterone-induced depression. (a, b) Behavioral assessments of depressive-like phenotypes in LPS-induced (a) (TST: two-tailed unpaired t-test, FST: two-tailed unpaired t-test: **p = 0.0049, SPT: two-tailed unpaired t-test: ***p = 0.0002, N = 10 for both groups) and corticosterone-induced depression mouse models (b) (TST: one-way ANOVA: *p < 0.0127, Dunnett post-hoc: corticosterone 3wk: *p = 0.0229, FST: one-way ANOVA: p = 0.0832, SPT: one-way ANOVA: *p = 0.0319, Dunnett post-hoc: corticosterone 3wk: *p = 0.0202, N = 10 for all groups). c, LDA plots of syllable frequency and transition indicate that behavioral structure in depression models diverges from respective baselines, whereas control groups cluster together across models. d, Confusion matrices for classification accuracy across depression models based on frequency (top) and transition patterns (bottom) of spontaneous behaviors. Brighter shading indicates higher classification confidence. e, Log-2 fold change in syllable frequencies in LPS- and corticosterone-induced models relative to their respective controls. Syllables with significant alterations are bolded. Statistical significances were denotes as follows: *p < 0.05, **p < 0.01, ***p < 0.001, and ****p < 0.0001. f, Venn diagram illustrating the number of syllable frequency changes in three depression mouse models to depict the shared and unique behavioral changes among the models. g, Syllable bigram networks for significantly up-regulated (red) and down-regulated (blue) transitions in LPS- and corticosterone-induced depression mice. Line thickness represents the transition probability. h, Hierarchically organized transition matrix heatmap summarizing the transition shifts across chronic stress-, LPS-, and corticosterone-induced models. \nComment 2. They claim CLOSER can show sex-specific effects but then only include male animals when testing predictive validity with the 2 antidepressant drugs.\nResponse:\nWe thank the reviewer for highlighting this weakness and agree that the sex-specific effects of antidepressants should be examined to fully describe the sex-dimorphic spontaneous behaviors in the depression model. Thus, we conducted an identical antidepressant experiment in female mice and added a new Supplementary Figure 16 to include the following result. We extended the Results section (line 462) to explain our findings for the antidepressant responsiveness of stressed female mice in terms of spontaneous behavior:\nSex-dependent behavioral fingerprints in naturalistic behavior for monoaminergic antidepressants\nTo elucidate sex-dependent antidepressant responsiveness and its impact on naturalistic behavior, we evaluated female mice with the same experimental paradigm previously applied to male mice using monoaminergic antidepressants. Unlike the male CUS model, female mice exhibited antidepressant efficacy exclusively in SPT and not in FST, highlighting substantial limitations in the predictive validity of traditional assays (Supplementary Fig. 11d, e).\nTo test if the behavioral structure identified by CLOSER could be applied to reliable discrimination of stress and antidepressant administration states within the female cohort (Supplementary Fig. 16a). Analysis based on syllable usage yielded a sensitivity of 47.22% and specificity of 82.41%, whereas classifiers leveraging transition profiles demonstrated enhanced performance, achieving sensitivity and specificity values of 61.11% and 87.04%, respectively, showing an analogous trend observed in males. Subsequent SHAP analyses elucidated antidepressant behavioral fingerprints, revealing significant sex-dependent differences (Supplementary Fig. 16b). In contrast to the rearing behaviors pivotal in males, female fingerprints predominantly featured head movements and sniffing behaviors, with cluster 22 exhibiting decreased frequencies following monoaminergic antidepressant treatment, while clusters 14 and 29 differentiated fluoxetine from imipramine (Supplementary Fig. 14c). Although cluster 4 displayed analogous antidepressant responsiveness across sexes, disparities in other syllable fingerprints underscored sex-specific differences in responses of naturalistic behavior to depressive disorders and their pharmacological interventions.\nNext, we investigated drug-induced on- and off-target effects within naturalistic behavior syllables (Supplementary Fig. 16c). Chronic stress significantly altered five behavioral clusters; fluoxetine and imipramine successfully reversed three to baseline states (Supplementary Fig. 16d). Notably, off-target effects were substantially pronounced in females, with fluoxetine altering eleven syllables and imipramine influencing nineteen. Cross-sex comparisons revealed minimal overlap between male and female responses to fluoxetine (two common syllables) and none for imipramine, underscoring sexual dimorphism in behavioral responses due to divergent baseline behavioral structures post-stress induction. Moreover, fluoxetine consistently exhibited more off-target effects than imipramine across both sexes.\nFinally, analysis of syllable bigram transitions revealed marked differences between female and male responses to antidepressants. Fluoxetine robustly normalized stress-induced transition modifications toward non-stressed states, whereas imipramine treatment predominantly induced off-target transitions (Supplementary Fig. 16e). In conclusion, while off-target effects on syllable usage were more evident in females, off-target transitions within syllable sequences were dominant in males, further emphasizing substantial sex-dependent variability in antidepressant pharmacodynamics.\nSupplementary Fig. 16 ǀ Behavioral fingerprints in naturalistic behavior of female treated with monoaminergic antidepressants. a, Confusion matrices of classification performance for stress condition and antidepressant type, derived from syllable frequency (left) or transition probability (right). b, Mean SHAP values highlight the relative importance of individual syllables (x-axis) in a random forest classifier trained to predict the administered antidepressant in female mice. c, Venn diagram showing shared and unique alterations in syllable frequency across different treatment paradigms in females. d, Relative expression of discriminant-relevant syllables, normalized to non-stressed female mice, reveals distinct antidepressant-driven modulation of syllable frequency in stressed mice. (two-tailed unpaired t-tests between non-stressed saline-treated mice and stressed groups: CUS: cluster 4: *p = 0.0239; cluster 41: *p = 0.0400; cluster 22: *p = 0.0252; cluster 7: *p = 0.0101; cluster 9: *p = 0.0108, fluoxetine: cluster 22: ***p = 0.00056; cluster 7: *p = 0.0210, imipramine: cluster 22: ***p = 0.00052; cluster 9: *p = 0.0288, N = 9 for all groups) e, Fluoxetine in females restores non-stressed transition patterns and imipramine induces broader off-target shifts, while males display dominant off-target transitions within syllable sequences, which underscores robust sex-specific pharmacodynamic variability.\nComment 3. They claim that CLOSER has better predictive validity than classical tests for depression but then only test drug classes that actually perform well in the classical tests too (monoaminergic drugs).\nResponse:\nWe appreciate the reviewer's constructive comment about including data on non-monoaminergic antidepressants in addition to our previously reported monoaminergic findings. We agree that evaluating these additional compounds helps address the limitations of conventional depressive-like behavior tests and further illustrates the broader applicability of the CLOSER pipeline, which enhances its potential for revealing subtle antidepressant actions. In this revised manuscript, we have integrated agmatine, scopolamine, and rapastinel into our experiments and extended the analysis to compare them directly with fluoxetine and imipramine. Specifically:\n1) Experimental design and drug selection\n We added three non-monoaminergic antidepressants (agmatine, scopolamine, and rapastinel) to complement our existing monoaminergic group.\n This expanded set of compounds allowed us to investigate whether broader mechanistic classes exhibit distinct or overlapping behavioral signatures.\n2) Behavioral assessment and CLOSER-based analysis\n The non-monoaminergic group was subjected to the same chronic stress paradigm and machine learning (CLOSER) pipeline used for the monoaminergic group, ensuring consistent data collection and analytical methods.\n Our approach revealed drug-specific on- and off-target effects that highlight how non-monoaminergic antidepressants differ from (or sometimes parallel) monoaminergic treatments, particularly in subtle psychomotor behaviors not captured by classical tests.\n3) Comparative findings\n Results demonstrated that non-monoaminergic compounds often induce broader off-target changes than the monoaminergic group, underscoring their potentially different modes of action.\n Certain behavioral clusters remained unique to non-monoaminergic treatments, while others showed partial overlap or mirrored patterns in fluoxetine or imipramine, offering deeper insights into the heterogeneity of antidepressant effects.\nWe trust that including these non-monoaminergic antidepressants addresses your recommendation and provides a more comprehensive understanding of how diverse antidepressant classes influence psychomotor behavior.\nBelow, we share the revised Figure 5, and the main document is included in the Results section, ‘CLOSER can quantify the pharmacological effects of monoaminergic and non-monoaminergic antidepressants on psychomotor symptoms (line 398)’.\nComment 4. Also, “rodents” would imply that they tested more than one species, which they did not (they only tested mice; it is untested if CLOSER would work in rats).\nResponse:\nWe fully agree with the reviewer’s opinion that experiments were performed on mice, and therefore, we needed to use the precise term for experimental subjects. We have rewritten the manuscript to specify that we tested on mice, not on rodents.\nComment 5. The extended figures were not provided to me, so I could not review them in detail. However, statistical summaries were present and allowed some conclusions. Due to the excellent quality of the rest of the data presented, I would not expect major mistakes in these graphs, but please provide them in the next round.\nResponse:\nWe recognized that some supplementary figures were named with an extended figure in the supplementary data for statistics, and have now fixed them to the proper label.\nI recommend a major revision and suggest that the authors (i) add another (genetic, not stress-based) depression model to strengthen their results or (ii) add a non-monoaminergic antidepressant (e.g., ketamine) to show that CLOSER could perform better in screenings, where the classical depression tests fail.\nResponse:\nWe thank the reviewer for this excellent question and have now further strengthened this aspect in the revised manuscript. Specifically, we have addressed this point in Comment 1 for various depression models and in Comment 3 for non-monoaminergic antidepressants. We hope this explanation resolves all the concerns raised by the reviewer.\nComment 6. Also, I strongly advise removing overreaching statements,e.g. that CLOSER has enhanced translational value – this statement is only based on the presence of psychomotor symptoms (slowed speech, decreased movement, and impaired cognitive function) in human patients. There are no data in this manuscript allowing a conclusion regarding the translational value of CLOSER. Psychomotor retardation scoring in humans is more complex, limb-trunk movement is only as small part of it. The translational value of the elegant, but also simplified, scoring of CLOSER has still to be elucidated. So please phrase this with a little bit more caution.\nResponse:\nWe agree with the reviewer and have removed all statements regarding translational relevance to avoid any overstatement of clinical value. Moreover, we have expanded the Discussion section to specify this limitation, along with the considerations raised in Comment 7 below.\nComment 7. Also, Manifestations of psychomotor symptoms are common in patients with melancholic depression and those with psychotic features – in other words, psychomotor symptoms are not a common feature in depression. There are various forms of depression, including atypical, seasonal, melancholic, psychotic, catatonic, double depression (comorbid dysthymia), and anxious/agitated forms of depression. This limitation should be mentioned!\nResponse:\nWe thank the reviewer for raising this important point regarding the subtype-specific manifestation of psychomotor symptoms. Although it is well recognized that the psychomotor symptoms are prominent in melancholic depression, our primary aim was to demonstrate that naturalistic behavioral patterns offer a robust framework for detecting hidden alterations induced by stress, and our approach uncovers such latent features that were heterogeneous across different contexts. Psychomotor symptoms were one strong evidence for the relevance of the psychiatric state and motion pattern, with neurological linkage between the amygdala and cerebellum. In other words, we still roughly expect the potential alteration in behavioral structures of human patients for various subtypes of depression, which cannot be quantified by single motion factors such as body speed, speech tone, eye movements, or self-touching. Nonetheless, we acknowledge that their clinical relevance across the full spectrum of depressive disorders remains to be determined. In response, we have revised the Discussion section (line 566) to address this limitation, with additional references that highlight the variability in psychomotor symptoms across depression subtypes:\n“For broader application of CLOSER in depression diagnosis, an action recognition efficiency and diagnostic accuracy must be validated across different species and ultimately in human patients. It is important to note, however, that the psychomotor retardation and agitation are primarily characteristic of melancholic or catatonic depression (Buyukdura, J. S., McClintock, S. M., & Croarkin, P. E., 2011) (Bennabi, D., Vandel, P., Papaxanthis, C., Pozzo, T., & Haffen, E., 2013) (Northoff, G., Koch, A., Wenke, J., Eckert, J., Böker, H., Pflug, B., & Bogerts, B., 1999), and the relationship between motion patterns and other subtypes, for example, atypical, psychotic, and seasonal depression, remains elusive (Sobin, C., & Sackeim, H., 2025). Thus, despite our results, which demonstrate that naturalistic behavioral profiles captured by CLOSER reveal distinct psychomotor signatures associated with stress, additional studies on the clinical relevance of psychomotor symptoms with the full spectrum of depressive subtypes are needed to prove the applicability of CLOSER and the translational potential of these behavioral markers in human depression diagnosis.”\nMinor comments.\nThere are several minor issues, which I will summarize according to chapter/lines:\nResponse:\nWe thank the reviewer for reading through our literature and for the assessment that has strengthened our manuscript. Your comments and suggestions helped us substantially improve our manuscript.\nIntroduction\nComment 1. General: I’m missing a section describing how psychomotor retardation is scored in humans (MARS, SRSS, etc) and how the authors take into account that for example in MARS five major body categories including eyes, face, voice, limbs, and trunk are used, while CLOSER tries to do the same with only limbs and trunks in mice.\nResponse:\nWe appreciate the reviewer for spotting the scoring method for human psychomotor retardation and the discrepancy in body region coverage between these methods and CLOSER. However, while we agree that the linkage between human psychomotor scoring and mouse pose tracking could strengthen the diagnostic relevance of our approach, the AVATAR pipeline in this study is configured with a fixed set of keypoints. Therefore, adapting our system to reflect the major body categories used in human scoring lies beyond the scope of the current work. Nonetheless, we have revised the Introduction (line 64) to clarify the psychomotor scoring strategies employed in clinical settings to show the conceptual similarity of human motion scoring and mouse pose estimation:\n“In human mental diseases, the motion-associated symptoms, or psychomotor activity, have been assessed through behavioral observation of limb movements, trunk posture, voice, facial expression, and eye movements, in parallel with identifications of circuital and functional connectivity between the amygdala and motor cortex. […]”\nComment 2. Line 49-52: References 3-5 underscore the efforts to discriminate depression subtypes; none of them supports the claim that “motion parameters … are among the most symbolic symptoms of psychiatric diseases.” Please add a Reference there.\nResponse:\nWe have updated the references underscoring the clinical significance of psychomotor symptoms in the diagnosis and treatment of major depressive disorder. Moreover, we have revised the corresponding sentence to temper the generalization of these symptoms, considering the selective manifestation in melancholic depression subtypes (Introduction, line 49):\n“[…] Recently, researchers have tried to discriminate genetic and pathophysiological depression subtypes, while no attempt has been made to classify depressive episodes according to the motion parameters, which are the symbolic symptoms of psychiatric diseases. (Schrijvers, D., Hulstijn, W. & Sabbe, B. G. C., 2008) (Gupta, R. K., 2009) (Buyukdura, J. S., McClintock, S. M., & Croarkin, P. E., 2011) (Bennabi, D., Vandel, P., Papaxanthis, C., Pozzo, T., & Haffen, E., 2013)”\nComment 3. Line 56-57: The forced swim test has high predictive validity for monoaminergic antidepressant drugs (SSRIs, SNRIs, as well as MAO-I and tricyclic actually reduce the immobility in the FST), but not non-monoamine compounds (e.g., ketamine, neuropeptides). I would formulate that a little bit more precisely (or less harshly), especially as you are not testing CLOSER with non-monoamine compounds.\nResponse:\nWe agree with the reviewer that a detailed description of the shortcomings of current depressive-like behavior tests in mice to emphasize the necessity of our framework. As the reviewer suggested, despite high predictive validity, FST has suffered from low specificity (high false positives), little construct validity, and poor effectiveness in the discovery of novel antidepressants. Indeed, in our experiment (Supplementary Figure 10), FST was extremely sensitive to the administration of antidepressants in male mice. Therefore, we have removed the term ‘predictive validity’ from our revised manuscript and, instead, emphasized the high false positive rate in drug discovery to precisely describe the imperfection of traditional behavioral tests. Below is the updated paragraph (Introduction, line 54):\n“Traditionally, behavioral screening tests in rodents (e.g., the tail suspension test, forced swim test, and sucrose preference test) have been used to evaluate depressive-like symptoms, such as learned helplessness and anhedonia. Unfortunately, they show low construct validity, making them incapable of reflecting the complex psychiatric symptoms and prognoses of MDD. The highly controlled environments and oversimplified conditions used to elicit non-spontaneous behaviors in these tests have limited the development of the most common antidepressants, resulting in high false-positive rates for novel antidepressant discovery. Moreover, the over-reliance of the tests on manual scoring introduces subjectivity and inter-observer variability, further limiting the accuracy and reproducibility of depressive disorders models.”\nWe hope this revision provides clearer insight into the limitations of the FST for evaluating non-monoaminergic antidepressants and better addresses the reviewer’s concerns.\nComment 4. Line 76: Please add one sentence to explain the concept of contrastive learning.\nResponse.\nWe have adjusted the introduction to include a brief explanation of contrastive learning as follows (Introduction, line 76):\n“[…] A popular unsupervised algorithm, contrastive learning, is a self-supervised model that learns behavioral representations to capture relevant features in the data20-22, by contrasting pairs of augmented views from the same sample against unrelated negative samples. […]”\nResults:\nComment 5. Line 17-129: something is wrong here; the written numbers do not match the graph Fig 2b. Please correct.\nResponse:\nWe thank the reviewer for the feedback on the potential confusion regarding the description of the main figure 2. Upon careful re-examination of both the figure and the manuscript, we confirm that the information was accurate. We believe that the confusion may stem from a statistical aspect, as longer segments disproportionately influence to the mean value of the dataset. To address this issue, we have considered an additional statistical measure—the mode (i.e., the most frequently occurring value within the dataset)—which is represented by the peak of each distribution (Figure 2b).\nFig. 2 | […] b, The distribution of durations of the behavioral motifs recognized by CLOSER (red), VAME (blue), Keypoint-MoSeq (gray), and by manual annotation (green, dotted line). The data are presented as a fraction of the total syllable length for each frame length, where 30 frames equal 1 second. The dotted lines denote the mode of each distribution across analytic methods. […]\nComment 6. Line 167-169: as this was no real Open field test (20x20cm arena was too small), Lux illumination?? I suggest removing the time in the center graph (Fig 3e) as conclusions regarding anxiety cannot be drawn in this setting; Furthermore, please split the distance moved into 5 min segments as you did with the clusters (Fig 3j-k); the difference in the clusters could be driven by alterations in locomotion, please show if this is the case.\nResponse:\nThe reviewer raises a relevant point that was not clearly described in the previous manuscript. We measured the light illumination (20 lux) and added the result in the Methods section (line 640). Next, we agree that a 20 x 20cm arena may not be ideal for the measurement of anxiety-like behavior, as the smaller open-field area can artificially inflate the proportion of time spent in the center relative to the edges, considering the size of the mice. Thus, we have removed references to the anxiety index and now focus on locomotion data throughout the manuscript.\nRegarding the suggestion to examine movement over time, we performed a time-bin analysis in 5-minute intervals to investigate whether differences in locomotion drive the observed behavioral patterns. The results showed no significant divergence in total distance traveled between stressed and non-stressed mice over the entire test period, reinforcing that variations in spontaneous behavior are not simply oriented by locomotive differences.\nComment 7. Line 209 / Suppl Fig 2d-f: estrous cycle analysis is missing here, which could explain the low correlational values that might be mentioned in the text.\nResponse:\nWe appreciate the critical feedback regarding the potential effects of the estrous cycle on female behavioral outcomes, particularly in the tests for mood disorders. However, our primary objective was to compare overall male-female differences in naturalistic and depressive-like behaviors rather than to investigate hormonal fluctuations in depth. Truly, many studies on depressive model mice do not routinely measure the estrous cycle unless they intend to explore specific hormonal mechanisms5-7. Moreover, we considered that individual differences in the baseline exploratory pattern of the mouse might have a greater influence than estrous status8, prompting us to normalize post-experimental data to the baseline of individuals.\nNevertheless, we recognize that estrous cycle fluctuations may account for lower depressive-like behaviors in TST and less distinctive features in naturalistic behaviors. Therefore, we have revised the Results (line 197) of the Figure 3, emphasizing the potential impact of hormonal variability on female mouse behavior.\n“[…] The reduced appearance of depressive-like behavior in females during TST sessions may be attributed to hormonal influences associated with estrous cycle fluctuations, which were not measured in this experiment.”\nComment 8. Line 318-333: Or this means that the antidepressants didn’t work in all animals. Did you check for individual variety within your dataset? According to your classical behavioral results, you have some “non-responders” to the UCMS protocol what about the antidepressants (they should be in the extended dataset that wasn’t provided so i couldnt check)? It would be great if CLOSER could predict non-responders!\nResponse:\nWe appreciate the reviewer’s insightful comment regarding the potential of our platform in the prediction of antidepressant responsiveness based on innate behavioral characteristics. We agree that such an approach is highly novel and holds significant potential, as it could enable the tailoring of therapeutic interventions according to individual behavioral phenotypes. However, in our study, it is challenging to strictly delineate responder and non-responder groups for antidepressants, as mice exhibited statistically significant improvements in at least one conventional depressive-like behavior following drug administration. For instance, in male mice, a significant antidepressant effect was observed in FST for both fluoxetine and imipramine (Supplementary Figure 11a), suggesting that the treatment was effective, though its impact across other behavioral assays was comparatively modest. Notably, the extent of behavioral recovery appeared to vary depending on both the class of antidepressant and the sex of the animal (Supplementary Figure 11).\nOne major limitation is the absence of a universally accepted guideline for categorizing drug responders versus non-responders in preclinical models. While there is a consensus on defining stress susceptibility in rodents, a similar consensus for antidepressant response classification is lacking. Although biomarker-based approaches to predict antidepressant response have been explored in humans (Leuchter, Andrew F., et al.. Current psychiatry reports, 2010), these efforts are largely aimed at distinguishing between treated and non-treated patients, rather than stratifying responders within a treated cohort.\nDespite these issues, we attempted to classify the non-responders in fluoxetine- and imipramine-treated groups based on behavioral tests for depressive-like phenotypes. In our dataset, the result revealed that conventional behavioral assays performed post-antidepressant treatment failed to show a consistent correlation across tests when attempting to categorize drug-treated male mice into responder and non-responder, as illustrated below. Indeed, other groups have performed a task aimed at predicting antidepressant responsiveness, involving the ‘pre-treatment recording’ of brain activity during emotional conflict regulation and the HAMD (Hamilton Depression Rating Scale) scores in humans after antidepressant treatment (Fonzo, Gregory A., et al. Nature human behavior, 2019). In contrast, to predict spontaneous behavior-based antidepressant efficacy in our condition, an alternative experimental design is required, which extracts the behavioral structure of mice following depression induction but prior to antidepressant treatment. However, in our experimental setup, behavioral recordings were performed only after both depression induction and antidepressant administration, precluding this type of analysis.\nNonetheless, we want to point out that our current dataset does reveal the distinct effects of each antidepressant on normalizing specific subsets of altered syllables. This suggests the feasibility of a symptom-guided approach, where particular antidepressants could be selected based on the pattern of syllable alterations observed in depression models. We have now included this point in the revised Discussion section, with the potential for future studies utilizing the CLOSER framework to predict responders and non-responders based on intrinsic behavioral structures as below (line 540).\n“Understanding the mechanisms by which antidepressants modulate depressive-like behaviors has traditionally centered on an ideational domain. While this has elucidated key aspects of the neurobiology of psychiatric disorders, the interpretability of data remains constrained by the divergence between rodent behavioral paradigms and self-reported human symptomatology. We resolved this limitation by benchmarking the pose estimation algorithm for human motion tracking to construct a generalizable diagnostic framework.\nOur framework identified the innate efficacy of each antidepressant in restoring distinct subsets of behavioral syntax and architecture and, thus, enables a subtle assessment of antidepressant effects, including quantitative evaluations of both on- and off-target actions. This insight allows for the automated prediction of drug indications tailored to the recognized symptom clusters within heterogeneous depressive disorders, offering refined therapeutic strategies with improved predictive power. A promising direction for future studies could leverage intrinsic behavioral structure under depressive states to infer treatment responsiveness, which substantially strengthens the importance of spontaneous behavior as a determinant in prescription and potential application to clinical frameworks.”\nMethods:\nComment 9. Please add the License number of the animal experimental license approving the experiments\nResponse: We added the approval number for this study in the Methods section (line 589).\nComment 10. Please add that both sexes were used and the total n number, estrous cycle measurement for females.\nResponse:\nWe appreciate these helpful suggestions. We have specified (1) the sex and age of the mice used for each behavioral test in the Methods, and (2) the number of animals (n) for each group in the figure legends. Regarding the estrous cycle, please refer to our minor Comment 7, as it was not specifically measured in this study.\nComment 11. Point out that drug tests were performed only on male animals (this is a pity as here there would have been the chance to show a potential advantage of CLOSER as the classical tests don’t work well in females!)\nResponse:\nWe appreciate the reviewer for the perfect suggestion, and the analysis of the antidepressants on female mice has now been added in the main Figure 4 and corresponding supplementary figures. We have addressed this point in conjunction with our response to major Comment 2. We hope this explanation resolves all the concerns raised by the reviewer.\nComment 12. Please define your room temperature in °C.\nResponse: This has been adjusted accordingly (Methods, line 593).\nComment 13. How many batches of recordings? Are these litter mates?\nResponse:\nWe gladly added the information for experimental batches and the number of mice per batch to the new Supplementary Table 4.\nSupplementary Table. 4 ǀ Summary of Experimental Batches and the number of mice per batch\nMoreover, we have revised the Methods (line 594) to include the information for littermates and mouse distribution.\n“Animals. […] We used mice from at least three litters, and mice were randomly allocated into groups to balance potential genetic or developmental variance.”\nComment 14. Was the analysis repeated with a separate experiment to verify that the clusters identified by CLOSER can be reproduced in a different set of animals?\nResponse:\nThe reviewer raised an important point regarding the reproducibility of our behavioral analysis results in the depression model mice. As mentioned in response to minor comment 13, we performed more than three batches for the previous experiments and two batches for experiments conducted in the revision process. To further validate the consistency, we grouped the behavioral structures from two independent batches of male mice exposed to the chronic stressor and compared the representative variables, syllable frequency and transition. The result revealed that overall syllable distributions are similar between batch groups, and only two syllables (syllables 45 and 34) showed statistical differences. The transition bigram exhibited a few weak transition differences between the batch sets. These findings indicate that, while some minor batch-to-batch variations exist, the differences are modest relative to those observed between stress conditions.\nComment 15. -the very small open field (20 x 20cm) likely results in your high center values (approx. 35-50% center time: in this small arena, the exploration of the walls perpendicular to the wall direction will most probably result in detection of the animal in the center;)\n-OF: center plus wall area are nearly all of the arena, wall area usually defined as + 8cm distance from wall, center defined as 10 x 10cm at a complete size of 20 x 20 the apparatus is much too small for analyzing center time! Please remove all statements regarding anxiety. See guidelines for Mouse Behavior phenotypes from https://mousebehavior.org/ethogram-comparison/.\nResponse:\nThe reviewer raised important points, and we agree with the reviewer that the description of anxiety-like behavior in the small open-field test box could confuse the careful interpretation of mice behavior. Therefore, we have now excluded any description of anxiety-like behavior based on the time spent in the center area of the recording studio.\nComment 16. First, there are only 2 recordings (prior and post-UCMS); then, there is one recording per week for Figure 4. Please clarify the details (n mice, sex, age, how you dealt with repeated FST/TST, etc).\nResponse:\nWe are glad to clarify the suggested concerns and have added all the more details to the Results, Methods, and figure legend of the manuscript.\nUCMS Protocol:\nComment 17. 13 degrees Celsius water for 5 min?! That is a very strong stressor - shallow water exposure is what is used normally and at 24-26 degrees! (e.g. JOVE Burskin & Doron 2018), TST can be part of UCMS but is problematic here because it's also used as a readout for food/water deprivation, which is unusual in UCMS, cold exposure, and sleep deprivation as well. Very strong stressors for a UCMS! I wouldn’t get this protocol through the ethics committee in my country (in Europe). Please consider that for follow-up experiments.\nResponse:\nWe acknowledge the reviewer’s concern regarding the intensity of certain stressors in our experimental design and have accordingly revised the terminology from 'chronic unpredictable mild stress (CUMS)' to 'chronic unpredictable stress (CUS)'. However, we strongly emphasize that there is substantial variability in the stressors employed in existing CUMS protocols, with many published studies using cold swim exposure at temperatures lower than ours (13°C) or for longer durations than 5 minutes: (i) 30 min, 8°C (Leng, Lige, et al., Neuron, 2018)9, (ii) 5 min, 8 °C (Fernandes, Joneth, and Girdhari Lal Gupta., Behavioural brain research, 2019)10, (iii) 5 min, 4 °C (Song, Qiqi, et al. Neuropharmacology, 2019)11, (iv) forced swimming in ice water at 4 °C for 5 min, hot water at 45 °C swimming for 5 min (Li, Hui, and Yaping Wang., Biochemical and biophysical research communications, 2016)12, (v) 5 min, 4 °C (Chen, Guanghui, et al.. Behavioural brain research, 2015)13, and (vi) 30 min forced swimming (Liu, Meng-Ying, et al., Nature protocols, 2018)14. Indeed, many standard CUMS protocols described in these literatures apply significantly harsher conditions than those utilized here.\nNext, we recognized the similarity of the cold swim and tail pinch to the FST and TST assays, respectively, and therefore deliberately implemented distinct contexts and apparatus to differentiate these stressors from the depressive-like behavioral assessments. In detail, the cylinder for the cold swim was different in size and material from the cylinder used in FST, with different water temperatures. Likewise, the tail pinch involved holding the tail gently on a surface, with all four paws of the mouse fully supported, clearly differentiating this from the suspended posture in the TST. These procedures are not uncommon in literature9-12, and our approach maximized the unpredictability and variability of the stressors, which was our primary objective.\nRegarding sleep deprivation, we utilized an apparatus with shallow water sufficient only to prevent mice from comfortably lying down but still allowing them to remain seated. Importantly, the procedure was conducted during the night, whereas mice are nocturnal animals that primarily sleep during daytime hours; thus, our sleep deprivation does not constitute a full deprivation and does not represent an excessively severe stressor. We have now described the sleep deprivation methodology in the revised manuscript (Methods, line 603):\n“Chronic Unpredictable Stress (CUS) model. […] (viii) 12 h of sleep deprivation with a cylindrical pedestal (2.5 cm diameter and 3 cm height) placed on the cage floor filled with 1 cm-deep tap water to prevent sleep during the dark phase, […]”\nFurthermore, we reassure the reviewer that our ethical approval process thoroughly reviewed and approved these protocols, and thus, ethical considerations are fully satisfied and not a cause for concern.\nComment 18. sucrose preference – why additional food deprivation? Normally not necessary, just take the data from your ‘habituation phase’? Does this indicate that you did not get nice results there and increased the drive via food deprivation? Please at least state if the bottles were switched in the “real test” after 12h to exclude side preference influences. sawdust? Does this mean the animals didn’t have proper bedding depth (that’s a stressor that will influence your results)\nResponse:\nWe appreciate the reviewer raising important concerns regarding our sucrose preference test (SPT) methodology. However, we would like to strongly emphasize that our experimental protocol closely follows the most widely accepted and commonly employed methods for SPT in the literature14,15, including a 24-hour testing period covering both light and dark phases, 1% sucrose concentration, and adequate adaptation to water and sucrose bottles without prior deprivation. The \"food deprivation\" described in our initial manuscript may have caused unintended confusion. To clarify explicitly, we did not impose any additional deprivation beyond standard methodology; rather, mice had food withheld only during the 24-hour sucrose preference measurement to ensure accurate and reliable intake assessment. Furthermore, we validated the physiological impact of this standard food-withholding period by monitoring mouse body weight before, after, and one day post-SPT. As anticipated, the animals experienced a transient reduction in weight following the test but fully recovered within one day, indicating minimal stress impact.\nRegarding side preference, we agree that our previous description was incomplete and have now stated in the Methods section (line 659) that bottle positions were switched after 12 hours to eliminate potential side preference biases.\n“Sucrose preference test. […] The positions of the bottles were switched after 12 hours of testing to exclude the potential effects of side preference.”\nLastly, the reviewer’s concern regarding sawdust bedding is understandable; however, the sawdust was minimally applied to detect possible bottle leakage and did not compromise the standard living conditions of mice or induce meaningful stress. Control mice, tested under identical conditions, exhibited significantly different sucrose preferences and depressive-like behaviors compared to stressed groups, further indicating that bedding depth had negligible effects on the overall well-being and behavior in our experimental condition.\nComment 19. Performing tail suspension test and forced swim test on the same day might have affected the FST data, you would for example expect some pain in the tail after the TST that can affect swimming performance;\nResponse:\nWe agree with the reviewer that the physical pain from the TST might affect the behavioral results in the FST. However, we wish to emphasize that the impact of tail suspension was minimal on the swimming performance. Firstly, the FST was conducted after allowing a sufficient interval (at least 4 hours) following the completion of the TST, which we have now noted in the revised manuscript (Methods, line 673):\n“Forced swimming test. The forced swimming test (FST) was conducted to assess behavioral despair, at least four hours after the TST to minimize any potential effect of physical discomfort caused by tail suspension. […]”\nSecondly, when removing the adhesive tape used for tail suspension, we carefully used a razor blade to minimize any possible tail injury or pain. Moreover, mice experiencing physical discomfort typically exhibit noticeable twitching or abnormal movement patterns in water; such behavioral indications were not observed in our FST video recordings. Indeed, there was a significant difference in immobility in the FST between stressed and non-stressed groups, which proves the validity of our experimental schedule.\nComment 20. In addition, the UCMS protocol already contains something very similar to the FST, and this will affect your FST results too, especially if the 2 are temporally close (how did you deal with this in the depression progression protocol where you did FST and TST every week?)\nResponse:\nWe have now clarified the Methods section to describe the experimental schedule. Specifically, during the test week (week 4), we omitted the cold swim and tail pinch to prevent repetitive exposure to similar stressors and following habituation effects. Additionally, for the weekly assessment of depressive-like behaviors, the TST and FST were conducted at the beginning of the week, and there was no cold swim and tail pinch in the CUS paradigm (Methods, line 608)\n“Chronic Unpredictable Stress (CUS) model. […] During the test week (week 4), we omitted the cold swimming and tail pinching to prevent repetitious exposure and habituation effect.”\nDiscussion\nComment 21. line 409-410: Overstatement, only one depression model was used.\nResponse:\nWe agree with the reviewer’s opinion that this argument would benefit from supporting experimental evidence. In response, we have added a new section presenting behavioral analysis data from two additional depression mouse models (Supplementary Figure 4) and altered the text to emphasize the broader applicability of our findings (Results, line 283).\nComment 22. line 416: You are stating that this algorithm is used in humans? Where is the reference for that?\nResponse:\nWe have cited three references that applied contrastive learning on human 3D skeleton for action recognition. These citations were included in the Introduction to emphasize the broader applicability of the machine learning algorithm in mouse contexts, thereby supporting our statement regarding its usage in humans.\nComment 23. line 419 – overstatement, where did you show a translational relevance of CLOSER? There is no human data in this manuscript\nResponse:\nThe reviewer raises a valid point, and we agree with such shortcomings of our manuscript. Therefore, we have revised the sentence to eliminate descriptions on the translational relevance of CLSOER and have added further arguments in the manuscript (Discussion, line 566)\nComment 24. Limitations need to be formulated more precisely (no other depression models, only 2 drugs, mice only, etc)\nResponse:\nWe have revised the Results section to include the experimental outcomes from various depression models (CUS, LPS, and corticosterone models), non-monoaminergic antidepressants, and drug efficacy in female mice, addressing major comments 1, 2, and 3. In addition, we have expanded the Discussion to incorporate limitations —depression subtype-specific expression of psychomotor symptoms, responder/non-responder discrimination, and lack of validation in other species—in response to major comments 4 and 7 and minor comment 8. We hope this explanation resolves all the concerns raised by the reviewer.\nFigures\nComment 25. General: Please make sure that n numbers can be found in the figure legends, e.g., missing in Fig5\nResponse: We have revised all relevant figure legends to indicate the number of mice included in each experiment, ensuring that sample sizes are now explicitly stated throughout the manuscript.\nComment 26. Figure 3 legend\nLine 1017: typo\nLine 1019: I think you mean decreased, same in 1020; or better, gained less weight\nResponse: These are two helpful comments. The legend was changed to clarify the change in the body weight of stressed male or female mice.\nComment 27. Line 1044-1045: This doesn’t match what you stated in the beginning. Please correct and explain why there are sometimes more and sometimes fewer animals in the behavioral tests (e.g., Female TST 22 vs 21, FST 24 vs 24, SPT 24 vs 24). Are these different experimental batches?\nResponse:\nThe reviewer raised an important issue regarding the inconsistencies in sample sizes across different behavioral tests. Originally, we conducted experiments in main figure 3 on 22 male and 24 female mice. However, in the TST, several mice remained motionless, hanging onto their tail, which introduced ambiguity in determining whether such postures should be classified as immobility, since we commonly define immobility as the duration during which the mouse remains motionless with only minimal movements necessary to maintain balance. Thus, mice clinging to the tail presented a borderline scenario, and we initially decided to exclude these individuals from the TST analysis under the rationale that their hanging might not reliably reflect behavioral despair. Furthermore, as those same mice displayed depressive-like behaviors in the FST and SPT, we presumed that such exclusion would not compromise the overall dataset.\nNonetheless, we acknowledge the reviewer’s concern that this approach may cause confusion and result in nonuniform sample sizes. Therefore, we re-examined our criteria, considering relevant literature and alternative implementations such as measuring tension changes on the bar where the tail is suspended (Steru, Lucien, et al., Psychopharmacology, 1985) (Crowley, James J., et al., Neuropsychopharmacology, 2006). Based on these considerations, we concluded that it would be more consistent and transparent to include those mice in the TST result, treating the passively suspended posture as immobility. Consequently, we have reanalyzed the data and updated the corresponding figures, legends, and statistics in the manuscript to ensure uniform sample sizes across the tests. We also revised the Methods section for TST to specify our immobility criteria, explicitly stating that remaining completely motionless, whether clinging to the tail or not, is categorized as immobility (line 669):\n“ Tail suspension test. […] Mice were considered immobile when hanging motionless, swinging passively, or remaining completely motionless.”\nSupplements\nComment 28. Line 60: delete SPT statistics in figure legends (they are there 2x)\nResponse:\nWe thank the reviewer for bringing this to our attention. However, there were no duplications in SPT statistics. This confusion may be oriented from a complicated parsing strategy for graph statistics. We revised the legend to indicate that two-tailed unpaired t-tests for FST and TST were conducted on stressed and non-stressed mice in the fourth week of depression induction.\nComment 29. Suppl Figure 4e-h, clarify what’s the CTRL in e-f) in comparison to the data of the control in g-h). I guess these are separate experiments, but please clarify.\nResponse:\nWe thank the reviewer for this important comment. We found that the x-axis labels were miswritten, and they have now been revised in Supplementary Figure 9 (which was originally Supplementary Figure 4). In addition, we added the title for each graph for clear information to readers.\nStatistics table\nComment 30. Typo – Cluter instead of Cluster nearly everywhere\nResponse:\nWe recognize that this can be frustrating and have adjusted the part accordingly.\nComment 31. Fig 4 stats: why are there significant values for Cluster 6, 16, etc., but not shown in the graph? even if only the results not shown in the graph are summarized here (and I would have expected otherwise), cluster 7 is on both??? please check this; there is something wrong.\nResponse:\nWe apologize for the inconvenience and have corrected the number of clusters in the statistics table. The mismatch occurred due to the omission of a step that adjusts cluster numbers to start from 1.\nStatistics\nComment 32. General: Do you test for normal distribution before performing a t-test? If so, please add it to the Statistics paragraph.\nResponse:\nWe thank the reviewer for spotting the statistical issue. We have gladly added a thorough description of the t-test to the statistics section, considering the distribution of sample data. The corresponding part (Methods, line 1031) now reads as:\n“[…] Two-group comparisons, including those for syllable usage and syllable subset analysis, were performed using two-tailed unpaired t-tests (parametric) if the group passed the Anderson-Darling (AD) normality test. Otherwise, we used a Mann-Whitney test (non-parametric) to deal with non-Gaussian distribution.”\nReviewer #4\nReviewer #4 (Remarks to the Author):\nThe manuscript presents CLOSER, a self-supervised contrastive learning framework designed to analyze spontaneous rodent behavior in a chronic mild stress model of depression. This approach represents a step forward in behavior quantification by integrating 3D skeleton tracking and unsupervised learning. CLOSER provides a tool for advancing preclinical psychiatric research, particularly in identifying nuanced depressive-like behaviors and tracking responses to antidepressant treatments in a more ecologically valid setting.\nKey contributions of the manuscript include the following:\nFirst, CLOSER enhances behavior quantification by integrating 3D pose skeleton data, which captures detailed motion patterns and adds granularity to behavioral segmentation. This approach enables a more refined quantification of behavior, which is especially beneficial for nuanced analyses in psychiatric and behavioral neuroscience research.\nSecond, the framework leverages a self-supervised contrastive learning approach, organizing behavioral representations effectively without reliance on labeled data. While contrastive loss is common in other areas of ML it was not applied to animal behavior quantification yet.\nThird, CLOSER applies its analytical power to the Chronic Mild Stress (CUS) model, identifying behavior changes associated with chronic stress in rodents and revealing sex-specific and time-dependent behavior patterns. These findings offer insights into the differential impact of chronic stress on behavior, and the model’s application to pharmacological studies demonstrates its potential for preclinical drug screening.\nFinally, the model shows potential for creating standardized, reproducible behavioral assays, with the ability to quantify behavioral changes linked to both depressive-like phenotypes and antidepressant responses. This represents a promising advancement for translational research and reproducibility in animal models of psychiatric disorders.\nCLOSER’s implementation builds upon the MoCo (Momentum Contrast) framework by Meta, incorporating several similar components for contrastive learning, such as a dual encoder system, momentum-based updates, and a projection head. The model architecture mirrors MoCo’s query-key structure, with two encoders to process different behavioral data modalities, stabilized by a momentum mechanism that supports consistent embeddings over time.\nThe originality in CLOSER lies in its domain-specific adaptations, such as using 3D skeletal and kinematic data and tailoring augmentations to the behavioral nuances of rodents. However, the modifications are incremental rather than fundamentally new: they focus on adapting MoCo’s principles to the nuances of behavior analysis rather than introducing novel learning techniques or architectural elements.\nDespite these contributions, there are several points that raise questions and could benefit from further clarification or refinement.\nResponse:\nWe appreciate the constructive feedback and hope to have addressed the reviewer's concerns in our answers below.\nMajor comments:\nComment 1. First, while the reported accuracy increase over VAME is an improvement, it remains marginal. The absence of error bars or confidence intervals for the reported benchmarks raises questions about the robustness and generalizability of these findings, especially considering potential dataset-specific effects or stochastic variations. Including error bars, confidence intervals, or statistical tests would support the accuracy claims and enhance clarity regarding model generalizability across different datasets.\nResponse:\nWe appreciate this helpful suggestion that the performance assessment requires comprehensive statistics. Together with this comment, we revised the performance measurement for three behavioral recognition models (CLOSER, VAME, and keypoint-MoSeq) in an unbiased condition. In the revision process, we carefully examined the model performance by adjusting four parts of the measurement of behavior prediction accuracy1) employment of the embedding distance-based classifier, 2) introduction of independent test dataset, 3) subtle hyperparameter tuning in VAME, 4) specification of hyperparameters in CLOSER, and 5) an adaptation of PCA and UMAP embedding.\nWe described a detailed analysis of the performance comparison below, which is the same as the response for reviewer 2 (major comment 2).\nIn detail:\n1) We discarded an action module using a simple multi-layer perceptron (MLP) decoder. Commonly, self-supervised learning is operated in pre-training models using encoder architectures, then fine-tuned for specific downstream tasks after the addition of decoders. Accordingly, we adopted an exemplary algorithm in the BERT framework to investigate whether the amount and quality of information in embeddings are sufficient to be decoded into action modules using a simple multi-layer perceptron (MLP), and trained a two-layer MLP with a hidden dimension size of 100 and a RELU activation function for action module prediction. However, we found that the structure of the embedding space cannot be accurately estimated in the presence of a decoder if K-means clustering or Hidden Markov Model (HMM) are performed directly on the embedding structure. Fortunately, the k-nearest neighbor (k-NN) probing exhibits a high efficiency without any distribution assumption and is a reliable indicator of downstream task performance2. Indeed, many studies have employed the k-NN classifier to assess the quality of the latent space3,4. Hence, we revised our validation, adopting the k-NN classifier in embedding space qualification throughout the study.\n2) To ensure the fairness of the comparison step, we generated an independent test dataset that was never seen during the training process. Similar to the original data production, we assigned one of thirteen behavior labels in Supplementary Table 2 to each video frame for a freely moving mouse.\n3) Regarding the hyperparameter tuning in the VAME framework, we listed the default and optimized hyperparameters of VAME in the table below. It must be highlighted that we modified the VAME structure to develop an optimal environment comparable to our framework. The VAME structure, which originally managed 2D skeletal input, was adjusted to receive a 3D action skeleton while complying with the ratio of hidden dimensions. Also, we reshaped the 30-frame reconstruction / 15-frame prediction training format into a 10-frame reconstruction / 5-frame prediction to compare with other methods without bias. In addition to former optimizations, we further modulated the learning rate and the number of maximum epochs in the revised manuscript, allowing us to monitor the effect of hyperparameter manipulation. The classification performance of VAME trained with two learning rates (5e-4 or 1e-3) and two maximum epochs (500 or 20) was tested.\nHyperparameters of VAME Default Optimized\nInput dimension 12 27\nEmbedding dimension 30 108\nFrame lengths for reconstruction 30 10\nFrame lengths for prediction 15 5\nInput normalization Yes Yes\nOptimizer Adam Adam\nBatch size 256 256\nLearning rate 5e-4 5e-4 or 1e-3\nMax epochs (patience step) 500 (50) 500 (50) or 20\nThe details of hyperparameters are now added to the Methods section (line 923) as follows:\n“Variational Animal Motion Embedding (VAME). For the VAME, we modified the input structure to accept 3D behavioral segments as input through the extension of the number of features, from 12 to 27, and the embedding dimension, from 30-dimension to 108-dimension. In model training and inference, we adjusted the frame lengths of reconstruction and prediction methods as ten and five frames, respectively. As the variational autoencoder architecture of the VAME consists of an encoder and a decoder, each using Gated Recurrent Units (GRUs), we trained the model with the given parameters for reconstruction, prediction, and KL divergence, which shows the maximal performance in behavioral classification. […]”\n4) To clarify the hyperparameter information for the CLOSER pipeline, we categorized any hyperparameters used on our platform. Specifically, the momentum of a key encoder and temperature for the InfoNCE loss and distillation calculations originate from the MoCo framework, which we utilized in the CLOSER. The maximum angle for the ‘ChangeAngle’ augmentation module was determined after a comparison of linear evaluation tests for various degrees. The corresponding table is now included as Supplementary Table 3.\n5) In the previous analysis, we employed a performance evaluation to assess the quality of behavior representations, excluding the segmentation step. Based on this framework, we initially believed that it was reasonable to use the 13-dimensional PCA projection of individual frames (i.e., the input to the HMM) as the behavior representation of keypoint-MoSeq in performance estimation. To account for sequential context, we concatenated 10 consecutive frames of these PCA projections to create 10-frame length behavioral segments. However, we recognize that evaluating the input of keypoint-MoSeq alone does not fully capture the motion recognition efficiency of the entire tool. To address this limitation, we now provide an alternative comparison utilizing dimensionality reduction methods (PCA and UMAP) on raw data—comprising 270-dimension vectors representing the 3D coordinates of keypoints over 10 consecutive frames—instead of the keypoint-MoSeq input. We have included the result of the performance comparison with raw pose data (270 dimensions), PCA (13 dimensions, identical to Keypoint-MoSeq input dimension), and UMAP (13 dimensions), and addressed the detailed description. Moreover, we have revised the Results section (line 150) to clarify the employment of the PCA and UMAP embedding as follows:\n“[…] Since keypoint-MoSeq employs an HMM that learns transition probabilities between states, behavioral syllables IDs, and emission probabilities from a given 13-dimensional Principal Component Analysis (PCA) projection of motion sequences, we leveraged 270-dimension representation with 13-dimension PCA and Uniform Manifold Approximation and Projection (UMAP) embedding for 10-frame length behavioral segments, instead of keypoint-MoSeq. […]”\nAlong with the modification of the benchmarking embedding of Keypoint-MoSeq, we re-analyzed the performance of the behavioral classification across different methods. As a result, our framework exhibited the highest balanced (66%) and top-1 (55%) accuracy. Moreover, additional metrics revealed that CLOSER consistently showed strong precision, recall, and F1 scores. We have revised Figure 2c and the corresponding Results section (line 155) to reflect the updated performance metrics as follows:\n“[…] In this analysis, CLOSER provided the highest balanced (66%) and top-1 accuracy, accompanied by consistently strong precision, recall, and F1 score. […]”\nFig. 2 ǀ The CLOSER framework recognized temporal and behavioral characteristics for classifying behavior segments into semantic syllables. […] c, Normalized confusion matrix (top) and associated performance metrics (bottom) showing the accuracy of k-nearest neighbor classifier based on the latent spaces of different embedding methods. The classifiers were trained on the top of the embedding spaces with an unseen manual annotation dataset to predict the behavior labels for given segments. Accumulated performance over the fivefold cross-validation is shown. The classification rate is represented in grayscale, with white for a classification rate of 1 and black for a classification rate of 0. Box plots show 25th-75th percentiles of the precision, recall, and F1 score for methods. […]\nMoreover, addressing additional comments as well (Reviewer 2, Comment 1a and 1b), we have added further intrinsic evaluation of latent spaces for different embedding methods using Pearson Correlation Coefficient, which measures intra-cluster and inter-cluster, in new Supplementary Figures 1 and 2 to strengthen the validation of embedding qualities.\nComment 2. Secondly, the use of 3D skeleton data indeed enhances behavioral segmentation, yet this could likely be achieved by combining DANNCE or similar 3D tracking tools with VAME. This integration raises questions regarding the necessity of developing an entirely new model to incorporate 3D capabilities. Comparing CLOSER’s performance with a 3D-augmented VAME (i.e. running DANNCE and adding the 3D data as input to VAME) would clarify whether CLOSER’s use of contrastive learning advantages over an adaptation of existing frameworks, or if the improvement stems only by the fact that more data (3D coordinates) are fed to the model.\nResponse:\nThe reviewer is correct that the VAME version using 3D action skeleton coordinates should be tested for a fair comparison. In fact, we had already modified the VAME framework to train with 3D coordinates. We understand that the confusion may have arisen from the lack of detailed descriptions in the previous manuscript. We have addressed this point in conjunction with our response to Comment 1. We hope this explanation resolves all the concerns raised by the reviewer.\nComment 3. Third, while the findings on chronic stress-induced behavior changes are compelling, it remains unclear if these insights are due to the unique capabilities of CLOSER or simply the result of analyzing a new benchmark model. Including baseline results with existing models for the same chronic stress, data would help establish the unique value of CLOSER’s findings in this context.\nResponse:\nWe thank the reviewer for the valuable comment and share the same interest. As reviewer 2 (major comment 3) raised the same concerns considering the comparison of depression diagnostic power, we note that the response below is identical:\nWe share the same interest and had anticipated that analyzing performance in the diagnosis of mouse depressive disorder using three different motion recognition frameworks could underscore the necessity of our tool. However, the GPU-RAM demand of VAME exceeded our computational resources due to the HMM processing on the massive dataset. Fortunately, Miller, Stephanie R., et al. recently introduced a k-means algorithm for latent space clustering, which has been shown to improve the classification accuracy of the genetic Alzheimer’s disease model (Miller, Stephanie R., et al., 2024). Building on this advancement, we used the keypoint-MoSeq and k-means version of VAME segmentation to explore the effectiveness of alternative segmentation methods in the detection of subtle behavioral changes in stressed mice.\nFollowing the analogous approach to the examination of diagnostic accuracy shown in Figure 3h, we investigated the predictive performance of individual pipelines for depressive disorder in mice. To demonstrate this, we partitioned the motion sequences of non-stressed and stressed mice into 5-minute segments for both males and females. Behavioral syllables and summaries were then extracted employing alternative methods, which were subsequently used to train random forest classifiers for diagnosis. We have now incorporated these results in the main manuscript (Results, line 223) with a new Supplementary Figure 3. The relevant section and the figure are added below:\n“To further evaluate the efficiency of CLOSER in depression diagnosis, we applied the same analysis on male and female groups using the syllable combinations identified by VAME and keypoint-MoSeq. Classification accuracy indicated that CLOSER outperformed VAME and keypoint-MoSeq in the male mice, achieving superior predictive performance for both classifiers trained using syllable frequency of transition (Supplementary Fig. 3a, b). In contrast, for non-stressed and stressed females, the random forest classifier leveraging syllable frequency from keypoint-MoSeq demonstrated greater performance compared to other behavior recognition pipelines (Supplementary Fig. 3c); nevertheless, the syllable transition calculated by CLOSER consistently yielded the highest accuracy (Supplementary Fig. 3d). Notably, behavioral deconstruction in males offered higher classification rates than in females for all segmentation methods, aligning with the observed sex-dimorphic alterations of syllable frequency and transition (Fig. 3i, l).”\nSupplementary Fig. 3 ǀ Diagnostic accuracy of behavioral parsing pipelines in a mouse model of depressive disorder. (a, b) Confusion matrices illustrating the classification accuracy of CLOSER (left), VAME (middle), and keypoint-MoSeq (right) pipelines in male mice. The matrices compare the diagnostic performance based on syllable frequency (a) and syllable transition (b), where brighter colors along the diagonal represent a better classification rate. The CLOSER pipeline consistently outperforms VAME and keypoint-MoSeq in the male group. Box plots indicate 25th-75th percentiles of the precision, recall, and F1 score for each analytic pipeline, based on a 5-fold validation step of the random forest classifier, and all individual outcomes are displayed as data points. (c, d) Confusion matrices and box plots describing classification performance and statistics for non-stressed (CTRL) and stressed (CUS) female mice, using classifiers trained with frequency (c) and transition (d).\nComment 4. Finally, the paper does not provide a detailed comparison of the findings related to the CUS model with existing literature to establish their novelty in the neuroscience field. Without such a comparison, it's challenging to determine whether these behavioral patterns represent new discoveries in neuroscience or if they align with previously reported observations. Have there been perhaps some new substantial findings using the new method that were not known so far?\nResponse:\nWe sincerely appreciate the reviewer’s critical comment regarding the novelty and significance of our findings compared to existing literature. Psychomotor symptoms in humans are well-established clinical features, particularly within melancholic depression subtypes16,17. However, current mouse models predominantly rely on tests with known limitations, including low construct and predictive validity (e.g., sucrose preference test and tail suspension test), and high sensitivity but low specificity (e.g., forced swim test). This has led to the predictive validity of current preclinical drug screening using traditional mouse models remaining unsatisfactory. Indeed, recent literature pointed out a need for enhanced preclinical frameworks, which evaluate the animal behavior in semi-naturalistic settings to capture complex, ethologically relevant behaviors for high translational potential and predictive validity in drug discovery18.\nTo date, comprehensive machine learning-based frameworks to evaluate naturalistic behaviors across various depression models and antidepressant treatments remain unexplored. Sturman et al. (2022)19 tried to discover alterations of naturalistic behavior in the predator stress mouse model, while they didn’t estimate diagnostic performance based on their behavior profiles, in addition to limitations in comprehensive evaluations across different etiologies, pathological stages, and antidepressant efficacy. In another study, Dolensek et al. (2020)20 employed tail suspension as a sole stressor and analyzed the naturalistic behavior in this mouse model without any post-hoc investigation. Numerous studies have utilized motion characteristics to differentiate other disease states such as Parkinson’s disease21, Alzheimer’s disease22, and epilepsy23. Yet, comprehensive and high face validity models addressing chronic stress-induced depression through naturalistic behaviors and drug efficacy screening have not been established.\nAfter thorough revision in response to reviewers’ constructive feedback, we have now significantly improved our manuscript by:\n Employment of distinct depression models: stress, inflammation, and hormone-induced models (new Supplementary Figure 7, line 283)\n Estimation of antidepressant effects on naturalistic behavior structures of female mice (new Supplementary Figure 16, line 462)\n Comparison between traditional monoaminergic antidepressants and novel non-monoaminergic antidepressant candidates (Figure 5, line 398)\n Validation of high-quality embedding in our CLOSER pipeline designed for depression diagnostics (new Supplementary Figures 1 and 2, line 117)\nBased on our findings, we would like to highlight two significant discoveries of this manuscript.\n1) Sex-dimorphic depressive symptoms and drug actions: Conventional depressive-like behavior tests could not illustrate differences between male and female depression models. Our naturalistic behavior analysis, however, distinctly reveals pronounced sex-dependent differences in behavioral structures and antidepressant efficacy. Considering the sex-independent drug prescription process in human depressive disorder, this finding suggests sex-dependent prognosis and underscores the incorporation of gender considerations into clinical antidepressant prescription.\n2) Antidepressant-induced recovery of depressive phenotypes in naturalistic behaviors: While previous studies only emphasized general behavioral changes induced by antidepressants, our results demonstrate how antidepressants restore depression-induced alterations in behavioral structure, highlighting the capability of our framework to quantify “on-target” drug efficacy.\nWe acknowledge that translational relevance from mouse models to human clinical settings necessitates further validation. Nonetheless, our manuscript provides critical evidence addressing whether motion features have diagnostic value in the depression mouse model. Therefore, we think that we established a novel approach for depressive disorder and provided a strong rationale for future translation application of motion profiles in diagnosis by demonstrating heterogeneous behavioral features in the depression model mice through our depression-optimized frameworks.\nComment 5. In terms of reproducibility, while the code is available, the absence of example datasets limits the ability of other researchers to replicate the findings or to test the model on new datasets. Providing a sample dataset or detailed data specifications would allow others to reproduce the results for reviewers AND readers and evaluate CLOSER’s effectiveness across diverse datasets.\nResponse:\nWe sincerely apologize for omitting the sample datasets, including the mouse 3D coordinates used in our study. We have now uploaded the whole raw dataset at the provided Zenodo link to guarantee the utility of the CLOSER.\nMinor comments.\nComment 1. The manuscript presents confusion matrices for classifier accuracy, but the interpretation of these matrices is limited without details on error patterns. This limitation restricts insights into potential misclassifications. Adding evaluation metrics like precision, recall, and F1 scores, along with error bars, could illustrate model reliability and provide a more comprehensive understanding of classification performance.\nResponse:\nWe agree with the reviewer that confusion matrices themselves cannot provide sufficient information on the classification performance. Therefore, we have added detailed classification statistics for all confusion matrices in the manuscript.\nComment 2. The use of LDA to visualize antidepressant effectiveness could be strengthened by a rationale for its selection over other dimensionality reduction techniques. Including a comparison with alternatives like t-SNE or UMAP would offer a broader perspective on clustering patterns and further support the use of LDA.\nResponse:\nWe understand the reviewer’s point about the utilization of other dimensionality reduction (DR) methods for the representation of behavioral structures to prove the effectiveness. However, our key rationale for using LDA, a supervised DR method, is to find a projection that best separates the data based on known class labels, while t-SNE and UMAP, unsupervised algorithms, cannot reflect the label of experimental samples. Given that the group labels were known in advance, we do believe that LDA was the appropriate choice to highlight group separability in the embedding space.\nNonetheless, additional visualization of behavioral structures using t-SNE and UMAP could strengthen the discriminability of CLOSER in mouse conditions. Thus, we performed t-SNE and UMAP on the dataset of Figure 3, which included behavioral syllable frequencies and transition probabilities from stressed and non-stressed male, female groups.\nHowever, these unsupervised approaches didn’t clearly separate the groups in the reduced two-dimensional space, which indicates that although group-discriminative information exists in the higher-dimensional space, simple preservation of topological structure in 2D is not sufficient to capture these distinctions. Although these are intriguing results showing the difference of DR methods on behavioral structures, we did not include them in the revised manuscript, as we think that a comparison of various DR methods is beyond our scope, while LDA analysis is sufficient to address the structure of naturalistic behaviors in mice.\nComment 3. The robustness and interpretability of SHAP summary plots for antidepressant efficacy would benefit from both global and individual feature importance, helping to identify consistent features across different conditions. Presenting individual SHAP values for specific animals or conditions could clarify which features are universally important or context-dependent in the assessment of antidepressant efficacy.\nResponse:\nWe agree with the reviewer that highlighting individual feature importance for each drug or stress condition could enhance the distinctiveness of the behavioral fingerprint of each antidepressant. In our original manuscript, we presented impactful syllables for each drug, but we recognized the need to emphasize top-five SHAP values at both global and individual scales to better capture the distinctive features of each drug. Accordingly, following the reviewer’s comment, we have added individual SHAP plots for these top-5 impactful features in the new Supplementary Figure 14, with a detailed description of how they reflect key differences across antidepressant efficacy (Results, line 430):\n“[…] Utilizing SHAP analysis for deeper behavioral profiling, we identified syllables that critically influence antidepressant classification, indicative markers reflecting antidepressant heterogeneity (Fig. 5e). Different variants of \"rear up\" behaviors emerged prominently as discriminative features. Specifically, increased occurrences of clusters 48, 47, and 4 and decreased cluster 3 activity characterized positive predictions for non-monoaminergic antidepressants, while inverse patterns distinguished monoaminergic treatments (Supplementary Fig. 12a). Within groups, fluctuations in cluster 19 contributed to precise drug identification. These findings underline that the CLOSER pipeline effectively leverages profound behavioral compositions of mice to distinguish psychiatric drug actions, offering potential for personalized therapeutic recommendations grounded in behavioral phenotypes.”\nSupplementary Fig. 14 ǀ SHAP analysis reveals treatment-specific alterations in behavioral syllable contributions in mouse models of chronic stress and antidepressant interventions. a, SHAP summary plots of most impactful syllables for individual prediction of monoaminergic or non-monoaminergic antidepressant treatments in male mice. b, SHAP values for male mice treated with saline, fluoxetine, or imipramine in the absence of stress exposure. Each table lists the top five impactful behavioral syllables along with corresponding annotations. c, Explanation of the random forest classifier predictions for stressed female mice administered with monoaminergic antidepressants. In adjacent SHAP plots, the x-axis denotes the SHAP feature value, reflecting the magnitude and direction of each syllable’s contribution to classification outcomes. The sign of the SHAP feature value indicates the decision tendency of the classifier for a given group. Each dot represents an individual animal, with color gradients indicating the relative expression level of each behavioral syllable.\nComment 4. The interpretability of the results for drug classifications based on confusion matrices is unclear. Supplementing these matrices with precision-recall metrics or analyzing specific types of errors (e.g., false positives vs. false negatives) could offer insights into the model’s strengths and weaknesses in classifying stress conditions and drug responses. Clarifying why certain drugs or conditions are more challenging to classify would make the results more actionable.\nResponse:\nWe appreciate the suggestion for careful interpretation of the classification of drugs. Before stating the classification, we would like to inform you that we have now included non-monoaminergic antidepressants (agmatine, scopolamine, and rapastinel) in addition to previously reported monoaminergic agents (fluoxetine and imipramine) to broaden our comparison of distinct antidepressant mechanisms. Monoaminergic antidepressants target neurotransmitters such as serotonin, norepinephrine, and dopamine, while the non-monoaminergic compounds act through alternative pathways, for example, muscarinic acetylcholine receptors (mAChRs) or N-methyl-D-aspartate receptors (NMDARs).\nNext, following the reviewer’s suggestion on the interpretability, we investigated the false negatives of the random forest classifier by categorizing errors into three categories—vehicle, intra-group, and inter-group—thereby clarifying how naturalistic behavioral patterns differentiate drug classes. This fine-grained error analysis helped us understand the strengths and weaknesses of our model in capturing subtle distinctions between monoaminergic and non-monoaminergic effects. Moreover, we incorporated sensitivity and specificity for a qualitative assessment of the classifier’s performance following the minor Comment 6 below. We believe that these additional metrics and the expanded set of non-monoaminergic comparisons enhance the interpretability of our classification approach. We have revised our Results (line 413) and added a new Supplementary Figure 12 as follows:\n“CLOSER can quantify the pharmacological effects of monoaminergic and non-monoaminergic antidepressants on psychomotor symptoms. […] As the initial step of our ML-guided behavioral evaluation, we implemented a machine learning-guided behavioral evaluation to discern subtle alterations within exploratory behaviors. A random forest classifier, informed by syllable frequencies and transitions, yielded sensitivities of 40.35% and specificities of 88.00% (Fig. 5b). Incorporating transition information enhanced performance, raising sensitivity to 50.07% and specificity to 90.01%. We examined the classification errors, which revealed predominant confusion within drug groups; however, inter-group and vehicle-associated misclassifications remained moderate (Supplementary Fig. 12).”\nSupplementary Fig. 12 ǀ Distribution of false negative rate in random forest classification of vehicle, monoaminergic, and non-monoaminergic groups. False negative rates in random forest classification of drug types using syllable frequency (left) or (right) syllable transition features (right). Each stacked bar shows the proportion of false negatives due to confusion with a vehicle (gray), confusion within the same antidepressant group (intra-group; orange), or confusion between different antidepressant groups (inter-group; blue).\nComment 5. The syllable usage and transition analyses would benefit from statistical validation to support claims of generalizability. Adding statistical tests, such as permutation tests, and validating the results on additional datasets would reinforce the claims of generalizability.\nResponse:\nWe agree with the reviewer that the generalizability of our platform would benefit from statistical tests and validation on additional datasets. To clarify, all reported alterations in syllable frequency and transition within spontaneous behavior were statistically validated using either an unpaired t-test or ANOVA, followed by appropriate post-hoc tests, as addressed in the Statistics of the Methods section. Moreover, the details of these statistical methods are fully described in the figure legends and supplementary statistical data.\nAs the reviewer suggested, we conducted additional permutation tests on the behavioral structures of male and female mice from Figure 3. We then illustrated the result of permutation tests by labeling significant alterations in syllable frequency and transition calculated by the unpaired t-test or the permutation test below. The permutation tests yielded significance patterns similar to the original analysis, although slightly more syllables or bigrams were identified as significant alterations due to the lack of distributional assumptions. Since we have already employed the unpaired t-test as a statistical method, we decide not to include the results of permutation tests on depressive behavior analysis.\nTo further validate the consistency, we grouped the behavioral structures from two independent batches of male mice exposed to the chronic stressor and compared the representative variables, syllable frequency and transition. The result revealed that overall syllable distributions are similar between batch groups, and only two syllables (syllables 45 and 34) showed statistical differences. The transition bigram exhibited a few weak transition differences between the batch sets. These findings indicate that, while some minor batch-to-batch variations exist, the differences are modest relative to those observed between stress conditions.\nComment 6. The reliance on balanced accuracy as the sole metric for classifier accuracy may not fully capture the model’s performance in distinguishing depressive symptoms. Including metrics like sensitivity and specificity would offer a comprehensive evaluation and ensure that the classifier’s performance reflects accuracy across various symptom categories.\nResponse:\nWe thank the reviewer for highlighting this weakness, and we strengthened the interpretation of classifier accuracy through sensitivity-specificity analysis. We have addressed this point in conjunction with our response to minor comment 4.\nDetailed points (citations from the draft):\nDraft: In the estimation of model prediction accuracy, the algorithm incorporating Scale, ChangeStartPoint, ChangeAngle, and ChannelMask presented the best performance for both Top-1 (73%) and balanced accuracy (70%) in the classification of manually labeled mouse behavior segments\nComment 7: This is absolutely unclear, how should the reader know what the algorithm incorporating Scale, etc, means?\nResponse:\nWe recognize that the usage of the function name can confuse readers. We have revised the terminology to intuitively represent the type of augmentations with the best performance. The changed text in the main manuscript (Results, line 111) is as follows:\n“[…] In the estimation of model prediction accuracy, the algorithm incorporating size scaling, joint angle distortion, motion crop, and x- or y-axis mask presented the best performance on all evaluation metrics for the classification of manually labeled mouse behavior segments.”\nDraft: To determine and categorize behavioral syllables, which are minimal motion units involving ethological context, we leveraged K-means clustering in 128 dimensions to partition behavior segments into 50 clusters.\nComment 8: How were these numbers picked?\nResponse:\nWe appreciate the insightful comment regarding the selection of the number of clusters, which was not addressed in the previous manuscript. To determine an appropriate granularity, we varied the number of clusters (from 10 to 100 in increments of 10) and investigated whether a single cluster represents a single behavioral pattern or if identical behavioral motifs split into multiple clusters. Through this evaluation and subsequent parameter tuning, we concluded that the final choice yields (50 clusters) behavioral syllable sets that are independent and unique, thereby providing an optimal balance for specificity and interpretability in our behavioral sequence embeddings.\nHowever, we agree with the reviewer that determining the optimal number of clusters should be estimated in a quantitative manner. Here, we evaluated clustering quality using both the Silhouette score and Davies-Bouldin index across a range of cluster numbers (from 10 to 100 in increments of 10). The results indicated a general trend of improved clustering quality as the number of clusters increased. This suggests that our embedding space focuses on distinctions of fine-grained behaviors.\nAlthough clustering performance continued to improve beyond 50 clusters, we chose 50 as a practical compromise, since it sufficiently captures the structural diversity of the ethogram. As shown in the annotation results, 50 clusters are enough to represent diverse behavioral categories. Moreover, we would like to emphasize that our software allows users to adjust the number of clusters in the k-means clustering step, which enables flexible tuning of the analysis to suit specific experimental needs.\nDraft: Given the continuous nature of mouse behavior, a behavioral syllable was denoted as a consecutive frame with the same cluster label.\nComment 9: No minimum length was picked or no smoothing or filtering applied?\nResponse:\nWe thank the reviewer for spotting this issue. As the reviewer mentioned, we didn’t apply filtering or smoothing the segment according to the length, including the minimum length condition. The main reason for such an algorithm is that the distribution of frame lengths for behavior syllables calculated from a majority voting was comparable to other existing approaches, VAME and keypoint-MoSeq (Figure 2b). To provide a clearer indication of the length of identified syllables, we revised the corresponding section to include the time information in addition to the frame length (Results, line 141).\nDraft: Cluster labels were determined by majority voting on a maximum of ten cluster candidates collected at the first learning step (Fig. 1a).\nComment 10: Unclear, how were the ten cluster candidates collected?\nResponse:\nWe understand the reviewer’s point and make sure to clarify the concept of dynamic temporal segmentation in Figure 1a. A central idea behind our approach is that as one frame belongs to up to 10 behavioral segments, which are obtained from a sliding window of size 10 frames with a stride of size 1 frame, we can do majority voting on cluster labels assigned to those behavioral segments. The corresponding main text (Results, line 130) is as follows:\n“[…] Then, a cluster label for each frame was determined by majority voting on candidates assigned from respective sliding windows at the first learning step (Fig. 1a). Given the continuous nature of mouse behavior, a behavioral syllable was denoted as a consecutive frame with the same cluster label.”\nDraft: Observing the transition of behavior motifs longer than 10 frames with z-scored keypoint change-scores, we found that the start frame\nComment 11: It would be much more scientific to report seconds/milliseconds, than the number of frames so that the reader can compare this value directly to other publications without reading up on which frame rate you were using.\nResponse:\nWe appreciate this helpful comment and have changed all texts containing frame information, including the sentence above.\nDraft: ... and segments created by keypoint-MoSeq (27.93 ± 34.95 frame) but longer than those created by Variational Animal Motion Embedding (VAME) (7.75 ± 12.52 frame) ...\nComment 12: which configurations have you used for both packages?\nResponse:\nWe clarified the configuration of CLOSER in the new Supplementary Table 3. Additionally, hyperparameters used in VAME are listed in the table included in major comment 1. The table also incorporates hyperparameters used in the optimization process.\nDraft: In this analysis, CLOSER provided a higher balanced accuracy (73%) than VAME (68%) or keypoint-Moseq (37%).\nComment 13: Could you provide this as +/- STD over multiple videos to see the degree of error in generalization?\nResponse:\nTo address the reviewer’s comment, we assembled an independent test set composed of mice and video sessions not used during training. We then evaluated classification performance for CLOSER, VAME, and pose representations (including PCA and UMAP) in major comment 1. These results confirm that CLOSER not only achieves the highest average accuracy but also maintains general performance across independent sessions, demonstrating its generalization to unseen data.\nDraft: Initially, we performed an LDA projection for syllable frequency and transition summaries of five-minute intervals to address the sexual disparity in stress-induced ethograms (Fig. 3g).\nComment 14: So far, the LDA and transition summaries have not been introduced, hence is hard to follow here.\nResponse:\nWe apologize for the absence of an explanation for the LDA and syllable transition diagram. The Result section (line 208) was extended as below with additional citation:\n“[…] Initially, we performed a Linear Discriminant Analysis (LDA) to transform the syllable frequency and transition summaries of five-minute intervals into a low-dimensional subspace (Fig. 3g), which enhances the expressiveness of stress-induced ethograms to address the sexual disparity (Fig. 3g). As LDA searches the axes showing the best classification performance, the larger distances between samples from different classes indicate the higher disparity in original data. […]”\nDraft: In both males and females, CLOSER-identified syllables and sequences enabled the classifier to predict depressive disorder,\nComment 15: To make sure, has this been tested on unseen data?\nResponse:\nFor all confusion matrices in the main manuscript, we adjusted 5-fold validation and prepared the dataset by partitioning 20-minute sample recording videos into 5-minute sections (Methods, line 993). We adopted such a training and test set methodology to overcome the limitation of the number of behavioral datasets.\nDraft: In the hierarchical structure of defined behavior syllables, clusters assigned analogous labels were located adjacent to one another in the hierarchy table\nComment 16: How come there are so many labels related to “sniff”? It seems to me that these are labels that have been randomly assigned during labeling but not a concise dictionary of behaviors defined a priori (ideally comparable to different publications).\nResponse:\nThe reviewer correctly points out the issue regarding the annotation of sniffing behaviors. In the original manuscript, we labeled “sniff” for investigations, which include both “stretched attend” and “undirected sniffing” (https://mousebehavior.org/ethogram-comparison/). To distinguish two different investigation modes, we have now revised the manuscript and corresponding figures, so that the sniffing syllables are assigned solely for undirected sniffing behavior.\nDraft: …with discrete distribution patterns of behavior segments in embedding spaces for global and local components built with LatentOut.\nComment 17: What is LatentOut? Please don’t use code functions in an article, that’s really hard for the reader to interpret and make sense of if the code is not known of not of interest.\nResponse:\nWe thank the reviewer for this important comment. Anomaly detection is the task that identifies anomalous instances without explicit training. Among various anomaly detection algorithms, we applied LatentOut, developed by Fabrizio Angiulli, et al., which enables an anomaly detector to train on a corrupted dataset. Therefore, we confirm that LatentOut is not merely a function but an anomaly detection algorithm. However, we acknowledge that additional clarification is necessary to enhance readability and ensure that readers can follow the subsequent analysis. To address this, we rewrote the sentence with the corresponding reference (Results, line 37), which now reads:\n“[…] with discrete distribution patterns of behavior segments in embedding spaces for global and local components built with latent space-based anomaly detection26 (Supplementary Fig. 3i, j).”\nDraft: This finding indicated that the antidepressants only partially alleviated the depressive symptoms in spontaneous behavior.\nComment 18: It’s OK to say “indicated”, but making some judgments based on LDA embeddings is rather naive.\nResponse:\nWe recognized the inaccuracy of our interpretation and omitted the sentence from the main manuscript.\nDraft: We addressed this problem by applying a self-supervised contrastive learning algorithm with augmentation modules that force the machine to embrace the underlying behavioral structure while ignoring irrelevant features\nComment 19: How do you define “irrelevant features”?\nResponse:\nThe reviewer raises an important point. In the previous manuscript, we meant to emphasize a characteristic of contrastive learning that output embeddings where similar instances are located closer in the embedding metric space (Charlotte Loh et al., 2022), and used the ‘irrelevant features’ to express factors with marginal effect on the syllable segmentation of mice (for example, the size of individuals). We acknowledge the ambiguity of terminology and have altered the text (Discussion, line 514) using technical terms:\n“[…] We addressed this problem by applying a self-supervised contrastive learning algorithm with augmentation modules that force the machine to embrace the underlying behavioral structure using invariance and contrast.”\nDraft: This leads to the development of transferable and generalizable ML agents\nComment 20: How is your codebase an ML agent?\nResponse:\nWe agree with the reviewer that an ML agent is only appropriate for reinforcement learning, where agents learn optimal strategies for specific tasks, and have replaced the word with ‘toolkit’ for the self-supervised algorithm.\nDraft: Indeed, the algorithm generated high-quality embedding for naturalistic behavior compared to other recent unsupervised toolboxes, with clear recognition of novel motion patterns imperceptible on visual inspection.\nComment 21: How can you say that if your score was marginally better than for the competition? This needs to be re-assessed with a larger scope and at least an additional dataset in comparison.\nResponse:\nWe thank the reviewer for bringing this point to our attention. We strengthen the performance measurement including the unseen, independent dataset as mentioned above (major Comment 1). We believe that the improved analysis provides a comprehensive assessment of efficiency comparison for individual behavioral recognition toolkits.\nFinal Verdict\nWhile CLOSER is a carefully implemented application of contrastive learning for analyzing rodent behavior, its contributions appear primarily incremental. The model heavily relies on the established MoCo framework, with adaptations that are domain-specific but do not introduce fundamentally new methodologies or architectural innovations. These modifications—such as using 3D skeletal data and behavior-specific augmentations—tailor MoCo’s principles to the nuances of rodent behavior analysis but lack technical novelty.\nFurthermore, the manuscript does not clearly establish whether its primary contribution lies in the methodological adaptation or in producing new insights within the neuroscience of depressive behavior. Although it reports findings of sex-specific and time-dependent behavior changes in a chronic stress model, the novelty of these results for neuroscience is not convincingly discussed (hopefully this will be confirmed with reviewers with more understanding of the particular field). Without detailed comparisons with existing literature, it is challenging to assess whether these behavioral observations offer new understanding or align with previously known patterns.\nFor these reasons, the paper would benefit from a clearer positioning of its main contribution—whether as a novel behavioral quantification tool or a source of new biological insights—and from substantial additional justification for the technical and scientific novelty of its findings. The authors should aim to clearly establish unique methodological contributions that distinguish CLOSER from existing models like MoCo, provide detailed comparative benchmarks to demonstrate added value over simple adaptations of MoCo and strengthen generalizability through robust statistical validation.\nResponse:\nThroughout the revision process, we have aimed to develop the manuscript into a comprehensive framework explicitly designed for depression diagnosis and drug indication through naturalistic behavior, addressing critical translational gaps in preclinical research, with demonstration of broader and stronger applicability, as described in major Comment 4. We believe that the improvements made substantially enhance the clarity in the scientific relevance of our work, in addition to methodological distinctiveness.\nReviewer #4 (Remarks on code availability):\nIn terms of reproducibility, while the code is available, the absence of example datasets limits the ability of other researchers to replicate the findings or to test the model on new datasets. Providing a sample dataset or detailed data specifications would allow others to reproduce the results for reviewers AND readers and evaluate CLOSER’s effectiveness across diverse datasets.\nResponse:\nWe sincerely apologize for omitting the sample datasets, including the mouse 3D coordinates used in our study. We have now uploaded the whole raw dataset at the provided Zenodo link to guarantee the utility of the CLOSER."}, {"round": 2, "reviewer_comments": {"reviewer_1": "In their revision, the authors have completely failed to take account of my original comment - which they agreed with - that the forced swim test and the tail suspension test do not measure depression, behavioral despair, or any other, similar anthropocentric, subjective construct. They state \"We thank the reviewer for raising this important issue about the interpretation of conventional behavioral tests (FST, TST, and SPT), and fully agree that these tests have significant limitations in construct validity ....\" Unfortunately, they then go on to make the contradictory and erroneous statement \"yet they remain essential for validation of the depression models in mice.\" These tests do not have construct validity or face validity, and are therefore irrelevant for validation of valid mouse models of behavioural processes relevant to depression. For example, there is no objective evidence that swimming/climbing or not swimming/climbing during placement for a few minutes in water has anything to do with any neurobehavioural process that underlies or contributes to depression. Some antidepressants induce an acute increase in swimming/climbing behaviour, but to then misinterpret this as meaning that the mouse would have otherwise been depressed or in a state of behavioural despair is complete nonsense.\nWhat I think the authors are trying to argue is that they are vindicated in using the FST and TST because many other groups have done so in the past and continue to do so. But to continue making the same mistake is not a basis for progress in science. At some point, authors - and editors - need to take a stand and stop using and publishing these tests.\nThe very least that the authors need to do in the current manuscript is to apply appropriate, objective vocabulary: this means that they need to use physical descriptions of what was observed in tests rather than nonsensical, subjective interpretations.\nTherefore:\n\"Moreover, while FST and TST are widely used to represent behavioral despair1, we agree that these tests alone could not capture learned helplessness. Thus, we have removed the term ‘learned helplessness’ from our manuscript and replaced any use of the phrase ‘depressive symptom’ with the more precise term ‘depressive-like behavior’ or ‘depressive-like phenotypes’ to avoid overstating the clinical implications of our findings.\"\nThese tests do not represent behavioural despair and do not capture learned helplessness. Behavioural despair is such a complex emotional state it is ridiculous to infer it here. Do not use the terms behavioural despair or depressive-like; instead, state, objectively, whether one group showed less or more immobility in the test, and then state that the effect of stress was opposite to that induced by an acute dose of an antidepressant. Al least then, it will be clear that the tests were used because of their sensitivity to antidepressants, whatever this may indicate.", "reviewer_2": "I applaud the authors on the new version of the manuscript, which is significantly improved. However, two of my major concerns still need to be addressed before this is suitable for publication.\nMajor point 1\nIn the last round I was concerned that the authors did not directly validate their contrastive learning method. In response to this concern, the authors nicely show that the embedding space of data samples in the same cluster are more correlated than those assigned to different clusters. However, this does not address whether they are learning an invariant representation, this only addresses whether the clustering algorithm is working properly. In fact, a common problem in clustering animal behavior is that the same behavior can be assigned different clusters simply due to size variation. If the embedding space were highly sensitive to size, then the clustering algorithm would take, e.g., rears in small mice and rears in large mice and assign them different clusters. The inter-cluster distance would be larger than the intra-cluster distance *because the embedding space is not size invariant*. In other words, the author’s result does *not* confirm size invariance.\nThe more incisive analysis is in Supplementary Fig. 2, where the authors show that two clusters have strong intra-cluster correlations with mice different in weight, size, or individuality. If I understand this correctly (important detail: I could not easily find what was done here described in the legend or in the methods so please fix this). It does look like VAME and CLOSER do a better job than PCA or UMAP. However: (1) the analysis is only shown for two hand-selected behaviors, and (2) as far as I can tell the correlations are not shown for mice with matched weight, size or individuality, which would be a suitable comparison.\nI recommend the following. First, explicitly plot the weights, sex and individual balances for all behaviors. If the representation is indeed invariant to these quantities you should see weights, sexes and individuals all equally represented in all clusters. Second, repeat your correlation analysis for all behaviors and you must compare different weights, sexes and individuals against the correlations with these variables matched. This will tell us how much the correlation drops as a function of weight, sex and individuality. Third, run a decoding analysis to see if you can decode weight, sex or individuality from either the embedding space, the cluster frequency, or the transition matrix. If CLOSER truly is invariant the decoding should be close to chance. At the moment the reader still cannot tell how well their contrastive learning pipeline actually performs at its stated objective.\nMajor point 2\nI raised concerns about the scale augmentation previously. In response the authors show an example of scale being applied. Again, I am glad to see this but it does not directly address the concern. The concern is whether variation in mouse size can be explained by a single scale parameter uniformly applied to x, y, and z. Can the keypoints of a large mouse be explained by simply multiplying the x, y, and z coordinates of a smaller mouse by one number?\nHere, the authors need to take the skeleton of a single example mouse and show that you can apply scale to the x, y, and z coordinates and get a reasonable match to the skeleton of another mouse different in size (and/or sex). To estimate how good the fit is, the authors can estimate a null distribution by taking the discrepancy across skeletons of mice from the same sex and size. The authors need to quantify how reasonable their approach is since it is central to the method.\nMinor points\nI am not expecting the authors to conduct a complete analysis of AVATAR in this manuscript, I agree it is beyond scope. That said, they are using AVATAR as the basis for all work in this manuscript and it is not standard in the field, so some description is required. Can the authors cite quantitative evidence that AVATAR provides better labeling consistency, and is more robust in localization and with respect to annotation errors as they claim in the rebuttal? Hopefully I didn't miss this, but I could not easily find direct evidence for these claims in the AVATAR preprint (at the moment the supplementary figures do not appear to be accessible). There should be enough information for the reader to have a sense for how noise characteristics, accuracy, etc. compare to standard trackers like SLEAP, DLC and Lightning Pose, and whether this could impact the results of the manuscript.\nI found typos. I am listing a couple of examples here, but please thoroughly check for typos and writing clarity.\nLine 757 typo (JoinJittering)\nLine 923 typo (For “the” VAME)", "reviewer_4": "The authors have answered all my questions, except the following points:\nRegarding comment 1 point (3):\na) The hyperparameter table for VAME notes \"Max Epochs\" (the maximum number of training epochs) to be 500 or 20. As VAME is a self-supervised method using RNNs, the setting of 20 seems to be far to little to guarantee convergence. Can you clarify which of the max epochs criteria was used in the results presented in the manuscript? In case you used both settings, it would be helpfull to see the confusion matrix in 2)c) to be plotted for both settings in the supplementary material.\nb) the table provides a great overview of original and adapted hyperparameters. The textual version of the table that is available in the new manuscript in lines 923-930 is helpful, but does not carry all the details of the table. I would ask the authors to also add the table with original vs altered parameters for VAME and KeyPoint Moseq in the supplementary material of the manuscript and link it from the method section for interested readers.\nc) Please also clarify shortly how data was fed into VAME and KeyPoint Moseq, i.e. were all time series concatenated together, then clustered together, or have you done that in batches, etc. This is also important for the readers to understand how you used those methods.\nRegrading Minor comment 15:\no) As you now clarified that you have used 5-fold cross fold validation I would expect all reports on scores (f1, accuracy, ...) to be reported as mean +/- std in the whole manuscript as well as in all figures. Please make sure that this is implemented in the whole manuscript."}, "author_response": "Response:\nWe understand the feedback raised by the reviewer regarding the interpretation and terminology used in the forced swim test (FST) and tail suspension test (TST). We would like to clarify and carefully address each point raised by the reviewer. First, we would like to clarify that our previous manuscript did not intend to completely dismiss the relevance of these tests in the representation of ‘behavioral despair’. Rather, our earlier response intended to indicate that the FST and TST are widely and conventionally utilized in the field to reflect behavioral despair, acknowledging their limitations, particularly regarding the concept of learned helplessness. To avoid potential misinterpretation, we had decided to remove the term ‘learned helplessness’ from our manuscript.\nWe carefully considered the reviewer’s comments regarding the terminology used: ‘behavioral despair’ and ‘depressive-like’. While the word ‘behavioral despair’ is currently and widely used for the conceptual description of behavioral patterns observed in the FST and TST, we acknowledge that the interpretation of immobility behavior in the FST and TST as indicative of behavioral despair symptoms could be an oversimplification, as the immobility could reflect the adaptive strategy to an experimental environment1,2. We therefore adopted a more scientifically rigorous and objective approach by replacing the interpretation of immobility as behavioral despair with a strict description using immobility time (TST, FST) and sucrose preference (SPT) as below.\n Line 54: Traditional behavioral screening tests in rodents (e.g., the tail suspension test, forced swim test, and sucrose preference test) have been used to evaluate depressive-like symptoms that are affected by stress exposure and antidepressant treatment.\n Line 202: In male CUS mice, we observed a significant increase in immobility at Tail Suspension Test (TST) and Forced Swim Test (FST) (Fig. 3a, b, left) and decrease in sucrose preference (Fig. 3c, left), as well as a decrease in body weight (Fig. 3d). Although female CUS mice showed reduced sucrose preference (Fig. 3c, right) and a notable loss of body weight (Fig. 3d), we detected immobility increase only with the FST, not the TST (Fig. 3a, b, right).\n Line 270: In addition, syllable frequency was more closely related to stress-related behaviors in TST and FST (Supplementary Fig. 7a, b, d, and e) than anhedonia-like behavior in SPT (Supplementary Fig. 7c, f) […].\n Line 300: In the LPS-induced depression model, mice exhibited hallmark stress-induced behaviors, as evidenced by increased immobility in the FST and reduced sucrose preference (Supplementary Fig. 7a). In the corticosterone paradigm, stress-related behaviors in TST and SPT emerged exclusively after 3 weeks of administration, […].\n Line 694: The forced swimming test (FST) was conducted to assess the behavioral effect of stress exposure, at least four hours after the TST to minimize any potential effect of physical discomfort caused by tail suspension. […]"}, {"round": 3, "reviewer_comments": {"reviewer_1": "The authors have clearly thought about and taken on board the feedback I have provided and the scientific objectivity of their manuscript has improved subsanially as a resul of this. However, they need to do this consistently throughout the manuscript. The abstract contains the sentence:\n\"In offering a comprehensive analysis of exploratory behaviors, CLOSER proposes the standardization of depressive disorder diagnosis in mice, thereby unifying preclinical and clinical assays for psychiatric drug discovery.\"\nMice do not suffer from depressive disorder, and the authors need to apply the same level of objective, scientific writing throughout their manuscript.", "reviewer_2": "Thank you to the authors for their detailed reply. They have completely addressed my criticism about uniform scaling. However, the analysis of the latent space invariance only partially addresses my main critique: the authors are still not directly addressing whether the latent space is or is not invariant to variables like weight, sex, and overall size. This is the central claim of their paper. The authors must directly address this critique in a simple, straightforward way or the paper is not suitable for publication in Nature Communications.\nFirst, the authors provide a heatmap of cluster proportions for mice in week 0 of the experiment (male only) and week 4 (male and female). This is great and helps the reader get a sense of how well groups of mice map onto the cluster proportions. However, the plot is not grouped by weight, so we are only left to guess what the weights of these mice are. This is important since weight can vary significantly among 12-week-old males and females. Additionally, 8-week-old males can be similar in weight to 12-week-old females. Furthermore, even though weight is not directly shown here, there are cases where clusters appear to not be uniformly distributed across groups (e.g. 14, 15, 16, 29, 30, and 50). Would these trends become more pronounced if mice were explicitly grouped by weight? The reader does not have enough information to make this determination.\nSecond, the quantification of the heatmap appears to have very little dynamic range. All of the chi square and entropy values only vary a small amount and all p-values exceed .9 . This would not be a problem if the heatmap matched this result, but there is apparent clustering by group for a non-trivial number of clusters. This could be due to the authors running their chi square test by individual and not by group or weight class or sex. Running the chi square test against individuals here to address sensitivity to weight does not directly address this question since multiple individuals can be the same size. As the authors are aware, there is significant weight variability in both 8- and 12-week old mice (https://www.jax.org/jax-mice-and-services/strain-data-sheet-pages/body-weight-chart-000664). This needs to be fixed to be interpretable. Additionally, these numbers are not shown for other methods that the authors claim are not invariant, e.g. VAME, raw trajectories, or PCA. Again, the reader does not have enough information to validate what the authors claim.\nThird, the authors present decoding analysis, which would directly show if the clustering based on the latent space locations has information about mouse sex, weight, or age. Here, again the authors are on the right track, but then present incomplete information with no comparison to other methods. A large confusion matrix is shown, and it is apparent that the decoder can extract information about mouse size (week 0 vs week 4) and sex. Again, there is large size variation in 8-week-old and 12-week-old mice so this analysis should be noisy even if CLOSER is highly sensitive to size. The amount of information extracted by the decoder is not given, and CLOSER is not compared to other methods. So again, the authors have done more analysis, but the reader still cannot validate the author’s claim.\nI should note that an alternative, possibly simpler decoding analysis would have been to fit a regression model to latent space coordinates to predict weight. I do not know how the authors implemented their knn classifier (see minor comment below).\nFourth, the supplementary figures are still not available for the AVATAR preprint (as of June 26th, 2025), so the data presented by the authors is the only data readers will have in trying to compare AVATAR to commonly used tools like SLEAP, DLC, and Lightning Pose. So please ensure that details about performance characterization are self-contained and complete, e.g. I assume the RMSE given in the methods section is 2D and not 3D? Since all analysis is done in 3D it is important that the reader has a sense for how accurate the method is in 3D relative to a tool like DANNCE. Additionally, please include complete details in how AVATARNET goes from body part segmentation to 3D coordinates. Moreover, concerningly, the figure in the response included what appears to be inappropriate language in referring to the hindquarters. I strongly encourage the authors to double check all language used in the manuscript for appropriateness and accuracy.\nIn summary, to be clear, I am not asking the authors to prove that their method is perfectly invariant; the reader simply needs a concrete demonstration that it is more invariant than alternative methods since this is a central claim of the manuscript.\nMinor notes:\n1. The authors use an entropy threshold of .5999 in their heatmap analysis without stating why this is a reasonable threshold. Did they conduct simulations?\n1a. Also is entropy in nats or bits?\n2. Very little detail is given about the knn decoding analysis. Please present enough information for others to understand what was done.\n3. Related, For supplementary Fig. 3 there is no detail given on how mice were determined to be “different in variables”. The reader must have enough information to exactly replicate what the authors did here. This analysis might show invariance, but the reader does not have enough information to fill in the gaps.\n3a. Again, for Fig. S3, show the result for alternative methods and quantify it directly."}, "author_response": "Response:\nWe thank the reviewer for the constructive feedback and insights on the terminology for psychiatric disease and its model system. We have now revised the Abstract (line 38) to precisely describe the psychiatric state of the depression model mouse as follows:\n“[…] In offering a comprehensive analysis of exploratory behaviors, CLOSER proposes the standardization of depressive phenotype assessment in mouse models, thereby bridging preclinical and clinical diagnostics for psychiatric drug discovery.”"}, {"round": 4, "reviewer_comments": {"reviewer_2": "First, let me thank the authors for their hard work in addressing my critiques from the previous round. They have addressed all of my major concerns, and this manuscript is now suitable for publication in Nature Communications. This article will be well received by the community and will be a valuable resource moving forward.\nTwo minor things:\n1. Supp. Fig 4 is new, but it was not highlighted in the combined files for reviewers\n2. I would encourage the authors to include KP-MoSeq in Supp. Fig 4 since this is a useful comparison for the community.\nAdditionally, I still cannot run the code to completion. Specific files needed to run the demo are still missing in the zip file I was given (e.g., \"CTRL_1wk_#01.mp4.txt.csv_fix.csv\"). Additionally, I had to modify the demo notebook to install all required libraries since a cell linked to a private github repository. I strongly encourage the authors to carefully scrutinize their code and dataset to ensure that it can be run from someone outside of their group prior to publishing."}, "author_response": "Response:\nWe thank the reviewer for the constructive feedback across rounds. The comments prompted additional validation of the CLOSER framework and improved clarity throughout the manuscript."}]

Decision letter
Peer Review File
AI-driven decoding of naturalistic behaviors enables tailored
detection of depressive-like behavior in mice
Corresponding Author: Professor Won Do Heo
This file contains all reviewer reports in order by version, followed by all author rebuttals in order by version.