Identifying the factors governing internal state switches during nonstationary sensory decision-making

Domain: physics
Published: 2025-11-27
Editorial outcome: major_revision

Abstract
Traditional models of perceptual decision-making fail to capture dynamic strategy switching in non-stationary environments, and the factors governing these switches remain unknown. To address this gap, we developed an advanced internal state model with input-driven transitions and observations. Our approach employs a hidden Markov model (HMM) coupled with two sets of per-state generalized linear models (GLMs): a Bernoulli GLM for state- and stimulus-dependent choices, and a multinomial GLM for input-dependent transitions between states. We applied our model to a decision-making task in a non-stationary environment, analyzing hundreds of thousands of trials from a cohort of mice, and found that their behavior can be accurately described by a four-state model. This model identified two engaged states with low biases relative to the stimulus and two disengaged states with pronounced biases relative to the stimulus. Our analyses revealed that mice preferentially used left-bias strategies during left-bias stimulus blocks, and right-bias strategies during right-bias stimulus blocks, achieving high performance even in disengaged states by biasing choices toward the side with greater prior probability. Our model showed that past choices and past stimuli predicted transitions between left- and right-bias states, while past rewards predicted transitions between engaged and disengaged states. In particular, greater past reward predicted transition to disengaged states, suggesting that disengagement may be associated with satiety. Our approach uncovers links between animal behavior, input regressors, and state transitions, highlighting the complexity of adaptive strategies. This provides a foundation for future research in dynamic decision-making models.

Introduction
Understanding the brain’s orchestration of behavior1–9 and decision-making strategies10–14 is fundamental to deciphering the complex computations driving adaptive behavior in dynamic environments. Extensive research11,14–23 has characterized the computational mechanisms governing sensory decision-making across various species and tasks. Bayesian learning and probabilistic modeling techniques have also found practical applications in the study of mammalian and murine behavior24–30. Nevertheless, all of these works either assume that animals make decisions using a single, consistent strategy or fail to account for the non-stationary features of their decision-making environments.

A powerful approach for identifying these strategies and their transitions from behavioral time series data involves a hidden Markov model (HMM) with states corresponding to distinct decision-making rules31–38. Each of these rules is parameterized by a generalized linear model (GLM), whose weights describe how the animal weighs sensory inputs and other task covariates (such as recent choices and rewards) towards a decision in that state. The resulting modeling framework, known as GLM-HMM, has been used to reveal state-dependent decision-making during courtship behavior in flies31 and to show that mice rely on distinct neural circuits in different states34. Recent work has also revealed lawful relationships between states identified with GLM-HMM and an animal’s level of arousal and uninstructed movements38.

However, these studies have not thoroughly explored the correlation between animal behaviors and input regressors or state transitions. One notable limitation is their challenge in identifying the factors driving state transitions, especially in non-stationary tasks where stimulus statistics change. Identifying the factors causing an animal to switch from an engaged state to a disengaged or biased state, and vice versa, remains crucial. Standard HMMs, which rely on fixed transition probabilities, fail to capture these dynamics because they assume static probabilities over time. Furthermore, while previous work has modeled time-varying behavior using continuous updates to decision weights, such as PsyTrack23, these approaches primarily focus on learning dynamics during training, capturing gradual changes in decision strategies. In contrast, our study examines behavior after learning has stabilized, applying a GLM-HMM framework to capture behavioral strategy shifts in the absence of learning, providing a complementary perspective on how trained animals dynamically engage with or disengage from the task.

Our research addresses this gap by revealing significant fluctuations in animal behavior across session phases, which allows for the identification of distinct observation and transition strategies in dynamic contexts. Using a multinomial GLM instead of a fixed transition matrix, our approach shows that animal strategies are influenced by internal state changes. This study explores flexible decision-making in a non-stationary environment using a model with two GLMs, incorporating various independent and distinct transition and observation inputs. This model implements flexibility in decision policy by recognizing that the tuning covariates differ across various behavioral contexts and phases.

We applied this GLM-HMM to a mouse decision-making task with time-varying statistics. In this task, the stimulus probability alternated randomly between left-biased and right-biased blocks, with an average block length of 50 trials39. We fit models with different numbers of latent states and found that a four-state model provided an accurate yet interpretable description of decision-making behavior. We found that mice employed left-bias strategies during left-bias stimulus blocks and right-bias strategies during right-bias stimulus blocks. They achieved good performance even in disengaged states by favoring the side with greater prior probability. Notably, past choices and stimuli predicted bias state transitions, while past rewards predicted engagement/disengagement transitions, suggesting a link between disengagement and satiety. This study leverages extensive data from tens of mice and hundreds of thousands of trials, ensuring both generalizability and reliability of our findings. Our results contribute significantly to the sensory decision-making field, especially in environments where stable stimulus-action associations are absent. This highlights the need for models that account for rapid internal state shifts and provide insights into the complexity of behavioral dynamics.

Results
An internal state model of sensory-decision-making
In this study, we employed the GLM-HMM framework31–34,40,41, referred to as an input-output HMM, to analyze behavioral data obtained from mice participating in a decision-making experiment. Our model of state-dependent decision-making incorporates two distinct generalized linear models: an observation GLM (GLM-O), which provides the state-conditional probability of the animal’s choice on each trial, and a transition GLM (GLM-T), which provides a vector of transition probabilities for the state transitions after each trial. Notably, we specify independent regressor sets for the transition and observation components of the GLM-HMM.

The observation GLMs seek to describe how various task variables (e.g., sensory input, past rewards) affect choice in each state, and thus correspond to distinct decision-making strategies. The probability of a rightward choice in state k, based on observation covariates at trial t, is given by:

p(z_t=right|x_t^ob, z_t=k) = 1 + exp(-(w_k^ob)^T x_t^ob) (1)

where x_t^ob denotes the observation GLM covariates, w_k^ob is the observation GLM weights for state k, and y_t reflects the animal’s decision.

The transition GLM seeks to describe how task covariates (e.g., elapsed time in session, accumulated reward, previous choices) affect transitions between states. For state transitions, the multinomial GLM predicts the probability of moving to state k at trial t, based on transition covariates:

p(z_t=k|x_t^tr) = P_j^K exp((w_k^tr)^T x_t^tr) / P_j^K exp((w_j^tr)^T x_t^tr) (2)

highlighting the role of K total states and the transition weights w_k^tr in modeling transitions based on the trial’s context. Here, x_t^tr denotes the transition GLM regressors. Because we expected the factors governing choices to be different from those governing state transitions, we used different sets of regressors for the observation and transition GLMs (see Methods).

To limit model complexity while still capturing covariate-driven transitions, we used one transition weight vector per destination state (w_j^tr), along with a baseline term for each state pair (B_ij). Therefore, the probability of transitioning from state i to state j at time t is modeled as:

p(z_t=j|z_{t-1}=i, x_t^tr) / exp(B_ij + (w_j^tr)^T x_t^tr) (3)

This structure allows transitions to depend on both the previous state through B_ij, a learned baseline transition logit from state i to j- and the upcoming state, enabling covariate-driven transitions without requiring separate filters for each origin state. We verified that transitions into a given state did not differ significantly depending on the preceding state, supporting this modeling choice (Supplementary Fig. S11).

Mice employ different strategies in a non-stationary perceptual task
We analyzed the mice decision-making dataset from the International Brain Lab (IBL)39. In this task, mice detect the location of a Gabor patch on the screen and subsequently turn the wheel to the right or left to indicate the stimulus location42. Figure 1a shows an illustrative diagram of the IBL sensory decision-making task. Each experimental trial encompasses the presentation of a sinusoidal grating, characterized by gradient values ranging from 0% to 100%. This grating stimulus is selectively presented on either the left or right periphery of the visual display. Subsequently, mice are mandated to discriminate the spatial location of the grating and communicate their decision via rotational manipulation of a wheel, resulting in a left or right turn, which corresponds to the perceived location of the grating stimulus. Successful execution of this task merits a water reward. For further insights into this experiment, please refer to the detailed description of the IBL task39,42.

In this IBL task, after training mice in the foundational purely sensory task, they were introduced to an advanced paradigm where optimal performance necessitated the fusion of sensory perception with recent experience. Specifically, block-wise biases were incorporated into the probability distribution of stimulus locations, thereby influencing the more probable correct choice. Each session commenced with an unbiased trials block, offering an equal 50:50 probability of left versus right stimulus locations. The length of the unbiased block was 90 trials for all sessions of the task (Fig. 1b). Subsequent trial blocks alternated variably and exhibited biases toward the right and left. The probability distribution skewed at a 20:80 and an 80:20 ratio for the right and left sides, respectively. The length of these biased blocks ranged from 20 to 100 trials. The transition between these biased blocks was not overtly indicated, necessitating the mice to extrapolate a prior estimation for stimulus location based on recent task statistics. This intricate task compels the mice to assimilate information across multiple trials, strategically employing their prior knowledge to inform their perceptual decisions.

Figure 1c illustrates the distribution of biased data blocks within the IBL experiment, specifically focusing on blocks that extend beyond a defined length n. These extended biased blocks varied in duration, spanning from 20 to 100 trials, with an average length of approximately 58 trials. It’s important to note that a typical experimental session consists of several such data blocks.

Within this modeling framework, two distinct sets of inputs were incorporated, encompassing transition and observation inputs as presented in Fig. 1d. In this context, the observed output corresponded to the animal’s decision, manifesting as a binary value indicative of the direction in which the wheel was turned-either right or left. A multitude of covariates were additionally integrated into the model and will be elucidated herein. Specifically, the stimulus parameter was defined as the contrast level of a sinusoidal grating, spanning luminance variations between 0% and 100%. For normalization, this stimulus was divided by the standard deviation of trials across sessions. Conversely, the stimulus side parameter characterized the mouse’s behavior upon receiving a reward: continued execution of the same choice after reward receipt or a change in behavior in its absence. This binary stimulus side was represented as -1, +1. Furthermore, past choice was established as a binary variable, taking values of -1 or 1, denoting left or right prior choices by the mouse, respectively. The previous reward (pr) value was defined as -1 or 1, corresponding to incorrect or correct decisions, respectively. Moreover, three bases were introduced as linearly independent vectors for the initial 100 trials of each session, capturing the animal’s warm-up effect within the model. In the course of this study, the term filtered covariate was employed to reference a covariate subjected to exponential filtering, facilitating the consideration of a temporally filtered variant of the regressor.

Taking into account this description, we included the stimulus, past choice, stimulus side, and bias as regressors for the observation model. Additionally, the covariates influencing GLM transition include a filtered version of the previous choice, previous stimulus, previous reward, and three basis coefficients to capture the warm-up effect. Incorporating an exponential filter for the transition model enables the integration of a temporally filtered regressor. This innovative approach allows for a detailed evaluation of the temporal impact of model regressors on state transitions, enhancing our understanding of the dynamic nature of these transitions.

The GLM-HMM framework used for this analysis is presented in Fig. 1e, which, as mentioned, includes an HMM with a Bernoulli GLM for observations and a multinomial GLM for transitions. We used the IBL data, consisting of extensive data evaluating 123 mice and analyzing all sessions from 37 of them, encompassing hundreds of thousand of trials, to ensure generalizability. From this dataset, we selected 37 mice to fit the GLM-HMM framework with a different number of states. Our animal selection criteria involved including those with a minimum of 30 sessions and incorporating sessions characterized by a low number of error trials, where the animal either did not make a choice or timed out.

In fitting the model, we employed the Maximum A Posteriori (MAP) approach, utilizing the Expectation Maximization (EM) algorithm. The model parameters, denoted as Θ={w_tr, w_ob, π}, include the transition weights w_tr, observation weights w_ob, and the initial state distribution π. Subsequently, we evaluate the model’s log-likelihood based on this approach (see Methods).

A 4-state model captures decision-making behavior
To identify the optimal model, we fit the GLM-HMM with varying numbers of states and computed the cross-validated log-likelihood on a set of held-out test sessions. After evaluating the log-likelihood values, the GLM-HMM with four states demonstrated notably superior performance compared to the 1-state model. Therefore, we focus on the 4-state model because it comes close to the maximum but is more parsimonious and interpretable than the 5-state model. These four distinct states correspond to four distinct decision-making strategies. The findings suggest that mice consistently employed all states for multiple consecutive trials, with each session seeing the utilization of various states. It was observed that in the GLM-HMM fit of both unbiased and biased data, covariates such as the stimulus (Δ contrast), past choice, stimulus side, and others played crucial roles in predicting the animal’s choice.

To conduct this comparison, following the fitting of the model with different numbers of states and computed the log-likelihood of the test data. The EM algorithm was employed for the fitting procedure on the training data sessions. Subsequently, in the testing step, the likelihood of the remaining data was calculated based on the model parameters obtained from the training procedure. By summing over all states, the log-likelihood of the test data is stated as:

L = log Σ_{k=1}^K α_{T,k} (4)

in which α_{T,k} is the posterior probability of the mice's decisions from trial 1 to T and was computed solely on the held-out sessions. We can express α_{T,k} as:

α_{T,k} = p(Y, z_k | fX_ob, X_trg) (5)

where X_ob = {x_ob_1, ..., x_ob_T} is the observation input vectors, and X_tr = {x_tr_1, ..., x_tr_T} presents the transition input vectors. The animal choices for the specified trials are denoted as Y = {y_1, ..., y_T}.

First, we performed the global fit in which the model was applied globally to the entire IBL dataset, and the normalized log-likelihood (NLL) for this data was calculated. The defined NLL, measured in bits per trial (bpt), provides a more intuitive understanding of the model’s performance and facilitates meaningful comparisons across various models and datasets. This can be expressed as:

L_bpt = (L_t - L_0) / (T_t log(2)) (6)

In this equation, L_0 represents the log-likelihood of the baseline model, and T_t is the number of trials in the test set. The equation has been divided by T_t log(2) to present the value per trial. The difference between L_t and L_0 represents the enhancement in log-likelihood due to the performance of the GLM-HMM.

Analyzing the Test NLL plot (Fig. 2a), it is evident that the four-state model demonstrates better performance and interpretation, particularly when considering the structured nature of both biased and unbiased data. So, derived GLM weights within the context of the four-state model unveil distinct patterns, which are shown in Fig. 2.

In Fig. 2a, for comparative analysis of different models, we computed the log-likelihood for all animal data. The computation strategy involved 5-fold cross-validation with reserved sessions. In this model, when evaluated on the held-out data, the four-state GLM-HMM performed much better than the GLM itself (1-state model). Additionally, the GLM-HMM successfully captured the temporal pattern of inhibition influencing the animal’s decision-making process. Also, a comparison was drawn in terms of the Test Normalized Log Likelihood (NLL) between the model solely incorporating the Bernoulli GLM and the model encompassing both the GLM-O (Observation) and GLM-T (Transition). This is presented in Fig. 2a. We can see similar effects and improvement for the 5-state model as shown in Supplementary Fig. S1a. Our analysis revealed an enhanced performance of NLL in the presence of the transition GLM (GLM-T) within our model, as opposed to the scenario wherein the transition GLM was absent. Although the difference between the two models with and without GLM-T (Fig. 2a) is not very large, this is because the results are shown in log-likelihood per trial (in bits). Even a very small improvement in log-likelihood per trial can result in a much better model fit (refer to Eq. 32 in the Methods, but in this case comparing models with and without GLM-T instead). For example, for the 4-state model, the model with GLM-T is ~0.015 bits per trial higher than the model without GLM-T. This means that for a dataset with 400 trials, the data would be about 2^6 ≈ 64 times more probable under the GLM-T than without it. For a dataset with 4000 trials, the data would be about 2^60 ≈ 1.15 × 10^18 times more probable under the GLM-T model. This exponential scaling highlights how even small per-trial improvements translate into dramatically better fits for larger datasets.

Although the NLL plot shows a slight increase for the 5-state model, our focus here remains on the 4-state model for the sake of simplicity and interpretability. This choice is motivated by the fact that, in the 5-state model, four states closely resemble those in the 4-state model and the new state, named Past-choice, emphasizes the role of past choice. However, our emphasis in this work revolves around the stimulus and bias roles, which are crucial for analyzing biased block data. A comprehensive description of the 5-state model is available in Supplementary Figs. S1 and S2, providing insights into the observation and transition weights, and analysis of all five states.

In Fig. 2b, state 1, Engaged-L, exhibits a substantial weight attributed to the Δ contrast (stimulus) and a moderate weight associated with the left bias. State 2, designated as the Engaged-R state, features a significant weight related to the stimulus and a moderate right-bias weight. Conversely, states 3 and 4, denoted as Biased-L and Biased-R, respectively, exhibit reduced stimulus weights. Nevertheless, bias weights generate a strong left bias for state 3 and a right bias for state 4. Additionally, in these states, a nominal weight is assigned to the previous choice factor.

These state-specific weights influence the shape of the psychometric curves, a fundamental tool in psychophysics43,44 and decision-making modeling. Typically characterized by a sigmoid-shaped function, the psychometric curve is intricately linked to a linear representation of the stimulus (Δ contrast) and augmented by a bias term. This mathematical framework is widely adopted to capture the relationship between stimuli and an individual’s responses. Here, the psychometric curve graphically represents the choice probability (right side) relative to stimulus contrast. State-specific psychometric curves were meticulously generated within the framework of a four-state GLM-HMM presented in Fig. 2c. The resulting curves serve as intricate depictions of the behavioral responses observed within each state. The psychometric curves for Biased-L and Biased-R states exhibit shallower inclines, indicative of notable leftward and rightward biases, respectively.

In Fig. 2d, the green triangles correspond to the experimental choice data of the mouse (alongside 95% confidence intervals). Also, to derive the solid black line, a temporally sequenced dataset was generated to match the trial count of the example mouse. This process involved using the meticulously fitted parameters of the GLM-HMM specific to this animal and the actual sequence of stimuli presented during its trials. For each trial iteration, the probability of making a rightward choice (p(R)) was calculated for each of the nine potential stimuli, regardless of the actual stimulus presented. This calculation entailed averaging the per-state psychometric curves, as illustrated in Fig. 2d while adjusting their weights according to the pertinent row in the transition matrix, as this adjustment was contingent on the latent state sampled in the preceding trial.

Modeling the dynamics of state transitions in animal behavior
Figure 3a represents both the stimulus and the animal’s choice on the same trial, complemented by the transition regressors of the model. These transition regressors include filtered stimulus side, filtered choice, and filtered reward, providing a comprehensive illustration of the multinomial GLM inputs in the model.

The multinomial GLM weights for transition between states are shown in Fig. 3b. For states 1 and 3, characterized by a left-biased component, negative weights are assigned to the filtered choice and filtered stimulus side. This suggests that these factors contribute to the transition towards left-biased states (Engaged-L or Biased-L). Conversely, for the right-biased states, including states 2 and 4, positive weights are assigned to the filtered choice and filtered stimulus side, indicating a tendency towards right-oriented choices. Additionally, in disengaged states (states 3 and 4), the positive weight of the filtered reward signifies its role in decision-making, as the animal is less attentive to the stimulus in these states. This pattern is reversed in engaged states (states 1 and 2), where the stimulus is pivotal, resulting in a negative weight for the filtered reward. Therefore, past rewards were found to predict transition to disengaged states, suggesting a connection to satiety. Also, transitions between left- and right-bias states were influenced by past choices and stimuli. Notably, the filtered covariate here refers to transition regressors subjected to exponential filtering, facilitating the integration of temporally filtered versions of these regressors in the analysis.

On the other hand, Supplementary Figs. S2 and S3 present an analysis of transition weights and transition patterns between different states, as well as the dwell time distribution for the model with 5 states. The calculation of dwell time in each state involves utilizing the diagonal elements of the inferred transition matrix of the 5-state model. This allows for a nuanced examination of the temporal dynamics and the relationships between states, contributing valuable insights into the overall understanding of the 5-state model.

Additionally, we introduced a set of three temporal basis vectors to capture the time dependence of state transitions at the beginning of each session. The temporal modulation of the bases’ effect is illustrated in Fig. 3c. These bases were strategically integrated to capture the gradual adaptation of the animal during the initial 100 trials of each session, effectively accounting for the warm-up effect within the model. Furthermore, as depicted in Supplementary Fig. S4, we present the impact of transition weights and the associated effects of these bases, showing the transition weight corresponding to each basis in a specific state for both the 4-state and 5-state models. This representation provides insights into the influence and weight of each basis in the warm-up effect.

Figure 3e presents the inferred transition matrix pertaining to the four-state GLM-HMM, designed to accommodate the full IBL dataset. Evident within this matrix are notable magnitudes along the diagonal, serving as clear indications of a heightened probability for the system to sustain its presence within the same state during transitions. This prominence underscores the significant propensity for the model to exhibit persistence and stability within individual states. Figure 3d displays the transition bias for pooled IBL data, derived from the intercept terms of the GLM-T model (prior to input modulation). These values represent the model’s default expectations about the likelihood of remaining in the same state or transitioning to another, independent of trial history or covariates. Visualizing these biases provides insights into the intrinsic stability of each behavioral state-for example, higher diagonal values indicate stronger self-transitions and more persistent states.

To compare models with different filtering parameters and choose the best one, we compared the normalized test Log-Likelihood (test LL) across models with various numbers of states, varying the values of τ. Here, τ represents the time constant of the exponential filter applied to the transition covariates and is associated with the duration of the history considered for these covariates. Figure 3f illustrates the normalized test LL for different values of τ while adjusting the number of states. In Fig. 3g, a similar comparison is shown, but for models with different numbers of states. It is evident that the model with the specific value of τ = 4 yields the best test log-likelihood results, and we have chosen this value for filtering the transition covariates of the model.

Exploring the temporal patterns within the model
Our model, harnessing the power of the fitted GLM-HMM, captures the temporal dynamics and diverse decision-making strategies of mice, enhancing our understanding of their cognitive processes. The findings regarding the dynamics of the model over time are presented in Figs. 4 and 5. This model allowed us to calculate the posterior probability of the mouse’s hidden state throughout all the trials. These state trajectories represent our best estimates of the mouse’s internal state on each trial, considering the entire sequence of observed inputs and choices during a session.

The plots showcase the posterior probabilities and correct/incorrect animal choices associated with their corresponding sessions in four distinct states, as presented in Fig. 4a. These visualizations incorporate a background color scheme that serves to distinguish biased blocks: the color pink corresponds to right-biased blocks, while the blue shade denotes left-biased blocks. This color-coded representation enhances the comprehensibility of the data, facilitating the identification of patterns within the biased segments of the experiment. This figure illustrates that states exhibiting right bias, namely Engaged-R and Biased-R, demonstrated a greater likelihood within blocks characterized by a right-biased orientation (designated by a light pink background color). Left-oriented blocks with a blue background hue exhibited a similar observation, where states influenced by left bias exhibited higher probabilities. Also, as we can see Fig. 4a, what emerged is a clear pattern: in most trials, one state stood out as significantly more probable than the others, signifying a strong level of confidence in our understanding of the mouse’s internal state based on the observed data.

We made a histogram to elucidate the distribution of the first transition into states aligning with related bias within each data block (Fig. 4b). This analytical approach is geared towards capturing the number of trials spanning from the commencement of a biased data block until the occurrence of the first transition to the corresponding biased states. For instance, in the context of a right-biased data block, this entails quantifying the number of trials required for transitions to the states characterized by a right bias, namely Engaged-R, and Biased-R. A similar assessment is made for left-biased data blocks and their respective states, with a high value on the left bias weight. The analysis underscores that the median value of this histogram stands at 9 trials, while the maximum value observed is 6 trials. This implies that, on average, it takes approximately 9 trials for mice to transition from the start of a biased data block to a state matched with the prevailing bias.

In Fig. 4c, the fractional occupancy of the four distinct states across the entirety of trials for right-biased and left-biased data blocks was analyzed. Each trial was assigned to the state that exhibited the highest likelihood, and the proportion of trials designated to each respective state was computed.

As evident from the analysis (derived from data of all 37 mice) for each bias-related plot, the corresponding states were observed to encompass a substantial portion of the entire trial set, indicating a significant representation of the mouse’s behavioral responses within the experimental framework. Remarkably, the analysis uncovered that the mice spend ~72% of their time in related biased states (e.g., Engaged-R and Biased-R in right-biased blocks), with the engaged state having a slightly higher occurrence chance. In stark contrast, the mice allocated a relatively smaller fraction of their trials, ~28%, to the other unrelated states (e.g., Engaged-R and Biased-R in left-biased blocks).

In the plots of Fig. 5a, we depict the multiplication of transition weights and transition inputs, providing a visual representation of the temporal patterns exhibited by the weighted transition regressors. The specific data for these plots corresponds to the same three sessions of a single mouse as in Fig. 4a. These visualizations offer insights into the nuanced dynamics of the relationship between transition weights and inputs across the given sessions.

Also, a histogram was created to show the frequency distribution of inferred state changes per session across all 58 sessions of data for this mouse in Fig. 5b. In this plot, we considered sessions with lengths more than T trials and analyzed the first T trials of each session. Here, T represents the average duration of all sessions conducted for the respective animal, which was 821 trials. Notably, in about 18% of all sessions, the mouse exhibited fewer than 10 state changes, leading to an average state duration of 154 trials (calculated by dividing 821 trials by 5-state changes). This observation highlights a consistent tendency for this mouse to remain in specific states for extended periods, particularly in high-performance states. These prolonged periods of stability were often interrupted by state changes, primarily occurring when the mouse either adopted a new strategy or shifted its attention to different covariate effects.

On the other hand, in the majority of sessions, approximately 63%, the mouse underwent more dynamic behavior with more than 40 state changes, averaging around 15 trials per state (derived by dividing 821 trials by 55 state changes). This frequent state-switching behavior reveals a decision-making process where the mouse continuously adapted its strategies and attention throughout the session, while still maintaining an average of 15 trials per switch. The notable variation within and between sessions highlights that these different rates of state changes might be due to varying strategies, environmental factors, or animal behaviors on separated days or in different sessions.

Furthermore, our analysis delved into the dynamics of state transitions by utilizing the diagonal components within each transition matrix. These components allowed us to calculate the expected duration of residence, referred to as dwell time, for each animal within distinct states, as illustrated in Fig. 5c. The median duration of residence in the Engaged-L and Engaged-R states was approximately 15 and 10 trials, respectively, while for the Biased-L and Biased-R states, it was 10 and 12 trials. This observation indicates that the animal spent considerable periods in all four states, shedding light on the persistence of specific behavioral states in different strategies.

Analyzing each individual animal yields consistent findings
To gauge the universality of our findings, we applied the GLM-HMM to the choice data obtained from all animals in the IBL dataset separately (37 individual fits for all mice). As a result, two sets of GLM weights (observation and transition weights) for these animals are presented in Fig. 6a, b. A notable level of substantial agreement was distinctly apparent upon analyzing the fits of the four-state GLM-HMM in the study. This consensus was particularly pronounced, as a significant majority of the mice showcased discernible states, identified as Engaged-L, Engaged-R, Biased-L, and Biased-R (Fig. 6a, b). This alignment in the identification of states underscores the robustness and consistency of the applied GLM-HMM framework in capturing these behavioral patterns across the population of interest for both observation (Bernoulli GLM) and transition weights (multinomial GLM).

The transition and observation weights for the 5-state model, resulting from individual fits for all animals, are thoroughly presented in Supplementary Fig. S5. As we can see, these weights exhibit mostly similar patterns and in the near ranges across the various animals. Notably, this consistency extends to the fifth state, Past-choice, where the observed patterns remain consistent in both observation and transition analyses. This shows valuable insights into the universality of the observed patterns within the studied population.

Figure 6c presents an analysis of the test log-likelihood variation considering different numbers of states for each mouse in the studied population. Each line on the graph represents an individual mouse’s data, providing an overview of log-likelihood changes across different state configurations. The solid red line shows the average across all animals, serving as a reference point for comparison. The findings in Fig. 6c are consistent with trends observed across the entire cohort, indicating a strong pattern. Notably, the four-state GLM-HMM consistently outperformed the single-state GLM during cross-validation. This trend was consistent across all 37 mice in our study, reinforcing the reliability and validity of our multi-state GLM-HMM framework as a powerful tool for understanding decision-making behavior in mice across diverse individuals. Similarly, Fig. S6 presents the GLM weights of the 5-state model applied individually to all IBL animals, showing both observation and transition weights. The global fit is shown as a solid black line, with an example mouse fit overlaid as a dashed black line.

Model performance analysis and comparison
Figure 7 presents a detailed analysis of model performance and a comparison of models with and without GLM-T. Figure 7a shows the difference in T_90 (90th percentile response time; disengaged - engaged) across mice, demonstrating that the model with GLM-T (purple) achieves higher separation of engaged and disengaged states in terms of response times for all mice, indicating better performance in this aspect. Figure 7b illustrates the average state probabilities across trials, with sessions interpolated to 100 points and then averaged across all sessions and animals. It highlights an initial warm-up effect in which mice start in disengaged states (orange) before transitioning to engaged states (green). Also, this plot suggests a link between disengagement and satiety, as past rewards predict transition to disengaged states, leading to increased disengagement state occupancy toward the end of the session when mice have accumulated more rewards. Figure 7c further supports this relationship, showing a positive correlation between filtered reward and the probability of disengagement states.

To assess the effectiveness of GLM-T in capturing trial-dependent dynamics, we compared model performance using synthetic data. Figure 7d presents the test log-likelihood comparison for models fitted to data sampled from a generative model without GLM-T (left) and a model with GLM-T (right). When trial-dependent transitions are present in the data, the GLM-T model performs better, improving by 0.009 bits per trial. Conversely, when transitions are independent of trials, the non-GLM-T performs almost similarly to the model with GLM-T. This advantage becomes particularly significant in large datasets, making the GLM-T model more suitable for capturing structured behavioral dynamics. This improvement of 0.009 bits per trial when the data has trial-dependent transitions mean that for a dataset with 5000 trials, the data would be ~3.52 × 10^13 times more probable under the GLM-T model than under the non-GLM-T model. Figure 7e compares the inferred states from both models, with most data points lying above the y=x reference line (red dashed), indicating that the GLM-T model assigns higher probabilities to states compared to the non-GLM-T model, demonstrating its confidence in detecting states and its improved performance.

These results collectively demonstrate that incorporating GLM-T leads to better performance and more structured and behaviorally relevant state inferences, making it a more effective framework for modeling decision-making dynamics. Fig. S7–S10 further support this by showing that GLM-T improves the capture of structured state transitions, enhances the stability and clarity of posterior state probabilities, and enables dynamic, trial-dependent transition probabilities that align with block structures, whereas models without GLM-T fail to capture these behavioral dynamics effectively. Furthermore, GLM-O weights were similar across models with and without the GLM-T component (Fig. S7a). This shows that the GLM-T regulates transitions between behavioral states without substantially modifying the states themselves, as described by the GLM-O. Fig. S11 shows that transition probabilities into a given state are nearly identical across different preceding states, for IBL data. This confirm that our key conclusions are not sensitive to this aspect of the model.

Methods
Hidden Markov Model
An HMM is a statistical model for time series data that is governed by hidden or latent factors that cannot be directly observed. The events that we observe are called observations, and the underlying, unobservable factors driving them are referred to as latent states29,30,77. Therefore, HMM consists of two stochastic components: one governing the latent states, and another governing the observations.

The latent process component satisfies the Markovian property. A transition probability matrix A ∈ R^{K×K}, where, at trial t, the element corresponding to state j and state k presents the transition probability between those two states and can be written as:

P(z_{t+1} = k | z_t = j) (7)

and an observable state-dependent component {y_t}_{t=1}^T, for which we have:

P(y_t | y_1, y_2, ..., y_{t-1}, z_t) = P(y_t | z_t) (8)

where T is the number of considered trials. Generally, in HMM, the probability distribution of the observed symbols is based on the underlying, unobserved states of the system, following the principles of a Markov chain. In many cases, observations can be grouped into different classes, which can provide more insightful information than the individual observations themselves. In such situations, it becomes advantageous to model these observations using both the observable and unobservable aspects of HMM.

Bernoulli GLM
The Bernoulli Generalized Linear Model (GLM) is a statistical model designed for binary data, where the response variable can take values of 0 or 1. It belongs to the broader GLM family, which includes models like Poisson, Gaussian, and Dirichlet. The primary purpose of using Bernoulli GLMs here is to model the relationship between a mouse’s expected decision and the relevant regressors for each trial.

In the Bernoulli GLM, the response variable follows a Bernoulli distribution, which is a discrete probability distribution. It takes the value 1 with probability p (representing success) and 0 with probability 1-p (indicating failure). Estimating the probability p in the Bernoulli GLM involves predictor variables, and a link function connects the mean of the Bernoulli distribution to the linear predictor.

The linear predictor is formed as a linear combination of the predictor variables and their respective coefficients. Subsequently, the link function is used to transform the linear predictor to the probability scale. The logit link function stands as the most widely adopted link function for a Bernoulli GLM, and it can be expressed as log(p/(1-p)) = Fβ, where F corresponds to a predictor variable matrix, and β represents a vector of coefficients.

In this study, we employed a Bernoulli GLM to analyze the animal’s strategies, presented in Fig. 1e, in relation to various experiment regressors. It can map the binary values of the animal’s decision to the weighted representations of the considered covariates. These weights serve to depict the inputs of the model in relation to the output, which is the animal’s choice on each trial. Consequently, we can describe an observational GLM using the following equation, where the animal choice, denoted by y, can take a value of 1 or 0, indicating the mouse turning the wheel to the right or left-side, respectively:

p(y | z_t=k, x) = 1 / (1 + e^{-(w_k^ob)^T x}) (9)

In this equation, as indicated by the notations, the presented GLM is solely associated with the observation covariates, x, and observation weights w_k^ob for state k.

The fitting procedure of the model involves utilizing a penalized maximum likelihood estimation. This estimation minimizes the sum of the priors on the transition and observation weights, in addition to a negative log-likelihood function, often referred to as the log-posterior. The prior corresponds to a normal distribution over the weights with a mean of zero and a variance of σ^2. The negative log of this prior can be represented as 1/2(w)^T w. The purpose of this prior is to impose a penalty on the model weights, thereby regularizing the model by discouraging excessively large weight values for the regressors78. Consequently, a relevant loss function can be defined as:

Loss(w) = -log p(Y | w, X) + λ 1/2 w^T w (10)

Here, log p(Y | w, X) represents the conditional probability of the output, which corresponds to the decisions made by the animals, given the model regressors. Also, the symbol λ assumes the role of a hyperparameter that governs the regularization term’s influence on the model. The log-likelihood function can be mathematically defined as follows:

log p(Y | w, X) = Σ_{t=1}^T log p(y_t | w, x_t) = Σ_{t=1}^T (y_t w^T x_t - log(1 + e^{w^T x_t})) (11)

In this context, Y represents the observations from trial 1 to T, while X corresponds to the regressors for the GLM applied to the same trials.

The GLM can be fitted using the maximum likelihood or MAP estimation, and the resulting coefficients can subsequently be used to make predictions on new data. To assess the performance of the fitted model, we use cross-validation on held-out test data.

Multinomial GLM
Another form of the GLM is the multinomial GLM, which we will employ to model how external covariates influence transitions between different states. This multinomial logistic regression, also known as softmax regression or maximum entropy classifier, serves as an extension of logistic regression to handle data with multiple categories. Multinomial GLMs are GLMs possessing the capability to analyze data from more than one category simultaneously. By modeling the relationship between independent variables and categorical dependent variables, this approach allows for the determination of the likelihood associated with each category. In the field of neuroscience, multinomial GLMs are frequently employed to analyze the link between brain activity and behavior, facilitating the investigation of neural processes underlying different types of behavior.

The primary difference between Bernoulli and multinomial GLM lies in the number of categories the models handle. Bernoulli GLM is used for binary outcomes, modeling the probability of success (1) or failure (0), commonly employed in binary classification. Multinomial GLM, on the other hand, deals with data featuring more than two categories, modeling the probabilities of each category using a multinomial logistic function. The choice between them depends on whether you’re working with binary or multiclass categorical data. In multinomial GLM, the expression for the conditional probability of observing a particular outcome y, given input variables x and a set of model parameters w, is typically formulated as follows:

p(y=c | x, w) = exp((w_c)^T x) / Σ_{j=1}^C exp((w_j)^T x) (12)

where w_j corresponds to the parameters associated with j-th outcome and C is the number of possible outcomes in which for multinomial GLM C is more than 2. The multinomial equation finds frequent application within the domain of multinomial logistic regression, serving as a means to model the probability distribution across numerous discrete outcomes or categories. This mathematical framework is foundational in a multitude of machine learning and statistical contexts, including but not limited to tasks such as text classification, image recognition, and addressing multiclass classification challenges.

In this paper, we delve into the exploration of the GLM-HMM, GLM-HMM, with multinomial GLM outputs, a method capable of estimating the likelihood of the next state (presented in Fig. 1e). Notably, the GLM-HMM framework offers the advantage of enabling each state to possess a multinomial GLM that effectively captures the intricate relationship between transition covariates, such as previous choice and previous reward, and the associated transition probabilities.

Therefore, we consider the animal’s behavior at trial t and proceed to calculate transition probabilities to various states at trial t+1 utilizing a multinomial GLM. The model is adeptly structured to establish associations between the vector of model transition inputs and the unnormalized log probability of each potential future state. So our model presents transition probabilities using a multinomial GLM, and emission probabilities using a Bernoulli GLM, each defined by its own set of parameters.

Structure of the GLM-HMM
GLM-HMM, or Generalized Linear Model-HMM, is a sophisticated probabilistic model in which the core structure combines the principles of HMM and GLM31–38. At its essence, GLM-HMM is defined by a dual-layered structure. The first layer comprises an HMM, a stochastic process with hidden states that transition over time. These hidden states capture latent information about the underlying dynamics of a system. The second layer incorporates GLMs, which govern the mapping from inputs to outputs, and this mapping is influenced by the current hidden state of the HMM. This dual-layer structure enables GLM-HMM to effectively model complex, sequential data where the relationship between observed outputs and input variables varies depending on the underlying, unobservable state. This capacity to incorporate state-dependent relationships within a probabilistic framework makes GLM-HMM an appropriate tool for analyzing and understanding temporal data in various domains.

In this manuscript, we present a GLM-HMM framework that incorporates both GLM observation and GLM transition models with independent covariates. This approach enables us to capture the temporal patterns of animal transitions between states, enhancing our understanding of decision-making dynamics in mice within non-stationary environments. Specifically, we use a Bernoulli GLM for observations and a multinomial GLM for transitions. The Bernoulli GLM captures the roles of different regressors in animal choices, establishing the relationships between observed and predictor factors by defining the conditional distribution of the observed parameter based on its predictors. The multinomial component captures the intricate patterns associated with state transitions, characterizing the transition probabilities between the hidden states. Instead of a fixed transition matrix between states, our model uses a set of vectors, one for each state, to capture transitions into that state, with the averaged transition probability matrix presented in Fig. 3e. At each trial or time point in a dynamic environment, these probabilities govern the likelihood of transitioning from one state to another.

Therefore, in the context of a multinomial GLM, for the transition model, the GLM output reflects different probabilities for transitions between states. In this case, x_t^tr presents the transition covariates at trial t and w_k^tr is the transition weights associated with state k. So for the transition probability to state k at trial t, denoted as z_t, we can write:

p(z_t=k | x_t^tr) = exp((w_k^tr)^T x_t^tr) / Σ_{j=1}^K exp((w_j^tr)^T x_t^tr) (13)

here, K represents the total number of states.

Furthermore, the probability of transitioning from state i to state j at time t, given transition covariates x_t^tr, is modeled as:

p(z_t=j | z_{t-1}=i, x_t^tr) / exp(B_ij + (w_j^tr)^T x_t^tr) (14)

where B_ij is a learned baseline transition logit from state i to j. Also, w_j^tr is a vector of covariate weights that modulate all incoming transitions to state j.

The log-likelihood function in a multinomial GLM is rooted in the multinomial distribution. It quantifies the probability of observing categorical outcomes based on the predictor variables and model parameters. This log-likelihood function is conventionally formulated as follows:

log L(w^tr) = Σ_{i=1}^N log (exp((w_k^tr)^T x_i^tr) / Σ_{j=1}^K exp((w_j^tr)^T x_i^tr)) (15)

= Σ_{i=1}^N ((w_k^tr)^T x_i^tr - log Σ_{j=1}^K exp((w_j^tr)^T x_i^tr))

in which N is the total number of outputs and x_i^tr represent the transition regressors for the i-th output. The other details of the mathematical description of this model will be explained in the upcoming sections. This comprehensive framework provides a robust tool for analyzing flexible decision-making and the underlying mechanisms in non-stationary environments.

Model inputs
In this framework, we incorporated several covariates into the model, some of which are shown in Figs. 1d and 3a, and will be elaborated upon in detail here.

Observation model covariates: For the GLM observation, the covariates were stimulus, previous choice, previous stimulus and bias. The stimulus, in the experiment, was defined as the contrast of a sinusoidal grating, varying between 0% and 100% brightness. To further normalize the stimulus, it was divided by the standard deviation of the variables across sessions. Also, the previous choice was defined as the mouse’s previous choice was left or right, respectively. On the other hand, the previous stimulus was defined as the behavior of the mouse when rewarded. If the mouse received a reward, it continued to make the same decision; however, when no reward was given, it changed its behavior. The previous stimulus values were binary, denoted as -1, +1. Finally, bias is a covariate that captures the animal’s inherent tendency to choose left or right, representing an internal preference or offset that exists independently of external factors such as the stimulus or choice history, and its value was set to 1 for all trials.

Transition model covariates: The covariates for the GLM transition included a filtered version of the previous choice, previous stimulus, previous reward, and three basis coefficients. Throughout this paper, the term filtered covariate denotes a covariate filtered with an exponential filter, enabling the consideration of a temporally filtered regressor to consider the impact of the regressors in the transition between states. Therefore, the filtered previous choice and previous stimulus are temporally filtered versions of similar GLM observation covariates. The previous reward value was set to -1 or 1, based on whether the previous animal’s decision was correct or not, respectively. Furthermore, three bases were defined as linearly independent vectors to represent the initial 100 trials of each session. These covariates aimed to capture the animal’s warm-up effect within the model.

Setting the MAP prior on the model parameters
We set the MAP prior on the model parameters, denoted as Θ={w_tr, w_ob, π}. These parameters consist of the initial state distribution, transition weights, and observation weights for all states. The observation weights are shown in Fig. 2b, and the transition weights are presented in Fig. 3b, c.

To implement our approach, we employed the EM method, as introduced by Dempster et al.79. This iterative technique enables the determination of parameters within a given model by maximizing the likelihood of the data, given the specific parameters. Previous works31,40,41 have successfully applied the EM approach to HMMs integrated with external regressors. The parameter estimation using the EM method involves an iterative process with two essential steps: the expectation step, where parameter expectations are computed, and the maximization step, optimizing data likelihood based on those parameters. This iteration continues until optimal parameter values are determined.

In this study, we incorporated a Dirichlet prior to model the initial state distribution π. As for the GLM, we employed independent zero-mean normal priors with variance σ_ob^2 and σ_tr^2, respectively. Larger values of these variances signify a flatter prior distribution. The entire set of model choices and inputs is expressed as F = {Y, X_ob, X_tr}. Here, X_ob = {x_ob_1, ..., x_ob_T} represents the observation input vectors, and X_tr = {x_tr_1, ..., x_tr_T} corresponds to the transition input vectors. The animal choices for the specified trials are denoted as Y = {y_1, ..., y_T}. The prior distribution considered for our GLM-HMM, p(Θ), is as follows:

p(Θ) = p(π) p(w_ob^k) p(w_tr^k) (16)

In this context, the model parameters encompass three distinct categories, namely, the initial state distribution, the state-specific observation weights and state-specific transition weights denoted by w_ob^k and w_tr^k, respectively. Here, k denotes the number of states, ranging from 1 to K.

The prior distributions for the observation and transition weights in the GLM are both assumed to follow normal distributions, as expressed in the following equation:

p(π) = Dirichlet(π | γ_π), p(w_ob^k) = Π_{j=1}^K N(w_ob^j | 0, σ_ob^2 I), p(w_tr^k) = Π_{j=1}^K N(w_tr^j | 0, σ_tr^2 I) (17)

The prior value for the initial state distribution was assigned as γ_π = 1. To determine the optimal hyperparameter value for the model prior on weights, we conducted a grid search over a range of values, specifically σ ∈ {0.25, 0.5, 1, 2, 4, 8, 16}. We employed a held-out validation set to compare the NLL values for the different σ values within the specified set. Subsequently, the value of σ yielding the highest NLL on the IBL data was selected, and it was found to be σ = 4.

The GLM-HMM employs an HMM, encompassing distinct sets of weights for both transition and observation aspects. Within the model, each state is governed by a state-specific Bernoulli GLM and a multinomial GLM, representing the animal’s decision-making behavior concerning choice probability and the probability of transitioning between states, respectively. In this context, we used the notations GLM-O and GLM-T to represent GLM observation and GLM transition, respectively.

In this context, we used the notations GLM-O and GLM-T to represent GLM observation and GLM transition, respectively.

The transition multinomial GLM involves weights that associate relevant regressors, denoted as x_t^tr, with the probabilities of transitioning between states. These transition probabilities are not fixed values and are contingent upon the combination of related regressors and the current state. They are represented by a matrix A ∈ R^{K×K}, where, at trial t, the element corresponding to state j and state k presents the transition probability between those two states and can be written as:

α_jk = P(z_{t+1} = k | z_t = j, x_t^tr) (18)

It is pertinent to acknowledge that the observation Bernoulli GLM plays a crucial role in determining the observation weights and characterizing the decision behavior as a function of the input observation regressors. By defining the Bernoulli GLM distribution as p(y_t | z_t=k, x_t^ob, w_k^ob), we obtain the following equation for the expected log-likelihood during the E-step:

log P(Θ | F) = log P(Y | X_ob, X_tr, Θ) + log P(Θ) + const. (19)

Model fitting
The GLM-HMM employs the EM algorithm40 to optimize model parameters for maximizing the likelihood of observed data78. This iterative algorithm consists of two main steps: the E-step, where the expected complete-data log-likelihood is computed based on parameter estimates and observed data, and the M-step, which maximizes model parameters based on these expectations. The likelihood is calculated using the forward-backward approach78, a dynamic programming algorithm. The EM algorithm continues iteratively until convergence is achieved, ensuring an accurate parameter estimation process.

To elaborate further, during each trial and based on the specified GLM-HMM parameters, we compute the joint probability distribution encompassing both the states and the animals’ decisions (left or right). Subsequently, the log-likelihood of the model is evaluated using this joint probability distribution. This relationship can be expressed in the following manner:

log p(Y | θ, X_ob, X_tr) = log Σ_Z p(Y, Z | θ, X_ob, X_tr) (20)

in which as mentioned, X_ob = {x_ob_1, ..., x_ob_T} represents the observation covariates, and X_tr = {x_tr_1, ..., x_tr_T} represent the transition covariates. Additionally, we have a set of latent states denoted as Z = {z_1, ..., z_T}, and corresponding observations for these states denoted as Y = {y_1, ..., y_T}.

In the model, X_ob and X_tr capture relevant information related to the observations and transitions, respectively. These covariates play a crucial role in characterizing the underlying dynamics of the system under consideration. The latent states, Z, represent unobservable or hidden variables that drive the observed data. They are essential components of the model, as they provide insights into the underlying processes governing the observed phenomena. The observations, Y, are the data collected from the system, corresponding to each specific latent state in Z. These observed data points are used in the model to estimate and infer the hidden states and the model parameters.

In the GLM-HMM, state-dependent GLM-T weights represent transition regressor weights, and GLM-O weights represent the significance of observation covariates. These weights’ patterns differ across distinct states. In the Bayesian context, where prior information exists for unknown parameters, the EM algorithm can be used to compute the mode of the posterior probability distribution, facilitating parameter estimation.

In the context of probabilistic graphical models and variational inference, the E-step is often associated with variational lower bounds. Specifically, when dealing with intractable posterior distributions or complex models, the E-step aims to maximize a lower bound on the log-likelihood, rather than the log-likelihood itself. This lower bound is often referred to as the Evidence Lower Bound or the variational lower bound. Here, during the E-step of the EM algorithm, a lower bound on the right-hand side of the objective function referred to as Eq. (20)79,80 is maximized. Subsequently, in the M-step, the expected log-likelihood obtained from the E-step is maximized with respect to the GLM-HMM parameters. In the following sections, we will detail the calculations for both the E-step and M-step of this estimation approach. These steps play a crucial role in iteratively refining the parameter estimates until convergence is achieved, enabling the determination of the mode of the posterior probability in the Bayesian setting.

Ultimately, in the context of mice decisions, along with transition and observation covariates, the primary goal of the EM algorithm is to optimize the log posterior of the model parameters. The log-posterior can be mathematically expressed as follows:

Σ_Z p(Z | F, Θ^{(t-1)}) log p(Y, Z | Θ, X_ob, X_tr) (21)

In this equation, we can express the model joint distribution, p(Y, Z | Θ, X_ob, X_tr), as:

p(z_1) p(y_1 | z_1, x_1^ob) Π_{t=2}^T p(z_t | z_{t-1}, x_t^tr) p(y_t | z_t, x_t^ob) (22)

To calculate this expected log-likelihood, the E-step uses a forward-backward approach81 to estimate the single and joint posterior state probabilities. We are going to explain each one separately here. So the process involves calculating the single posterior state probability at trial t, given by:

p(z_t = k | Y, X_ob, X_tr, Θ^{(t-1)}) (23)

We show this probability by ϕ_{t,k}. In the forward-backward approach, the E-step iteratively calculates the posterior probability of the mice decisions by going through a loop. As a result, the single posterior state probability can be decomposed into the following equation:

ϕ_{t,k} = p(Y_{[0:t]}, z_t = k | fX_ob, X_trg_{[1:t]}, Θ^{(t-1)}) p(Y_{[t+1:T]} | z_t = k, X_ob_{[t+1:T]}, Θ^{(t-1)}) / p(Y | X_ob, Θ^{(t-1)}) (24)

in which Θ^{(t-1)} is the parameters of the model at trial t-1.

In the E-step of the estimation process, the forward-backward algorithm is used to compute the expectation of the desired function. This two-stage message-passing algorithm is also known as the filtering process during the forward pass40. The forward-backward algorithm, through its filtering process, computes the posterior probabilities for each time point, enabling the estimation of the latent states’ influence on the observed data. In the following, we will explain the forward and backward passes of the algorithm.

Forward pass: In the E-step, the forward-backward algorithm is employed to obtain the expectation of the desired function. This is a two-stage message-passing algorithm40 and the forward pass is called filtering. By assuming we have GLM-HMM parameters and observation data, the forward pass is used to calculate the posterior probabilities for each time point. The objective of the E-step is to infer the hidden state sequence, given the available observations and the current parameter estimates. In the above equation for the first trial, the posterior probability is obtained by π_k p(y_1 | z_1 = k, x_1^ob, w_k^ob). Here, p(y | z=k, x_ob, w_ob) is the Bernoulli GLM distribution for observations (GLM-O). In this equation, if we consider the posterior probability of the mice decisions, α_{t,k}, from trial 1 to t as:

α_{t,k} = p(Y_{[1:t]}, z_t | fX_ob, X_trg_{[1:t]}) (25)

Based on this definition, for the first trial, we can call the posterior probability as α_{1,k}. Then using earlier calculations, for upcoming trials until T, we can compute the posterior probabilities, α_{t,k}, as:

Σ_{j=1}^K α_{t-1,j} α_{jk} p(y_t | z_t = k, x_t^ob, w_k^ob) (26)

Backward pass: In the forward-backward algorithm, the backward step complements the forward pass, utilizing previously obtained information to update latent state probabilities. Named backward due to its reverse chronological order, it calculates updates essential for the M-step. The backward pass relies on the joint posterior distribution of two latent states, contributing to refined GLM-HMM parameter estimates. Its primary goal is to determine the posterior probability of observed mouse data for all states and trials beyond the current trial. This probability is represented as β_{t,k}, where t denotes the trial index and k refers to a specific state. This can be written as:

p(Y_{[t+1:T]} | z_t = k, X_ob_{[t+1:T]}) (27)

Therefore, for trial T, we have β_{T,k} = 1. Also, we can compute the posterior probability, β_{t,j}, for trials {T-1, ..., 1} as:

Σ_{k=1}^K β_{t+1,k} α_{jk} p(y_{t+1} | z_{t+1} = k, x_{t+1}^ob, w_k^ob) (28)

Here, it should be considered that a Bernoulli GLM distribution for observations can be written as p(y_{t+1} | z_{t+1} = k, x_{t+1}^ob, w_k^ob). By incorporating the information from both the forward and backward passes, the model gains a better understanding of the system’s underlying dynamics and is better equipped to estimate the latent states’ influence on the observed data, leading to improved parameter estimation in the M-step of the EM algorithm. So, by employing the forward-backward approach, we have:

ϕ_{t,k} = α_{t,k} β_{t,k} / Σ_{k=1}^K α_{t,k} β_{t,k} (29)

On the other hand, for the joint posterior state probability μ_{t,j,k}, we can write:

α_{t,j} α_{jk} β_{t+1,k} p(y_{t+1} | z_{t+1} = k, x_{t+1}^ob, w_k^ob) / p(Y | X_ob_{[1:T]}, Θ^{(t-1)}) (30)

and by substituting the denominator using the definition of posterior probabilities, we have:

α_{t,j} α_{jk} β_{t+1,k} p(y_{t+1} | z_{t+1} = k, x_{t+1}^ob, w_k^ob) / Σ_{k=1}^K α_{T,k} (31)

Therefore, to calculate the expected log-likelihood, we should use the results of the posterior state probabilities which were obtained using the forward-backward algorithm. Therefore, for a given trial t and trial t+1 is given by μ_{t,j,k} = p(z_{t+1} = k, z_t = j | F, Θ^{(t-1)}). In conclusion, by considering the single and joint posterior state probabilities, ϕ_{t,k}, μ_{t,j,k}, we can rewrite the Eq. (21) as:

Σ_{t=1}^T Σ_{j=1}^K Σ_{k=1}^K μ_{t,j,k} log α_{jk} + Σ_{t=1}^T Σ_{k=1}^K ϕ_{t,k} log p(y_t | z_t = k, x_t^ob, x_t^tr, w_k^ob, w_k^tr) + Init (32)

In this equation, the term related to the initialization and is equal to Σ_{k=1}^K ϕ_{1,k} log π_k.

M-step: The M-step of the Expectation-Maximization algorithm updates the GLM-HMM parameters by utilizing the posterior probabilities calculated during the E-step. The objective of the EM algorithm is to minimize the negative log-likelihood function, which is augmented with a prior on the GLM-HMM weights.

L-BFGS-B (Limited-memory Broyden-Fletcher-Goldfarb-Shanno with Bound constraints) is a popular optimization algorithm used to solve unconstrained and bound-constrained nonlinear optimization problems. It’s an extension of the BFGS algorithm, which is a quasi-Newton method for unconstrained optimization. L-BFGS-B is particularly useful when dealing with optimization problems where the variables have certain bounds or constraints. These constraints can be upper and lower bounds on the variables, and L-BFGS-B is designed to handle such constraints. So L-BFGS-B is a second-order optimization method that approximates the Hessian matrix, leading to faster convergence and accurate parameter estimates. Here, the EM algorithm employs the L-BFGS-B approach, which belongs to the class of Quasi-Newton optimization methods.

During the M-step, the EM algorithm maximizes the expected log-likelihood, computed through the forward-backward algorithm during the E-step. This optimization process aims to identify the best parameters for the GLM-HMM. Consequently, in each iteration of the EM algorithm, the initial state distribution π is updated to refine the model’s representation of the latent state dynamics.

By iteratively performing the E-step and M-step, the EM algorithm iteratively refines the parameter estimates until convergence is achieved, enabling the determination of the mode of the posterior probability in the Bayesian setting.

Fitting the psychometric curve. The psychometric function was derived by plotting the percentage of choices made by the animal against various stimulus values. In this paper, the psychometric curves illustrate the animal’s rightward choice probability as a function of stimulus intensity and is presented in Fig. 2c. To fit the psychometric curve to the sigmoid function, maximum likelihood estimation was used, resulting in the following formulation:

p(choice=R | SC) = 1 / (1 + exp(-w_ob_1 x_ob_1)) (33)

in which, the SC represents the stimulus contrast. To minimize the loss function, the study employed the Python package ‘optimize.minimize’ from the scipy library. This package provides numerical optimization tools that enable accurate minimization of the loss function, facilitating the estimation of the model parameters and the fitting of the psychometric curve.

K-fold cross-validation for GLM-HMM
A cross-validation approach was employed to evaluate the model’s performance, involving random splitting of the data into 5 folds for training and testing. Approximately four-fifths of the data sessions were used for training the model, while the remaining randomly selected sessions were held out for testing the fitting performance. The sessions from all participating animals were evenly considered to ensure that mouse-to-mouse variability did not influence the cross-validation process.

This analysis revealed that a GLM-HMM with 3 to 5 latent states yields appropriate log-likelihood estimations. Following the fitting of the model with 1 to 5 states, we computed the log-likelihood of the test data, depicted in Fig. 2a. Also, the test log-likelihood was calculated for all individual fits and is presented in Fig. 6c. During the fitting procedure, the EM algorithm was executed during the training data sessions. Subsequently, in the testing stage, the likelihood of the remaining data was calculated based on the model parameters obtained from the training procedure. The forward pass was performed only once during this stage, and the likelihood of the test data was obtained by summing over all states as l = Σ_{k=1}^K α_{T,k}, where α_{T,k} was computed solely on the held-out sessions. It is important to note that the GLM-HMM parameters were solely calculated during the training procedure and used for evaluating the model’s performance on unseen data during testing.

In our analysis, Fig. 2, we have calculated the log-likelihood using the described procedure, which is the log of the calculated likelihood and we call it L = log(l). In some results, we have used the unit term “bits per trial". This L with this unit is acquired by calculating L of the test data on held-out sessions as described, and then subtracting the log-likelihood of the identical dataset considering a Bernoulli model for observed data. This is a baseline model and its corresponding log-likelihood is shown by L_0. Then we divide it by T_t log(2) in which T_t is the test set size (the number of trials). The predictability of test sets can be greatly enhanced by even tiny amounts of log-likelihood when expressed in bits per trial.

In our analysis, we calculated the log-likelihood using the described procedure, denoted as L = log(l) and in some of the results, we used the unit term “bits per trial" to quantify the log-likelihood.

To obtain the log-likelihood in bits per trial (L_bpt), we followed a specific process. Firstly, we computed L for the test data on held-out sessions, as described previously. Then, we determined the log-likelihood of the same dataset under the assumption of a Bernoulli model for observed data and multinomial GLM for transition data, which we refer to as the baseline model. We denoted the baseline model corresponding log-likelihood as L_0. The difference between L and L_0 represents the enhancement in log-likelihood due to the GLM-HMM performance. Finally, to express this enhancement per trial, we divided it by T_t log(2), where T_t is the size of the test set (the number of trials).

The resulting value in bits per trial provides a measure of predictability for the test set. Even minute increments in log-likelihood, when expressed in bits per trial, can lead to significant improvements in the predictability of the test data compared to the baseline model.

To illustrate this, consider a numerical example where the log-likelihood difference estimate is 0.02 bpt. This implies that the test data is approximately 10^47128.54 times more likely to have originated from the GLM-HMM compared to the baseline model when the test data comprises 1000 trials.

This representation in bits per trial, referred to as NLL, allows for a more intuitive understanding of the model’s performance and enables meaningful comparisons between different models and dataset. So we can write it as:

L_bpt = (L_t - L_0) / (T_t log(2)) (34)

in which bpt is an abbreviation for bit per trial unit.

Synthetic data
We simulated data (Fig. 7d) using the best-fitting model for each individual animal from the IBL dataset. For each animal, we reconstructed the model and generated simulated data on a session-by-session basis. In each session, we used the true stimulus values presented to the animal and generated corresponding choices and latent states, along with the covariate values, following the model structure. Specifically, for each trial, latent states were sampled using the transition model-either a fixed transition matrix (model without GLM-T) or a multinomial GLM with input-driven transitions (model with GLM-T). Conditioned on the latent state, choices were then sampled from a Bernoulli GLM. This process ensured that trial-by-trial choices and latent states were newly simulated while preserving the structure of the task, enabling the use of ground-truth labels for model recovery evaluation. We simulated data with K=4 latent states, using the original session lengths and recursively updating the covariates to reflect the simulated trial histories. The observation model was a Bernoulli GLM with covariates including past choice, stimulus side, and previous reward. These covariates were updated trial-by-trial based on the simulated choice and feedback (e.g., reward outcome and correct side). Simulations were run with fixed random seeds to ensure reproducibility. For the model without GLM-T, transitions were static and governed by a fixed transition matrix without covariate inputs. In contrast, the model with GLM-T had a multinomial GLM with input-driven transitions, where transition probabilities were modulated by exponentially filtered versions of the different behavioral covariates.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This manuscript by Mohammadi et al. is an interesting and needed generalization of the GLM-HMM framework, which has become an useful tool for analyzing diverse internal states during decision making. It extends the “standard” GLM-HMM proposed by Ashwood et al. (2022) by incorporating non-stationary transition probabilities among the assumed states. More specifically, the extended GLM-HMM infers dynamically changing transition probabilities that depend on prior history, suggesting that variables such as choice and reward history can have significant impact on state transitions. One related notable finding is that higher reward rate is associated with transition into “biased” or disengaged states.\nOverall, the extended model significantly improves the fit of choice behavior over the previous model by combining Bernoulli and multinomial GLM in a novel way. However, the study lacks clarity on what additional insights it offers beyond the previous standard GLM-HMM, aside from simply providing a better fit to the behavioral data. In other words, we believe there is still room to better demonstrate the full utility of the presented/extended model. The transition GLM (“GLM-T”) weights are presented, but are not sufficient in illustrating how they impact the choice behavior overall. The study presents evidence on how prior trial history affects state transitions, but does not explore the reverse: how the newly inferred states, which are presumably more accurate, provide new insights into the behavior they produce.\nOne possible analysis is to compare how the observation GLM weights (“GLM-O,” Fig. 2b) differ when GLM-T is excluded or included.Another crucial piece of information missing from the manuscript is the analysis of behavioral correlates for each state estimated with the GLM-T. For example, We wonder if the inclusion of GLM-T enables better distinction between states in terms of performance (probability correct), reaction time, and actual rewards collected. The authors suggest the link between disengagement and satiety from transition weights alone, but this could be more exhaustively tested; for example, by looking at whether occupancy of those states also tend to increase toward the end of the session (when mice have accumulated overall more rewards over the experiment). Also we expect that (filtered) reward rate is correlated with probability of occupying biased (disengaged) states, but this has not been explicitly tested. These analyses could make the main claim of the study more convincing.\nIn addition, there are certain places where the assumptions of the extended GLM-HMM model need to be checked. One issue is the question of how it truly differentiates from the previous GLM-HMM model with fixed transition probabilities. Although transition probabilities are fixed for the standard GLM-HMM, there might be scenarios where it’s difficult to distinguish it from non-fixed cases, since the posterior state probabilities are after all estimated on a trial-by-trial basis. This raises the possibility that if transition probabilities (for the extended model) are not dynamic enough, the resulting behavior might not be so distinguishable from the standard model. We suggest that authors should find a way to directly address this issue, perhaps through model recovery from simulated data. Relatedly, could authors plot how the (inferred) transition probabilities change over time, instead of their averaged values (Fig.3e), to connect with the information shown in Fig.5a more intuitively?\nLastly, if we are not mistaken, the covariates that are included for the transition GLM are assumptions that were pre-selected for the model rather than those that survive among other candidate covariates. Could the authors provide a rationale for selecting these covariates and not others? And could they comment on whether there might be other potential covariates that were not tested here but could be important?\nMinor comments\nEither Fig. 3f or Fig. 3g for the point corresponding to 5-state & Tau=8 seems to be incorrect. We assume these two plots should be different representations of the same data. In Fig. 3f for 5-state case, Tau = 8 (green) has slightly higher LL than Tau = 2 (purple). But looking at Fig. 3g 5-state graph (in red), Tau = 2 has a higher LL than Tau = 8.\nIn Fig.6b, the X-axis has 6 points but the labels only have five. We were wondering if these were labeled incorrectly because “filtered reward” for biased states are not predominantly positive.\nFig. 3d (transition bias) is not mentioned anywhere in the main text and could use some explanations (the same for Fig. S6).\nFig.4a is somewhat confusing. We suggest adding axis labels in each panel and clearly marking them as “example session 1”, “example session 2”, etc.\n“Stimulus side,” as explained in the manuscript (#124-125) is equivalent to what’s commonly known as “win-stay/lose-switch.” We were wondering why the former term was adopted and not just called the latter. For this variable, isn’t there no relation with specific stimulus side as the mice can WS-LS on either side?\nIn line #367-368, “solid black line” should be “solid red line” according to Fig. 6c figure caption.\nLine #388-389 states: “For the transition model, filtered choice, stimulus side, and reward were the most influential covariates, ….” Related to the comment above, this statement could be toned down a little because these factors are assumptions that were pre-selected for the model rather than those that survive among other candidate covariates.\nThe Methods section (Section 5) for describing the dataset and the exclusion criteria could be a little more detailed. It seems like the current study is selecting the same 37 mice (out of 123) that were used in the previous study by Ashwood et al. (2022), which we found out only after referring to it for more information. The authors could either explicitly state this or make the Methods more self-contained, since the readers might not be aware of the previous work.\nReferences\nLe et al., which is cited in #36, has been published in PLOS Comp. Bio.:\nLe, N. M., Yildirim, M., Wang, Y., Sugihara, H., Jazayeri, M., & Sur, M. (2023). Mixtures of strategies underlie rodent behavior during reversal learning. PLOS Computational Biology, 19(9), e1011430.\nSimilar to above work, it might be worth also citing the following study, which applies HMM to decision making:\nLi, J. J., Shi, C., Li, L., & Collins, A. G. (2024). Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making. Journal of Mathematical Psychology, 119, 102842.", "reviewer_2": "In the present manuscript, the authors present an adapted hidden Markov model that allows for various covariates from the experiment to guide the transitions between decision making states, and (potentially) separate covariates to guide the decision making ‘policy’ within those different states. They then apply their model to some rat perceptual learning data in which the probability of left or right decisions (i.e., a bias) should increase or decrease across blocks of trials. They find some apparent evidence of a four state model in which left or right bias appears to be crossed with engaged or not (so engaged left, engaged right, left bias, right bias).\nI think this is an interesting manuscript in that it develops a useful statistical model to capture important trends in the decision making style across trials. These HMMs with transitions across decision states are conceptually easy but very difficult to fit to data. Adding to this complexity is the possibility of four different sets of GLM weights to predict choices. Finally, the authors use a nice data set to illustrate the utility of their model – not only is there left and right biases, but these are very long experiments that would almost certainly catch the rat off task from time to time (and as their model suggests, they do engage in off task behavior).\nAll of that said, the novelty of the research seems to be a bit low for this particular outlet in my opinion. According to the authors, this division between two separate generalized linear model structures to guide both the transition between the states and the decisions within those states is the main novelty of their research. To me, the main contribution of their modeling efforts is that they allow for two very general (literally GLM) structures to modulate the two main components of their model, one being a transition among the states and one being the decisions within those states. While this is interesting, methodologically speaking this is not a significant leap forward. Furthermore, the evidence for a need of a separate transition GLM seems quite low. Maybe it is just a scaling issue, but Figure 2a is not really convincing that having a transition model is actually necessary.\nAlthough the results are also interesting, it is perhaps not that surprising that they would find a model that supports the two real conditions of the experiment (left or right bias) crossed with ‘engaged’ or ‘not engaged’. To be honest, it was a little unclear what the inputs were to the GLM observation, despite a whole section for it in 4.5. For example, they say that ‘bias’ was in the set of observational covariates, but it is not defined in 4.5. It is hard to understand how a variable called ‘bias’ could meaningfully contribute to the fits if there are already regressors for the stimulus itself and the past choice and previous stimulus. In any case, I would worry that they are finding evidence for a four state model simply because the models that explain decisions are far too simplistic. It is possible that a more detailed model of the decision itself might imply that there are actually fewer states. For example, potentially even having a regressor for ‘number of trials performed in this block’ might soak up a lot of within-state variability to remove the need for two of those four states. Because questions like this were not really fully examined, the results surrounding the transition GLM’s contribution are pretty minimal. However, even if they were significant, it is not clear that an incremental methodological advancement in a well established field of HMMing makes sense at this particular outlet.", "reviewer_3": "The authors investigated behavioral strategies used by mice performing a binary decision-making task with a time-varying distribution over correct choices. To do so, they fit two sets of GLMs: one that predicts observed choices as a function of sensory inputs, task covariates, and latent state, and another that predicts transition probabilities to each latent state. Crucially, this setup allows the transition probabilities to vary over time, in contrast with traditional HMMs, and thus is a sensible choice for analyzing behavior in a non-stationary environment. In sum, they find four interpretable states that govern behavior during this task, which are broken down into “engaged” states, in which animals appear to rely on the observed stimulus, and “biased” states, in which animal’s choices seem heavily biased towards one direction regardless of stimulus. They characterize various task-relevant factors that influence transitions between these states, and provide a detailed account of how transition rates are modulated over time by the underlying evolving task structure.\nStrengths:\nThe authors do an excellent job demonstrating that the behavioral data is well-explained by their proposed four-state model, and their interpretation of these states is solidly supported by breakdowns of GLM weights and psychometric curves.\nConcerns/Questions:\nThe relationship between this work and the cited “Extracting the dynamics of behavior in sensory decision-making experiments” (Roy et al., 2021, Neuron) should be explicitly discussed. In particular, both make inferences about animal behavior from the same dataset, and both employ a fitting method that permits a time-varying observation model. The authors should make an effort to underscore what findings are uniquely highlighted by their GLM-HMM approach that couldn’t be captured by the model used there.\nI am somewhat puzzled by why state transition probabilities are modeled by the transition GLM in the way they are. If I am understanding correctly, only the probabilities of transitioning to a particular latent state are parameterized, regardless of starting latent state, for a total of K transition probabilities (this interpretation is consistent with the exposition in the text as well as Fig. 5). This is quite a restrictive choice compared to standard HMMs, which parameterize all K^2 pairs of from-and-to. In this case, I would expect that transitions “Engaged R -> Biased R” would be significantly more probable than “Engaged L -> Biased R”. Indeed, other works that use time-varying transition probabilities, such as the cited work Calhoun et al. 2019, fit a separate GLM for the transition probabilities out of each latent state. I would like to see that the conclusions drawn are not overly dependent on this particular modeling choice.\nWhile directions for future work are discussed thoroughly, interpretations and caveats of the presented findings seemed a tad scarce in the discussion. For example, the authors allude to occupancy of disengaged states being related to satiety. The authors could expand on this, possibly by commenting on the robustness of this conclusion, especially when considering that the delineation appears less clear for the 5 state model as shown in Fig. S2b.\nMinor/miscellaneous:\nIs there a relationship between the model’s predicted probability of the animal being in a biased/disengaged state during some time window and the number of incorrect choices observed? The authors state that performance remains high even in the biased states because of imbalanced prior probabilities, but I don’t see this quantified anywhere. Reporting a prevalence-independent performance metric like balanced accuracy could also be insightful here.", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": "Dear Reviewers,\nWe sincerely appreciate your time and effort in reviewing our manuscript. Below, we have\nprovided detailed responses to each of your queries and comments, addressing them point by\npoint. Your constructive feedback has been invaluable in improving the clarity and rigor of our\nwork.\nReviewer #1 (Remarks to the Author):\n1. This manuscript by Mohammadi et al. is an interesting and needed generalization of the\nGLM-HMM framework, which has become an useful tool for analyzing diverse internal states\nduring decision making. It extends the “standard” GLM-HMM proposed by Ashwood et al.\n(2022) by incorporating non-stationary transition probabilities among the assumed states. More\nspecifically, the extended GLM-HMM infers dynamically changing transition probabilities that\ndepend on prior history, suggesting that variables such as choice and reward history can have\nsignificant impact on state transitions. One related notable finding is that higher reward rate is\nassociated with transition into “biased” or disengaged states.\nWe thank the reviewer for this positive assessment of our mansuscript’s contributions.\n2. Overall, the extended model significantly improves the fit of choice behavior over the previous\nmodel by combining Bernoulli and multinomial GLM in a novel way. However, the study lacks\nclarity on what additional insights it offers beyond the previous standard GLM-HMM, aside from\nsimply providing a better fit to the behavioral data. In other words, we believe there is still room\nto better demonstrate the full utility of the presented/extended model.\nWe thank the reviewer for these insightful comments and appreciate their recognition of our\nmodel’s improvements. We agree that it is important to highlight the additional insights our\nmodel provides. Below, we describe the additional analyses and figures we have included to\naddress the reviewer’s specific suggestions.\n3. The transition GLM (“GLM-T”) weights are presented, but are not sufficient in illustrating how\nthey impact the choice behavior overall. One possible analysis is to compare how the\nobservation GLM weights (“GLM-O,” Fig. 2b) differ when GLM-T is excluded or included.\nThanks for this suggestion. We have now added plots comparing the GLM-O weights obtained\nfrom models with and without GLM-T. These results demonstrate that including the GLM-T\nmodel does not significantly alter the GLM-O weights, confirming that the two components\noperate independently. This aligns with our expectations, as GLM-T is not designed to modify\nobservation weights or choice behavior substantially.\nHowever, the addition of GLM-T enhances the model’s ability to capture behavioral dynamics,\nas reflected in the higher log-likelihood (LL) value in the 4-state model compared to the 3-state\nmodel used in Ashwood et al. (2022). Notably, Ashwood et al. (2022) model only included\nGLM-O weights for the unbiased part of the data (the first 90 trials of each session), where\nenvironmental dynamics were not present. In contrast, our 4-state model, which explicitly\nincorporates bias values, provides a more accurate description of the changing environment,\nsuggesting that state transitions play a key role in behavioral adaptation.\nThis result highlights that our model, which integrates both GLM-T and GLM-O, governs how\nbehavioral states evolve over time while capturing choice behavior patterns, whereas GLM-O\nalone defines within-state choice behavior. The consistency of GLM-O weights across models\nfurther supports the idea that the transition process (GLM-T) does not interfere with\nmoment-to-moment decision weightings but instead regulates long-term behavioral shifts. We\nhave added this plot to the supplementary materials.\n4. The study presents evidence on how prior trial history affects state transitions, but does not\nexplore the reverse: how the newly inferred states, which are presumably more accurate,\nprovide new insights into the behavior they produce.\nAnother crucial piece of information missing from the manuscript is the analysis of behavioral\ncorrelates for each state estimated with the GLM-T. For example, We wonder if the inclusion of\nGLM-T enables better distinction between states in terms of performance (probability correct),\nreaction time, and actual rewards collected.\nThank you for your thoughtful feedback and for highlighting the importance of analyzing\nbehavioral correlates of the inferred states.\nWe compared the performance of both models. The left scatter plot and the right density plot\nboth show that states inferred under the model with GLM-T generally have higher probabilities\ncompared to those from the standard GLM-HMM. The upward shift of many points above the\ndiagonal (y = x) indicates that, for the same states, the model with GLM-T assigns a higher\nprobability. This suggests that the inclusion of GLM-T leads to more confident state assignments\nand better performance. The density plot further supports this pattern, showing that the most\nfrequent state probabilities tend to be higher under GLM-T. These findings indicate that the\nmodel with GLM-T provides more confident state estimations.\nFurthermore, to address the reviewer’s point about how the newly inferred states provide\ninsights into behavior, we plotted the posterior state probabilities across trials for the standard\nGLM-HMM (top), the model with GLM-T (middle), and their combined visualization (bottom).\nThese plots show that incorporating GLM-T results in higher confidence in state assignments. In\nthe model without GLM-T, multiple states often have moderate probabilities, leading to greater\nuncertainty in state identity. In contrast, with GLM-T, states reach higher probabilities, reducing\noverlap and making transitions between them more distinct.\nAdditionally, the model with GLM-T better captures transition patterns that align with the task\nstructure, particularly biased block switching (e.g. left-biased states appear more frequently in\nblocks where the stimulus is more probable on the left side).\nOverall, the model with GLM-T provides greater confidence in state estimations—not only\nbecause state probabilities are generally higher, but also because it more successfully captures\nthe bias-block structure and accurately tracks switches between them. For example, as shown\nin the lowest panel with green squares, the states inferred by the model with GLM-T (solid lines)\nconsistently align with the expected transitions, further demonstrating its improved ability to\nreflect behavioral adaptations to stimulus probability changes.\nOn the other hand, below we have plotted the reward rate for both the model with GLM-T and\nthe model without GLM-T (engaged states in green and disengaged states in orange). Since the\nobservation model parameters remained largely unchanged between the two models, the\nreward rate also showed minimal differences. This suggests that the inclusion of GLM-T did not\nsignificantly alter overall reward outcomes, reinforcing the expectation that the estimated states\nin our model do not strongly influence choice behavior.\nFurthermore, we analyzed response times instead of reaction times here, as reaction time data\nwas unavailable for most IBL data sessions. Response time is defined as the interval between\nstimulus onset and the moment the mouse receives feedback on its decision. In contrast,\nreaction time in the IBL task refers to the time from stimulus onset to the mouse’s initial\nmovement of the wheel. While response times varied widely, ranging from a few hundred\nmilliseconds to as long as 10 seconds in extreme cases, typical reaction times (estimated from a\nlimited number of sessions with reliable data) were consistently in the range of a few hundred\nmilliseconds. Compared to other tasks, the IBL task is characterized by relatively short reaction\ntimes.\nWe analyzed the difference in response times between disengaged and engaged states across\nall animals. Specifically, we computed the difference in the 90th percentile response time for\neach state in every mouse and plotted the results, including 95% bootstrap confidence intervals\n(n = 2,000). This analysis allows us to quantify how distinct these behavioral states are in terms\nof response time.\nTo further assess the impact of GLM-T, we compared this difference across models with and\nwithout GLM-T. As shown in the below figure, for all animals, the difference in 90th percentile\nresponse time between disengaged and engaged states was larger in the model with GLM-T.\nThis suggests that the inclusion of GLM-T enhances the distinction between these states,\nmaking disengaged trials more clearly separated from engaged trials in terms of response\nlatency. The larger response time gap under model with GLM-T indicates that this model better\ncaptures behavioral differences associated with engagement and disengagement.\nTo better address the reviewer’s question regarding the performance of the models, we plotted\nthe fractional occupancy of the four discrete states in the model without GLM-T for all IBL data,\nsimilar to Fig. 4c. This analysis was conducted across all trials, covering both right-biased and\nleft-biased blocks. The results are displayed separately for each block type. As seen in the plot,\nthe background stimulus bias was not captured by the model without GLM-T.\nFurthermore, to evaluate performance in relation to switching patterns in both models, we\nanalyzed the switching data. The below figure compares the first switching probability across\ndifferent trials, representing the time it takes for animals to switch states after encountering a\nnew biased block of data. The results are shown for models without GLM-T (left) and with\nGLM-T (right). The model without GLM-T shows no specific peak, indicating that it fails to\ncapture structured state transitions following block changes. The model with GLM-T exhibits a\ndistinct peak around trials 6-8, indicating that animals tend to switch states after detecting a\nblock transition, with a slight delay. This suggests that incorporating GLM-T allows the model to\naccount for the bias block, understand it, and adjust the states accordingly with a short delay, so\nthat it better aligns with behavioral dynamics, as animals appear to systematically adjust their\nstate based on changes in stimulus blocks. These findings further support the idea that GLM-T\nenhances the model’s ability to capture meaningful behavioral patterns, making it a more\nbehaviorally relevant and interpretable framework for decision-making in dynamic environments.\nWe have included these figures in either the paper or the supplement.\n5. The authors suggest the link between disengagement and satiety from transition weights\nalone, but this could be more exhaustively tested; for example, by looking at whether occupancy\nof those states also tend to increase toward the end of the session (when mice have\naccumulated overall more rewards over the experiment) . Also we expect that (filtered) reward\nrate is correlated with probability of occupying biased (disengaged) states, but this has not been\nexplicitly tested. These analyses could make the main claim of the study more convincing.\nThank you for these suggestions to clarify the relationship between disengagement and satiety.\nAs the reviewer suggested, we examined the relationship between them by plotting the\nprobability of occupying engaged and disengaged states throughout the session. We\ninterpolated 100 points from each session and then averaged them across all sessions for all\nanimals. The plot shows that at the beginning, there is a clear warm-up effect, where mice\ntransition from disengaged to engaged states, indicating an initial adjustment phase before\nstable task performance. Toward the end of the session, the occupancy of disengaged states\nincreases while engaged states decrease, supporting the idea that as mice accumulate more\nrewards, they become less engaged, potentially due to satiety. This provides initial evidence for\nthe proposed link between disengagement and satiety.\nFurthermore, to study the relationship between disengagement and satiety, we generated two\nanalysis plots. The first figure shows the relationship between filtered reward (gray) and the\nprobability of being in the disengaged state (orange) across trials. The shaded regions\nrepresent different biased blocks throughout the session. For most trials, as the filtered reward\nrate increases, disengagement probability also starts to increase, suggesting a direct correlation\nbetween the two.\nThe second figure further confirms this relationship by plotting P(disengaged) vs. filtered reward\nacross trials. The red trend line shows a positive correlation. So we found a statistically\nsignificant positive moderate correlation between the filtered reward rate and the probability of\ndisengagement (r ≈ 0.30, p < 0.001), where r = 0.30 (Pearson correlation coefficient) indicates a\nmoderate positive correlation, and p < 0.001 (p-value) confirms that this relationship is highly\nunlikely to be due to chance. This moderate correlation suggest that as the reward rate\nincreases, disengagement probability also tends to increase, but other factors likely contribute\nas well, supporting the idea that reward rate plays a role in disengagement dynamics.\n6. In addition, there are certain places where the assumptions of the extended GLM-HMM\nmodel need to be checked. One issue is the question of how it truly differentiates from the\nprevious GLM-HMM model with fixed transition probabilities. Although transition probabilities are\nfixed for the standard GLM-HMM, there might be scenarios where it’s difficult to distinguish it\nfrom non-fixed cases, since the posterior state probabilities are after all estimated on a\ntrial-by-trial basis. This raises the possibility that if transition probabilities (for the extended\nmodel) are not dynamic enough, the resulting behavior might not be so distinguishable from the\nstandard model. We suggest that authors should find a way to directly address this issue,\nperhaps through model recovery from simulated data. Relatedly, could authors plot how the\n(inferred) transition probabilities change over time, instead of their averaged values (Fig.3e), to\nconnect with the information shown in Fig.5a more intuitively?\nAlso, to address the reviewer’s point regarding how the extended GLM-HMM differentiates from\nthe standard model, please see the figure below, which is a supplementary figure (Figure S5)\nfrom the paper employing synthetic data. This figure illustrates the fractional occupancy of\ndifferent states for models with and without GLM-T, separated by left-biased and right-biased\ntask conditions. The comparison shows that when GLM-T is included (panel a), the state\noccupancy distributions adapt more distinctly to task biases, whereas without GLM-T (panel b),\nthe state occupancies appear without being affected by data bias blocks. This further supports\nthat the inclusion of trial-dependent transitions allows the model to capture dynamic shifts in\nbehavioral states, distinguishing it from the standard GLM-HMM with fixed transition\nprobabilities.\nTo address the reviewer's question, we conducted two model comparisons between models with\nand without GLM-T using two types of synthetic data.\n● First, we simulated data from a model without GLM-T (4-state) and fitted the GLM-T and\nnon-GLM-T models to this data (left side of the figure below). In this case, both models\nperformed almost similarly.\n● Second, we simulated data from a model with GLM-T (4-state) and then fitted both the\nGLM-T and non-GLM-T models to this data (rigth side of the figure below). The test\nlog-likelihood comparison shows that the model with GLM-T performs better, achieving\nan improvement of 0.009 bits per trial over the non-GLM-T model. This demonstrates\nthat when the true generative process involves trial-dependent transitions, the model\nwith GLM-T can better capture behavioral dynamics.\nThis improvement of 0.009 bits per trial when the data has trial-dependent transitions means\nthat for a dataset with 5000 trials, the data would be approximately 3.52×1013 times more\nprobable under the GLM-T model than under the non-GLM-T model.\nThe difference in performance between these two cases highlights the impact of capturing\ndynamics effectively. This further highlights the substantial advantage of the GLM-T model in\ncapturing trial-dependent dynamics, especially in behavioral data, where datasets typically\nconsist of a large number of trials.\nPer the reviewer’s request, we plotted the transition probabilities (instead of their averaged\nvalues (Fig.3e)) for both models—with and without GLM-T—to highlight their differences over\ntime.\nAs shown in this plot, when GLM-T is included, transition probabilities exhibit trial-by-trial\nfluctuations, adapting dynamically across different states and conditions. In contrast, without\nGLM-T, transition probabilities remain fixed, lacking the flexibility to adjust to behavioral shifts.\nThis visualization confirms that the extended model (GLM-T) captures dynamic transitions,\ndifferentiating it from the standard GLM-HMM, where transitions are static and do not evolve\nwith trial history or task conditions. Also, the main diagonal plots (high-probability states) follow\na similar pattern to Fig. 5a (top session) and are connected to it.\nWe have included these figures in either the paper or the supplement.\n7. Lastly, if we are not mistaken, the covariates that are included for the transition GLM are\nassumptions that were pre-selected for the model rather than those that survive among other\ncandidate covariates. Could the authors provide a rationale for selecting these covariates and\nnot others? And could they comment on whether there might be other potential covariates that\nwere not tested here but could be important?\nWe sincerely appreciate the reviewer’s insightful comment. In selecting the covariates for the\ntransition GLM, we aimed to include those that based on both our reasoning and the literature,\nwould have the most significant influence on making decision and strategy switching.\nSpecifically, we incorporated history-dependent factors such as the animal’s choice history,\nstimulus side history, and reward history, as these are well-established determinants of\ndecision-making strategies. Additionally, we accounted for warm-up effects using basis functions\nto capture early-session adaptation in transition probabilities. To ensure that our model\nappropriately captured the temporal dynamics of these covariates, we searched for the optimal\ntime constant that best described their influence on transitions (Figures 3f and 3g). This involved\nfiltering the covariates with different time constants and selecting the one that most accurately\ncaptured the relevant history and period of effect. Even this process required significant time\nand computational resources due to the large dataset and the complexity of the model. Given\nthese constraints, our approach focused on refining a predefined set of covariates rather than\nconducting an exhaustive search over a broader range of predictors.\nThat said, we acknowledge that additional covariates could be relevant to strategy switching.\nFor example, response time could reflect decision urgency or hesitation, movement vigor (e.g.,\nvelocity, acceleration) might indicate engagement or motivation, and inter-trial interval could\ncapture the influence of short-term breaks on strategy shifts. Additionally, neural activity in\nrelevant brain regions, if available, could provide insight into strategy adaptation mechanisms,\nwhile task engagement metrics (e.g., number of licks, pupil dilation) may indicate attentional\nfluctuations affecting transitions. Exploring a broader set of covariates and implementing a more\ndata-driven selection approach would be a valuable direction for future work. We have added\nthis to the future work section of the paper in orange as follows: “On the other hand, it would\nbe useful to explore other covariates that may influence strategy switching but were not\nincluded in our current model. While we selected covariates based on prior literature and\ntheoretical reasoning, a more data-driven approach could help identify additional\npredictors, such as response time, movement vigor, or task engagement metrics, to\ncapture decision-making dynamics in more detail.”\nMinor comments\nEither Fig. 3f or Fig. 3g for the point corresponding to 5-state & Tau=8 seems to be incorrect.\nWe assume these two plots should be different representations of the same data. In Fig. 3f for\n5-state case, Tau = 8 (green) has slightly higher LL than Tau = 2 (purple). But looking at Fig. 3g\n5-state graph (in red), Tau = 2 has a higher LL than Tau = 8.\nWe sincerely appreciate the reviewer’s careful examination of our figures and for catching this\ndiscrepancy. We have now corrected the figure to ensure that both panels accurately reflect the\nsame underlying data.\nIn Fig.6b, the X-axis has 6 points but the labels only have five. We were wondering if these were\nlabeled incorrectly because “filtered reward” for biased states are not predominantly positive.\nWe have rotated the x-axis labels appropriately, so they are now correctly displayed.\nFig. 3d (transition bias) is not mentioned anywhere in the main text and could use some\nexplanations (the same for Fig. S6).\nWe explained both in the paper.\nFig.4a is somewhat confusing. We suggest adding axis labels in each panel and clearly marking\nthem as “example session 1”, “example session 2”, etc.\nWe edited the labels to make the figure clearer.\n“Stimulus side,” as explained in the manuscript (#124-125) is equivalent to what’s commonly\nknown as “win-stay/lose-switch.” We were wondering why the former term was adopted and not\njust called the latter. For this variable, isn’t there no relation with specific stimulus side as the\nmice can WS-LS on either side?\nWe thought that win-stay/lose-switch referred to the strategy itself, but \"Stimulus side\" is actually\nthe name of the regressor that influences it. If the weight was negative, the strategy would\ninstead be win-switch/lose-stay. So we have chosen ‘Stimulus side’ as the name of the\nregressor.\nIn line #367-368, “solid black line” should be “solid red line” according to Fig. 6c figure caption.\nGood catch! We just edited the words.\nLine #388-389 states: “For the transition model, filtered choice, stimulus side, and reward were\nthe most influential covariates, ….” Related to the comment above, this statement could be\ntoned down a little because these factors are assumptions that were pre-selected for the model\nrather than those that survive among other candidate covariates.\nThank you for the suggestion—we have revised the statement to clarify this.\nThe Methods section (Section 5) for describing the dataset and the exclusion criteria could be a\nlittle more detailed. It seems like the current study is selecting the same 37 mice (out of 123)\nthat were used in the previous study by Ashwood et al. (2022), which we found out only after\nreferring to it for more information. The authors could either explicitly state this or make the\nMethods more self-contained, since the readers might not be aware of the previous work.\nThanks, we have added the explanation to Section 5 to make the dataset selection criteria more\nexplicit and self-contained.\nLe et al., which is cited in #36, has been published in PLOS Comp. Bio.:\nLe, N. M., Yildirim, M., Wang, Y., Sugihara, H., Jazayeri, M., & Sur, M. (2023). Mixtures of\nstrategies underlie rodent behavior during reversal learning. PLOS Computational Biology,\n19(9), e1011430.\nSimilar to above work, it might be worth also citing the following study, which applies HMM to\ndecision making:\nLi, J. J., Shi, C., Li, L., & Collins, A. G. (2024). Dynamic noise estimation: A generalized method\nfor modeling noise fluctuations in decision-making. Journal of Mathematical Psychology, 119,\n102842.\nThanks for pointing this out. We have edited reference number 36 and added the recommended\nrelated reference (number 37).\n------------------------------------------------------------------------------\nReviewer #2 (Remarks to the Author):\nIn the present manuscript, the authors present an adapted hidden Markov model that allows for\nvarious covariates from the experiment to guide the transitions between decision making states,\nand (potentially) separate covariates to guide the decision making ‘policy’ within those different\nstates. They then apply their model to some rat perceptual learning data in which the probability\nof left or right decisions (i.e., a bias) should increase or decrease across blocks of trials. They\nfind some apparent evidence of a four state model in which left or right bias appears to be\ncrossed with engaged or not (so engaged left, engaged right, left bias, right bias). I think this is\nan interesting manuscript in that it develops a useful statistical model to capture important\ntrends in the decision making style across trials. These HMMs with transitions across decision\nstates are conceptually easy but very difficult to fit to data. Adding to this complexity is the\npossibility of four different sets of GLM weights to predict choices. Finally, the authors use a\nnice data set to illustrate the utility of their model – not only is there left and right biases, but these\nare very long experiments that would almost certainly catch the rat off task from time to time\n(and as their model suggests, they do engage in off task behavior).\nWe appreciate the reviewer’s thoughtful summary of our work and their recognition of the\ncomplexity involved in fitting HMMs with transition-dependent decision-making states, especially\nto this large dataset collected across several labs, which adds to the complexity. The reviewer\nhighlights several key strengths of our approach, including its ability to capture distinct\ndecision-making states and the challenge of modeling transitions across these states in a\ndata-driven manner. We particularly appreciate the reviewer’s recognition that our method is\nwell-suited for long experimental sessions, where animals may disengage or shift strategies\nover time. Our model is designed to address exactly these kinds of behavioral dynamics, which\nwould be difficult to capture using traditional, fixed-transition HMMs.\nAdditionally, we acknowledge the reviewer’s point that incorporating different models with\ndifferent sets of GLM weights adds further complexity to model fitting. However, we view this\ncomplexity as a necessary step to ensure that the transitions between decision-making states\nare informed by meaningful, behaviorally relevant covariates rather than being treated as a\nstatic, unchanging process. By explicitly modeling both the transitions between states (GLM-T)\nand decision policies within each state (GLM-O), we provide a flexible framework that allows us\nto infer how behavioral states evolve dynamically. This separation enables us to distinguish\nfactors influencing long-term strategy shifts from those affecting moment-to-moment choices.\nAll of that said, the novelty of the research seems to be a bit low for this particular outlet in my\nopinion. According to the authors, this division between two separate generalized linear model\nstructures to guide both the transition between the states and the decisions within those states\nis the main novelty of their research. To me, the main contribution of their modeling efforts is that\nthey allow for two very general (literally GLM) structures to modulate the two main components\nof their model, one being a transition among the states and one being the decisions within those\nstates. While this is interesting, methodologically speaking this is not a significant leap forward.\nFurthermore, the evidence for a need of a separate transition GLM seems quite low. Maybe it is\njust a scaling issue, but Figure 2a is not really convincing that having a transition model is\nactually necessary.\nWe thank the reviewer for raising these points. While HMM-based approaches to\ndecision-making have been previously explored, our work extends these models by dedicating\ntwo distinct GLMs to separately capture within-state decision-making and between-state\ntransitions. Specifically, we use a multinomial GLM to model state transitions, allowing the\nprobability of switching between strategies to depend on past behavior and external covariates,\nand a Bernoulli GLM to model the stimulus-dependent decision process within each state. This\ndual-GLM architecture allows us to fully capture the decision-making process,\nsimultaneously estimating the animal’s decision state and determining how strategies evolve\ndynamically in response to changes in the environment. This framework also improves state\ninference confidence that indicates more reliable state identification, and explicitly accounts for\nengaged and disengaged states, which provides a richer description of real-world behavioral\ndynamics.\nIn addition, we conducted systematic validation using synthetic data, showing that the model\nonly detects dynamic transitions when they actually exist, further confirming its utility.\nThis separation is particularly valuable for modeling behavior in dynamic environments, where\nthe decision-making process is shaped not only by sensory inputs but also by the animal’s\nevolving internal state and memory of recent experiences. Importantly, decision-making in such\ndynamically changing environments has not been extensively studied, and incorporating the\nstatistics of environmental changes into the model adds another layer of novelty.\nFurthermore, using 5-fold cross-validation, we show that this model significantly outperforms\nsimpler models, such as one-state GLM-HMM (so a GLM), demonstrating the necessity of\nincorporating different states and two separate GLMs to fully capture the richness of the\ndecision-making process. This empirical validation highlights that the proposed framework is not\njust a conceptual extension, but a practical improvement that provides deeper insights into\nthe behavioral strategies employed by animals in adaptive environments.\nThe reviewer suggests that the need for a separate transition GLM is not well supported and\nthat Figure 2a does not convincingly demonstrate its necessity. To further illustrate its impact,\nwe have conducted additional analyses comparing models with and without GLM-T and\nhave added figures that show how inferred state probabilities change when transition\nprobabilities are fixed versus dynamic. These analyses clearly demonstrate that when\ntransition probabilities are allowed to vary based on behavioral history, the model provides a\nbetter fit to the data and more interpretable latent states.\nRegarding Figure 2a, we thank the reviewer for raising this point — it is indeed a scaling effect,\nand we have now revised the main text to explicitly explain how even small differences in\nlog-likelihood per trial, when accumulated across many trials, translate into substantial\nimprovements in overall model fit. This clarification highlights the importance of the transition\nGLM (GLM-T) and its contribution to capturing dynamic behavioral changes, which may not be\nimmediately apparent due to the compressed scale shown in Figure 2a. Even a very small\nimprovement in log-likelihood per trial can result in a much better model fit (refer to Eq. 32 in the\nMethods, but in this case comparing models with and without GLM-T instead). For example, for\nthe 4-state model, the model with GLM-T is approximately 0.015 bits per trial higher than the\nmodel without GLM-T. This means that for a dataset with 400 trials, the data would be\nabout 26≈64 times more probable under the GLM-T than without it. For a dataset with 4000 trials, the\ndata would be about 260≈1.15×1018 times more probable under the GLM-T model. We thank the\nreviewer for helping us clarify this point, we have added this explanation to the main text, where\nFigure 2a is explained, to make this clearer.\nTo directly address the reviewer’s question about the novelty and whether the transition GLM\n(GLM-T) is necessary and whether it provides clear added value, we highlight several points\nand present both new and existing analyses, as described below.\n1) First, we conducted a model comparison using two synthetic data generated (sampled) from\na model without GLM-T (4-state) and from a 4-state model with GLM-T (right side of the figure).\nAs shown in the figure below (and described in more detail in our response to\nReviewer 1), this comparison demonstrates that the GLM-T substantially improves model\nperformance when the true underlying process involves trial-dependent transitions. Importantly,\nwhen the data are generated from a model without GLM-T, the two models perform similarly,\nconfirming that GLM-T does not overfit or artificially inflate performance when trial-by-trial\nmodulation of transitions is unnecessary. This improvement of 0.009 bits per trial when the data\nhas trial-dependent transitions means that for a dataset with 5000 trials, the data would be\napproximately 3.52×1013 times more probable under the GLM-T model than under the\nnon-GLM-T model. This provides direct evidence that the GLM-T is both useful and\nnecessary when state transitions are indeed dynamically influenced by external factors, as is\nthe case in our behavioral data.\n2) Second, the model with GLM-T identifies latent states with higher probabilities compared to\nthe model without GLM-T, as shown in the figure below (left: scatter plot, right: density plot). This\ndirectly relates to the reviewer’s concern about whether the transition GLM (GLM-T) is\nnecessary. By assigning higher probabilities to the inferred states, the GLM-T provides more\nconfident and more stable state estimates, showing that trial-dependent transitions improve the\nability to uncover the underlying behavioral states. This provides direct evidence that allowing\ntransitions to depend on external covariates (such as trial history) makes the model\nbetter at tracking the animal’s true internal state.\n3) Third, as shown in Fig. 4a and 4c, the model with GLM-T captures the dynamics of state\nswitching much more effectively, reflecting the ability of trial-dependent transitions to track shifts\nin behavioral strategies over time. This further demonstrates that allowing transitions to depend\non external covariates enhances the model’s ability to uncover meaningful temporal structure in\nthe data.\n4) Additionally, we have included a response time plot, which shows that incorporating GLM-T\nleads to a greater difference in the 90th percentile of response time between disengaged and\nengaged states for all animals. This enhanced separation demonstrates that the model more\neffectively captures behavioral dynamics and engagement-related variability in response times.\nFor more details, please see our response to Reviewer 1 (section 4).\n5) Furthermore, the below figure provides further evidence supporting the link between satiety,\nreward history, and disengagement, which we uncovered by incorporating GLM-T into our\nmodel. The left panel illustrates a positive correlation between filtered reward and the\nprobability of being in a disengaged state, reinforcing the idea that as animals accumulate\nmore rewards, they tend to disengage from the task. This trend aligns with previous findings\nsuggesting that increased satiety leads to behavioral disengagement. The red trend line\nhighlights this relationship, showing a moderate but consistent increase in disengagement\nprobability with higher filtered reward.\nThe right panel further supports this link by showing the average probability of engaged and\ndisengaged states over the course of a session, with all sessions interpolated to 100 points and\naveraged across all animals. Initially, animals transition from disengaged to engaged states,\nindicating a warm-up period before reaching a stable level of task engagement. However, as\nthe session progresses, disengagement probability steadily increases while engagement\ndecreases, particularly toward the end of the session. This pattern aligns with the interpretation\nthat prolonged task engagement and increasing reward accumulation contribute to\ndisengagement, likely due to satiety effects, and further supports our findings.\nBy incorporating GLM-T, our model was able to reveal these structured, trial-dependent state\ntransitions, a novel achievement compared to traditional fixed-transition models. This approach\nallows us to capture how behavioral states evolve dynamically over time and provides a\nricher understanding of the interplay between reward history, disengagement, and satiety.\n6) Additional analysis of data with bias blocks (Figure S7b) and synthetic data analysis for both\nmodels, with and without GLM-T (Figure S5), demonstrate that GLM-T effectively captures the\ndynamics related to bias block data and the environment dynamics. This provides further\nevidence that incorporating GLM-T significantly enhances the model’s ability to comprehensively\nrepresent behavioral dynamics.\nAlthough the results are also interesting, it is perhaps not that surprising that they would find a\nmodel that supports the two real conditions of the experiment (left or right bias) crossed with\n‘engaged’ or ‘not engaged’.\nWe thank the reviewer for raising this point. While the existence of left/right bias and\nengagement states naturally reflects the external structure of the task, the core novelty and\npower of our model lies in its ability to reveal the internal structure of the animal’s\ndecision-making process — something that cannot be reduced to these simple task-defined\ncategories. Our approach is not limited to labeling trials as biased or engaged; rather, we\nexplicitly assess how past choices, sensory evidence, trial history, and reward feedback\ndynamically influence both within-state decision policies and the transitions between\nstates themselves. This allows us to track the full temporal evolution of strategy switching,\ncapturing subtle forms of sequential dependence and behavioral adaptation that would be\nentirely invisible in a simpler model with static transitions, or in approaches that only\nlabel strategies as either engaged or disengaged.\nImportantly, the states we identify are discovered from the data itself, not imposed by the\nmodel. It happens that in this task, these states strongly correlate with left/right bias and\nengagement because these reflect real behavioral strategies used by the animals — but the\nframework is fully general and could identify different states in different tasks (so not\nonly limited to disengaged or engaged labels), such as states driven by sensory uncertainty,\nmotivational changes, fatigue, or exploration-exploitation trade-offs. In fact, this flexibility is\none of the most valuable aspects of the approach: the same model structure can adapt to\nuncover distinct behavioral motifs across different experimental contexts. This makes the model\nbroadly useful and enables discovery-based behavioral analysis rather than simple\nclassification.\nFinally, while we respect the reviewer's perspective, decision-making is far more complex than a\nsimple binary distinction between ‘engaged’ and ‘not engaged.’ It is a complex, dynamic process\nshaped by multiple interacting factors, including sensory information, biases, memory of past\noutcomes, and evolving internal motivation. By combining a flexible state-space representation\nwith dynamic, covariate-driven transitions, our model provides a uniquely powerful tool to\nstudy how internal decision states evolve over time, and how they respond to the\nstatistics of the external environment. This is particularly critical in naturalistic and adaptive\nbehaviors, where decision-making cannot be understood as a simple, isolated process but as a\ncontinuously evolving dialogue between internal strategies and external contingencies.\nFor these reasons, we emphasize that our approach is much more than a slight refinement of\nleft/right bias and engagement labels — it is a complete modeling framework that provides a\nricher and more biologically realistic view of how decisions unfold in complex, dynamic\nenvironments.\nTo be honest, it was a little unclear what the inputs were to the GLM observation, despite a\nwhole section for it in 4.5. For example, they say that ‘bias’ was in the set of observational\ncovariates, but it is not defined in 4.5. It is hard to understand how a variable called ‘bias’ could\nmeaningfully contribute to the fits if there are already regressors for the stimulus itself and the\npast choice and previous stimulus.\nWe appreciate the reviewer’s comments regarding the inputs to the GLM observation model\n(GLM-O) and understand that the role of bias was not clearly defined in Section 4.5. The\nreviewer is correct that, without a clear definition, it can be difficult to interpret how \"bias\"\ninteracts with other covariates such as past choice and stimulus history. To clarify this for\nreaders, we have revised Section 4.5 to explicitly describe bias as a separate regressor, whose\nrole is to capture the animal’s inherent left/right preference that is present even in the absence\nof any stimulus or trial history effects. This type of bias term (sometimes referred to as an\n\"offset\") is standard in logistic regression and GLM-based behavioral models, where it\nrepresents a baseline tendency toward one choice, independent of external influences.\nImportantly, this term does not \"compete\" with stimulus, past choice, or previous stimulus\ncovariates — instead, it captures unexplained side preference that remains after accounting for\nthose external factors.\nTherefore, we have divided Section 4.5 into two subsections to improve clarity and added the\nfollowing explanation to explicitly describe bias as a separate regressor:\n“Finally, \"bias\" is a covariate that captures the animal’s inherent tendency to choose left\nor right, representing an internal preference or offset that exists independently of\nexternal factors such as the stimulus or choice history, and its value was set to 1 for all\ntrials.”\nIn any case, I would worry that they are finding evidence for a four state model simply because\nthe models that explain decisions are far too simplistic. It is possible that a more detailed model\nof the decision itself might imply that there are actually fewer states. For example, potentially\neven having a regressor for ‘number of trials performed in this block’ might soak up a lot of\nwithin-state variability to remove the need for two of those four states. Because questions like\nthis were not really fully examined, the results surrounding the transition GLM’s contribution are\npretty minimal. However, even if they were significant, it is not clear that an incremental\nmethodological advancement in a well established field of HMMing makes sense at this\nparticular outlet.\nThank you for raising this point. Our decision (observation) model design is aligned with similar\nliterature [31-34], and we respectfully do not agree with the reviewer that it is overly simplified.\nWe carefully selected reasonable covariates to ensure the model appropriately captures\nbehavioral complexity. We believe this model is well-suited to capturing the richness of\ndecision-making, particularly in adaptive environments where behavior reflects not only\nimmediate sensory inputs but also sequential history and environmental context.\nFurthermore, our model selection process itself was guided by rigorous 5-fold cross-validation,\nensuring that the choice of the model was based on empirical evidence rather than subjective\npreference. In our case, the GLM observation model (GLM-O) already incorporates stimulus,\npast choice, past stimulus, and bias, explicitly accounting for both sensory-driven decisions and\nsequential effects like choice and reward history — factors known to strongly influence behavior\nin perceptual decision-making tasks. This level of detail in the observation model reduces\nthe risk of spurious state discovery caused by missing within-state structure.\nMoreover, the four states identified by our model are not arbitrary categories introduced to\nartificially improve fit, but rather emerge consistently across animals and sessions, with\nclear behavioral interpretation. Importantly, the transitions between these states align with\ntask structure (block switching) and show systematic modulation by trial history covariates such\nas past reward and choice. This coherence between inferred states, task design, and\nbehavioral interpretation provides strong evidence that the four-state solution reflects\nmeaningful latent cognitive strategies, rather than compensating for a weak within-state\nmodel.\nFinally, we respectfully disagree with the characterization of this work as an \"incremental\nmethodological advance.\" While GLM-HMMs have been used before, the combination of\ndynamic, covariate-dependent state transitions (GLM-T) with a behaviorally rich observation\nmodel (GLM-O) is novel in its ability to capture how animals dynamically adapt strategies in\nnon-stationary environments — something that is essential for studying behavior in tasks with\nevolving contingencies. Also, this framework is broadly applicable beyond the specific task\nstudied here and offers a flexible, interpretable tool for understanding adaptive behavior\nacross diverse experimental paradigms. This conceptual and practical advance, supported\nby both synthetic and real data, goes well beyond fitting a better curve and provides\nnovel insights into how animals dynamically structure their behavior.\nIn summary, the four-state structure reflects a biologically interpretable hierarchy of behavioral\nmodes, supported by the data, the task design, and established principles of\ndecision-making behavior — not a byproduct of an overly simplistic within-state model.\nWe appreciate the opportunity to explain this key aspect of our approach. But we appreciate the\nreviewer’s point that adding more covariates could provide additional clarity and enhance the\nmodel’s interpretability (in addition to GLM-T and GLM-O). We have included a few lines in the\nfuture work section to address this.\nFinally, we have organized our response into three sections based on the reviewer’s concerns:\n1. The reviewer is concerned that the four-state structure might result from an insufficiently\ndetailed observation model rather than reflecting genuine behavioral states. The four-state\nmodel is empirically justified and emerges naturally from the data rather than being imposed\ndue to a limited GLM-O. Additionally, as mentioned, a 5-fold cross-validation was performed,\nensuring that the model selection was based on empirical evidence rather than subjective\npreference. Another important point is that incorporating GLM-T allows us to gain deeper\ninsights into state transition patterns—such as the effects of different covariates, their\nhistorical influence on transition probabilities, and how states evolve—without manipulating\nthe observation model.\na) Figures comparing state occupancy patterns in 4-state models with and without GLM-T, for\nboth real and synthetic data, demonstrate that with GLM-T, states align meaningfully with\ntask biases, whereas without it, state occupancies lack structure. To further address the\nreviewer’s question, we plotted the fractional occupancy of the four discrete states in the model\nwithout GLM-T for all IBL data (similar to Fig. 4c). This analysis was conducted across all trials,\ncovering both right-biased and left-biased blocks, with results displayed separately for each\nblock type. As shown in the plot, the background stimulus bias was not captured by the model\nwithout GLM-T.\nIn addition, please see the figure below, which is a supplementary figure from the paper\nemploying synthetic data. This figure illustrates the fractional occupancy of different states for\nmodels with and without GLM-T, separated by left-biased and right-biased task conditions. The\ncomparison shows that in a 4-state model when GLM-T is included (panel a), the state\noccupancy distributions adapt more distinctly to task biases, whereas without GLM-T\n(panel b), the state occupancies appear without being affected by data bias blocks. This further\nsupports that the inclusion of trial-dependent transitions allows the model to capture dynamic\nshifts in behavioral states.\nb) Additionally, posterior state probabilities across trials indicate that states inferred under\nGLM-T are assigned higher probabilities (Figure S9), demonstrating that the states are\nwell-defined and not arbitrary. Figures comparing scatter plots of inferred state probabilities\nshow that states remain well-separated and confidently assigned under GLM-T, which would\nnot be the case if the model were unnecessarily splitting states.\nFurthermore, as mentioned in the previous response, the state probability plot comparing the\nmodel with GLM-T to the model without GLM-T shows that inferred states under GLM-T exhibit\nhigher certainty and structure. This suggests that the model is detecting meaningful\nbehavioral states rather than artificially splitting states.\n2. The reviewer suggests that adding a within-state regressor could absorb variability that\ncurrently leads to the four-state model, potentially making fewer states sufficient. The four-state\nmodel captures discrete behavioral shifts that a continuous regressor like \"number of trials in\na block\" could not. To illustrate this, we plotted the transition probabilities for both models—with\nand without GLM-T—to highlight their differences over time. The plots show that incorporating\nGLM-T allows transition probabilities to vary dynamically from trial to trial, adapting to\ndifferent states and conditions in a way that a simple within-state regressor could not\ncapture. In contrast, without GLM-T, transition probabilities remain constant, failing to reflect\nbehavioral changes. This comparison underscores that the extended model effectively captures\nadaptive state transitions, distinguishing it from the standard GLM-HMM, which assumes fixed\ntransitions that do not evolve based on trial history or task structure.\nAdditionally, the response time plot referenced in the previous response further supports this by\nshowing better state separation between disengagement and engagement when GLM-T is\nincluded. Collectively, these analyses demonstrate that without GLM-T, state transitions lack\nstructure, providing evidence that the four-state model with GLM-T captures genuine\nbehavioral shifts rather than compensating for missing within-state regressors.\nFinally, please consider that GLM-T estimates transition probabilities between latent states,\nmeaning it determines how an animal switches between different behavioral modes. These\ntransitions are influenced by multiple covariates, including past choices, rewards, and task\nstructure, allowing for adaptive, history-dependent behavioral shifts. In contrast, \"number of\ntrials in a block\" is a continuous regressor that only tracks the progression of time within a\nblock and affects decision weights within a single state. It assumes that changes in behavior\noccur gradually within a state, rather than through distinct transitions between states. Thus,\nwhile \"number of trials in a block\" might account for a small portion of choice behavior\nchanges within a given state, it fails to capture the actual switching between states,\nwhich is the core function of GLM-T. Our model is significantly more powerful, as it not\nonly tracks within-state decision dynamics but also accurately models trial-dependent\nstate transitions, allowing for a much richer and behaviorally meaningful representation\nof adaptive decision-making.\n3. The reviewer questions whether adding a separate transition GLM is a meaningful innovation\nor just a minor refinement of existing HMM approaches. GLM-T provides a fundamentally\nimproved framework for capturing behavioral state transitions, validated through both synthetic\ndata and real experimental results. Figures from the model comparison using synthetic data\ndemonstrate that GLM-T improves performance only when trial-dependent transitions exist,\nconfirming its necessity. Additionally, log-likelihood improvement analyses show that even small\nper-trial gains translate into substantial benefits in large datasets. This was previously\nmentioned, but we reiterate it here for clarity. As shown in the figure below, an improvement of\n0.009 bits per trial when the data contains trial-dependent transitions means that for a dataset\nwith 5000 trials, the data would be approximately 3.52×1013 times more probable under the\nGLM-T model than under the non-GLM-T model.\nThis analysis validates that GLM-T improves model performance only when trial-dependent\ntransitions are present, demonstrating that it captures meaningful behavioral dynamics\nrather than serving as a minor refinement. This suggests that transitions between latent\nstates are not merely stochastic but are systematically influenced by past experiences,\nreinforcing the necessity of modeling them with a flexible, input-driven approach.\n------------------------------------------------------------------------------\nReviewer #3 (Remarks to the Author):\nThe authors investigated behavioral strategies used by mice performing a binary\ndecision-making task with a time-varying distribution over correct choices. To do so, they fit two\nsets of GLMs: one that predicts observed choices as a function of sensory inputs, task\ncovariates, and latent state, and another that predicts transition probabilities to each latent state.\nCrucially, this setup allows the transition probabilities to vary over time, in contrast with\ntraditional HMMs, and thus is a sensible choice for analyzing behavior in a non-stationary\nenvironment. In sum, they find four interpretable states that govern behavior during this task,\nwhich are broken down into “engaged” states, in which animals appear to rely on the observed\nstimulus, and “biased” states, in which animal’s choices seem heavily biased towards one\ndirection regardless of stimulus. They characterize various task-relevant factors that influence\ntransitions between these states, and provide a detailed account of how transition rates are\nmodulated over time by the underlying evolving task structure.\nStrengths: The authors do an excellent job demonstrating that the behavioral data is\nwell-explained by their proposed four-state model, and their interpretation of these states is\nsolidly supported by breakdowns of GLM weights and psychometric curves.\nWe sincerely appreciate the reviewer’s thoughtful feedback and recognition of our modeling\napproach. Your comments on the interpretability of the identified states and our framework’s\nability to capture behavior in a non-stationary environment are highly encouraging.\nConcerns/Questions:\nThe relationship between this work and the cited “Extracting the dynamics of behavior in\nsensory decision-making experiments” (Roy et al., 2021, Neuron) should be explicitly discussed.\nIn particular, both make inferences about animal behavior from the same dataset, and both\nemploy a fitting method that permits a time-varying observation model. The authors should\nmake an effort to underscore what findings are uniquely highlighted by their GLM-HMM\napproach that couldn’t be captured by the model used there.\nThank you for this point. While both our study and Roy et al. (2021) analyze behavioral\nstrategies in decision-making tasks using time-varying models for observation, the scope,\nmethodology, and focus of the two approaches differ significantly. Roy et al. (2021) focused on\nlearning dynamics, applying PsyTrack to track smooth, trial-by-trial fluctuations in\ndecision-making weights, such as sensory evidence weighting. Their method is particularly\nwell-suited for studying how decision strategies evolve as animals learn the task and adapt to\nchanging task statistics. PsyTrack continuously adjusts the model parameters over time,\ncapturing gradual learning-driven changes in decision-making behavior. They used the\ntraining data to model learning.\nIn contrast, our study examines behavior after learning has stabilized, focusing on how trained\nmice dynamically make decisions or transition between distinct latent behavioral states\nduring task performance. Rather than modeling continuous weight updates, we apply a\nGLM-HMM framework that infers discrete states (e.g., engaged vs. biased states) and analyzes\nhow transitions between these states are influenced by sensory inputs, past choices, and task\nstructure. This discrete-state representation allows us to investigate abrupt strategy shifts,\nproviding insight into how trained animals dynamically engage with or disengage from the task.\nAlso, prior work (Zoe et al., 2022, Nature Neuroscience) has shown that mice use discrete,\nrather than continuously evolving, decision-making states (GLM-HMM achieved\nsubstantially higher test log-likelihood than the PsyTrack model), further motivating our use\nof a GLM-HMM framework instead of a smooth-tracking approach like PsyTrack.\nIn terms of methodology, Nick Roy’s model employs a dynamic Bernoulli GLM, without any\nHMM, where psychophysical weights evolve smoothly over time to track how\ndecision-making strategies change from trial to trial. This approach allows the model to capture\ngradual adaptation by adjusting the influence of task variables (e.g., sensory stimuli, biases,\nchoice history) on choices. However, it lacks an explicit latent state structure, meaning it does\nnot model discrete shifts in behavioral strategies that might arise due to factors such as\nengagement level or block transitions. In contrast, our method integrates a GLM-HMM\nframework, which introduces latent behavioral states and allows for history-dependent state\ntransitions. Our model consists of an HMM and two separate Bernoulli and multinomial\nGLMs: GLM-O (observation model), which governs within-state decision-making based on task\nvariables, and GLM-T (transition model), which predicts how and when animals switch between\ndifferent decision-making states based on past choices, rewards, and other covariates. This\ndistinction is crucial, as it enables our model to capture both gradual within-state adaptation and\ndiscrete shifts between cognitive states, which a standard dynamic GLM like Roy’s cannot.\nAdditionally, our framework allows for trial-dependent transition probabilities, making it more\nsuitable for analyzing adaptive, non-stationary behaviors where animals dynamically shift\nbetween strategies in response to the task structure.\nThis distinction enables us to characterize global shifts in decision-making behavior, offering a\nnovel framework for understanding structured strategy adaptation in well-trained\nanimals—something that may not be fully captured by a smooth-tracking approach like\nPsyTrack. Importantly, our approach allows for discovery of latent behavioral states directly\nfrom data, rather than only tracking pre-specified weights. We have clarified these differences in\nthe manuscript (in introduction with orange color) and appreciate the opportunity to better\narticulate the unique contributions of our approach. We have provided a comparison table as\nbelow to address the different features of both methods more clearly.\nFeature PsyTrack (Roy et al., 2021) Our Model\nMain Goal Track continuous learning dynamics Capture discrete behavioral states and dynamic\nover training transitions during stable behavior after learning\nType of Model Dynamic GLM with time-varying Hidden Markov Model with Bernoulli GLM for\nweights within-state choices and multinomial GLM for\ntransitions\nFocus Evolution of psychophysical State switching (strategy shifts) and within-state\nweights across trials during training choice policies during steady-state behavior\nWhat is modeled Continuous weight trajectories that Discrete latent states (e.g., engaged, biased) and\nevolve smoothly across trials probabilistic transitions between them\nState Transitions No explicit states, so no transitions Explicit state transitions modeled via a multinomial\nGLM (GLM-T) informed by past trials\nTrial history Captured indirectly through smoothly Directly modeled as covariates in GLM-T (e.g., past\neffects evolving weights reward, past choice)\nFlexibility Can track arbitrary weights if Can discover distinct behavioral states, each with its\nincluded as covariates own decision policy, allowing for richer interpretation\nApplication Ideal for early learning and tasks Ideal for adaptive decision-making in stable, dynamic\nfocus with gradual adaptation environments after learning completes\nIn order to address the reviewer's question, we present a few additional analyses here:\n1. GLM-HMM Captures Discrete Behavioral States, Unlike Psytrack’s Continuous Weight\nEvolution: A key distinction between our GLM-HMM approach and Nick Roy’s Psytrack-based\nmodel is that our framework explicitly infers discrete behavioral states, rather than assuming\ndecision-making changes occur as a continuous weight evolution. To better address this, we\nexamined the fractional occupancy of the four discrete states in the model without GLM-T for all\nIBL data, similar to Fig. 4c. This analysis was conducted across all trials, covering both\nright-biased and left-biased blocks, and results were displayed separately for each block type.\nAs seen in the plot, the background stimulus bias was not captured by the model without\nGLM-T, demonstrating that without explicit state transitions, the model fails to correctly align\nbehavioral states with task structure. This reveals a fundamental limitation of Psytrack: much\nlike a GLM-HMM without GLM-T, it tracks gradual changes in decision weights but fails to infer\nstructured latent states that represent discrete shifts in decision-making strategies. In contrast,\nGLM-HMM provides a richer representation by capturing discrete cognitive states and\ntheir transitions, which are essential for understanding dynamic behavioral patterns in\nnon-stationary environments.\n2. GLM-T Captures Trial-Dependent Transitions That Psytrack and Standard GLMs\nCannot: GLM-T provides a fundamentally improved framework for modeling behavioral state\ntransitions, as validated through both synthetic data and real experimental results. Unlike\nPsytrack, which assumes gradual, continuous changes in decision-making behavior, GLM-T\nexplicitly models trial-dependent state transitions, allowing it to capture discrete shifts\nbetween behavioral strategies rather than smooth weight updates.\nWe compared model performance using synthetic datasets generated from two different\nsources: one sampled from a model without GLM-T (left side of the figure) and another from a\n4-state model incorporating GLM-T (right side of the figure) . As illustrated in the figure below\n(and further detailed in our response to Reviewer 1), this analysis confirms that GLM-T provides\na significant advantage when trial-dependent transitions are present in the underlying process.\nHowever, when the data originate from a model without GLM-T, both models perform\ncomparably, demonstrating that GLM-T does not introduce overfitting or artificially enhance\nperformance in cases where trial-by-trial transition modulation is unnecessary.\nAs previously discussed, an improvement of 0.009 bits per trial when the data contains\ntrial-dependent transitions means that for a dataset with 5000 trials, the data would be\napproximately 3.52×1013 times more probable under the GLM-T model than under the\nnon-GLM-T model. In contrast, Psytrack does not model state transitions at all, reinforcing that\nour approach offers a significant methodological advancement.\n3. GLM-HMM Provides More Confident and Structured State Estimates: We compared\nmodels with and without GLM-T, and the results show that the model with GLM-T assigns higher\nprobabilities to inferred states, as seen in the figure below (below scatter plot). This indicates\nthat GLM-T provides more confident and stable state estimates, reinforcing its ability to uncover\nmeaningful behavioral states. Our model captures structured, discrete state transitions better\nthan models without a transition component. In contrast, models without GLM-T, similar to\nPsytrack, lack a mechanism to explicitly model state-to-state transitions and instead\nassume that behavior changes gradually and continuously. If a purely continuous process\nwere sufficient, we would not observe such high-probability state assignments in the model with\nGLM-T.\nAdditionally, the response time analysis (Fig. 7a) further supports the behavioral relevance of\nthese inferred states, showing that the separation between engaged and disengaged states is\nclearer when GLM-T is included. This reinforces the idea that decision-making is not just a\ncontinuous adaptation but involves distinct transitions between cognitive strategies.\nTogether, these findings demonstrate that GLM-HMM with GLM-T provides a more structured\nand confident inference of behavioral states compared to models without an explicit transition\ncomponent, such as Psytrack.\nI am somewhat puzzled by why state transition probabilities are modeled by the transition GLM\nin the way they are. If I am understanding correctly, only the probabilities of transitioning to a\nparticular latent state are parameterized, regardless of starting latent state, for a total of K\ntransition probabilities (this interpretation is consistent with the exposition in the text as well as\nFig. 5). This is quite a restrictive choice compared to standard HMMs, which parameterize all\nK^2 pairs of from-and-to. In this case, I would expect that transitions “Engaged R -> Biased R”\nwould be significantly more probable than “Engaged L -> Biased R”. Indeed, other works that\nuse time-varying transition probabilities, such as the cited work Calhoun et al. 2019, fit a\nseparate GLM for the transition probabilities out of each latent state. I would like to see that the\nconclusions drawn are not overly dependent on this particular modeling choice.\nWe appreciate the opportunity to clarify this point. Our transition model does not impose a single\nset of transition probabilities across all states. Instead, we fit a separate multinomial GLM\n(GLM-T) for each latent state (see sections 4.3 and 4.4 of Methods), similar to Calhoun et\nal. (2019). However, a key difference in our approach is that the covariates for GLM-T are\nindependent from those used in the observation GLM (GLM-O), allowing to distinguish between\nfactors driving within-state choices and those influencing state transitions.\nThis framework ensures that transition probabilities remain state-dependent, meaning the\nprobability of entering a given state varies based on both the current latent state and external\ncovariates, such as past rewards, past choices, and session warm-up effects. By modeling\ntransitions with a multinomial GLM rather than a fixed K × K transition matrix, our approach\nnaturally captures asymmetries in state transitions, enabling behaviorally meaningful shifts\nbetween engaged and biased states. Furthermore, this design allows us to better model\ndecision-making in non-stationary environments, where animals must dynamically adjust their\nstrategies in response to changing reward contingencies. This level of flexibility is not present in\nstandard HMMs, which assume fixed transition probabilities over time.\nWhile directions for future work are discussed thoroughly, interpretations and caveats of the\npresented findings seemed a tad scarce in the discussion. For example, the authors allude to\noccupancy of disengaged states being related to satiety. The authors could expand on this,\npossibly by commenting on the robustness of this conclusion, especially when considering that\nthe delineation appears less clear for the 5 state model as shown in Fig. S2b.\nWe appreciate this thoughtful point and agree that further elaboration on the interpretation and\nrobustness of our findings would strengthen the discussion. Our results for 4-state show that\npast rewards are a predictor of transitions to disengaged states, aligning with the idea that\nsatiety may contribute to disengagement. Importantly, while satiety provides a compelling\nexplanation, disengagement may also incorporate other factors such as temporary reductions in\ntask engagement or shifts in decision strategy. We will clarify these points especially for the\n5-state model in the discussion to highlight the robustness of our findings while acknowledging\nthe broader behavioral context in which disengagement occurs.\nAlso, to address the reviewer’s question regarding the interpretation of disengaged state\noccupancy and its relationship to satiety, we provide additional analysis supporting this link. The\nleft panel of the figure below demonstrates a positive correlation between accumulated reward\n(filtered reward) and the probability of being in a disengaged state. We found a statistically\nsignificant positive moderate correlation between the filtered reward rate and the probability of\ndisengagement (r ≈ 0.30, p < 0.001), where r = 0.30 (Pearson correlation coefficient) indicates a\nmoderate positive correlation, and p < 0.001 (p-value) confirms that this relationship is highly\nunlikely to be due to chance. This trend suggests that as animals receive more rewards over\ntime, they are increasingly likely to disengage from the task, aligning with the hypothesis that\nsatiety drives disengagement. This result reinforces the robustness of our interpretation, as it\nis consistent with previous findings that reward accumulation influences behavioral state\ntransitions.\nThe right panel further supports this by illustrating the average probability of engaged and\ndisengaged states over the course of a session, with all sessions interpolated to 100 points and\naveraged across animals. Initially, animals exhibit a transition from disengaged to engaged\nstates, reflecting an early adaptation or warm-up period before stable engagement. However, as\nthe session progresses, the likelihood of disengagement increases, particularly toward the end\nof the session, while engagement probability decreases. This pattern is consistent with the idea\nthat extended task participation and accumulating rewards lead to disengagement, likely\ndue to satiety effects.\nMinor/miscellaneous:\nIs there a relationship between the model’s predicted probability of the animal being in a\nbiased/disengaged state during some time window and the number of incorrect choices\nobserved? The authors state that performance remains high even in the biased states because\nof imbalanced prior probabilities, but I don’t see this quantified anywhere. Reporting a\nprevalence-independent performance metric like balanced accuracy could also be insightful\nhere.\nWe thank the reviewer for their insightful comment. To address this, we present the normalized\nerror rate across all sessions of all animals for both engaged and disengaged states. While the\nerror rate in the disengaged states is slightly higher than engaged states, it remains relatively\nlow overall, demonstrating that the model continues to perform well even in these periods. The\nuse of a normalized error rate ensures that differences are not driven by variability in trial\nnumbers, providing a clearer comparison. This analysis directly reflects the relationship between\nan animal being in a biased/disengaged state during the experiment and the rate of incorrect\nchoices observed. Our results support the explanation that performance remains high in the\ndisengaged state due to imbalanced prior probabilities, as errors do not increase drastically and\nprovides meaningful quantification of state-dependent performance, reinforcing our\ninterpretation.\n------------------------------------------------------------------------------\nReviewer #4 (Remarks to the Author):\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is\npart of the Nature Communications initiative to facilitate training in peer review and to provide\nappropriate recognition for Early Career Researchers who co-review manuscripts.\nThank you for your contribution to the review process. We appreciate the thoughtful feedback\nand the opportunity to improve our manuscript through this collaborative review.\nThank you again for your comments and helpful suggestions.\nSincerely,\nZeinab Mohammadi, Zoe C Ashwood, The International Brain Laboratory, and Jonathan W\nPillow"}, {"round": 2, "reviewer_comments": {"reviewer_1": "We would like to thank the authors for thoughtfully and thoroughly addressing our concerns. Below, we have outlined a few minor final points.\nRebuttal point 3:\n\"... This result highlights that our model, which integrates both GLM-T and GLM-O, governs how\nbehavioral states evolve over time while capturing choice behavior patterns, whereas GLM-O\nalone defines within-state choice behavior. The consistency of GLM-O weights across models\nfurther supports the idea that the transition process (GLM-T) does not interfere with\nmoment-to-moment decision weightings but instead regulates long-term behavioral shifts. We\nhave added this plot to the supplementary materials.\"\nThank you for running this additional analysis. We now see that the GLM-O weights do not really differ between the models with or without the GLM-T component. We were wondering which part of the main text (if any) refers to these results. It might be worth explaining this point in a little more detail, because it was at first counterintuitive that the two weights should not influence each other.\nRebuttal point 4:\nThank you for following up with a comprehensive set of analyses. The result suggests that including GLM-T component leads to higher confidence in state assignments with distinct predictions for behavior such as RT and switching. The new figures were overall clear, with the exception of Fig.9c: the plots were too cluttered and the analysis here seems to rely on qualitative judgment. Could this be replaced, for example, with the entropy of the P(state) or showing just for the maximum probability instead, to avoid redundancy with panels (a)-(b)? Also we assume this is for an example session. The figure is also missing the legend for the colors of the plots.\nRebuttal point 5:\nThank you for testing this idea. We think the results are clear.\nRebuttal point 6:\nThank you for providing these extensive simulation results. The model recovery results and Fig.S10 were clear in demonstrating the utility of the transition model. We would just recommend clarifying the simulation conditions (e.g., parameters) for generating the synthetic data from two models in Methods, if this has not already been done.\nRebuttal point 7:\nThank you for the additional explanation.", "reviewer_2": "I don't really have any further comments. Although I appreciate the authors' efforts, the model they use here is more of a statistical approach (through the use of very general GLMs) and lacks a lot of theoretical interest, which is the type of approach I usually gravitate toward. The authors did a lot to justify their position, and while I appreciate their efforts, the manuscript remains in a similar state as the first version although some of the additional analyses are slightly more convincing statistically, it is still not clear to me whether the results are sort of guaranteed from the structure of the task. We continue to disagree about the novelty of their approach for this particular high-impact journal, but it is not up for me to decide so hopefully the AE can make a decision here.", "reviewer_3": "I thank the authors for their additional analyses and for the expanded discussions in response to my points. I have one remaining comment.\nRe: Response to “I am somewhat puzzled by why state transition probabilities are modeled by the transition GLM in…”\nIf a separate multinomial GLM is indeed being fit for each latent state, I would interpret that as meaning a separate set of K weight vectors is used for modeling the transition probabilities of leaving each state. However, Based on equation (2) and lines 90-94, as well as equation (12) and lines 607-611, the multinomial GLM used for state transitions parameterizes probabilities of transitioning to states with just one set of K weight vectors. What is still not clear here is how these probabilities take into account the starting state from which the state transition is occurring. Is starting state information somehow implicitly embedded in x_t^{tr}? If not, then the problem of indistinguishability of P(Engaged R -> Biased R) vs P(Engaged L -> Biased R) remains.\nThis point remains my biggest concern. If K separate multinomial GLMs are being fit for transition probabilities, the quoted equations/text should be updated to clarify this.", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts."}, "author_response": "Dear Reviewers,\nThank you once again for your thoughtful and constructive feedback on our manuscript. We\nappreciate the opportunity to address this second round of comments. Your insights have\nhelped us further refine the clarity and depth of the work. Below, we provide point-by-point\nresponses and describe the corresponding changes made to the manuscript.\nREVIEWER COMMENTS\nReviewer #1 (Remarks to the Author):\nWe would like to thank the authors for thoughtfully and thoroughly addressing our concerns.\nBelow, we have outlined a few minor final points.\nThank you for your careful review and constructive feedback. We have addressed the\nremaining points as outlined below and made the corresponding edits to the manuscript\nand figures.\nRebuttal point 3:\n\"... This result highlights that our model, which integrates both GLM-T and GLM-O, governs\nhow behavioral states evolve over time while capturing choice behavior patterns, whereas\nGLM-O alone defines within-state choice behavior. The consistency of GLM-O weights\nacross models further supports the idea that the transition process (GLM-T) does not\ninterfere with moment-to-moment decision weightings but instead regulates long-term\nbehavioral shifts. We have added this plot to the supplementary materials.\"\nThank you for running this additional analysis. We now see that the GLM-O weights do not\nreally differ between the models with or without the GLM-T component. We were wondering\nwhich part of the main text (if any) refers to these results. It might be worth explaining this\npoint in a little more detail, because it was at first counterintuitive that the two weights\nshould not influence each other.\nThank you for this suggestion. We agree that the relationship between the GLM-T and\nGLM-O components warrants clearer explanation in the main text. As noted in the rebuttal,\nthe consistency of GLM-O weights across models with and without GLM-T indicates that\nGLM-T governs transitions between latent behavioral states without altering state identities\nthemselves, as captured by GLM-O.\nTo clarify this, we have now added a sentence in the Results section (Lines 430–433):\n\"Furthermore, GLM-O weights were similar across models with and without the GLM-T\ncomponent (Supplementary Fig. S7a). This shows that the GLM-T regulates transitions\nbetween behavioral states without substantially modifying the states themselves, as described\nby the GLM-O.\"\nRebuttal point 4:\nThank you for following up with a comprehensive set of analyses. The result suggests that\nincluding GLM-T component leads to higher confidence in state assignments with distinct\npredictions for behavior such as RT and switching. The new figures were overall clear, with\nthe exception of Fig.9c: the plots were too cluttered and the analysis here seems to rely on\nqualitative judgment. Could this be replaced, for example, with the entropy of the P(state) or\nshowing just for the maximum probability instead, to avoid redundancy with panels (a)-(b)?\nAlso we assume this is for an example session. The figure is also missing the legend for the\ncolors of the plots.\nWe thank the reviewer for the helpful suggestion. We have revised Fig. S9c to reduce visual\nclutter by replacing the original plot with a plot of the maximum posterior probability over\ntime, which more directly reflects confidence in state assignments. As shown in the\nupdated panel, Fig. S9c now presents a direct comparison between the models with and\nwithout GLM-T. The model with GLM-T (solid purple line) mostly assigns higher confidence\nto latent state inferences than the model without GLM-T (dashed line), with confidence\nquantified as the maximum posterior probability per trial. This change avoids redundancy\nwith panels (a)–(b), which already depict the full state probability profiles. We have also\nclarified in the figure caption that the plot is based on an example session, added a legend\nfor state colors, and updated the caption accordingly.\nRebuttal point 5:\nThank you for testing this idea. We think the results are clear.\nWe thank the reviewers for their positive feedback. We are glad the results are clear and\nsatisfactory.\nRebuttal point 6:\nThank you for providing these extensive simulation results. The model recovery results and\nFig.S10 were clear in demonstrating the utility of the transition model. We would just\nrecommend clarifying the simulation conditions (e.g., parameters) for generating the\nsynthetic data from two models in Methods, if this has not already been done.\nWe thank the reviewers for their helpful feedback and are glad the simulation and model\nrecovery results were clear. In response to your suggestion, we have now added a\ndescription of the simulation conditions to generate synthetic data from the two models, as\ndescribed in the Methods section titled “Synthetic Data” (Section 4.9).\nRebuttal point 7:\nThank you for the additional explanation.\nWe appreciate the reviewer’s feedback and are glad the additional explanation was helpful.\nReviewer #2 (Remarks to the Author):\nI don't really have any further comments. Although I appreciate the authors' efforts, the\nmodel they use here is more of a statistical approach (through the use of very general\nGLMs) and lacks a lot of theoretical interest, which is the type of approach I usually\ngravitate toward. The authors did a lot to justify their position, and while I appreciate their\nefforts, the manuscript remains in a similar state as the first version although some of the\nadditional analyses are slightly more convincing statistically, it is still not clear to me\nwhether the results are sort of guaranteed from the structure of the task. We continue to\ndisagree about the novelty of their approach for this particular high-impact journal, but it is\nnot up for me to decide so hopefully the AE can make a decision here.\nWe thank the reviewer for their comments and for acknowledging the added weight of the\nadditional statistical analyses. We recognize that our statistical modeling approach differs\nfrom the kinds of theoretical models the reviewer might prefer, but we respectfully disagree\nwith the assertion that the results are guaranteed from the structure of the task. For\nexample, we do not feel it is obvious that mice would exhibit both engaged and disengaged\nstates with left and right biases, respectively. It might have been that their engaged states\nreflected a sensitivity to the left and right blocks, but that they had only a single disengaged\nstate with no bias to either side. Similarly, we felt it could have come out that the factors\ndriving transitions into the engaged states could have been an increase (rather than a\ndecrease) in recent reward. (Our colleagues who train animals have told us they sometimes\ntry to drive an animal back into an engaged state by giving them an extra large reward).\nMore generally, the fact that we can recover stable, interpretable states across animals —\nalong with consistent within-state decision weights and behaviorally relevant transition\ndynamics — represents a novel finding that we feel has not been predicted (to our\nknowledge) from previous normative accounts of behavior. We hope the editor will consider\nthis broader theoretical contribution, especially given the growing recognition of statistical\napproaches in modeling for decision-making. (Although we are very grateful to the reviewer\nfor their respectful expression of their difference in opinion about this issue!).\nReviewer #3 (Remarks to the Author):\nI thank the authors for their additional analyses and for the expanded discussions in\nresponse to my points. I have one remaining comment.\nRe: Response to “I am somewhat puzzled by why state transition probabilities are modeled\nby the transition GLM in…”\nIf a separate multinomial GLM is indeed being fit for each latent state, I would interpret that\nas meaning a separate set of K weight vectors is used for modeling the transition\nprobabilities of leaving each state. However, Based on equation (2) and lines 90-94, as well\nas equation (12) and lines 607-611, the multinomial GLM used for state transitions\nparameterizes probabilities of transitioning to states with just one set of K weight vectors.\nWhat is still not clear here is how these probabilities take into account the starting state\nfrom which the state transition is occurring. Is starting state information somehow\nimplicitly embedded in x_t^{tr}? If not, then the problem of indistinguishability of P(Engaged\nR -> Biased R) vs P(Engaged L -> Biased R) remains.\nThis point remains my biggest concern. If K separate multinomial GLMs are being fit for\ntransition probabilities, the quoted equations/text should be updated to clarify this.\nWe thank the reviewer for this comment, and we apologize for the confusion over this point.\nThere was a miscommunication between the authors during the previous round of review,\nand it appears that our previous response about the transition GLMs was incorrect. We\nsincerely apologize for this oversight!\nThe reviewer is correct that we used only K linear weight vectors for the GLM-T model, one\nfor the incoming transitions to each state. However, the probability of transition also\ndepended on an additive constant, one for each pair of states. Thus, the probability of\ntransitioning from state i to state j at time t, given covariates xtr , is modeled as:\nt\nThis means:\n● B is a learned baseline transition logit from state i to j (as in a standard HMM).\nij\n● wtr is a vector of covariate weights that modulate all incoming transitions to state j.\nj\nWe have added this as Equation (14) in the Methods section and Equation (3) in the Results\nsection in order to clarify the model structure.\nIf K denotes the number of states and d denotes the number of covariates, then the model\ncontains K2 parameters for B , and K×d parameters for the transition weight vectors wtr.\nij j\nThis leads to a total of K2+Kd transition parameters. So considering K=4 states and dim(x) =\n6 (i.e., d = 6) transition covariates, we have 40 transition parameters in total. (The number of\ndegrees of freedom is actually K(K − 1) + (K − 1)d, due to the fact that the probabilities for\neach starting state i must sum to 1.)\nBy contrast, a standard HMM contains K2 parameters, so only 16 parameters here for K=4\n(or K(K-1) degrees of freedom since every row of the transition matrix must sum to 1).\nTherefore, our GLM-based transition model is strictly more expressive and flexible — the\nstandard HMM is a special case of our model when all transition GLM weights are zero.\nWe note that other works such as Calhoun et al. (2019) used GLMs with separate weights\nfor each transition (i, j) which allows for greater flexibility, but also requires more data and is\nmore susceptible to over-fitting. Our formulation thus offers a middle ground: it allows\ncovariate-driven transitions without requiring separate filters per origin state. Even though\nwtr does not depend on i, the presence of B ensures that the transition probability does\nj ij\ndepend on the previous state.\nHowever, to address the reviewer’s original concern and to validate this modeling choice, we\nperformed a new analysis of the state transitions inferred by our model. In particular, we\ncomputed the conditional probability of all transitions to a state given the previous state\n(e.g., to assess whether it is more likely to transition to the Biased-R state from the\nEngaged-R state than from the Biased-L state). This analysis, shown below, revealed that\nthe transition probabilities into a given state are similar across different preceding states.\nThis suggests that the model's transition dynamics were not heavily influenced by the\nspecific prior state, thereby suggesting that using different transition filters for each starting\nstate would not substantially improve the model.\nFor IBL data:\nWe have included this analysis as Supplementary Figure S11 in the revised manuscript,\nalong with an explanation in the main text (lines 433–435) as below:\n“Fig. S11 shows that transition probabilities into a given state are nearly identical across\ndifferent preceding states, for IBL data. This confirms that our key conclusions are not\nsensitive to this aspect of the model.”\nWe also included this in the Results section for added clarity (lines 97 to 106):\n“To limit model complexity while still capturing covariate-driven transitions, we used one\ntransition weight vector per destination state (wtr ), along with a baseline term for each state\nj\npair (B ). Therefore, the probability of transitioning from state i to state j at time t is modeled\nij\nas:\nThis structure allows transitions to depend on both the previous state—through B , a learned\nij\nbaseline transition logit from state i to j—and the upcoming state, enabling covariate-driven\ntransitions without requiring separate filters for each origin state. We verified that transitions\ninto a given state did not differ significantly depending on the preceding state, supporting this\nmodeling choice (Supplementary Figure S11).”\nAnd similarly we explained it in the method section (Equation (14)).\nReviewer #4 (Remarks to the Author):\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports.\nThis is part of the Nature Communications initiative to facilitate training in peer review and\nto provide appropriate recognition for Early Career Researchers who co-review manuscripts.\nWe thank Reviewer #4 for their contribution to the review process as part of the Nature\nCommunications initiative to support Early Career Researchers. We appreciate the time and\neffort they dedicated to co-reviewing our manuscript.\nA point-by-point response to the reviewers' comments\nWe are grateful to the Editor and Reviewers for the time and care devoted to our manuscript.\nYour comments helped us sharpen the presentation and improve clarity. We appreciate the\nconstructive feedback throughout the review process.\nReviewer #1 (Remarks to the Author):\nThe authors have adequately addressed our questions and comments, and we have no further\nremarks.\nResponse: We thank the reviewer for this positive assessment.\nReviewer #1 (Remarks on code availability):\nI did not test the code but the it seems well documented.\nResponse: Thank you. The public repository includes a concise README with step-by-step\ninstructions, versioned dependencies (Conda environment), and figure-level scripts to reproduce\nresults.\nReviewer #3 (Remarks to the Author):\nThank you for your response. I don't have any more comments or questions.\nResponse: We thank the reviewer for their careful evaluation and are pleased that no further\nissues remain."}, {"round": 3, "reviewer_comments": {"reviewer_1": "The authors have adequately addressed our questions and comments, and we have no further remarks.", "reviewer_3": "Thank you for your response. I don't have any more comments or questions."}, "author_response": ""}]

Decision letter