Title: Mechanistic Interpretability of RNNs emulating Hidden Markov Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mechanistic Interpretability

Recurrent Neural Networks

Hidden Markov Models

Stochastic Resonance

Reverse-Engineering

Behavioral Neuroscience

Computational Neuroscience
Abstract: Recurrent neural networks (RNNs) provide a powerful approach in neuroscience to infer latent dynamics in neural populations and to generate hypotheses about the neural computations underlying behavior. However, past work has focused on relatively simple, input-driven, and largely deterministic behaviors - little is known about the mechanisms that would allow RNNs to generate the richer, spontaneous, and potentially stochastic behaviors observed in natural settings. Modeling with Hidden Markov Models (HMMs) has revealed a segmentation of natural behaviors into discrete latent states with stochastic transitions between them, a type of dynamics that may appear at odds with the continuous state spaces implemented by RNNs. Here we first show that RNNs can replicate HMM emission statistics and then reverse-engineer the trained networks to uncover the mechanisms they implement. In the absence of inputs, the activity of trained RNNs collapses towards a single fixed point. When driven by stochastic input, trajectories instead exhibit noise-sustained dynamics along closed orbits. Rotation along these orbits modulates the emission probabilities and is governed by transitions between regions of slow, noise-driven dynamics connected by fast, deterministic transitions. The trained RNNs develop highly structured connectivity, with a small set of “kick neurons” initiating transitions between these regions. This mechanism emerges during training as the network shifts into a regime of stochastic resonance, enabling it to perform probabilistic computations. Analyses across multiple HMM architectures — fully connected, cyclic, and linear-chain — reveal that this solution generalizes through the modular reuse of the same dynamical motif, suggesting a compositional principle by which RNNs can emulate complex discrete latent dynamics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper reverse engineers RNNs trained to match emission probabilities of HMMs. They train the RNNs with Sinkhorn divergence to match the distribution of sequences instead of individual traces. They find that the RNNs switch between two stable regions corresponding to the two outputs that vary in probability. Switches occur when enough noise accumulates in the stable region to push the RNN state to a \"kick-zone\" that pushes the state through a deterministic transition region to the other stable region. In one example RNN, they isolate these computations to particular neurons, specifically two triplets of \"kick-neurons\" that drive the kick-zone to transition dynamics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The paper is an interesting case study on reverse engineering RNNs implementing probabilistic computations, where most works in this vein have focused on deterministic tasks. The paper analyzes the population latent dynamics in-depth but also explores mechanistic implementation of these dynamics by single units.\n\n**Weaknesses:** As noted by the authors, the HMMs used here are quite simple and thus it is not entirely clear how these insights transfer to more complex, realistic models. In addition, I feel that the analysis is still lacking in explaining the emulation of switching between discrete states and how these transition probabilities are embedded in the network solution. As such, I feel like the explanation of how the RNNs implement the HMMs is pretty incomplete.\n\nQUESTIONS:\n1. If you were to infer HMM parameters from outputs from the trained RNNs, would you recover the original HMM transition and emission probabilities? I would find this result to strengthen the evidence that the HMMs are successfully embedded in the RNNs beyond the summary statistics that are currently used.\n2. How are the transition probabilities implemented by the RNNs? Is there a way to determine the current discrete state of the RNN?\n3. For the mechanistic analysis, how consistent are the results of the existence \"kick-neurons\", etc. across the individual RNNs trained to match the two-state HMM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work shows how RNNs can learn the stochastic dynamics of 4 different HMMs where the transition graph between hidden states is a line (the number of hidden states ranges from 2 to 5). The authors show that trained RNNs can reproduce to stochastic state transitions of these HMMs. Examining the projections onto the first two PCs of network activity, they find that the latent dynamics of the RNNs always produce “noise-sustained limit cycles” that are reminiscent of self-sustained stochastic resonance. Then, they examine single-neuron activity and perform targeted perturbation experiments identifying groups of neurons with different causal roles in these “noise-sustained limit cycles”.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe modeling setup the authors consider in this work is simple and clearly presented; its rationale is convincing. The text is well organized and the logic of the arguments are easy to follow. Overall, the text is well written. \n\nI find the insights provided by this work on how RNNs can produce dynamics that are seemingly far away from standard RNN dynamics valuable. \n\nOn an algorithmic level, I find it impressive that they can effectively train RNNs using a loss function that compares probability distributions estimated through 30,000 randomly generated sequences.\n\n### Weaknesses\n\n1. The theoretical analysis of the latent dynamics of the trained RNNs appears to contain an important self-contradiction, and the theoretical arguments the authors present are sometimes incomprehensible.\n    1. There is a contradiction between the first and the second paragraph of Section 4.1. In the first paragraph the author say that, in the absence of stochastic inputs, the latent dynamics of the RNNs suggest that there is a unique stable fixed point and no oscillatory dynamics; oscillatory dynamics (what the authors call “limit cycles”) only appear when stochastic inputs are added. This is coherent with Section 6, where the authors claim that this behavior is reminiscent of self-induced stochastic resonance. However, in the second paragraph of Section 4.1, the authors claim that oscillatory dynamics appear because the RNNs go through a Hopf bifurcation. This seems to contradict the explanation of the first paragraph. There are two other issues with the authors’ “Hopf bifurcation” explanation. First, in Muratov et al. (2005), the paper the authors refer to when introducing the notion of noise-induced stochastic resonance (SISR), Muratov et al. write explicitly than SISR is a phenomenon that appears *before* the Hopf bifurcation. Second, I do not understand the analysis presented in Appendix A.6. I have never heard of input-dependent stability analysis (the input being stochastic in this case) and I am not convinced by this approach. (Related to this point, I do not understand what is the y-axis on Fig. 4(c).)\n    2. I could not follow Appendix B, despite spending a fair amount of time trying to understand the arguments. This could simply be due to poor writing. For example, $\\delta h^{(1)}$ appears on line 484 without having been previously defined. It seems that a clear definition of what $\\delta h^{(1)}$ and $\\delta h^{(2)}$ is missing.\n2. On the methodological and theoretical sides, I cannot tell whether this work is original or not. The authors do not say explicitly in what aspects their methods or theoretical analyses are original. \n3. Some words with a precise meaning in dynamical systems are used with a much weaker meaning in the text. For example, the use of the term “limit cycle” when describing the latent dynamics of the RNNs is abusive. The dynamics of the latent variable only *resemble* that of a noisy limit cycle.\n4. The authors mentioned some preliminary results on HMMs with more diverse transition graphs but do not show them.\n5. The presentations of some figures could be improved.\n\nQUESTIONS:\nWeakness 1.1 above is the most important weakness of this work. Addressing this point satisfactorily, either by adding/updating current explanations or by completely removing all reference to the “Hopf bifurcation” explanation throughout the text, could change my rating from \"reject\" to \"weak accept\".\n\nRegarding weakness 1.2, can the authors fix Appendix B and explain, in a self-contained manner, what $\\delta h^{(1)}$ and $\\delta h^{(2)}$ are? I also do not understand what allows the authors to write \"without loss of generality\" on l. 479. Shouldn't the input be inside the nonlinearity $\\phi$? Clarifying these points would also positively affect my final rating.\n\nCan the authors address weaknesses 2, 3, and 4?\n\nRegarding weakness 5, here are some suggestions:\n- Fig 3: add legends for each rows of panels\n- Fig. 14: add titles, and legend to each rows\n- Fig 15: The labeling of the panels is confusing. In addition, a better choice of colors for the lines in the bottom panels would improve readability.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors train Gaussian noise-driven RNNs with deterministic and continuous dynamics to behave like stochastic discrete HMMs of a specific class. RNNs are trained directly on the outputs of the HMMs with varying number of hidden states with a distributional loss function. Analyses of trained RNNs show a transition from input-free fixed point to noise-sustained limit cycle globally, with distinct clusters corresponding to the output states of the HMM. Locally, there appears to be 3 distinct types of “zones” that are biased to be within vs. move across the clusters. Finally, the authors identify and causally test single neuron mechanisms that, coupled with the noise input, appear to be responsible for kicking the system state into one of the clusters corresponding to the discrete HMM outputs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- it’s in my opinion an interesting investigation into the equivalence of two popular classes of dynamics, RNNs and HMMs, and the mechanisms that enable the former to behave like the latter, i.e., stochastic discrete dynamics\n- the analyses are thorough, covering multiple “levels” of investigation from population dynamics to single neuron mechanisms, as well as causal perturbations\n- the writing and presentation are in general clear and crisp\n\nWeaknesses\n- the findings, while interesting, are based on a very narrow family of HMMs with a specific transition and emission probability structure, and thus unclear how generalizable the insights are (e.g., results in 4.1)\n- the writing is very clear at the sentence level. But structurally, it was difficult to understand and follow the whole picture. Given the density of results and a lack of high-level signpost (perhaps at the end of the intro), it read to me like a list of disparate findings.\n- I’m ultimately not sure what the implication or significance of these results are. I suppose it’s not a given that RNNs can be trained to behave like HMMs. But even accepting that this is a novel finding, in addition to the in-depth investigation into its mechanisms, the fact that it’s a purely modeling study employing a very narrow class of HMMs leaves me wondering what this means for brain circuits.\n\nQUESTIONS:\n1. Is it not relevant to discuss HMMs, stochastic RNNs, and “hybrids” like switching LDS in the related works section? Similarly, perhaps a brief mention of the relationship between HMMs and linear Gaussian systems (stochastic linear RNNs) may be instructive?\n2. What’s the rational for using the specific class of HMMs defined in eq 1? The overall motivation for this is not clear, as well as specific choices, e.g., why reduce the generality of the HMM model, e.g., by limiting the transition matrix to being band-diagonal? (line 76) What happens when these assumptions are violated?\n3. what happens if the middle output class (O2) is removed?\n4. observed state labels in Fig1d shows O1,2,3, whereas everywhere else in text is labeled as 0-2?\n5. line 82: I’m confused what is meant by “the same 0 to 2 continuum is partitioned…” Does this refer to the latent state, which increases with M and is hence further partitioned, or the observation space, which seems to be always 3 discrete states?\n6. Based on the schematic shown in Figure 2, how is the system different from the normal bistable / double well potential? \n7. could you please clarify how Fig 3 row iii shows less skewed distribution as HMM state increase?\n8. I’m not sure about the generality of the SISR claim: does this not depend on the exact structure of the transition and emission matrices? The current setup, even in the 5-hidden state HMM, has a strong preference for the two extreme output classes, both through high-probability transitions into the 1st and last hidden state, and the fact that the middle output state O2 always has very low probability. Do all these findings hold for flatter distributions?\n9. there is some discussion of stochastic dynamics in the brain in the conclusion section, but it’s not at all clear to me how that relates to the current study. Is the idea that brain circuits can be well-modeled by continuous dynamics RNNs, but behavior has been observed through methods such as MoSeq to be composed of discrete set of syllables, so we need to find the equivalence? This is briefly described as the motivation of the study in the introduction, which is informative, but I think much more can be said throughout the paper to concretely relate the current findings to that overarching idea / goal, though I understand that this is difficult to do with a purely computational modeling study.\n\nLIMITATIONS:\nIn my opinion, there is an inadequate discussion of the setup and assumptions underlying the current study, in particular the narrow range of HMMs used and therefore possible observed behaviors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors describe their efforts to train and then reverse engineer vanilla RNNs to reproduce the probabilistic dynamics of an HMM.  The authors utilize a range of mathematical techniques from nonlinear dynamics, perturbation theory, dimensionality reduction, machine learning, and computational neuroscience to describe in detail the mechanisms that the trained RNNs deploy and the training dynamics that they display.

STRENGTHS AND WEAKNESSES:
This paper is exceptional in every regard. It is surprisingly easy to read despite being extremely dense. The authors were both careful and complete in both their experiments and their analysis. The paper addresses an interesting and important question regarding the learning of a stochastic process by an RNN, and the analyses in Sections 4-6 were a feast of methods and details that were seamlessly integrated into a mechanistic description of the RNN dynamics from the population, connectivity, and single-neuron levels. The ease and simplicity of paper belies a mountain of work both in the analysis and the writing.

QUESTIONS:
None.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03