Title: Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-context learning; LLM; Hidden Markov models; Neuroscience; Decision-making; Animal behavior.
Abstract: Hidden Markov Models (HMMs) are fundamental tools for modeling sequential data with latent states that follow Markovian dynamics. 
However, they present significant challenges in model fitting and computational efficiency on real-world datasets. 
In this work, we demonstrate that pre-trained large language models (LLMs) can effectively model data generated by HMMs through in-context learning (ICL) — their ability to learn patterns from examples within the input context.
We evaluate LLMs' performance on diverse synthetic HMMs, showing that their prediction accuracy converges to the theoretical optimum. We discover novel scaling trends influenced by HMM properties and provide theoretical conjectures for these empirical observations. Furthermore, we present practical guidelines for scientists on using ICL as a diagnostic tool for complex data. Applied to real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts. Our results demonstrate potential for advancing understanding of LLMs' capabilities while opening new avenues for scientific discovery of biological mechanisms and hidden structures in real-world phenomena.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the ability of a black-box Large Language Model to learn Hidden Markov Models in-context. This investigation is conducted primarily through empirical evidence derived from simulations, and through the lens of a prediction task, rather than by directly addressing probabilities. The paper showcases that LLMs can be used as efficient and capable predictors of the next token for data coming from a Markov process. The gap the paper aims to close concerns the \"hidden\" setup, where the states are only observed through emissions. Moreover, the paper somewhat evaluates (see comments below) interesting real-world data, which is considered to come from a Markovian process.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n\nThe paper is very well-written and easy to follow, contributing to its accessibility. Its design is equally well-done, featuring clear and illustrative figures that greatly aid comprehension. Furthermore, the evaluation plots are nicely presented, enhancing the clarity of the results. The authors have thoughtfully addressed most intriguing aspects of Markov models (not as much of the \"hidden\" aspects, see Weaknesses), and their methodology for constructing the benchmark simulations is robust and well-conceived. Finally, the inclusion of the real-world data is particularly interesting and adds a valuable dimension to the empirical investigation.\n\n# Weaknesses\n\nSummery of the main weaknesses :\n1) The paper focuses on an LLM's ability in a *prediction* task for data sampled from an HMM. It's unclear how this relates to the LLM's ability to genuinely \"learn\" the underlying Markov model. In this case, the novelty is questionable, given its already known that LLM can do ICL for much more complex temporal data.\n2) The paper aims to address the \"hidden\" aspect of HMMs, but this is largely unaddressed in both the text and the simulations.\n3) Following (1) and (2), the scope is very limited. \n\nAbout point (1) : \n\nThe paper suggests LLMs can be efficient next-token predictors for data sampled from a Markov process, but this doesn't equate to learning a Markov model. Predicting the next state (maximum likelihood) is distinct from inferring probabilities (reconstruction), which is a primary function of Markov models. Specifically here, The \"overconfidence\" of LLMs, which might make them better next-token predictors, could paradoxically make them less convincing at reconstructing true Markov models - where the distribution of the next token should come from the transition probabilities. This appears to be the case in the evaluations where LLMs perform best on low-entropy transition matrices (where the most likely token has a high probability).\n\nMoreover on the same point, given the task is predicting next token, the LLM's memory bandwidth can allow it to use past observations (not the current) with lower sample complexity with comparison to traditional reconstruction algorithms which assume the data is Markovian. This also appears to be the case in the evaluations where LLMs easily learn models with fast mixing, indicating ample \"signal from the past\" in finite sequences.\n\nAbout point (2) : \n\nThe paper claim is to close the gap for \"hidden\" Markov models, but nonetheless, the paper barely address the hidden aspect of the HMM. There's a lack of analysis on how emission probabilities or the loss from misclassifying observations to hidden states affect predictor results. Also it's unclear whether sequential information is truly critical, or if the LLM is simply better at picking states (which is often more important in practice).\n\nAlso, the benchmarked graphs ignored the more interesting Markov process like a jumping processes or a cyclic processes which have unique properties.  \n\nAs a summery, the claim that LLM \"learns\" a HMM in ICL is not something the paper shows, but rather that an LLM can be a good next token predictor in ICL for a data sampled from an \"easy\" distribution like a simple Markov Model (no jumping process etc..).\n\nSmaller points : \n* The assumption that the real-world data example originates from a Markov process is not justified. It would be more robust to learn a Markov model from the real data and then sample from that as a benchmark.\n* The main paper doesn't contains enough details about the experimental setup, specifically regarding the emissions and their prior distributions.\n* Theorem 1 appears to be quite obvious for any learner, and it's unclear what makes an LLM particularly sensitive to these parameters.\n* In context of Markov models, an insightful experimental design would involve giving the model a sequence to learn, and then providing a single new sample to predict the next, rather than always providing the full context.\n\nQUESTIONS:\nAddressed in Weaknesses\n\nLIMITATIONS:\nAddressed in Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether pre-trained large language models (LLMs) can implicitly learn and predict the behavior of Hidden Markov Models (HMMs) through in-context learning (ICL), that is, by simply being provided with sequences generated by HMMs without explicit parameter updates or task-specific training. The authors systematically evaluate LLMs on a range of synthetic HMMs with varying properties, as well as on real-world animal behavior datasets.\n\nThe authors show that LLMs can converge to predict the next observation in HMM-generated sequences compared to an oracle defined with the Viterbi algorithm applying on the HMMs that were used to generate the sequences, given sufficient context. They also show the LLMs’ performance and convergence trends are closely linked to intrinsic HMM properties such as entropy and mixing rate. And they have also conducted additional experiments with real neuroscience datasets, and provide practical guidance for using LLM in-context learning as a diagnostic tool for exploring sequential data.\n\nIn summary, the study demonstrates that general-purpose LLMs possess an interesting capacity for modeling complex, latent-state-driven sequences and opens new avenues for applying ICL in scientific discovery and data analysis.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\nThis paper presents a comprehensive and careful empirical study, systematically evaluating large language models (LLMs) on both synthetic HMMs and real-world datasets, and comparing their in-context learning performance against strong statistical and neural baselines.\nWhile the empirical work is thorough and the main results are well-supported, one main downfall in my opinion is that ICL does not generalize HMMs for usuall applications for speech and handwriting recognition where HMMs are trained are generative models with multiple sequences a time.\n\nClarity\nThe paper is clearly written and well-organized, with a logical flow from motivation and background to synthetic experiments, scaling law analysis, and real-world application. The main contributions and findings are explicitly stated, and well-designed figures make the experimental results easy to interpret. \nHowever, some important implementation details, such as prompt formatting and LLM configuration, are only available in the appendix, making it difficult to understing the methodology without reading all the material\n\nSignificance\nThe paper brings to the table an interesting discussion on the easiness of modeling sequencial data with LLMs. \nOn the other hand, LLMs are generally more complex than usual architectures for HMMs, which can make this an utterly more complex approach to solve problems that could be easily solved with some tweking in HMM topology.\n\nOriginality\nThe paper introduces a novel experimental paradigm, systematically probing whether LLMs can perform HMM inference purely through in-context learning. \nHowever, the paper does not introduce new learning algorithms or theoretical frameworks, and the most original contributions are in empirical observation and framing rather than methodological invention.\n\nQUESTIONS:\n- How are the initial HMM parameters (initial state, transition, emission probabilities) initialized for Baum-Welch training?\n- What stopping criterion or tolerance is used for Baum-Welch training, and how many iterations does it typically require?\n- How are HMM-generated sequences presented to the LLMs? Are there any special formatting or tokenization strategies used to maximize in-context learning?\n- What decoding strategy and parameters (e.g., temperature, top-k, greedy) are used for LLM next-token prediction during evaluation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that pre-trained LLMs can learn Hidden Markov Models given in-context. Interestingly, they achieve near-optimal prediction accuracy that converges to nearly the Viterbi algorithm optimum with true parameters.. The authors systematically vary HMM properties (entropy, mixing rate, state dimensions) and show that convergence speed depends critically on these characteristics: faster mixing and lower entropy lead, for faster convergence. Comparing against baselines (Baum-Welch, LSTM, n-gram), LLMs consistently outperform traditional statistical approaches. The authors provide theoretical conjectures linking ICL behavior to spectral learning algorithms. Finally, the authors test on real-world animal decision-making datasets, where ICL achieves competitive performance with expert-designed models on mouse choice behavior and meaningful performance on complex rat reward-learning tasks. The work then establishes practical guidelines positioning LLMs as \"zero-shot statistical tools\" for scientific discovery.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The paper is very clear and well written. All figures, terms and definitions are easy to follow and well motivated.\n\nS2: The paper spans theoretical grounding, synthetic experiments and real data experiments. The findings connect well.\n\nS3: The paper validates their finding using multiple models(appendices) and using multiple metric and multiple HMMs.\n\nS4: The paper goes further and provides guidelines for using LLMs for HMM identification.\n\n---\n\nW1: While the paper demonstrates some connection to spectral learning theory, the predicted learning curves are not explicitly fitted/compared to the empirical in-context learning curves.\n\nW2: The connection of the theoretical section to the rest is relatively weak\n\nQUESTIONS:\nQ1: In Fig 4 caption, isn't smaller lambda 2 fast mixing and large lambda 2 slow mixing?\nQ2: There must be limits, right? Are there extreme cases where LLMs cannot figure out the hidden state well?\nQ3: Have you observed any sudden transitions? i.e. the LLM \"grokking\" that there exists a hidden state?\n\n---\n\nQ4: This is a pure curiosity driven question and will not affect the score at all negatively if ignored: What is you gave the context then natural language prompted the model (+CoT) to predict the next state?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work evaluates the ability of pre-trained LLMs to model data generated by Hidden Markov Models (HMMs). The work evaluates LLMs' performance on diverse synthetic HMMs, varying parameters including mixing rate, entropy level, and number of hidden states and observations. They find by leveraging in-context learning (ICL) abilities, LLMs are able to converge on theoretically optimal prediction performance in certain settings (i.e., low entropy and high mixing rate), and suffer in settings where optimal prediction is poor as well. The experiments also show that pre-trained LLMs are able to outperform state-of-the-art across various HMM settings. The work provides theoretical conjectures to help understand the performance of these pre-trained LLMs and provides practical suggestions for how to use pre-trained LLMs with ICL as diagnostic tools for complex data. Finally, they show ICL achieves competitive performance for understanding model dynamics in real-world animal decision-making datasets in simple settings, but struggles in settings similar to when it struggled in synthetic experiments (i.e., complex settings).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The work provides a nice background of HMMs and sets up a clear story that is followed in the rest of the work. \n2. Overall, the empirical results for this paper are strong and clearly answer the research questions laid out by the work: Can LLMs predict next states in HMMs, what properties of HMMs affect the LLMs ability to do so, and how does this translate to real world data. \n3. The work accurately isolates entropy and mixing rates as interesting axes to vary in the HMMs, and do a good job of characterizing when LLMs fail in these different settings. \n4.  The work does a good job of comparing to a theoretically optimal model to show both 1) LLMs are able to approach this model in some settings, and 2) the places where LLMs suffer are also where optimality is not guaranteed. \n\n\n**Weaknesses**\n\n1. It is difficult to read the different graphs and result plots due to their small size and small legend text. Please consider larger images or tables for displaying results, especially those in Figure 3 and Figure 5. \n2. The relationship between using ICL with pre-trained LLMs and spectral learning remains unclear. The work provides a theoretical characterization of spectral learning for HMMs, but does not describe why we can view these results as a potential explanation for the ICL results due to a lack of connection between spectral learning and ICL using LLMs. Accordingly, Section 3.3 seems quite disconnected, and the theoretical results do not seem meaningful to the rest of the paper. \n3. Related to the point above, there is limited understanding of why LLMs are good at this task. Is it the training process? The transformer architecture? Size of the model? What inherent to LLMs is resulting in strong performance. More experiments ablating different parts of LLMs could be useful for answering this question. \n4. The findings from the reward-learning rats dataset are underwhelming. The work finds that LLMs using ICL are unable to perform well on this task compared to a baseline, and hypothesize that this is due to the complexity of the underlying dynamics. But in these complex settings, even optimal algorithms should suffer (as seen in the synthetic results). However, the baseline model performs noticeably better than the LLM, so I am not convinced it's simply too complex of a setting for the LLMs. \n5. The work generalizes beyond the findings provided in the paper often. For example, Section 4 provides guidelines for how scientists can use their findings to use LLMs as data-discovery tools in new use-cases outside of synthetic HMMs, without discussing whether their results may not hold in new settings or for new data-discovery problems. I would caution the authors from making such grand statements given their findings. \n\n**Comments**\n\n1. The caption of Figure 4 seems incorrect, as the slower mixing rates are with the higher $\\lambda_2$ values. \n2. I would caution the use of \"zero-shot\" for these models. Using in-context learning is inherently few-shot as examples of predictions/sequences are given to the model.\n\nQUESTIONS:\n1. Are LLMs able to discover the underlying HMM mechanisms well (i.e., stationary distribution, transition/emission matrices, etc.), or is their strong performance constrained to prediction? \n2. How can we guarantee findings on using LLMs for HMM prediction can translate to inputs/fields with respect to the guidelines on using LLMs as diagnostic tools for data structure and learnability? \n3. How do we expect transformer models to perform a a baseline for this task (using the sample training set-up as an LSTM)? Are the LLMs strong because of their training process, or is it simply the transformer architecture which is good for this prediction task? \n4. What is the relationship between ICLs with LLMs and spectral learning that underlies the theory in Section 3.3?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}