Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context
Authors: Yijia Dai
Zhaolin Gao
Yahya Sattar
Sarah Dean
Jennifer J. Sun
Venue: NeurIPS 2025 poster ()
Keywords: In-context learning; LLM; Hidden Markov models; Neuroscience; Decision-making; Animal behavior.
Primary area: deep_learning

Abstract: Hidden Markov Models (HMMs) are fundamental tools for modeling sequential data with latent states that follow Markovian dynamics. 
However, they present significant challenges in model fitting and computational efficiency on real-world datasets. 
In this work, we demonstrate that pre-trained large language models (LLMs) can effectively model data generated by HMMs through in-context learning (ICL) — their ability to learn patterns from examples within the input context.
We evaluate LLMs' performance on diverse synthetic HMMs, showing that their prediction accuracy converges to the theoretical optimum. We discover novel scaling trends influenced by HMM properties and provide theoretical conjectures for these empirical observations. Furthermore, we present practical guidelines for scientists on using ICL as a diagnostic tool for complex data. Applied to real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts. Our results demonstrate potential for advancing understanding of LLMs' capabilities while opening new avenues for scientific discovery of biological mechanisms and hidden structures in real-world phenomena.

Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context
Yijia Dai Zhaolin Gao
Yahya Sattar
Sarah Dean
Jennifer Sun
Corelll University
Abstract
Hidden Markov Models (HMMs) are foundational tools for modeling sequen- tial data with latent Markovian structure, yet fitting them to real-world data re- mains computationally challenging. In this work, show that pre-trained large language models (LLMs) can effectively model data generated by HMMs via in-context learning therr ability infer patterns from examples within a prompt. On a diverse set of synthetic HMMs, LLMs achieve predictive ac- curacy approaching the theoretical optimum. We uncover novel scaling trends influenced by HMM properties, and offer theoretical conjectures for these em- pirical observations. We also provide practical guidelines for scientists on using ICL as a diagnostic tool for complex data. On real-world animal decision-making tasks, ICL achieves competitive performance with models designed by human experts. To our knowledge, this is the first demonstration that ICL can learn to predict HMM-generated sequences- an advance that deepens our understanding of in-context learning in LLMs and establishes its potential powerful tool for uncovering hidden structure in complex scientific data. Our code is available https://githur www. www./aaaiiaai..eeaaea.e.a.....ie wawp/aaaaaiaaa///a/aaaaaaaaaaaaa 10000////chhaiii
Introduction
Many natural and artificial systems, from animal decision-making ecological processes to climate patterns, generate observations governed by underlying, unobservable states that follow Markovian dynamics [3, 17, 34, 48, 57]. Hidden Markov Models (HMMs) [39] provide powerful framework for studying such phenomena. However, accurately modeling these systems presents significant challenges. Parameter estimation and model fitting require complex algorithms like Baum- Weln- [5]. and Gibbs Sampling [ [16]. These methods are often computationally intensive and can be algorithmi- cally unstable, demanding extensive domain expertise [8, 9]. For scientists across disciplines, these accessibility and computational bottlenecks limit the practical applications of HMM modeling tools.
Recently, large language models (LLMs) [1, 18] have reshaped the landscape of AI. Trained vast amounts of sequential text data, they have achieved unprecedented performance across natural language processing tasks and exhibit remarkable in-context learning (ICL) capabilities-the ability to learn patterns and perform new tasks directly from examples provided in the input context, without explicit parameter updates [7]. While prior theoretical and empirical works [13, 33, 40] have demon- strated LLMs' capabilities as Bayesian learners and their ability to model fully observed Markov processes, their capacity to implicitly learn Hidden Markov Models-
latent states, complex transition dependencies, and observation emissions-remanns remains largely unexplored. Understanding this capacity could illuminate the mechanisms underlying in-context learning HMMs, and reveal new ways to leverage LLMs for analyzing complex sequential phenomena in scientific contexts.
Correspondence yd73@corne1l.. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"Hnobserued
HMMM
mrined Closon
reWeNd
X
X
HMM
O:-2
Oc-1
R R P L
Pre-trained LLM
Q2: Do HMMS properties
Q3: Can work on real-world HMMs?
Q1: Can ICL learn HMMs?
affect ICL performance?
Figure Overview of study. We start by studying whether ICL using pre-trained LLMss can converge to theoretical optimum on HMM sequences (Q1, Section 2), then study how HMMs properties affect the convergence rate/gap with theoretical conjectures (Q2, Section 3), and finally we demonstrate how these findings translate to insights on real-world datasets for studying behaviors in science (Q3, Section 4).
In this paper. we present comprchensive study on the ability of pre-trained LLMs to learn HMMs through in-context learning (Figure 1). revealing their surprisingly strong performance and offering actionable insights for real-world scientific experiments. A key finding is that pre-trained LLMs demonstrate a remarkable capacity learn HMMs nearly optimally, achieving performance that approaches optimal Bayesian inference and often surpasses traditional statistical methods. These results not only advance our understanding of the emergent capabilities of in-context learning, but also introduce a novel and practical framework for using LLMs as powerful, efficient statistical tools complex scientific data analysis. Our study makes three key contributions:
1.
We conduct systematic, controlled experiments synthetic HMMs and empirically show that pre- trained LLMs outperform traditional statistical methods such as Baum-Welch. Moreover, their prediction accuracy consistently converges to the theoretical optimum -s given by the Viterbi algorithm with ground-tra model parameters across a wide range of HMM configurations (Section 2).
2. We identify and characterize empirical scaling trends showing that LLM performance improves with longer context windows, and that these trends are shaped by fundamental HMM properties such as mixing rate and entropy. We further provide theoretical conjectures to explain these phe- nomena, drawing connections to to-and highlighting distinctions from classical HMM learning paradigms, including spectral methods. These findings offer important insights into the learnability of stochastic systems through in-context learning (Section 3).
3. We translate findings into practical guidelines for scientists, demonstrating how LLM in- context learning can serve as a diagnostic tool for assessing data complexity and uncovering underlying structure. When applied to real-world animal decision-making tasks, LLM ICL performs competitively with domain-specific models developed by human experts (Section 4).
Synthetic Experiments and ICL Convergence
We investigate the in-context learning capabilities of pre-trained LLMs on sequences generated by synthetic HMMs. We first review key HMM properties (Section 2. and outline our experimental setup (Section 2.2). We then empirically demonstrate that the prediction accuracy of pre-trained LLMSs consistently converges to the theoretical optimum (Section 2.3).
2.1 HMM Background
Hidden Markoy model: HMMs impose set of probabilistic assumptions on how sequences of data are generated. The elements of the sequence are called observations, denoted at each step by O, The observations depend hidden state denoted by Xx, which evolves according to

---

HMM
low
Entropy:
Mixing Rate:
high
learner needed context
Figure 2: Properties of HMMs that impact pre-trained LLMs in-context learning performance.
Markov chain. A HMM is characterized by the Markov chain's initial state distribution and its state transitions, along with the emission probabilities of observation given the hidden state. The key assumptions are that the state transition depends only on the previous state (Markov property), the observation depends only on the current hidden state (output independence), and both transition and emission probabilities are time-invariant (stationarity).
We focus the setting with finitely many states and observations. Without loss of generality, states take values in X' = (1.2..... M } while observations take values in O = (1.2..... L}. The initial state distribution is denoted as transitions describe by the matrix A E I M with elements aij the probability of transitioning to state j from state i, and the emission matrix B E RMXL contains 15 the probability of observing I when in hidden state j. The triple = (T, A, B) completely parameterizes a finite-alphabet HMM.
R R. with T the probability of starting in state j. the state
Stationary distributions:
Under certain conditions (see Appendix A), Markov chains are guaranteed to converge unique stationary distributions, which are given by the μ RM satisfying μ μ.A. [14]. The stationary distribution of the hidden state characterizes the long term behavior of the HMM, and therefore plays an important role both predicting future observations and learning HMM parameters. The rate of convergence is characterized by the mixing rate which for finite-alphabet HMMs is equal to X2, the second-largest eigenvalue of A. From any initial distribution, the hidden state distribution approaches the stationary distribution geometrically with multiplier X2. A smaller mixing rate indicates faster convergence to the stationary distribution.
Entropy: HMMs can describe processes which vary from deterministic to purely random, depending on how transition and emission probabilities are defined. Entropy is a measure of the randomness or unpredictability of random variable. By considering the average entropy over the stochastic processes of hidden state and observation, we can quantify the entropy of particular HMM by ((A) = - Ei, Hhaii log asj and H HB. (4) E,, Mzbbi log log We additionally define normal- ized entropies (A) = H (A)/ log M and H µ) = H(, ()) log L as metrics for visualization. A smaller entropy indicates more predictable process. See Appendix A for further explanation.
2.2 Experimental Setup
Experiment setting: Our experiment follows a three-step protocol: First, we specify the HMM parameters À= A,,
A, B) according our control variables (described below). Second, we generate observation sequences {0r, . ... from this parameterized model. Third, we evaluate the ability of candidate models to predict the next observation oc+1 given preceding observations 01:
We systematically vary five control parameters and consider 234 total HMM settings: (1) state and observation space dimensions, with M, L E {2, 4, 16, 32, 64); ; (2) mixing rate of the hid- den Markov chain, with A2 E {0.5, 0.75, 0.95, 0.99). where A2 is the second-largest eigenvalue of A; (3) skewness of the stationary distribution la (uniform or non-uniform); (4) entropy of the transition and emission matrices A and B, ranging from deterministic (zero entropy) to maximum entropy (random); and (5) initial state distribution (uniform or deterministic). While generat- ing π and B is straigh-ffowaad, for matrix A, we define a constrained optimization problem and solve using first order optimization. See Appendix for additional details. For each parameter configuration, we sample 4,096 state-observation sequence pairs, each of length 2,048. We as- sess model performance across context lengths ranging from 4 to 2,048 observations, specifically {4,8, 8, 16, 32, 64, 128, 256, 512, 1024, 2048). For each HMM setting, we report performance metrics averaged over the 4,096 samples. Our candidate models are open-source pre-trained LLMs (Qwen and Llama family). Note that we are not training the LLMs, only evaluating their capability for in-context learning. In the following sections, we evaluate L.LMs' performance by comparison to several other approaches (Table l).
3

---

Mothod
Input
Variable
Description
Viterbi [51]
Oua A
8
Finds most likely hidden state sequence
P(O Oc-ke))
Or-hai A
k
Direct conditional probability
Baum- Welch [5]
Baum-Welch
Outs M
B
EM algorithm for HMM parameter estimation
Spectral [21]
Out, M
6
Improper learning for spectral parameters (Section 3.3)
r-gram
Ou
Optimal (n- 1-th order Markov predictor
LSTM (RNN)
Out
Neural network with memory cells Neural network with multi-head attention
Transformer
Oua
Table 1: Methods for HMM prediction task.
LLM Viterbi Aecuracy
Viterbl- LLM vellinger
-0.90
Accuracy
-2.10
0.23
T
-2.10
6:3
Viterbi
LLM
-12)
9
Context Length
Context Length
Context Length
Figure (Left) We define as when LLM converges (see Appendix B for computation metric), and e as the final accuracy gap at sequence length 2048. (Middle) Examples when LLM accuracy converges Viterbi. Each curve represents a different HMM parameter setting. LLM ICL shows consistent convergence behavior. (Right) Examples of convergence in Hellinger distance (distance between two prohability distributions). LLM ICL. is not just "guessing" the most probable output, but converging distributionally.
2.3 ICL Converges to Theoretical Optimum
We define convergence as achieving prediction accuracy comparable to the Viterbi algorithm. The Viterbi algorithm, given ground-truth HMM parameters X. computes the most likely hidden state sequence Z1: from observations 01:t (see Appendix C for details). Since Viterbi has access to the true model parameters, its performance represents the theoretical optimum. Remarkably, ICL with pre-trained L.L achieves this near-optimal prediction accuracy across diverse HMM parameter configurations in our experiments. Figure illustrates examples and conditions under which this convergence occurs. Convergence occurs reliably when HMM entropy is low and mixing is fast.
For challenging conditions where LLM convergence fails proceeds exceptionally slowly-like the red areas shown in the left-hand side of Figure 4-Viteri algorithm also exhibits diminished prediction accuracy and requires substantially longer context windows to achieve reliable performance. This degraded performance reflects the fundamental limits of stochastic system learnability due to random dynamics and long-range dependencies, affecting even the optimal inference methods. See Appendix D for detailed examples.
Impact of HMM Properties on Convergence
Having established that ICL with pre-trained LLM converges to the theoretically optimal predictions, we now provide an in depth characterization of their performance across variable settings. We summarize the scaling trends in terms of key HMM properties, compare these patterns against other popular methods for HMM prediction, and conclude by providing theoretical conjectures.
3.1 In-context Scaling Trends
Neural scaling laws describe empirical power-law relationships that characterize how neural network performance improves with increases in key resources such as model size, dataset size, and compute [24]. For in-context learning [29], it describes trends between prediction accuracy and context

---

875
0.5
135
0.5
2.01
0.4
1.22
0.2
0.0
0.2
0.4
0.6
0.8
1.0 2.3
0.5
0.8
0.5
1.0
1.3
12
1.3
0.2
0.4
0.6
0.5
Entropy
Entropy
Emtropy
Figure 4: (Left) Convergence gap E increases with higher mixing rate (slower mixing) and higher entropy. This plot is showing results averaged across all HMM configurations we tested. (Right)
Slower mixing (A2 = 0.95, 0.99) shows delayed convergence compared to (Middle) fast mixing
(A2 = 0.5, 0.75) at similar entropy levels.
window length. We further show how these scaling trends depend on the underlying stochastic process characteristics: entropy, mixing rate, and state-space dimensionality.
Context window length: Given an observation sequence sampled from an HMM, LLM performance generally improves monotonically with increasing sequence length before plateauing. Representative examples are shown in Figure Fluctuations may occur when the entropy the sequence is high; additional examples are provided in Appendix D.
Entropy of transitions and emissions: Entropy determines the predictability of the observation sequence. The entropies of both the transition matrix A and the emission matrix B are positively correlated with the number of steps required for LLM convergence, as shown in the middle and right plots of Figure 4. However, this relationship is not strictly monotonic in practice.
We control the mixing rate of synthetic HMMs using λ2 the second-largess eigenvalue of A. Lower values of 22 indicate faster mixing. As illustrated Figure 4 (middle vs. right), for the same entropy level, convergence occurs significantly later when mixing is slow- that slower mixing delays L.MM learning.
Mixing rate:
The dimensionality of the state and observation spaces affects the maximum possible entropy. While larger state spaces intuitively allow for higher entropy, our experiments show that when entropy is held constant, varying the number of states does not impact LLM convergence rates. Detailed examples and discussion are provided in Appendix D.
Number of hidden states and observations:
3.2 Comparison to Baselines
We compare the in-context learning performance of pre-trained LLMs against several established methods commonly used for HMM prediction tasks, as summarized in Table 1. An oracle conditional predictor, P(O++1 Oz-ket), ), uses the true HMM parameters but truncates the history to model limited memory (cf. Viterbi-style oracle inference). For learning- baaning-based baselines, we include the classical Baum-Waod (BW) expectation- maximization algorithm, which remains the statistical state of the art for HMM parameter estimation [56]. Spectral methods have empirical evidence of converging to the theoretical optimum on a range of HMM prediction tasks [31, 42]. We also train two neural models: an LSTM (reflecting common RNN practice in neuroscience [54]) and a 2-layer, 4-head Transformer with self-attention architecture akin to pretrained LLMs; due to training cost, these results are averaged over 16 samples (vs. 4,096 elsewhere). Finally, n-gram baselines connect our HMM results to recent ICL findings Markovian settings [40]. Full algorithm descriptions and implementation details are provided in Appendix C.
expectaiion-maxiiizziioo
Across range of HMM configurations, we find that LLM consistently outperforms empirical learning baselines. A representative example is shown in Figure 5 left two panels. Among the n-gram models with n E (1...., 4). the bigram model performs best, aligning with role as maximum likelihood estimator for first-order Markov chains. However, because HMM observations are not Markovian when H(B)= > O, the bigram model inherently suboptimal. Trigram models converge more slowly than bigram due increased data sparsity and resulting estimation bias.
5

---

6475
Hellinger Distance
0.43
6.00
0.3
Ag 6:89
0390
1054 2014
2014
Context Length
Context Length
Context Length
Figure 5: HMM parameters M 8, 8, (A) 1.5, H 1. (Left) Accuracy comparison between LLM ICL (Qwen2.5-7B) and baselines Hellinger distance is between model and ground- truth predictive distributions. The dash line means the result averaged over 16 samples (vs. 4,096 elsewhere). (Right) The gap between P(O++1|0r) and Viterbi is small when mixing is fast. The line styles differentiate mixing rates.
Spectral learning algorithm converges to optimal prediction with longer context. Note that at early context its hidden-state belief update becomes numerically unstable because the sample complexity of spectral learning scales as (MML), which is 512 context length in the Figure setting. The Helm-Welch (BW) algorithm, while given the correct HMM structure and leveraging expectation- maximization, suffers from nonconvex optimization. Its global convergence is not guaranteed. Even when averaging across multiple random seeds and 4,096 samples per setting, BW converges slowly and often unreliably. LSTM and Transformer baselines, despite their flexibility, require significant computational resources and exhibit unstable accuracy across varying context lengths. Transformer performs better than LSTM, which suggests attention mechanism plays important role for learning to predict HMMs. Notably, LLMs via ICL demonstrate clearly superior behavior across all gggelines-achivviiee faster, more stable convergence the ground-truie truth distribution and highlighting their surprising efficiency in modeling HMMs.
On the other hand, in Figure right, the conditional predictor P(O++1 O.-kzt) can approach Viterbi-level performance, particularly when the mixing rate is low. This observation suggests that approximate prediction using a truncated observation history can be nearly as effective as statistically optimal inference in fast-mixing regimes, motivating the conjecture discussed Section 3.3.
3.3
A Possible Theoretical Explanation
In this section, we give a potential explanation for the ICL behavior of pre-trained LLMs for HMM sequence prediction by comparing with the spectral learning algorithm [21]. This is motivated by empirical evidences [31, 42] showing the convergence of spectral learning prediction to theoretical optimum (in limited settings). Furthermore, for partially observed linear dynamical systems. et al. [27] observes that transformers can learn statistically optimal predictions in-context when trained on many similar tasks in the meta-learning setting. These findings suggest that ICL by LLMs may exhibit similar performance characteristics to spectral learning algorithms.
A key idea in spectral learning literature is to compute the probability of observation sequences in terms of observation operators [21]: For any o E O, define A, := A didaaa(( I......... M)),)))
where [B] denotes the ij-th element of B. Then for any >>>
1 AA A AT AT J
P(O = 0++1 O1 = 01,...,,0 = A ... A.,
where 1 all one vector of appropriate dimension. this formulation, the conditional probability is estimated by first learning the spectral parameters (Appendix F) using training samples (in-context observations). Then, one can predict the next observation directly using these parameters along-with hidden state belief updates, without explicitly learning the matrices A, and B. This is therefore an example of improper learning, which has been extensively studied in related areas like linear dynamical systems [45]. The spectral learning algorithm is theoretically well understood. The

---

following theorem is obtained by extending the results by Hsu et al. [21] to single trajectory spectral learning
Theorem (Informal) Fix e, > 0. Let Σg denote the pairwise probability matrix of observations such that [E2ss] = =P(O++1 = = O2 =-=]] j). Suppose π > element-wise, and A. B are rank M. Suppose L 2 M, and let (-) denote the M-th largest singular value. Suppose the observation
operator A > 0 element-wise for all o E O, and
1
M (B)M(B) + 20M (B) (B)OM (E2)
ML
! log 8 3
12 1-)2( (c
(2)
Then, with probability at least 1 the next observation prediction IP(- O1:t) using spectral learning algorithum (detailed in Appendix F) satisfies the following upper bound in Hellinger distance,
H2 (P(O Ola -0, o.), P)), -.......................... Se
=o.......))
Theorem indicates that the scaling trends we observed in Section 3.1 are similar to those of spectral learning based predictions. Specifically, like our observations in Section 3.1, the prediction accuracy improves with more samples (i.e.. larger t). The mixing rate, captured by AA) affects ICL and spectral learning similarly. This occurs because spectral parameter estimation from single trajectory degrades with AA) tte faster the HMM mixes, the smaller the estimation error. Finally, the effect of entropy is captured by the observability conditions Theorem 1. Estimation error is maximized when the HMM is unobservable, which corresponds maximum entropy rate. The relationship between entropy and HMM observability has been well studied in literature [28, 32).
One of the practical limitations of spectral learning algorithm is the requirement of rank conditions of and B. Furthermore, the spectral learning algorithm is sensitive to the conditioning of the observed sequence, making its numerical performance robust only in limited settings. ICL by pre- trained LLMs seems to handle such issues more gracefully, pointing to an intriguing gap in our statistical understanding for learning HMMs.
4 Guidelines for Practitioners: How to (creatively) use LLMs for your data?
LLMs' capacity in deciphering complex sequential patterns in language can be repurposed: as demonstrated our synthetic experiments (Sections 2 and 3), pre-trained L.LMs can effectively model HMM-generated sequences through ICL,, achieving theoretically optimal prediction accuracy under favorable conditions. This section first translates these findings into practical guidelines for scientists, then demonstrates our observations real-world data in animal behavior (Section 4.1).
LLM in-context learning as diagnostic tool for data structure and learnability. Our synthetic experiments (Sections 2.3 and 3.1) reveal that key HMM mixing rate site--strongly influence LLM ICL convergence behavior. Practitioners can leverage this relationship as diagnostic tool for their own sequential data. If you observe that an LLM's ICL prediction accuracy on your data sequence steadily improves and saturates with increased context length (like Figure 3), this strongly indicates learnable, non-random underlying structure. Our findings show that L.LMSs achieve near-optimal prediction accuracy on HMMs with clear, learnable patterns (Section 2.3).
Guideline 1:
popeeties-notable entropy and
The characteristics of this convergence provide further insight: faster convergence and higher final accuracy in LLM ICL experiments are consistently associated with HMMs having lower entropy (less randomness) and faster mixing rates, as shown in Section 3.1 and Figure 4. Conversely, if LLM ICL on your data converges slowly, requires exceptionally long contexts, or plateaus at low accuracy, this suggests the underlying process has high entropy or slow mixing mrnamicc--chractterr inherently limit predictability and affect even optimal methods like Viterbi (Section 2.3). While calculating intrinsic HMM parameters from real-world data challenging, you can qualitatively assess your data's learnability by comparing its ICL convergence profile to our synthetic HMM experiments (Figures? and 4).
that
"This issue should not be insurmountable, similar to how (appropriately tuned) regularization can overcome poor conditioning in ridge regression [37]. However, we unaware of prior work which provides solution.
7

---

Bigaged
stimolas
Auarage Accuracy
CLN HMN
action
reward
Disenguzed
Chance 2016
LLM-ICA
GLM-HMIN HHMM
Sequence Length
Figure 6: IBL dataset mice decision- ceriision-making task. (Left) GLM-HMM model developed by neurosci- entists. (Middle) A cartoon illustration of the task. A mouse observes visual stimulus presented on one side of a screen, with one of six possible intensity levels. It then chooses a side, receiving a water reward if the choice matches the stimulus location. (Right) LLM ICL performance curve averaged across all animals, with 1-7 error har. Its prediction accuracy steadily increase with longer context window, exceeding the domain-specific model performance.
Guideline L.Ms are data efficient in giving accurate next observation prediction in-context. Pre-trained LLMs offer remarkably data-efficient and accessible approach for next-observation prediction through ICL, particularly valuable when rapid insights are needed or when data for training bespoke models is scarce. Our analyses (Section 3.2) show that LLM ICL achieves strong predictive performance with fewer domain-speciff specific assumptions than Baum- Welch (which faces non-convexity issues) and fewer training resources than specialized sequence models like LSTMs/RNNs (which require substantial data and careful tuning). LLMs therefore deliver immediate predictive capabilities with stable performance on limited data.
key practical advantage of LLM ICL is accessibility: while traditional methods require substantial computational expertise, applying pre-trained LLMs simply involves formatting data as text prompts, dramatically lowering barriers to sequence analysis. We are not positioning LLM ICL as a universal replacement for meticulously tuned, domain-specife models. Rather, its strength lies in providing strong, often surprisingly near-optimal predictions (Section 2.3) without any task-specific parameter updates fine-tuning, Our key observation is that general-purpos scientific data tasks for which they were not explicitly
prediictionno
LLMs can effectively model
HMM- -MMM-ggnerrted sequences and pre-trained on. This highlights vast untapped potential and suggests that future LLM ICL development could yield transformative scientific tools.
4.1 Real World Examples
We extend our synthetic HMM findings to real-world biological decision processes, focusing two extensively studied behavioral neuroscience datascts. Understanding how animals make decisions and learn efficiently remains fundamental challenge, with researchers investing tremendous effort in high-precision modeling to capture underlying cognitive mechanisms. These datasets serve as ideal testbeds given the neuroscience community's modeling efforts and the inherent connections between agentic decision- decision-making and HMMs (Figure 6). We represent animal decisions as discrete token sequences and compare LLM ICL performance against established domain-specifi models in predicting future actions.
Decision-making Mice Dataset: This dataset, developed by the International Brain Laboratory (IBL) [25], has gained significant traction for studying mouse behavior within the neuroscience community. A popular study [3] characterizes mice choice behavior an interplay among multiple interleaved strategies governed by hidden states in a HMM. Their GLM-HMM model (Figure 6 left) achieved an average prediction accuracy of 82.2%. outperforming standard approaches like the classic lapse model by 2.8%. For scientists investigating animal decision making, these performance improvements are significant for advancing model fidelity and experimental interpretation.
We compare GLM-HMM to in-context LLMs on data from 7 mice, following the descriptions in Ashwood et al. [3]. The experimental data consists of three components: stimulus, choice, and reward. For each trial, the mouse perceives a visual stimulus presented to their left or right, makes choice by turning steering wheel, and receives a water drop as reward when correct (Figure 6

---

LIMIOL
'iidden
internal R
CogFarCaarch
Average Accurct
Aiddon changing aeward
T. To
learn
lufor
Chonce level
predlction
04, ar... O2 Cs .0ul
Sequence Length
Figure 7: Rat reward- learning task. (Left) Analog agent learning to HMMs. (Middle) A cartoon illustration of the more challenging task. No stimulus presented on either side; instead, the reward probabilities for left and right choices evolve independently via random walks. As the optimal choice changes over time, the rat must learn and adapt its decisions based solely the history of past rewards. (Right) LLM ICL performance curve averaged across all animals, with I-o error bar. Its performance curve improves only marginally with increasing context length.
middle). Each mouse is described by one sequence, composed of trials. The trials are ordered sequentially as they occurred during experiments.
Remarkably, when provided with a context of more than 1000 trials, LLM ICL consistently achieved higher prediction accuracy (average of 86.2%) than the expert-devcloped GLM-HMM (Figure 6 right). More importantly, the convergence trend of LLM ICL mirrors the in-context scaling we observed in synthetic experiments, particularly when entropy is relatively low. This observation suggests that mouse gpcision-making processes contain learnable structures that LLMs, even without tase-specific training. can effectively identify and leverage for prediction.
Reward-learning Rats Dataset:
The dataset from Miller et al. [36] allows us to explore LLM ICL capabilities on more complex learning behaviors. This task presents a significantly greater challenge than the IBI. dataset for two primary reasons: first, animals receive no explicit stimuli guide their choices towards potential rewards; secondly, the dataset captures the entire dynamic learning process itself, rather than behavior after learning. Consequently, the underlying behavioral dynamics are expected to be more complex and less stationary. To benchmark LLM ICL in this scenario, we compare its performance against a state-of-the-art model from recent work [10], which employed code generation and evolutionary search to discover interpretable symbolic programs well-fitted to this dataset. It is important to note that this state-of thethh- model results from an extensive, computationally intensive evolutionary search, setting very high performance bar.
As shown in Figure 7, the prediction accuracy of LLM ICL. this dataset improves only marginally with increasing context length. This limited improvement parallels the ICI. behavior we observed for synthetic HMMs characterized by high entropy and slow mixing rates. L.M ICL exhibits substantial performance gap when compared to the specialized model. This outcome is consistent with our hypothesis that the underlying dynamics of this naturalistic learning process are complex, potentially pushing the limits of what current off-the-shelf LLMs can capture through ICL alone.
5 Related Works
LLMs and In-Context Learning. The surprising ability of LLMS to perform ICL [7] has led to significant interest in understanding its underlying mechanisms [11, 22, 53]. Several works [20, 52, 55] interpret ICL as implicit Bayesian inference, suggesting that LLMs naturally perform posterior updates through attention mechanisms. Theoretical works analyzing transformers' ability to model Markovian data [6, 13, 33, 40, 41] show that they can efficiently learn fully observed Markov chains. However, the transition from observable Markov sequences to aitent-variable models like HMMs remains underexplored. Recent studies also evaluate LLMS' predictive performance on structured tasks, including dynamical systems [29], density estimation [30], and time series forecasting [19, 49]. While these works explore LLMs empirical capabilities, they do not systematically analyze how intrinsic properties of underlying stochastic processes-sue as mixing time and entropy-aften ICL performance. Our work fills this gap by providing a controlled study on synthetic HMMs and offering theoretical conjectures for the observed scaling trends in ICL performance.

---

Spectral learning (SL) HMMs. SL algorithms have emerged as compelling tools for learning HMMs from observations, using method-of-ne oof-ooooooomttss learn the spectral parameters. Several works [2, 21, 42] construct matrices with observations, perform singular value decomposition and projection to obtain the beliefs of the HMM operators. Recent work [31] improves the practicality of SL. by projecting the probability beliefs onto simplex after every belief update. Despite these, SL algorithms have practical limitations and make assumptions on how observations carry information about the HMM dynamics, ICL seems to handle such issues by learning better observation operators without requiring the limiting assumptions of SL algorithms.
Neuroscience and Animal Behavior. Many neuroscience studies model animal behavior as HMMs [47, 50]. A common modern approach involves training data-specific RNNs and finding attractor dynamics [4, 23, 54], which can be highly inta-inefficient. Large generative models have accelerated neuroscience discoveries, from data processing [44, 46] to model discovery [10]. In our work, we present a novel approach, leveraging the frontier of AT to help scientists understand their data, focusing on discrete behaviors [36, 43].
Discussion & Takeaways
LLMs are surprisingly effective HMM learners through in-context learning: We observe that LLM prediction accuracy often converges towards theoretical optimum achieved by the Viterbi algorithm, which knows true model parameters. Contrasting with iterative and computationally intensive traditional HMM estimation algorithms or trained neural architectures (e.g.. LSTMs), LLM ICL offers simplicity as tool. As demonstrated by the competitive performance to domain- specific models on real-world animal decision-making tasks, LLM ICL offers a new avenue for rapid data exploration. LLMs can serve a "zero-shot statistical tool", enabling scientists to diagnose data complexity and generate future predictions without the overhead of extensive model development- addressing common bottleneck many scientific workflows.
Existing gaps: While we observe promising trends, our experiments also point to existing gaps in the broad application of LLM ICL. A primary bottleneck is the reliance discrete tokenization, which poses challenges for modeling continuous, real-valued, or high-dimensional observations- such as neural recordings-witte tokenization strategies for discrete sequences (see Appendix E.2 for ablations), adapting LLMs to handle continuous state-space models direct real-valued inputs remains open question. More- over, despite achieving high predictive accuracy, the inherently "black-box" nature of LLMs limits interpretabiliiyy. While our findings demonstrate that LLMS can effectively model HMM dynamics, extracting explicit and interpretable parameters-such as transition or emission probabilities ffrom the model's internal representations nontrivial. Yet, such interpretability is often central to the goals of scientists and practitioners seeking to understand underlying system dynamics. We hope this work lays the groundwork for future research into extending ICL to continuous domains and developing tools for extracting interpretable structure from LLMs.
the ICL framework. Although our experiments successfully employed
Call action: Realizing the full potential of LLMS and HMMs advance our understanding of complex systems demands a multidiseiplinary effort. There is a growing need for next-generation foundation models specifically designed to meet the challenges of scientific data-ranging from structured sequences to high-dimensional, continuous signals. Moving beyond adaptations of NL.P- focused models, such advancements are critical not only for enabling more effective scientific analysis, but also for deepening our understanding of in-context learning and the structure embedded within human language corpora. Ultimately, this progress will be essential to unlocking the transformative potential of LLMs in scientific discovery across broad range of disciplines.
Acknowledgements
YD thanks Kristin Branson and Kimberly Stachenfeld for insightful technical discussions, and Owen Oertell for their support. ZG is supported by Linkedln through the LinkedIn-Cornell Grant. This work was partly funded by NSF CCF 23 127744 NSF OAC-231 15221, NSF IIS-2442137, NSF IIS- Bowers CIS Strategic Partnership, and an AI2050 Early Career Fellowship program at Schmidt Science.
2505098. gift to the
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 007777774 2023.
[2] Animashree Anandkumar, Daniel Hsu, and Sham M Kakade. method of moments for mixture models and hidden markov models. In Conference on learning theory, pages 33-1. JMLR Workshop and Conference Proceedings, 2012.
[3] Zoe Ashwood, Nicholas A Roy, Iris R Stone, International Brain Laboratory, Anne E Urai, Anne K Churchland, Alexandre Pouget, and Jonathan W Pillow. Mice alternate between discrete strategies during perceptual decision- makiin.. Nature Neuroscience, 25(2):201-215 2022.
[4] Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector- based navigation using grid-like representations artificial agents. Nature, 557(7705):429-433 2018.
[5] Leonard E Baum, Ted Petrie, George Soules, and Norman Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. The annals of mathematical statistics, 41(1):164-171, 1970.
[6] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36: 1560-1588, 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[8] Olivier Cappé. Eric Moulines, and Tobias Rydén. Inference in Hidden Markov Models. Springer Series in Statistics. Springer, New York, NY, 1st edition, 2005. ISBN 978-0-3877442644-. doi: 10.100770-387-28992
[9] George Casella and Edward I. George. Explaining the gibbs sampler, 1992.
[10] Pablo Samuel Castro, Nenad Tomasev, Ankit Anand, Navodita Sharma, Rishika Mohanta, Aparna Dev, Kuba Perlin, Siddhant Jain, Kyle Levin, Noémi Éltetō, Will Dabney, Alexan- der Novikov, Glenn C Turner, Maria K Eckstein, Nathaniel D Daw, Kevin Miller, and Kimberly L Stachenfeld. Discovering symbolic cognitive models from human and animal behavior. bioRxiv, 2025. doi: 10.1100/2025.0 03.6673... URL httpp:///vet biorxiv. .2..5.2.000....... 05. 636732.
org/contente wtt//ot//ttttecccccc /aarrll/222//2//6//// 202.2222222222222222222222222
[11] Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers, 2022. URL https:://aat //ariiv////ii////aa/e/a www..oog////..eeee... arg/5 05055.
[12] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Viley-Interscceece, USA, 2006. ISBN 0471241954.
[13] Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains, 2024. URL https:
// //arxiv. org/ aaaaab///a /2402.2422222 11004.
[14] Y. Ephraim and N. Merhav. Hidden markov processes. IEEE Transactions on Information 2002. doi: 10. 11109/TT:20022.003888
Theory, (8(6))1518116
[15] Robert G Gallager. Discrete stochastic processes. Journal of the Operational Research Society, 1811):103-103 1997.
[16] Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEFE Transactions on pattern analysis and machine intelligence, (6): 721-741, 1984.

---

[17] Richard Glennie. Timo Adam, Vianey Leos-Barajee Théo Michelot, Theoni Photopoulou, and Brett McClintock. Hidden markov models: Pitfalls and opportunities in ecology. Methods in Ecology and Evolution, 14(1):43-56 2023.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.2 2024.
[19] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters, 2024. URL htttt:t//aaiiiii http://aaxivv..r/aba.a.a///a 2ttp://20...0.......
[20] Ritwik Gupta, Rodolfo Corona, Jiaxin Ge, Eric Wang, Dan Klein, Trevor Darrell, and David M Chan. Enough coin flips can make Ilms act bayesian. arXiv preprint vrXiv:2503.0722. 2025.
[21] Daniel Hsu, Sham M Kakade, and Tong Zhang. A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences,
2012.
[22] Yibo Jiang. Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers, 2024. URL https: //wxxvv.vvv//////////////////x org/ abs. 2406 18400.
[23] Michael I. Jordan. Attractor dynamics and parallelism in a connectionist sequential machine, page 112-127. IEEE Press, 1990. ISBN 0818620153.
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / http///xxvv.rr///////./ hwagg//bs///0 2001. 08361..
[25] The International Brain Laboratory, Valeria qaiillon-Rodrigues Dora Angelaki, Hannah Bayer, Niccolo Bonacchi, Matteo Carandini, Fanny Cazettes, Gaelle Chapuis, Anne K Churchland, Yang Dan, Eric Dewitt, Mayo Faulkner, Hamish Forrest, Laura Hactzel, Michael Häusser, Sonja B Hofer, Fei Hu, Anup Khanal, Christopher Krasniak, Ines Laranjeira, Zachary F Mainen, Guido Meijer, Nathaniel J Miska, Thomas D Mrsic-Flogel, Masayoshi Murakami, Jean-Paul Noel, Alejandro Pan- Maz-uzzueez Cyrille Rossant, Joshua Sanders, Karolina Socha, Rebecca Terry, Anne Urai, Hernando Vergara, Miles Wells, Christian Wilson, Ilana B Witten, Lauren Wool, and Anthony M Zador. Standardized and reproducible measurement of decision-making in mice. elife, 10:e6371 11, 1, may 2021. ISSN 2050-084X. doi: 10.7554/eLifee.637111 URL http:::/ddoi.rggre.... 10. 7554/ ee.ee.iffeee.. 63711
[26] David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathematical Soc., 2017.
[27] Yingcong Li. Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in-context learning. In International conference machine learning, pages 19565-19594, PMLR, 2023.
[28] Andrew R Liu and Robert R Bitmead. Observability and reconstructibility of hidden markov models: Implications for control and network congestion control. In 49th IEEE Conference on Decision and Control (CDC), pages 918-923. IEEE, 2010.
[29] Toni J. B. Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher Earls. Lims learn governing principles of dynamical systems, revealing an in-context neural scaling law, 2024. URL https://arze /ttp/aaaxvv////////////////aaa./ org/ abs/2402.00799
[30] Toni J. B. Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher J. Earls. Density estimation with lmms: a geometric investigation of in-context learning trajectories, 2025. URL https:
// //arxiv... org/ / aaaaab//244 /2400.222...0 05218.
[31] Xiaoyuan Ma and Jordan Rodu. Bridging the usability gap: Theoretical and methodological advances for spectral learning of hidden markov models. arXiv preprint arXiv:2302.07453 2023.
12

---

[32] John R Mahoney, Christopher J Ellison, Ryan G James, and James P Crutchfield. How hidden are hidden processes? a primer on crypticity and entropy convergence. Chaos: An Interdisciplinary Journal of Nonlinear Science, 21(3), 2011.
[33] Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains, 2024. htt.htttps//aatt
www.obs///////ae..ooo.ooooe 04161.
[34] Brett McClintock, Roland Langrock, Olivier Gimenez, Emmanuelle Cam, David L Borehers, Richard Glennie, and Toby A Patterson. Uncovering ecological state dynamics with hidden markov models. Ecology letters, 23(12))1878-1990 2020.
[35] Florence Merlevède, Magda Peligrad, and Emmanuel Rio. Bernstein inequality and moderate deviations under strong mixing conditions. In High dimensional probability V: the Luminy volume, volume 5, pages 273-293. Institute of Mathematical Statistics, 2009.
[36] Kevin J. Miller, Matthew M. Botvinick, and Carlos D. Brody. From predictive models to cognitive models: Separable behavioral processes underlying reward learning the rat. bioRxiv, 2021. doi: 10.1 10.1101/46129) URL hwtph:tt/////e //www//wwwwwwww biorxiv. org/ content/tenteeeeeeee teatteet/eearl/2221111 19/461129.
[37] Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. arXiv preprint arXiv:2003.0189) 2020.
[38] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo,, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.0m 00656, 2024.
[39] Lawrence R Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2)2257-266, 1989.
[40] Nived Rajaraman, Marco Bondaschi, Kannan Ramchandran, Michael Gastpar, and Ashok Vard- han Makkuva. Transformers markov data: Constant depth suffices, 2024. URL https:
/ //arxiv..t Bergg/bss/2440 17686.
[41] Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. An analysis of tokenization: Trans- formers under markov data. Advances in Neural Information Processing Systems, 37:62503- 62556, 2024.
[42] Jordan Rodu. Spectral estimation of hidden Markov models. University of Pennsylvania, 2014.
[43] Matthew Rosenberg, Tony Zhang, Pietro Perona, and Markus Meister. Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration. eLife, 10.e66175, jul 2021. ISSN 2050-084X. doi: 10.7554/eLife..o 66755. URL https:///oi..... org/ 10. 7554/ 66175.
[44] Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer Sun, Pietro Perona, David Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. eLife, 10:c63720, nov 2021. ISSN 2050-084X. doi: 2t555//e..ff.637/200 URL https: //doi. org/10. 7554/ eLife. 63720.
[45] Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non non-stochastic control. In Conference on Learning Theory, pages 3320-3436. PMIR, 2020.
[46] Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations, 2021. URL https:
/ //arxiv. wwwoorggaaa.eoooooeeooooeeeeow /0s./ 13917.
[47] Weinan Sun, Johan Winnubst, Maanasa Natrajan, Chongxi Lai, Koichiro Kajikawa, Michalis Michaelos, Rachel Gattoni, James E. Fitzgerald, and Nelson Spruston. Learning produces hippocampal cognitive map in the form of an orthogonalized state machine. bioRxiv, 2023. doi: 10. 10.11000/023.0..00 03.5519000 URL. https: //www//wwww//////// biorxiv. org/ content/eearly//203 08/07/2022 08.00.0.00000. 03. 551900.
13

---

[48] Atika Syeda, Lin Zhong, Renee Tung, Will Long. Marius Pachitariu, and Carsen Stringer. Facemap: framework for modeling neural activity based on orofacial tracking. Nature neuroscience, 27(1):187-195 2024.
[49] Mingtian Tan, Mike Merrill, Vinayak Gupta, Tim Althoff, and Tom Hartvigsen. Are language models actually useful for time series forecasting? Advances in Neural Information Processing Systems, 17760162-66999 2024.
[50] Diego Vidaurre, Laurence T Hunt, Andrew J. Quinn, Benjamin A.E. Hunt, Matthew J. Brookes, Anna C. Nobre, and Mark W. Woolrich. Spontaneous cortical activity transiently organises into frequency specific phase-coupling networks. bioRxiv, 2017. doi: 10. 10.1101/15067, URL https: htwp:///wwwwbbbb.i.oooooo org/
[51] A. Viterbi, Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260-26) 1967. doi: 10.1109/TIT. 1967.1054010.
[52] Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning, 2024. URL https: /axiv org/ abs/ 220/ 11916.
[53] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023. URL https: /tattps://axiv....// wwwg////g/////eeee/ee/ 03846.
[54] Thomas Wills, Colin Lever, Francesca Cacucci, Neil Burgess, and John O'Keefe. Attractor dynamics in the hippocampal representation of the local environment. Science, 308:873 876, 2005. URL https: / //ppi. semanticscholar. org/CorpusID: 13909966......
[55] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. explanation of in- context learning as implicit bayesian inference, 2022. URL 02080.
[56] Fanny Yang, Sivaraman Balakrishnan, and Martin J. Wainwright. Statistical and computa- tional guarantees for the baum baum-welch algorithm, 2015. URL https://arxiv..rr wwg./iii.org/abo. abs/1522...1... 1512. 08269.
[57] Walter Zucchini and Peter Guttorp. A hidden markov model for space-time precipitation. Water Resources Research, 1719)1917, 1991.
14

---

Appendices
Table of Contents
Appendix A: Additional Background on HMMs
Appendix B: Additional Details of Experimental Setup
Appendix C: Details of Benchmark Models
Appendix D: Additional Synthetic Experiment Results
Appendix E: Ablations LLMs
Appendix F: Spectral Learning HMMs for Prediction Task
Appendix G: Additional Real World Experiments
A Additional Background on HMMS
In this section, we define in detail the HMM settings we are interested in, including the conditions for Markov chains to converge to unique stationary distributions. Recall that HMM is characterized by the Markov chain's initial state distribution and its state transitions, along with the emission probabilities of an observation given the hidden state. With finitely many states and observations, without loss of generality, states take values in = (1,2.... M) while observations take values in O 21,2.... I.). The initial state distribution is denoted of starting in state the state transitions
I with иy the probability describe by the matrix A E MM M with elements d.j the probability of transitioning to state from state i, and the emission matrix B RMx contains byt the probability of observing / when in hidden state j. The triple = (T,, A, B) completely parameterizes finite- alphabet HMM.
Let X2.... denote discrete-time Markov chain taking values in X' with transition matrix A. Let Pij (m) =P(X++n j(X, i) denote the n-step transition probability between states i, E X.
State j is said to be accessible from state if there exists an integer 2 1 such that p.y (m) > 0. A
subset C S X' is called irreducible if every pair of states i, E C is mutually accessible. The period of state is defined as c(i) gndfn 2 1: Pis
(n) 0}. the greatest common divisor of all possible return times. State is aperiodic if c(i) = 1. A Markov chain is termed geometrically ergodic if is irreducible and aperiodic, which guarantees convergence a unique stationary distribution μ RM satisfying μ u.A.A The mixing rate E (0,1) is such that for all states i, E X, there exists a constant C 2 0 for which lp uy S Cp" for all n 2 1. For finite-alphabet HMM, p equals A2. the second-largen eigenvalue of A. We run experiments on few non-ergodic cases, while the majority HMMs are with ergodic state transitions to avoid dependence on the initial state.
The entropy H()) of discrete random variable X is defned as H(X) = - Nsx p(z) log p(x). fundamental property entropy is that conditioning reduces uncertainty: for any two random variables X and Y, we have H(XYY) < H()), with equality holding if and only if X and Y are statistically independent [12]. By applying the chain rule of entropy, the joint entropy of a stochastic process can be expressed as (X1,, X2....,
X1). For a Markov chain with stationary distribution μ,, the entropy rate is defined as H = lim 1 XX,,
= H(X,|X;=1,.
- -E., p.a.y logais, aij. which depends solely on the transition matrix A. We additionally define the entropy of the emission matrix B ΣS, Myybi bog which quantifies the average uncertainty in observations given the underlying states. Although the entropy rate the observation process in a HMM has no known closed-forn expression, can be bounded ...
HOO. Xn-1,-.. ........ O2, X1) S H(O)= < HOO,, O 01). from X, to X and B determines sampling O from X,, the entropies of and combined help us control the entropy lower bound of the sampled HMM sequence.
As A defines transitions
15

---

B Additional Details of Experimental Setup
Construet A with specific mixing rate, entropy, and steady state distribution. For ergodic Markov chain that converges to a unique stationary distribution, the stochastic matrix A can be decomposed into eigenvalues and eigenvectors with the ordering shown Figure 8, where I E RM vector of ones, 22 is the second-largest eigenvalue of A, and μ is the stationary distribution [15]. We leverage this decomposition to construct A with predefined A2 and Ja. To determine the remaining eigenvalues and eigenvectors, following requirements: (1) all entries of A are non-negative; (2) each row of A sums to I; (3) U- I; and all remaining eigenvalues have magnitudes not exceeding A2. The optimization problem has the following form, where we translate the constraints above into penality terms.
formulate optimization problem based on the
M
M
M
M
min
max - aig, 0} +
Σ Ø=1
(CE a:d) - -1)
(VU
- D
+
Σ t=3
max(A:
22,
As.ar 22
+1==1
+,=1
H VI
A Vdiag(1, )3, .., AM)U, == V2). U=
This is nonconvex problem, which we solve using first order methods with pytorch. We randomly initialize the free variables ..,, AM and V2 and then run 5000 iterations of Adam with step size 0.01 and default values for other parameters. After the optimizer terminates, we reject instances which not satisfy the constraints exactly. By initializing with multiple random seeds, we generate matrices spanning the desired entropy spectrum.
A UAU==
Figure 8: The singular value decomposition ergodic unichain Markov matrix A. The darker shaded
region is pre-defined for controlled experiments. The lighter shaded region is randomly initialized and calculated using a neural network.
Steady state distribution. We construct steady state distributions with varying skewness using the Beta distribution with a and different values of B. When a = and B 1, the resulting steady state distribution is uniform. As B increases, the distribution becomes increasingly skewed toward smaller state indices, Unless otherwise specified (Appendix D.3), we use a uniform steady state distribution the default configuration.
Entropy for visualizations. The entropy definitions H(A) and H p1) we introduced in Section 2.1 are used for constructing HMM parameters and sampling trajectories. For graphing Figure 4 (Left), we define normalized entropy considering both matrices:
H(A.B. H(A) + HB)
log M log L
We define ÍAA H(A) log M and (B, H(B, 1)/ log for Figure 4 (Middle) and (Right).
T' is when LLM converges to Viterbi. The concept of "convergence", though intuitive to human eyes, requires a specific numerical definition for plots like Figure 4. We define convergence as the point where two conditions are simultaneously satisfied: (i) the accuracy difference between Viterbi and LLM is within 0.025, and (ii) LLM achieves at least 95% of Viterbi's accuracy. We use both constant and relative thresholds ensure strict convergence definition that accounts for different baseline performance levels across experimental conditions.
Hellinger Distance. For two discrete probability distributions P. E TR.LL, the Hellinger distance is defined as
L Σ
D Dellinger (P,Q)
V -
/2
16

---

C Details of Benchmark Models
In this section, we provide descriptions and pseudocode for the benchmark models we use (Table 1). The executable code for all methods are included in supplemental materials.
Algorithm 1: Viterbi Algorithm
Input: States X' {1, ... M), initial distribution μ1, transition matrix A, emission matrix B, and observation sequence {01...... .r.))...........
Output: Most likely state sequence path = {z)....,.20 ,TT
Initialization: P[O][s] < sl[s] B[s[[oi] for all 8 E xx;
Forward recursion: for 1 to T
do
for 8 E X do
P[t[[s] ( t 1][r] A B[s||od]]:
red
Q[[[+] <- arg max reX 1| A [r][[s] B[s][o:]]:
end end Backtracking: path[T - 1| +- arg max P(T - 1|[s]; .EN
path(/) +- Q((+QQ11100 ||pphhh[/ ( + 1]] for 1 = T :
return path
Viterbi algorithm. The Viterbi algorithm is dynamic programming technique for efficiently finding the most likely sequence of hidden states in a Markov model, given a sequence of observations. iteratively computes the highest probability path to each state at time t by considering all possible predecessor states at time t - 1, their transition probabilities, and the emission probabilities of the current observation. Rather than exhaustively evaluating all M possible state sequences, Viterbi maintains only the M most promising paths at each time step, storing both their probabilities and the penultimate states that maximize these probabilities. After computing probabilities for all time steps, the algorithm traces backward from the most probable final state to reconstruct the optimal state sequence. We use the most probable final state and ground-truth A and B calculate the prediction distribution of the next observation.
Algorithm 2: Compute P(O++110(-k)
Input: States X {1, 2.................. .......... M). initial distribution μl, transition matrix A, emission matrix B, and observation sequence {or-ks.... .o._ks...,0 ..).
Output: Probability of next observation
(0:
Forward pass over observation window: [4-k(s] <- B [o]][o sl u|s] for all e X; [g[s] < B[s][o,] Σ [r][(s] 11-1(r] for =t + t, s E X:
FEX
Normalize to get posterior: ((8)0rkk)) t Evex क.. for s E X:
L
Prediction step: P(s)o:-k:t) + Erex [s)][s]] P(r for all s X:
Marginalize over states: P(oc41 E.ex B[s][or+1] p(s
return P(o++1 k0)
Optimal inference with truncated memory (10 The forward-based prediction al- gorithm computes the probability of the next observation in a hidden Markov model by using a three-step approach. First, it calculates the posterior distribution over current hidden states via the forward algorithm, recursively processing the observation window while accounting for transitions and emissions. Second, it projects this belief state forward by applying the transition matrix to com- pute the distribution over next possible states. Finally, it determines P(o (o:-ks)) by marginalizing over all possible next states, weighting each by its emission probability.
Baum Welch algorithm. The Baum-Welcc algorithm an estimating hidden Markov model parameters. It iteratively alternates between computing state posteriors 7x (s) and transition posteriors (<(8, r) via forward--ackwee recursion (E-step), and updating parameters maximize likelihood (M-step); setting the initial distribution to 71, transition
maximization method for
17

---

Algorithm 3: Kelm- Algorithm
Input: States X' 2,, ...,, M) observations y {1, 21,2..... ..., L.), observation sequence
{oI,..., o. initial parameters (0)) A A B and threshold c.
Output: Refined parameters μ. A, B
Initialize: + 1 (0) A +
(0) B t B(0)
prev t -00;
repeat
Lpece t- L;
E-Step:
Forward pass: 01 [8]
t B [o:] pls) for all E X:
ag(s) < B [o Σ, Ariss Q--1 fort 2...., S
t E. A
Backward pass: B [s] + for Ex:
[ 4 Σ. As][r] B B 341l for = - T 1,-1.... 1.... 1, E Ex;
Expected counts: n < 0, for
Ee[s][r]+ + B(le)))
for I...... 8, X:;
M-Step: [=[s] < [ for 8 e X:
[+][[]] < >x - <.(el[r] for all 8, E EX;
21 7.1a]
B[s|[u] <- Ei=i [.[6]
for all X, );
until |C Lpmu <
return f1, A,
probabilitics to normalized expected transitions, and emission probabilities to normalized observation counts per state. This process continues until the loglliklddoood converges, yielding locally optimal parameters that maximize the probability of generating the observed sequence. We use the Iearned parameters predict next observation similar the Viterbi algorithm.
Algorithm 4: n-gram Based Next-Observation Prediction
Input: Observation sequence = {0i,..., ........ OT), context length 1, smoothing parameter ò Output: n-gram model for predicting
Count extraction: counts,, counts, + empty associative arrays;
for
1 do
context + [o:- (n..).n...,.... o111
Increment counts,, (cunts, @ or] and -ounts,-[/contexx/
end
Model construction with smoothing: V number of unique symbols in O;
for each observed context e in 1 do
for each unique observation in O do
modellc, o) + 1
counts, cool+
end end
Back-off for unseen contexts: (onif(o) f- y for all o;
(ndel[[o:-(n-1))--- - 1 o.),
if context observed
(x-
Punis (o:),
otherwise
return model
n-gram. R-gram models provide an elegant, computationally efficient framework for next-observation prediction Markov chain processes by directly estimating conditional probabilities from observed sequences. These models embody the Markov assumption that P(O = P(Or=110r-n+a
making them particularly effective for stochastic processes where future states depend only on limited history of previous states. For first-order Markov chains, bigram models (n = 2) precisely capture the underlying transition dynamics, while higher-order dependencies can modeled by increasing
18

---

Algorithm 5: Trained Neural Networks for Single Sequence Prediction
Input: sequence O (01,...;or]) vocab size V: context window K; model family F;
parameters 0: optimizer A: epochs E
Output: trained 9 approximating P(o++1 01:()
Model: Token embedding E [l,.... VJ -RR optional position map II {I,..., K} RR:
Core network fo (R.) - >R' (e.g. LLT.////R/TTTTatoor///LLLLL with causal constraint);
Output distribution pe(: 2) = softmax (fo(z));
loss CE (cross-entropy);
Training: for epoch 1 to E' do
for = 1 to T do
$ max (1, K+K+ x E(O @ IIll 1
// embed (+ positions)
y Oc+1:
{ t fo(r):
// logits for next token optimizer step
L. + CH((oftmax(()) y):
A A(O, VoL.);
end
end
Inference:;
Given prefix O1., return P(ou+1 O1a) softmax(f((E(Oe)) @ II)) with == m1 ++);;
return 0
RNN LSTM. LSTM networks are specialized recurrent neural architectures designed to model sequential data through memory cells regulated by input, forget, and output gates. These gates control information flow, allowing LSTMs to selectively retain relevant historical patterns while discarding irrelevant information. LSTMs excel next-observation prediction tasks by capturing both short-term correlations and long-tern gergttermmoo dependencies in the observation history. The network processes a window prior observations sequentially, updating its hidden state to encode temporal patterns, then projects this state through a softmax layer generate probability distribution over possible next observations--making LSTMs particularly effective for forecasting future values in time series where the prediction depends complex patterns spanning multiple time scales.
In our experiments, we the number of observations the vocab size, use a two-layer LSTM with an embedding dimension of 16 and hidden dimension of 8, and train for 10 epochs using the Adam optimizer with a learning rate of le-3. The results are averaged over 16 sequences.
Transformer. Transformers predict the next observation using causal self-attention, which reweights the visible prefix without recurrence. Given embedded inputs X (tokens + positions), define XWQ. K = XWR, V XWr (queries, keys, values as learned linear projections). Attention is
Attn(Q, K, V) softmax
QK
+ +M) vih
where M is a causal mask that blocks future positions. Multi-head attention computes this in parallel across heads and concatenates the results. A feed-forward layer then maps the attended representations, and the state at position t is projected to logits; the next-token distribution is P(O++1 O1:) softmax(logits.).
In our experiments, we set the number of observations as the vocab size, use two-layer Transformer with embedding dimension of 16, hidden dimension of 8, and 5 attention heads, and train for 10 epochs using the Adam optimizer with a learning rate of 1e-3. The results are averaged over 16 sequences.
19

---

D Additional Synthetic Experiment Results
This section presents additional results from synthetic experiments. All methods are evaluated using the average performance over 4,096 sequences, with the exception of LSTM and Transformer, which are evaluated on 16 sequences due their high computational cost. Consequently, the LSTM and Transformer results exhibit higher variance. Nonetheless, in metrics such as Hellinger distance-which which account the full output distribution rather than relying solely on the argmax for AcCUracy-LSTM and Transformer underperforms compared to the LLM most of the time.
D.1 Varying Entropy of A
In this section, we present detailed results varying the entropy of A matrix over 4/8/16 states and emissions, reporting accuracies and Hellinger distances.
Wenl
PM
Gwed
ESTH
Treckerne"
Spertiy
Diserodees
Clarrattinn
2
PREENCE
Conbest Length
Figure 9: Accuracies of six methods across different A entropy, B entropy, number of states, and number of emissions with Xg 0.75 and uniform steady state distribution.
20

---

Vires
BW
3wM25530
LTT
Tanefimee Sprera
petttopp: 0
Benntopy: 1
Bertioyy
Berrory
Berroly
B eetoor
COTTET 1.3
65
030
870
055
0.5
0.61
0.45
820
068
60
035
0010
DR0832
0120001
Context Length
Figure 10: Hellinger distances of six methods across different A entropy, B entropy, number of states, and number of emissions with X 0.75 and uniform steady state distribution.
21

---

D.2
Varying Mixing Rate of A
In this section, we present detailed results on varying the mixing rate (A2) of A matrix over 4/8/16 states and emissions, reporting accuracies and Hellinger distances.
Winti
Pio.. HA
BW
LSMM
Tetufarma
Soutiral
2 entropy: 0
entrops: 1
Bantropy:
B antroay:
B entroay
B enrray 1
0.5
65
0.6
30
2
2
Contest Langth
Figure 11: Accuracies of six methods across different mixing rates (A2), B entropy, number of states,
and number of emissions with uniform steady state distribution and (1, 3) A entropy for (4, 8, 16) states respectively.
22

---

Vinel
0.e225100
Cbecraaisne:
Gpseraztinn
Bennosy
Berzory
Benoorx
065
020
650
635
6.6
12
R20
875
071
045
040
048
850
D02000
Context Longth
Figure 12: Hellinger distances of six methods across different mixing rates (A2), B entropy, number of states, and number of emissions with uniform steady state distribution (1, 2, 3) A entropy for (4, 8, 16) states respectively.
23

---

D.3 Varying Steady State Distribution of A
In this section, we present detailed results on varying the steady state distributions of matrix over 4/8/16 states and emissions, reporting accuracies and Hellinger distances. We construct steady states with different skewness using Beta distribution with a 1. Notably, with 1 and 1, the steady state distribution is uniform. As we increase 3, the distribution becomes more skewed. We test with 3, representing uniform, skewed, and very skewed respectively.
gram
0wer2.5-0
Tenatimer
Specta
Otserationy.
Osarraatonz
Ozseraatomn
Opaerazione
Otzendlizne.2
1.0
Snary State urttum
67
SO0 stry
SOW fore
Qwarty SEW Gwnent
Figure 13: Accuracies of six methods across different steady state distributions, B entropy, number of states, and number of emissions with (0, 0.5. 2) A entropy for (4, 8, 16) states respectively and (0.99, 0.95, 0.75) Ag for (4, 16) states respectively.

---

Vitrll
0wee2.500
LETH
Ocseraatinn
bentragyy
Bentooyy:1
Bentooy
Berzory
Qranty SORT
60
Sounty
5
Strasy Grownd
2 2E2EE C
Steedyy COON Grwend
Context Longth
Figure 14: Hellinger distances of six methods across different steady state distributions, B entropy, number of states, and number of emissions with (0, 0.5, 2) A entropy for (4, 16) states respectively and (0.99, 0.95, 0.75) Ag for (4, 8, 16) states respectively.
25

---

D.4 Discussions
When LLMs fail to converge, While LLMs converge to Viterbi performance efficiently under most HMM parameter settings (scaling trends summarized in Section 3.1), we identify two conditions where convergence fails or proceeds exceptionally slowly. First, when entropy of A or approaches its maximum (log M or log I. respectively), the prediction accuracy gap e at context length 2048 remains substantial. For instance, in the last row of Figure 9 with M 16 and the entropy of A 3.5 (near the maximum of log 16 4), the LLM (Qwen2.5-7B) exhibits gradual convergence with a persistent gap. Second, when mixing is slow (A2 approaches 1), such as the third-to-last row of Figure 11 with M 16 and A2 0.95, performance gap persists even maximum context length.
Importantly, the Viterbi algorithm also struggles under these challenging conditions. Under high entropy. as shown in the last row of Figure 9, Viterbi accuracy barely exceeds random prediction (0.25/0. 00.25/0.125/0.00 225/0.0625 for L 4/8/16). Under slow mixing, such as in the fourth row of Figure 11 with M = 8 and X2 = 0.99, Viterbi algorithm requires context length 512 to achieve peak performance. These results demonstrate that L.LM performance degradation under high entropy and slow mixing conditions reflects fundamental limits of stochastic system arrmaailtty-riiiinng from random dynamics and long-range dependencies- that affect even optimal inference methods.
Monotonicity of LLM performance with respect to context length. We observe that LLM performance almost always improves monotonically with longer context length-a property notably absent in other learning baselines. Even excluding LSTM from this comparison (due to high variance from averaging over fewer sequences, as discussed in the first paragraph in Appendix D), both Baum- Welch and bigram models lack monotonic convergence behavior. For Baum Baum-Welch, the accuracy graphs (Figures 9, 11, and 13) reveal multiple cases where performance "dips" and recovers, or deteriorates as context length increases. The Hellinger distance graphs (Figures 10, 12, and 14) provide clearer evidence that both BW and bigram exhibit non-monotonic learning patterns. In most cases, LLM Hellinger distance decreases monotonically, while BW and bigram display erratic behavior: sometimes experiencing early-cone early-context "bumps", other times starting very close to the ground truth emission distribution (occasionally even closer than the oracle Viterbi by empirical chance) before gradually converging to statistically sound distributions. Importantly, when BW or bigram achieve lower Hellinger distances, this does not necessarily indicate better performance- corresponding prediction accuracy graphs often show poor results, highlighting the distinction between distributional similarity and predictive capability.
When (normalized) entropy H is held constant, varying the number of states does not affect the LLM convergence rate. We provide concrete evidence for this claim in Figure 9, where rows 1, 3, and 6 all have the same normalized entropy H (A) 0.5. Across each column, the Qwen2.5-7B convergence curves for these three rows exhibit nearly identical shapes, demonstrating that the convergence rate depends primarily on normalized entropy rather than absolute state space size.
We emphasize that convergence rate differs from convergence target-the Viterbi performance. While the rate of improvement remains consistent across different state space sizes (when normalized entropy is fixed), larger state spaces result in lower achievable prediction accuracy due to increased task difficulty.
26

---

E
Ablations on LLMs
In this section, we provide the results on the families, sizes, and tokenization of the LLMs.
E.1
LLM Size
compare Qwen and Llama model families with seven different models. found that their performances similar, with slight degradation when the model size is small.
Uann 12-36
1.11-32310 0ww11.5-76
Gwnr2.5330
3wer25-1.50
5 >entropy. 0
B entrops:1 1
Bentroyy
B antrogy
B entrozy
D enbroyy
A
1.45
03
134
0,40
0.40
6300
0.00
63-
3.4
3.0
0.40
000
1:0
0.10
8.38
3275
0/10
3239
035
1399
612
0.1
000
453
624
Contast Langth
Figure 15: Accuracies of seven models across different A entropy, B entropy, number of states, and number of emissions with X = 0.75 and uniform steady state distribution. Lighter color represents smaller models. The two smallest models from each family have suboptimal performance.
27

---

5222522-5 Beriooy::
Owent5:18
Gean22-3
3ner232.08
bentrgy:
Beniory
Bennory
Berzory
C33
535
0.50
(40
044
033
0.5
541
6.4
654
035
0.65
375
465
650
655
640
652
X70
304
AettTEE 2.5
400
6.2
8710
0.9
6.5
0.45
365
0.955
0.89
0.00
0.09
377
0.6
0.70
654
662
8270
6700
055
Context Length
Figure 16: Hellinger distances of seven models across different A entropy. B entropy, number of states, and number of emissions with À = 0.75 and uniform steady state distribution. The models converge similarly, especially when entropy is high.
28

---

Obesnvttine 4
Otasnations:
Oepersatons:
Ooservatonz
Ghesnations 26
Ocasnvatons: 26
0.05
045
510
0.4
0.0
345
6.6
5.5
0.2
6.6
639
040
0.49
030
0.2
023
0.49
6.09
0.35
0.3
0.20
627
A30
0.39
0.30
3.5
0.00
3.0
879
8.0
0.05
0.05
532
8.0
A3C
023
022
Context
Figure 17: Accuracies of seven models across different mixing rates (A2), B entropy, number of
states, and number of emissions with uniform steady state distribution and (1, 2, 3) A entropy for (4, 8. 16) states respectively. The two smallest models from each family have suboptimal performance. especially when mixing is fast.
29

---

Lanad
Dear23
Gberratiore
entruyy:
Beniory
Benorry
Benzory
Benvor
0250
1150
CA6
0.03
0309
0:10
137
646
0315
0000
8.50
0.30
644
040
6.0
535
8.0
3.5
0.42
6:00
0.75
0.0
2.48
0.00
6.6
6.0
0.00
0.00
6.00
6.0
6.50
6.0
6.9
656
650
630
6.5
6.0
4750
6775
0:40
0700
620
8000
6600
630
6.74
6.7
6.0
6.6
6.10
670
0.00
8.100
6:4
6.6
6.0
0.43
6.44
2.5
6.6
6.88
6.6
630
6.2
0.43
3.4
6/0
653
650
Context Length
Figure 18: Hellinger distances of seven models across different mixing rates (A2), B entropy, number of states, and number of emissions with uniform steady state distribution and (1, 2, 3) A entropy for (4, 8, 16) states respectively. The models converge similarly, especially when mixing slow.
30

---

Otsenations bentropp:
Osserstionn
Didersatons
Osservatinn Brorosy
Disenvtians 16 B
Osserattons:
Ventross:1
Bennrsyy
0.0
68
6.0
Sreaty Sha
Steayy tran
Canediy So0n Grwend
Longth
Figure 19: Accuracies of seven models across different steady state distributions, entropy, number of states, and number of emissions with (0, 0.5, 2) A entropy for (4, 8, 16) states respectively and (0.99, 0.95, 0.75) A for (4, 8. 16) states respectively. The poor performance observed smaller models at short context length under low A&B entropy settings may be attributed to the filtering of repeated n-grams during pretraining, as discussed in Appendix E.2.
31

---

Owen25:75
Dean23
3ner232.30
bentrgy:
Bentnoay:
Bennoyy
Brenooyy
Crantyy SORE iefur
24
63
Oneedy Soon Greend
135
033
Steeny Coee deound
Sounty SAS Lertor
340
Strady Tiae Grownd
3.0
365
3.2
3.5
Steaay daun
0.4
025
2
Swenty E:Y unfum
3315
236
Steedy CONEN drwnd
820
224
6:00
3.50
4.0
Context Longth
Figure 20: Hellinger distances of seven models across different steady state distributions, B entropy, number of states, and number of emissions with (0, 0.5, 2) A entropy for (4, 16) states respectively and (0.99, 0.95, 0.75) X for (4, 8, 16) states respectively.
32

---

E.2 Tokenization
In this section, we evaluate three tokenization strategies: ABC, which encodes emissions as single letters; 123, which encodes them as single digits; and random, which maps emissions to random tokens from the LLM's tokenizer. For the random strategy, we specifically map emissions special tokens (1@#$). All experiments are conducted using the Qwen2.5-1.55 model, and the results are presented below.
We observe that all tokenization methods converge to similar performance levels in terms of accuracy, with ABC converging slightly faster when the entropy of A is large. This suggests that the choice of tokenization has limited impact on final performance. In our experiments, we adopt the ABC tokenization for maximum performance on the LLM. However, when the entropy of matrix A is low, ABC tokenization exhibits significantly lower initial accuracy and a higher Hellinger distance with short context length. We hypothesize that this is due to the increased likelihood of repetitive state sequences early in the sequence for example, 'AAAAA.... During pretraining, such repeated n-gram patterns are often filtered out, as they could cause loss spikes [38]. As a result, the model may have limited exposure to these patterns, leading to poor initial performance on such inputs.
ABC
125
random
Steady Uniform 0.75
Steady State: Uniform 0.75 A entropy: 1.5
Steady State: Uniform entroppy: 1
A enitropy::
0.30
0.475
Accuracy
0.35
0.430
0.50
0.425
0,400
0.45
0.30
0.375
Stuady State: Uniiform Aui 0.55
Steady State Siswed
Steady Sta to: Viry Skowed Ag: 0.99
Ag: 0.99
entrrop::
entropyy 0.1
embrapy.
0.45
0.75
0.75
0.40
0.70
0.70
0.35
0.30
0.60
0.60
24 12
61
Context LengthtLength
Figure 21: Accuracy of three tokenization methods across different mixing rates (A2), A entropy, and steady states with 4 states, 4 emissions, and 1 for B entropy.
ABC
123
random
Steady As: 0.75
Steady Uniffrm 0.75 A entropy: 1.5
Steady As: 0.99
A eentropy:
A eentropy:
Hoilingor Distance
0.450
0.38
0.46
0.36
0.84
0.400
0.34
0.375
0.32
0.40
zee
323
Steady State: Uniform Ap:
Steady State: Stowed
Strady State: Very Skewned Avt 0.99 entmopy
Ay: 0.90
knntropy:
enitrpy:
Hellinger Distannc Distance
0.45
0.3
0.40
0.2
0.35
0.1
204
124
Context Length
Figure 22: Hellinger distance of three tokenization methods across different mixing rates (A2), A entropy, and steady states with 4 states, 4 emissions, and 1 for B entropy.
33

---

Spectral Learning HMMs for Prediction Task
We use [X] to denote the element of matrix X at its i-th row and j-th column. The indicator function 1 (x=x) is only when and is 0 otherwise. We use 1, denote vector of all I's with dimension M. We the notation [Z] {1, 2,, I.]. ||-Il denotes the Frobenius norm for matrices, and depending on the context it denotes & or &g norm for vectors.
Notations:
Algorithm 6: Spectral Learning- Based Prediction
Input: Number of hidden states M, number of observations L, sequence (OI,
Output: Conditional probability distribution P(Ox+1 = 01:N)
Estimate empirical probabilities: for all combinations i,J; n E [T.]
Pils N
Ek=1 {og=i};
N
P Palss Lk=
N
[P Pa,njaj < N end
Ek=
(o.=i.0--10.00--22
Compute SVD for dimensionality reduction:
Ú left singular vectors of P2 corresponding to M largest singular values;
Estimate spectral parameters: b, +- UTP
b.x
U)
for each observation o E [L] do
C. - P PP)):
end
Hidden state belief update: bj initial belief;
for T = 1 to N do
6
C b, bI C.,
end
Conditional probability prediction: for each possible next observation ON+1 E [4] do
P(ON+1 ON+1101.N )1:N) b CaN bx+1
Σh. b% C.bx-1"
end return PIO =0.N)
F. Preliminaries
For Markov chain with transition matrix A, we let E R- denote the initial state distribution. We assume that π also the stationary distribution of the Markov chain. This can be achieved by taking samples after burn-in time which is proportional to T-Xx(A] Note that
(A') π is essentially a convex combination of rows matrix A', then by triangle inequality, we have || - T 1l S
max ((.)). Tool/1- Thus, for an ergodic Markov matrix A, we define the following to
quantify the convergence of || ग Il2. For an ergodic Markov matrix A E R let TMC > 1 and PMC E (A2(A), 1) be two constants [26, Theorem 4.9] such that
max (I(A))).) Too ll S TMCPic-- M
(4)
Furthermore, we define the mixing time of A as
(Mc(E) == min { t N max
II([A*)..) Tooll1 S (E
(5)
Note that 7(M) and TMC have similar roles except 7(M) usually used to study state matrices while TMC is for Markov matrices. For square M, we have |M*| S rMM)),(M)*, and for Markov matrix, have A 1M S TMC
34

---

F.2 Sample Complexity Analysis
In this section, we analyze the sample complexity of spectral learning algorithm (Alg 6) when the observation sequence is coming from single trajectory. Our proof builds on [21] by modifying their analysis in Appendix A to incorporate single trajectory learning. We only present the Sample complexity analysis here and refer the reader to [21] for the remaining proofs.
F.3 Proof of Theorem
Fix 2 < T < N, and recall from [21] that [P1]:
E
[P2k, (o
E[1 (or mi,or - 1=رب
[P3,,],, E (or= =ko 2 - for all [L], when the initial distribution π is the stationary distribution of the Markov chain. the following, we will present three different estimators for cach of these quantities and analyze their convergence.
Estimation P1: Let N == | J. and without loss of generality, suppose N is an integer. Suppose 3 be the i.i.d. samples obtained from N independent trajectories of the HMM. We define the following three estimators of P1.
for (ot) ..... of )
N 1 (OLT e=i)
k= Σ 1
P1] 2 (ox=
(P(0) KK=1
for
(2)
=i}
(6)
N
for all f= 0, - By triangle inequality, we have
P - P S P P [P( Pill
[21] showed that, with probability least 6, we have,
(iP(L)) Pal -
log 1/6)
In the
following, we will upper bound the term || p by considering entry-wise concentration of each f-th subtrajectory as follows: We have
Ek=1 1 ((oTT (000) P 1P
(8)
N
First, we observe that E (our- 1 (aff =1)
0. Moreover, (our 1 (0) - S
almost surely. However, the summation in (8) has weakly dependent terms. Therefore, we use the Bernstein type inequality for class of weakly dependent and bounded random variables proposed in [35]. Before that, we need to upper bound the variance of the summation in (8). Observing that
1 [P|
O, we have,
Var (IP1)) -
(P()). 11) = E
(P.
[P] 11)) 2
E
(Ip(0)
(IP(1)
27
2
[
(
(9)
In the following, we will upper bound each term in (9) separately. We begin with,
N
Ñ
(1099)
E
E
N2
N
E
1
2
Ñ
N
N
N
B
B T diag TT) Ik-Akk
(10)
N
N2
35

---

Next, we have,
N
E
(1Pth)))
E
Σ
Σ
1 -1)
N
(0p
10z
N
E
11.0z 22 113
N2
10z
k=
N
B TT
B x|
Σ
Σ
(0 = i, 1.0%² =1)
+ (N - 1)
(11)
2
N
N
1=1
Lastly, we have
N
(P\%).
(PTt)
E
N2
E
(oor-
Cour
(og
k=
==
NV
N
k=
Ñ
Ñ
(OuT- =i) -B m|.
IP
(12)
k=1k=1
Combining (10), (11), and (12) into (9), we get
Var (1O(9) -IP(1)))
B π
diag (T) 1k-
N2
B T|
(N + 1)
N
2
[ T
Ñ
B "'dii(()) llt
N
N2
B T|
(N - 1)
N
2(b
(b, )T
N
N
+
by diag(Tr) (AlR
1MTT) b,:
N2
K
6
T
b. π (b,T)² 7)²
(13)
N(1 Plac)
where b; denotes the i-th column of B and get the last inequality by choosing,
| T log (b.T (b. )1 PlIc) ) )(11p)
(14)
Hence, using the Bernstein type inequality for weakly dependent and bounded random variables (Theorem 1 in [35]), together with (13) (14), and the observations we made right after (8), with probability least 1 d, we have
IP (b. T (b.TT )) log 8
[P
IP
(15)
36

---

Union bounding over all i [L), and E {0, .o....... T 1}, with probability at least 1 - ô, we have
||P|0 - 1|
B π
B
log
LI
(16)
N
given T 2 max { log
|
//1 p). This further implies that, with
(b, - (b, -1-)0)
probability least 1 5, the same upper bound also holds for [P] P Combining this with (7) and [21], with probability at least 8, we have
log(1/o) N
I
B
B π
LT
[P PI
log
(17)
N
Estimation of P2: Here, we follow a similar line reasoning as above. We begin with defining the three estimators of P2 as follows,
[P2las Ek=1 1 (o-=f,0--1=/ P2- bag
P Σx=1
N
N
2k=1
(18)
N
Similar to Pt, we consider the entry-wise concentration of each Ł-th subtrajectory as follows,
-PP( Ex
(Morr (oor--e=hone -ncee-=-orrr--o (-1=)) ( (or o =)
1 - 11p(")1.5
(19)
N
Observing that E
[ 1a.j (1/2)
have,
Var (IP( [P (i) =
(IP! - 1o. )².
E
(PP |n,) ) + { (w))
2E
[p- 1n
(P
1w
(20)
In the following, will upper bound each term in (20) separately, We begin with,
2
1
E
E
N2
Ñ
N2
k= ===1
Σ
-
N2
N
π
D
LM
TDDAA
(21)
N
N2
where, given the i-th column b, and the j-th column b, of B, define
D j.a == diag (b,) A diag (b,).
(22)
37

---

Next, we have
N
N
E
[PP
11.9)
12
=
N2
11 10z 1
(4)
=ン}
104
=1
ff1 =/
R
NV
2
E
1
loy =1.07
(4) (8)
1= -1,00 =
k= V=1
N
N
Σ
Σ
P (0))) = = T-2 = j
10 (K)
i, 0 -f-1=
k= ===1
D
MM N
(N -1 ( J
DJ. 1
(23)
N
Lastly, we have
N
I [IP
(es)
(126
1
E
fof =1,0 (81) 1=st
N
N
E
2
2
ΣΣ
P
=i)
(T Dy.1M)
(24)
Combining (21), (23), and (24) into (20), we get
N
N
Var (P2/))ss
[P] (P(1)) (i.)))
2T D
D
DD D
1k 1
N
N2
( T (
(N + 1)
N
2 (TT D Dj,lM
(i D ()()
N
N
D3 (Kl--
IMT 1) DJLLM:
π
R'= k'tk
T
DjAIM (T D3.elM)²)
| DD,s Ml D 1M MIlITMOPic
T-1
V2
N
N PMcc)
M D
M (π
DUD 12)
(25)
where we get the last inequality by choosing,
π T D Dj.5 1 I
T 121+ log
/11 /(1-p). p)-
(26)
(
Dy, 1 Ms )2) (1 -1-
Hence, using similar line of reasoning as we did in the case of Pl, with probability at least 1 have
6, we
Et.- i T D
DjAlM (π T D, D2,1a)?)
1.27
|| (2)
log
(27)
N
8
38

---

given T' 1+ je(z] log ( D w-r D PP /1- p). This further implies
x| Dy.ill TaMe
that, with probability at least 8, the same upper bound also holds for ||P
P21 Combining
this with the triangle inequality and [21], with probability at least 1 - ō, we have
(1/8) N
Eny=1 (x D Diclar las
(π D 32)
I.2T
PP P2l
log
N
N
(28)
Estimation of P3: Here, follow a similar line of reasoning above. We begin with defining the three estimators of P3 as follows,
.V. Ex=1 1(0==i00-
Palls = Ex wow.-
PPa.alss =
N
N
Ex
(off) -0
n - =1]
[
k,
(29)
Following the same line of reasoning as we did in the case of P2, with probability least have
3,
PI.n
log (1/))
Engn=l (π T Djnn - 1M (r TD Jmn 1m)
L
log
N
N
8
(30)
provided that,
π D j.m.a II Dj 1 TMC
) p).
2 2 max (.
log
(31)
(T
D
j.n. las - ( D jan 1 (1- (1-
where, given the i-th column b, the j-th column by and the n-th column b, of B, we define
Djn.s = diag (by) diag (b,) diag (b;)
(32)
Finalizing the proof: Theorem follows by repeating the proof of Theorem 7 in [21], with the i.i.d. estimators replaced by the single trajectory estimators, and the values of é,, E2,1 and 63,x,1 replaced by,
log(1/8) N
I
B
I T 1
LT
1 S
log
N
log(1/8)
D El
Disls
L2T 5
2,1 S
log
N
N
log(1/5)
I
ΣL
Eij.m (Tr T D (α T (D)...))??)
L3T
63,2,1 5
log
N
N
8
where = |
The proof is completed by upper bounding the Hellinger- O (A))).
distance in terms of KL-distance.
39

---

G
Additional Real World Experiments
We design additional experiment using real-world datasets validate our findings. We artifi- cially simulate different emission entropy levels for the same underlying hidden transition process by controlling the amount of information included in the observation sequence. Using complete information corresponds to low emission entropy, while limiting information artificially increases emission entropy.
We use the IBL. decision-making mice dataset [25]. In our LLM in-context learning experiment, we implement four ablation conditions that vary the information presented in each trial: (i) "choice only"; (ii) "choice reward"; (iii) "stimulus choice"; (iv) "stimulus choice reward". Note that the baseline GLM-HMM uses all available information in condition (iv). These ablations describe the same underlying mouse decision-making sequences but with varying levels of environmental state detail.
Choice
100%
Stim & Choice
90%
Stim Choice Reward
GLM-HMM
Average Accuracy
80%
Chance level
500
1000
1500
2000
2500
3000
Sequence Length
Figure 23: LLM in-context learning prediction accuracy for mice decision decision-making task with varying types of information in the observed sequences. Each line averaged over mice, with 1-o error bar. The model we use is Qwen2.5-7B.
The results shown in Figure 23 reveal significant differences across ablation conditions: while "stimulus choice reward" achieves performance exceeding GLM-HMM, "choice reward" merely chance level with its convergence trend similar to the synthetic experiments when the transitions or emissions are random. This demonstrates that accurately modeling mouse decision-making in this task requires both stimulus and reward information.
These findings highlight broader principle: obtaining appropriate information (corresponding to low emission entropy) essential for successful task modeling. This experiment parallels woal-world experimental design, where scientists must choose which signals to collect when studying task structure. When researchers omit critical information needed to describe sequence, can easily lead to incorrect conclusions about the underlying process.
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims in Section 1 are accurately describing our primary claims of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in Section 6 our limitations, and explain our assumptions in Section
3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
41

---

Answer:
[Yes]
Justification:
The proof of our conjectures are in Appendix with detailed assumptions and
cross-references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
42

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: provide an anonymized version of data and code for as supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
Serriidee/CodeSummissee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httpssccccpbccccccccpec.ppcccclt /ttllii//iieeeeeee gewdee//eeeeee CooeeSubmi ssi aesonPooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detailed the experiment details in Section 2 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Figure 6 and Figure 7 are plotting statistical significance error bar. Figure and Figure are showing the average accuracy over 4096 per setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
43

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Resources for experiments are discussed in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///eerripp.eeeee www.errr...cpuuuiieeeeeeeeeeeeee http/hhhicciiieeeeee.e.e..
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 4 and Section discuss broader impact of our discovery to practitioners and scientists. We do not perceive negative societal impacts yet with our findings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
44

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper doesn't 't pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Models and datasets are cited in Section 2 and Section4.1.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode. com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
45

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper doesn't release physical assets. We introduce findings that easily replicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper doesn't it involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper doesn't 'ttt involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
46

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Though our paper is discussing in-context learning of pretrained LLMs, core method development does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https //t////eenpp///ee/e hwtpe//c/oooeeeceeee/aaaaaa /LLM) for what should or should not be described.
47


Reviews:
[{"review_id": "Kdc1xhRl4i", "reviewer": "Reviewer_1stE", "summary": "This work evaluates the ability of pre-trained LLMs to model data generated by Hidden Markov Models (HMMs). The work evaluates LLMs' performance on diverse synthetic HMMs, varying parameters including mixing rate, entropy level, and number of hidden states and observations. They find by leveraging in-context learning (ICL) abilities, LLMs are able to converge on theoretically optimal prediction performance in certain settings (i.e., low entropy and high mixing rate), and suffer in settings where optimal prediction is poor as well. The experiments also show that pre-trained LLMs are able to outperform state-of-the-art across various HMM settings. The work provides theoretical conjectures to help understand the performance of these pre-trained LLMs and provides practical suggestions for how to use pre-trained LLMs with ICL as diagnostic tools for complex data. Finally, they show ICL achieves competitive performance for understanding model dynamics in real-world animal decision-making datasets in simple settings, but struggles in settings similar to when it struggled in synthetic experiments (i.e., complex settings).", "questions": "1. Are LLMs able to discover the underlying HMM mechanisms well (i.e., stationary distribution, transition/emission matrices, etc.), or is their strong performance constrained to prediction? \n2. How can we guarantee findings on using LLMs for HMM prediction can translate to inputs/fields with respect to the guidelines on using LLMs as diagnostic tools for data structure and learnability? \n3. How do we expect transformer models to perform a a baseline for this task (using the sample training set-up as an LSTM)? Are the LLMs strong because of their training process, or is it simply the transformer architecture which is good for this prediction task? \n4. What is the relationship between ICLs with LLMs and spectral learning that underlies the theory in Section 3.3?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The work provides a nice background of HMMs and sets up a clear story that is followed in the rest of the work. \n2. Overall, the empirical results for this paper are strong and clearly answer the research questions laid out by the work: Can LLMs predict next states in HMMs, what properties of HMMs affect the LLMs ability to do so, and how does this translate to real world data. \n3. The work accurately isolates entropy and mixing rates as interesting axes to vary in the HMMs, and do a good job of characterizing when LLMs fail in these different settings. \n4.  The work does a good job of comparing to a theoretically optimal model to show both 1) LLMs are able to approach this model in some settings, and 2) the places where LLMs suffer are also where optimality is not guaranteed. \n\n\n**Weaknesses**\n\n1. It is difficult to read the different graphs and result plots due to their small size and small legend text. Please consider larger images or tables for displaying results, especially those in Figure 3 and Figure 5. \n2. The relationship between using ICL with pre-trained LLMs and spectral learning remains unclear. The work provides a theoretical characterization of spectral learning for HMMs, but does not describe why we can view these results as a potential explanation for the ICL results due to a lack of connection between spectral learning and ICL using LLMs. Accordingly, Section 3.3 seems quite disconnected, and the theoretical results do not seem meaningful to the rest of the paper. \n3. Related to the point above, there is limited understanding of why LLMs are good at this task. Is it the training process? The transformer architecture? Size of the model? What inherent to LLMs is resulting in strong performance. More experiments ablating different parts of LLMs could be useful for answering this question. \n4. The findings from the reward-learning rats dataset are underwhelming. The work finds that LLMs using ICL are unable to perform well on this task compared to a baseline, and hypothesize that this is due to the complexity of the underlying dynamics. But in these complex settings, even optimal algorithms should suffer (as seen in the synthetic results). However, the baseline model performs noticeably better than the LLM, so I am not convinced it's simply too complex of a setting for the LLMs. \n5. The work generalizes beyond the findings provided in the paper often. For example, Section 4 provides guidelines for how scientists can use their findings to use LLMs as data-discovery tools in new use-cases outside of synthetic HMMs, without discussing whether their results may not hold in new settings or for new data-discovery problems. I would caution the authors from making such grand statements given their findings. \n\n**Comments**\n\n1. The caption of Figure 4 seems incorrect, as the slower mixing rates are with the higher $\\lambda_2$ values. \n2. I would caution the use of \"zero-shot\" for these models. Using in-context learning is inherently few-shot as examples of predictions/sequences are given to the model.", "was_revised": true, "final_justification": "New experiments and justification for the impact of the paper through the author discussion. Score is contingent on authors clarifying the terminology of \"learning HMMs\" in the updated version of the work.", "rebuttal": "Thank you for your thorough review and thoughtful suggestions. We address your questions and concerns below.\n\n> **Q1**: Are LLMs... constrained to prediction?\n\nOur findings show that LLMs ICL performance extends beyond simple “argmax” prediction. Using distributional metrics such as Hellinger distance (Main Paper Fig. 5, right; App. D Fig. 3), we observe that LLM‑predicted distributions converge toward the ground‑truth next‑emission distributions of the underlying HMMs.\n\nMotivated by the needs of scientists analyzing real‑world dynamical systems, our goal is to assess pre‑trained LLMs as statistical tools for structured sequential data rather than to explain their internal mechanisms. Future work will investigate how LLMs internally approximate latent probabilistic structures, which may illuminate the mechanisms behind in‑context learning.\n\n> **Q2**: How can we guarantee... learnability?\n\nWe interpret this question as aligned with the reviewer’s concern in **W5** regarding whether the guidelines in Section 4 generalize beyond the specific findings in this paper.\n\nOur decision to use synthetic experiments on HMMs is directly motivated by the scientific use case we stated above. We deliberately chose HMMs for two reasons: \n1. HMMs are an expressive class of generative models that are broadly used in scientific modeling—many real-world discrete-time dynamical systems [1-3] can be represented as or reduced to HMMs.\n2. HMMs capture how real-world scientific data is often structured: scientists often start with hypotheses about latent dynamics and choose observables accordingly [4,5]. This results in the fact that we often only can obtain partially observable data—a setting for which HMMs are a natural formalism.\n\nWe also want to emphasize that the guidelines in Section 4 are intentionally cautious and observational, not prescriptive. For example, we state: *“If you observe that an LLM’s ICL prediction accuracy on your data sequence steadily improves and saturates with increased context length (like in Figure 3), this strongly indicates a learnable, non-random underlying structure.”* We do not claim this is a necessary and sufficient condition for learnability. Rather, the goal is to provide a useful heuristic for scientists who may not be experts in LLMs but are trying to assess whether their sequential data contains structure that can be exploited.\n\nLastly, regarding the possibility of guaranteeing the utility of LLMs for real data: in scientific domains like neuroscience, practitioners commonly use models such as LSTMs [6,7] to fit behavioral or neural sequences—despite limited theoretical guarantees. LLMs ICL plays a similar practical role, offering the additional benefit of being usable out-of-the-box without task-specific training.\n\n> **Q3**: How do we expect transformer... prediction task?\n\nThis is an excellent question that will be helpful for us to understand the internal mechanisms of LLMs ICL’s strong performance on HMMs. We have included an additional experiment here as a first step in response to this suggestion, and will add the full set of results to the final version of the paper. Specifically, we train a 2-layer transformer model using the same training setup as in our LSTM experiments (App. D) with the same hidden/feed forward dimension and the same embedding dimension. For each evaluation, we average performance over 16 training samples, as we do with LSTMs. We provide a comparative table below under the same setting as Figure 5 (8 states, 8 observations):\n\n|seq_len|4|8|16|32|64|128|256|512|1024|2048|\n|---|---|---|---|---|---|---|---|---|---|---|\n|LLM_acc|0.3962|0.4436|0.485|0.5095|0.5569|0.552|0.542|0.5457|0.5605|**0.564**|\n|LSTM_acc|0.1875|0.3125|0.3125|0.0625|0.375|0.5|0.375|0.375|0.3125|**0.5**|\n|TF_acc *(new)*|0.375|0.4375|0.5625|0.5|0.5|0.5625|0.4375|0.6875|0.375|**0.563**|\n|LLM_hellinger|0.512|0.4673|0.4436|0.4316|0.4162|0.4165|0.4157|0.4073|0.4026|0.4026|\n|LSTM_hellinger|0.5598|0.526|0.5146|0.5856|0.4804|0.5179|0.4846|0.4582|0.4811|0.4294|\n|TF_hellinger *(new)*|0.5391|0.5099|0.4786|0.5237|0.4721|0.4425|0.4503|0.4608|0.4435|0.4251|\n\nAs shown, the trained small transformer model performs more similar to the pre-trained LLM (Qwen2.5 7B) than the LSTM baseline does. While the trained models exhibit higher variance due to limited sample sizes, these findings suggest that the transformer architecture itself plays a meaningful role in performance. In parallel, large-scale pretraining likely enables LLMs to leverage in-context learning to approximate the benefits of task-specific training.\n\nWe appreciate the review’s suggestion and will incorporate the trained transformer baseline into the synthetic experiments section of the paper.\n\nRelatedly, in **W3**, the reviewer requested an ablation study on LLM size. We have conducted ablations on both model size and tokenization (App. E) from the perspective of tool usability. However, we did not emphasize these results in the main paper, as they did not reveal useful scaling trends.\n\n> **Q4**: What is the relationship... theory in Section 3.3?\n\nWe interpret this question associated with the reviewer’s concern in **W2**—that the paper does not clearly explain why the theoretical characterization of spectral learning for HMMs may provide insight into the ICL results.\n\nOur primary motivation for introducing spectral learning is the hypothesis that LLM ICL behaves as an input–output predictor, effectively performing a form of smart filtering over observation sequences. This contrasts with explicit parameter‑estimation methods like the non‑convex Baum–Welch (BW) algorithm, which may fail to converge to the global optimum. This perspective is supported by the distinct behavioral patterns we observed between LLMs and BW across experiments (Sec. 3.2, Fig. 5; App. D).\n\nThe second motivation is that we desire a notion of sample complexity relating to accurate prediction and complexity of the underlying dynamics. Spectral methods provide sample‑complexity guarantees for HMM prediction tasks in terms of the structural properties of the underlying process (e.g., mixing rate and emission rank). This gives a principled way to interpret the LLM ICL results (Sec. 3.1, Fig. 4), and motivates our derivation of a sample complexity bound for spectral learning on HMMs.\n\nTo further support this connection, we added an additional experiment by implementing Algorithm 6 (App. F), a variant of spectral learning with theoretical performance guarantees in Theorem 1, and evaluating it on the same synthetic HMM setting used in Figure 5 (8 states, 8 observations). The results are discussed in our response to reviewer oWyV (W1 & W2) and are consistent with our hypothesis: LLMs behave more like spectral learners than EM‑based methods like BW.\n\nWe acknowledge that this conceptual link between ICL and spectral learning could be emphasized more clearly in the main paper and plan to highlight it in a revision.\n\n> **W4**: The findings from the reward-learning rats...\n\nThis is a very thoughtful assessment.\n\nThe CogFunSearch baseline (App. H.2, “Rat FullModel” in [8]) represents a cognitive model with latent states reflecting mice’s internal strategies and emission probabilities corresponding to their decisions. This naturally maps to an HMM framework.\n\nImportantly, CogFunSearch achieves its strong performance through extremely computationally expensive search over model structures, fitting each mouse individually—taking weeks to complete—so its results are close to an “optimal algorithm” benchmark.\n\nWe want to further address the reviewer’s concern about the numerical performance gap:\n1. Neuroscientists typically evaluate models by normalized likelihood over full sequences, aggregated per animal. By this metric (used in [8]), LLM ICL achieves competitive performance for 3 out of the 20 animals (CogFunSearch 0.672±0.012). In our paper, we chose to visualize prediction accuracy versus sequence length to highlight that LLMs ICL curve is flat, which mirrors our synthetic experiments when the normalized entropy of transitions and emissions is high (e.g., App. D, Fig. 2: 8 states, 4 observations, A entropy ≈ 2.5).\n\n|Animal|M14|M23|M25|M29|M27|M46|M15|M17|M43|M37|M42|M22|M18|M40|M16|M31|M41|M47|M44|M39|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|LLM_nl|0.6773|0.6721|0.6671|0.6411|0.6378|0.6179|0.617|0.613|0.611|0.607|0.6041|0.6036|0.5913|0.5893|0.5882|0.5783|0.5726|0.5651|0.5542|0.5494|\n \n2. The dataset has only two possible observations, so even modest numerical gaps can appear visually large. The Figure 7 gap corresponds to a (0.864 - 0.758)/0.5 ≈ 0.21 over chance, which is comparable to our synthetic results comparable to our synthetic high‑entropy HMM results (e.g., above setting 0.05/0.25=0.2).\n\n> **W1** and Comments\n\nThanks for the suggestions on style improvements and catching the typo. We will surely update those, incorporating the additional experiments.\n\n---\n\n[1] Mor B et al. (2021). *A systematic review of hidden Markov models and their applications*. Arch Comput Methods Eng 28:1429–1448.\n\n[2] McClintock BT et al. (2020). *Uncovering ecological state dynamics with hidden Markov models*. Ecol Lett 23:1878–1903.\n\n[3] Noé F et al. (2013). *Projected and hidden Markov models for calculating kinetics and metastable states of complex molecules*. J Chem Phys 139:18.\n\n[4] Gershman SJ & Niv Y (2010). *Learning latent structure: carving nature at its joints*. Curr Opin Neurobiol 20:251–256.\n\n[5] Gershman SJ et al. (2010). *Context, learning, and extinction*. Psychol Rev 117:197.\n\n[6] Jordan MI. (1986). *Attractor dynamics and parallelism in a connectionist sequential machine*. Proc CogSci 8.\n\n[7] Wills TJ et al. (2005). *Attractor dynamics in the hippocampal representation of the local environment*. Science 308:873–876.\n\n[8] Castro PS et al. (2025). *Discovering symbolic cognitive models from human and animal behavior*. bioRxiv."}, {"review_id": "yIw7gdIzo5", "reviewer": "Reviewer_oWyV", "summary": "This paper shows that pre-trained LLMs can learn Hidden Markov Models given in-context. Interestingly, they achieve near-optimal prediction accuracy that converges to nearly the Viterbi algorithm optimum with true parameters.. The authors systematically vary HMM properties (entropy, mixing rate, state dimensions) and show that convergence speed depends critically on these characteristics: faster mixing and lower entropy lead, for faster convergence. Comparing against baselines (Baum-Welch, LSTM, n-gram), LLMs consistently outperform traditional statistical approaches. The authors provide theoretical conjectures linking ICL behavior to spectral learning algorithms. Finally, the authors test on real-world animal decision-making datasets, where ICL achieves competitive performance with expert-designed models on mouse choice behavior and meaningful performance on complex rat reward-learning tasks. The work then establishes practical guidelines positioning LLMs as \"zero-shot statistical tools\" for scientific discovery.", "questions": "Q1: In Fig 4 caption, isn't smaller lambda 2 fast mixing and large lambda 2 slow mixing?\nQ2: There must be limits, right? Are there extreme cases where LLMs cannot figure out the hidden state well?\nQ3: Have you observed any sudden transitions? i.e. the LLM \"grokking\" that there exists a hidden state?\n\n---\n\nQ4: This is a pure curiosity driven question and will not affect the score at all negatively if ignored: What is you gave the context then natural language prompted the model (+CoT) to predict the next state?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "S1: The paper is very clear and well written. All figures, terms and definitions are easy to follow and well motivated.\n\nS2: The paper spans theoretical grounding, synthetic experiments and real data experiments. The findings connect well.\n\nS3: The paper validates their finding using multiple models(appendices) and using multiple metric and multiple HMMs.\n\nS4: The paper goes further and provides guidelines for using LLMs for HMM identification.\n\n---\n\nW1: While the paper demonstrates some connection to spectral learning theory, the predicted learning curves are not explicitly fitted/compared to the empirical in-context learning curves.\n\nW2: The connection of the theoretical section to the rest is relatively weak", "was_revised": true, "final_justification": "The authors have answered the questions i raised adequately.\n\nThis paper is a thorough and complete investigation on using LLMs to \"fit\" hidden Markov models.\n\nI maintain my positive score.", "rebuttal": "Thank you for your insightful and helpful feedback. We respond to your individual questions below.\n\n> **Q1**: In Fig 4 caption, isn't smaller lambda 2 fast mixing and large lambda 2 slow mixing?\n\nYes. Thanks for catching this, we will update in the next version.\n\n> **Q2**: There must be limits, right? Are there extreme cases where LLMs cannot figure out the hidden state well?\n\nYes. During the synthetic experiments, we did observe cases (e.g., Appendix D, Fig. 2: 8 states, 4 observations, A entropy ≈ 2.5) where LLMs ICL failed to converge to the Viterbi algorithm’s performance. The detailed discussions are in Appendix D.4.\n\n> **Q3**: Have you observed any sudden transitions? i.e. the LLM \"grokking\" that there exists a hidden state?\n\nThis is an insightful question. Indeed, in-context learning is an emergent property, which appears during later stages of pretraining. A related theoretical work [1] studies how ICL emerges during pretraining. However, a key difference between this and our experiments is training versus inference.\n\nFrom our experiments, we do not observe any sudden transitions. Instead, the performance improvements of LLMs ICL are gradual and steady (as shown in Main paper Figure 3, Appendix D multiple Figures). Such behavior is one key reason that we hypothesize LLMs ICL operate as input–output predictors, effectively performing a form of smart filtering over observation sequences.\n\n> **W1** & **W2**: While the paper demonstrates some connection to spectral learning theory, the predicted learning curves are not explicitly fitted/compared to the empirical in-context learning curves.\n\nFollowing the reviewer’s suggestion, we added an additional experiment by implementing Algorithm 6 (Appendix F), a variant of spectral learning with theoretical performance guarantees in Theorem 1, and evaluating it on the same synthetic HMM setting used in Figure 5 (8 states, 8 observations). The comparative results are shown in the table below:\n\n| seq_len | 4 | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | 2048 | 4096 *(new)* |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| LLM_acc | 0.39624 | 0.4436 | 0.48999 | 0.50952 | 0.55688 | 0.552 | 0.54199 | 0.54565 | 0.56054 | 0.56396 | **0.58154** |\n| Viterbi_acc | 0.55762 | 0.54932 | 0.56348 | 0.54614 | 0.56567 | 0.55859 | 0.544189 | 0.54785 | 0.56006 | 0.56323 | **0.57983** |\n| BW_acc | 0.49609 | 0.42407 | 0.41406 | 0.40576 | 0.43262 | 0.44556 | 0.43823 | 0.44775 | 0.47363 | 0.49097 | **0.50171** |\n| Spectral_acc *(new)* | N/A | 0.20728 | 0.22974 | 0.28393 | 0.34253 | 0.41064 | 0.49072 | 0.51318 | 0.53296 | 0.54248 | **0.56494** |\n| LLM_hellinger | 0.51198 | 0.4673 | 0.44358 | 0.43160 | 0.41619 | 0.41653 | 0.41573 | 0.40733 | 0.40264 | 0.40261 | **0.39427** |\n| Viterbi_hellinger | 0.36416 | 0.36221 | 0.36225 | 0.36751 | 0.35555 | 0.36485 | 0.37032 | 0.36448 | 0.36376 | 0.36312 | **0.35438** |\n| BW_hellinger | 0.54587 | 0.57499 | 0.59736 | 0.58332 | 0.53185 | 0.4899 | 0.46357 | 0.43798 | 0.42707 | 0.41467 | **0.41107** |\n| Spectral_hellinger *(new)* | N/A | 0.51126 | 0.5459 | 0.55643 | 0.51654 | 0.4723 | 0.40133 | 0.37927 | 0.36355 | 0.35434 | **0.35608** |\n\nWhen the sequence length is short (≤ 64), the hidden‑state belief update b_\\hat​ becomes numerically unstable because the sample complexity of spectral learning scales as $\\mathcal O (M^2 L)$, which is 512 in our setting. Once the context length reaches ≥ 512, spectral‑learning predictions converge distributionally to the ground truth, and their performance aligns with that of the LLMs, unlike the BW‑based predictions. At very large context lengths, spectral learning yields smaller Hellinger distances to the true output distribution, whereas the LLM achieves slightly higher accuracy. This occurs because LLMs tend to over‑concentrate probability mass on the top choice—boosting accuracy but increasing divergence for the rest of the distribution.\n\n> **Q4**: What if you gave the context then natural language prompted the model (+CoT) to predict the next state?\n\nWe appreciate the thinking engagements of the reviewer! Although we didn’t include any experiments using natural language and reasoning models in this paper, we did experiment with this setup directly on neuroscience real-world data. What we found was that the LLMs struggle to do reasoning in the meaningful way that we hope. More importantly, when natural language prompts (e.g., the mice's experimental contexts) are mixed into the dynamical sequences, the prediction accuracy actually has decreased significantly. It is an interesting future work to understand what happens to the HMM sequential prediction task when natural language prompts are introduced.\n\n---\n\n[1] Edelman, Ezra, et al. \"The evolution of statistical induction heads: In-context learning markov chains.\" *Advances in neural information processing systems* 37 (2024): 64273-64311."}, {"review_id": "TST1B0pwWA", "reviewer": "Reviewer_aPua", "summary": "This paper investigates whether pre-trained large language models (LLMs) can implicitly learn and predict the behavior of Hidden Markov Models (HMMs) through in-context learning (ICL), that is, by simply being provided with sequences generated by HMMs without explicit parameter updates or task-specific training. The authors systematically evaluate LLMs on a range of synthetic HMMs with varying properties, as well as on real-world animal behavior datasets.\n\nThe authors show that LLMs can converge to predict the next observation in HMM-generated sequences compared to an oracle defined with the Viterbi algorithm applying on the HMMs that were used to generate the sequences, given sufficient context. They also show the LLMs’ performance and convergence trends are closely linked to intrinsic HMM properties such as entropy and mixing rate. And they have also conducted additional experiments with real neuroscience datasets, and provide practical guidance for using LLM in-context learning as a diagnostic tool for exploring sequential data.\n\nIn summary, the study demonstrates that general-purpose LLMs possess an interesting capacity for modeling complex, latent-state-driven sequences and opens new avenues for applying ICL in scientific discovery and data analysis.", "questions": "- How are the initial HMM parameters (initial state, transition, emission probabilities) initialized for Baum-Welch training?\n- What stopping criterion or tolerance is used for Baum-Welch training, and how many iterations does it typically require?\n- How are HMM-generated sequences presented to the LLMs? Are there any special formatting or tokenization strategies used to maximize in-context learning?\n- What decoding strategy and parameters (e.g., temperature, top-k, greedy) are used for LLM next-token prediction during evaluation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality\nThis paper presents a comprehensive and careful empirical study, systematically evaluating large language models (LLMs) on both synthetic HMMs and real-world datasets, and comparing their in-context learning performance against strong statistical and neural baselines.\nWhile the empirical work is thorough and the main results are well-supported, one main downfall in my opinion is that ICL does not generalize HMMs for usuall applications for speech and handwriting recognition where HMMs are trained are generative models with multiple sequences a time.\n\nClarity\nThe paper is clearly written and well-organized, with a logical flow from motivation and background to synthetic experiments, scaling law analysis, and real-world application. The main contributions and findings are explicitly stated, and well-designed figures make the experimental results easy to interpret. \nHowever, some important implementation details, such as prompt formatting and LLM configuration, are only available in the appendix, making it difficult to understing the methodology without reading all the material\n\nSignificance\nThe paper brings to the table an interesting discussion on the easiness of modeling sequencial data with LLMs. \nOn the other hand, LLMs are generally more complex than usual architectures for HMMs, which can make this an utterly more complex approach to solve problems that could be easily solved with some tweking in HMM topology.\n\nOriginality\nThe paper introduces a novel experimental paradigm, systematically probing whether LLMs can perform HMM inference purely through in-context learning. \nHowever, the paper does not introduce new learning algorithms or theoretical frameworks, and the most original contributions are in empirical observation and framing rather than methodological invention.", "was_revised": true, "final_justification": "I acknowledge that I have read the rebuttal and reviews and will raise my final score to \"accept\". I think this a good, interesting paper", "rebuttal": "Thank you for your valuable review of our paper. We address each of your points below.\n\n> **Q1**: How are the initial HMM parameters (initial state, transition, emission probabilities) initialized for Baum-Welch training?\n\nWe initialize the Baum-Welch (BW) HMM parameters with random seeds using the actual number of states. This already gives BW more information than the LLM, which does not know the number of states.\n\nEach setting is averaged over 4,096 sample sequences to reduce variance in performance (line 106). We do not initialize BW near the ground‑truth parameters, as this would artificially favor BW in low‑sample regimes by effectively giving it prior knowledge of the HMM model.\n\n> **Q2**: What stopping criterion or tolerance is used for Baum-Welch training, and how many iterations does it typically require?\n\nThe Baum-Welch algorithm in this implementation uses the change in log-likelihood between iterations as its stopping criterion, halting when the absolute difference falls below a specified tolerance of 1e-6. This ensures the algorithm stops once the parameter updates no longer yield meaningful improvement in fitting the observed sequence. The number of iterations required for convergence typically ranges from 10 to 100, depending on factors such as the number of hidden states, the length and complexity of the observation sequence, and the random initialization of parameters. A maximum iteration cap of 100 is also set to prevent excessive computation in cases where convergence is slow or unattainable.\n\n> **Q3**: How are HMM-generated sequences presented to the LLMs? Are there any special formatting or tokenization strategies used to maximize in-context learning?\n\nAll results in the main paper are using upper case letters like ‘ABC’ to represent emissions (Appendix E.2). For each unique emission, we represent it by an upper unique letter and a space. We ensure that each emission would only get tokenized into one token. We performed an ablation on different tokenization strategies for LLMs (Appendix E.2), and found no significant differences on the prediction task performance.\n\n> **Q4**: What decoding strategy and parameters (e.g., temperature, top-k, greedy) are used for LLM next-token prediction during evaluation?\n\nDecoding strategy does not apply in our setting since we directly use the normalized logits for prediction. For example, given a sequence “A B A” with 2 possible emissions (A/B) the LLM would predict 0.3 probability for A, 0.4 probability for B, and 0.3 probability for the rest of the tokens in the vocabulary. We normalize the probability for the possible emissions such that they add up to 1 (3/7 for A and 4/7 for B). For accuracy, we directly take the argmax of the probability and compare it with the ground-truth emission. For Hellinger distance, we compare the probability of all possible emissions (in this case, [3/7, 4/7]) with the ground-truth emission probabilities based on the current hidden state (Appendix B).\n\n> **W1** Quality: ICL does not generalize HMMs for usual applications for speech and handwriting recognition where HMMs are trained as generative models with multiple sequences at a time.\n\nFirst, we clarify that our goal is not to replace HMMs as a model architecture in classical applications like speech or handwriting recognition. Instead, this work explores pre-trained LLMs ICL as a complementary statistical tool for analyzing discrete sequential data, particularly in scientific settings where sequences are often unlabeled.\n\nSecond, we acknowledge an inherent limitation: pre-trained LLMs operate on discrete tokenizations. As discussed in Section 6 (“Existing Gaps”), extending our approach from discrete observations with underlying discrete hidden dynamics to continuous, real-valued, or high-dimensional observations remains an open and promising research direction.\n\nFinally, our contributions are orthogonal to traditional HMM applications—we focus on demonstrating that there exists a lightweight, forward-pass-based tool for exploratory sequence analysis rather than a replacement for established generative modeling pipelines.\n\n> **W2** Clarity: Some important implementation details, such as prompt formatting and LLM configuration, are only available in the appendix, making it difficult to understand the methodology without reading all the material.\n\nDue to the main paper space limits, we had to push the implementation details to Appendices for finishing our story. We will work on a better content organization in the revised version. Specifically, we plan to add more details of the experimental setup (Appendix B), a detailed discussion about when LLMs fail to converge (Appendix D.4), and results of ablations on LLMs sizes and tokenizations (Appendix E) to the main paper contents.\n\n> **W3** Significance: LLMs are generally more complex than usual architectures for HMMs, which can make this an utterly more complex approach to solve problems that could be easily solved with some tweaking in HMM topology.\n\nWe appreciate the reviewer’s concern and address it in three parts:\n\nFirst, beyond using LLMs as a predictive tool on HMM sequences, a key contribution of our paper is demonstrating that LLMs with ICL can achieve near-optimal prediction on HMM sequences. As the community continues to explore the mechanism of ICL, we believe this new empirical observation provides a valuable step toward understanding how LLMs acquire and leverage latent structure without explicit training.\n\nSecond, the strength of LLM ICL is not solely due to the transformer architecture but also to large-scale pretraining. In contrast, “tweaking HMM topology” is not straightforward in practice. Our synthetic experiments show that Baum-Welch often fails to match LLM ICL’s robust predictive performance without significant parameter fitting or prior knowledge of the true dynamics.\n\nThird, pre-trained LLMs require no additional training and perform ICL via simple forward passes. In contrast, classical methods require either strong initialization for Baum-Welch or a large dataset to train RNNs effectively. In our experiments, using LLM ICL on average takes 0.14 seconds for a sequence with 4096 emissions on a H100 while it takes 10.15 seconds for a Baum-Welch algorithm to converge a single time on an Intel(R) Xeon(R) Gold 6426Y CPU, and 13.21 seconds for an LSTM model to converge on a H100. This efficiency gap comes from the ability of pre-trained LLMs to perform parallel inference over previous tokens without training, while Baum-Welch and RNNs depend on iterative expectation-maximization or sequence-level training pipelines. From a practitioner’s perspective, this makes LLM ICL a fast, convenient, and low-overhead tool for exploratory sequence analysis.\n\n> **W4** Originality: The paper does not introduce new learning algorithms or theoretical frameworks, and the most original contributions are in empirical observation and framing rather than methodological invention.\n\nOur work is motivated by the needs of scientists analyzing real‑world dynamical systems. By assessing pre‑trained LLMs as statistical tools, we demonstrate that LLMs performing in‑context learning can achieve near‑optimal performance on HMM prediction, a highly challenging task. These findings provide a foundation for studying how LLMs internally approximate latent probabilistic structures, potentially shedding light on the mechanisms behind in‑context learning. Similar contributions have appeared at NeurIPS, including works such as [1, 2].\n\n ---\n\n[1] Garg, Shivam, et al. \"What can transformers learn in-context? a case study of simple function classes.\" *Advances in neural information processing systems* 35 (2022): 30583-30598.\n\n[2] Gruver, Nate, et al. \"Large language models are zero-shot time series forecasters.\" *Advances in Neural Information Processing Systems* 36 (2023): 19622-19635."}, {"review_id": "YyIBcDKdiY", "reviewer": "Reviewer_EN5o", "summary": "The paper investigates the ability of a black-box Large Language Model to learn Hidden Markov Models in-context. This investigation is conducted primarily through empirical evidence derived from simulations, and through the lens of a prediction task, rather than by directly addressing probabilities. The paper showcases that LLMs can be used as efficient and capable predictors of the next token for data coming from a Markov process. The gap the paper aims to close concerns the \"hidden\" setup, where the states are only observed through emissions. Moreover, the paper somewhat evaluates (see comments below) interesting real-world data, which is considered to come from a Markovian process.", "questions": "Addressed in Weaknesses", "limitations": "Addressed in Weaknesses", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "# Strengths \n\nThe paper is very well-written and easy to follow, contributing to its accessibility. Its design is equally well-done, featuring clear and illustrative figures that greatly aid comprehension. Furthermore, the evaluation plots are nicely presented, enhancing the clarity of the results. The authors have thoughtfully addressed most intriguing aspects of Markov models (not as much of the \"hidden\" aspects, see Weaknesses), and their methodology for constructing the benchmark simulations is robust and well-conceived. Finally, the inclusion of the real-world data is particularly interesting and adds a valuable dimension to the empirical investigation.\n\n# Weaknesses\n\nSummery of the main weaknesses :\n1) The paper focuses on an LLM's ability in a *prediction* task for data sampled from an HMM. It's unclear how this relates to the LLM's ability to genuinely \"learn\" the underlying Markov model. In this case, the novelty is questionable, given its already known that LLM can do ICL for much more complex temporal data.\n2) The paper aims to address the \"hidden\" aspect of HMMs, but this is largely unaddressed in both the text and the simulations.\n3) Following (1) and (2), the scope is very limited. \n\nAbout point (1) : \n\nThe paper suggests LLMs can be efficient next-token predictors for data sampled from a Markov process, but this doesn't equate to learning a Markov model. Predicting the next state (maximum likelihood) is distinct from inferring probabilities (reconstruction), which is a primary function of Markov models. Specifically here, The \"overconfidence\" of LLMs, which might make them better next-token predictors, could paradoxically make them less convincing at reconstructing true Markov models - where the distribution of the next token should come from the transition probabilities. This appears to be the case in the evaluations where LLMs perform best on low-entropy transition matrices (where the most likely token has a high probability).\n\nMoreover on the same point, given the task is predicting next token, the LLM's memory bandwidth can allow it to use past observations (not the current) with lower sample complexity with comparison to traditional reconstruction algorithms which assume the data is Markovian. This also appears to be the case in the evaluations where LLMs easily learn models with fast mixing, indicating ample \"signal from the past\" in finite sequences.\n\nAbout point (2) : \n\nThe paper claim is to close the gap for \"hidden\" Markov models, but nonetheless, the paper barely address the hidden aspect of the HMM. There's a lack of analysis on how emission probabilities or the loss from misclassifying observations to hidden states affect predictor results. Also it's unclear whether sequential information is truly critical, or if the LLM is simply better at picking states (which is often more important in practice).\n\nAlso, the benchmarked graphs ignored the more interesting Markov process like a jumping processes or a cyclic processes which have unique properties.  \n\nAs a summery, the claim that LLM \"learns\" a HMM in ICL is not something the paper shows, but rather that an LLM can be a good next token predictor in ICL for a data sampled from an \"easy\" distribution like a simple Markov Model (no jumping process etc..).\n\nSmaller points : \n* The assumption that the real-world data example originates from a Markov process is not justified. It would be more robust to learn a Markov model from the real data and then sample from that as a benchmark.\n* The main paper doesn't contains enough details about the experimental setup, specifically regarding the emissions and their prior distributions.\n* Theorem 1 appears to be quite obvious for any learner, and it's unclear what makes an LLM particularly sensitive to these parameters.\n* In context of Markov models, an insightful experimental design would involve giving the model a sequence to learn, and then providing a single new sample to predict the next, rather than always providing the full context.", "was_revised": true, "final_justification": "In general I believe non of my points was resolved, and been honest, they seem to mostly been avoided. \nMy justication lay in those comments : \n\nThis paper receives a low score due to two fundamental flaws that significantly limit its contribution:\n\nWhile the authors claim the only gap to be fill is the bridge between fully observed and hidden Markov models, they didn't addresed what makes states truly \"hidden.\" The paper provides little focus on emission probabilities and ablation studies on this critical component. This is particularly problematic since emission variance directly determines the degree of \"hiddenness\" this crucial distinction is ignored despite extensive ablation on transition matrices.\n\nThe core novelty claim is undermined by a fundamental confusion between HMM learning and next-token prediction. True HMM learning involves parameter estimation, or at least some validated claim about the model implicitly decode the transition probabilities. Or, even some way to inference those via calibration or any other method. But, the paper only demonstrates predictive performance on the most likely token, which is not novel by itself, specifically given we don't know the emissions and ICL prediction capabilities on Markovien data are proven already. The authors acknowledge that distinguishing next-token prediction on HMM emissions from Markov chains is non-trivial. But, I do believe that given the limited scope to begin with, narrowing it further to the hidden aspects, and then the lack of ablations (comment 1), leaves the paper with insufficient novelty.\n\nAlso, there are many other, what i believe, are an insufficient or \"out of the point\" analyses and evaluations, all described in my comment.", "rebuttal": "We thank the reviewer for their feedback—we want to clarify that several comments appear to stem from **misunderstandings** of the problem setup and our experiments, which we clarify below.\n\n> *Sum1*. This investigation... through the lens of a prediction task, rather than by directly addressing probabilities.\n\nOur study does address probabilities. We evaluate Hellinger distance between the LLM‑predicted next‑emission distribution and the ground‑truth HMM (Main Fig. 5 right; App. D Fig. 3).\n\n> *Sum2*. The paper showcases that LLMs can be used as efficient and capable predictors of the next token for data coming from a Markov process.\n\nOur experiments are on HMM emissions, not simple Markov chains (Sec. 2.1; App. A). The distinction between next‑token prediction on sequences from a Markov chain versus on HMM emissions is non‑trivial. There is no existing theory that provides finite‑sample complexity or error bounds for next‑token prediction given a single finite trajectory of emissions from an HMM.\n\n> *Sum3*. Moreover…interesting real-world data, which is considered to come from a Markovian process.\n\nOur real‑world animal data are commonly modeled with HMMs [1,2], but are not assumed to be Markovian.\n\n> **W1**: It's unclear... complex temporal data.\n\nUnlike Baum‑Welch (BW), which uses non‑convex EM to infer HMM parameters, LLM ICL achieves near‑optimal prediction (Sec. 2.3). This suggests LLMs act like spectral methods—directly mapping sequences to next‑emission distributions. Empirically, LLMs behave similarly to spectral learning (reviewer oWyV W1 & W2). We provide theoretical guarantees for spectral learning (Sec. 3.3), explaining LLM performance trends (Sec. 3.1–3.2).\n\nEvaluating LLMs on theoretically grounded tasks like HMMs is non‑trivial and novel. Even in the simpler case of **fully observed** Markov models, the performance evaluation of LLM’s or transformers has received significant attention from works published at top conferences [3-5]. Our results show LLMs outperform non‑convex BW and align with direct methods like spectral learning.\n\nFinally, although LLMs are known to handle complex temporal sequences, the community remains interested in when and how ICL succeeds. This is why benchmarking LLMs on tasks with clear theoretical baselines—even relatively simple tasks like linear regression [6,7]—is considered valuable. Our work follows this philosophy.\n\n> **W1.1**: This appears... low-entropy transition matrices (where the most likely token has a high probability).\n\nIt is expected that low‑entropy transition matrices naturally produce high‑probability next emissions. However, our evaluation is **not** based on raw accuracy approaching 1, but rather on the **gap to the theoretical optimum** for each HMM (see Main Paper Fig. 3).\n\n> **W1.2**: LLM's memory bandwidth... ample \"signal from the past\" in finite sequences.\n\nWe acknowledge the reviewer’s observation that LLMs can leverage long‑range context through attention. For a fully observed Markov sequence, this indeed provides an advantage because induction heads could perform $n$-gram‑style prediction.\n\nHowever, in our setting with **hidden** Markov models, the emissions themselves are not Markovian. Memorizing recent emissions alone is insufficient for optimal prediction. Leveraging longer context is statistically valid and necessary, as classical methods such as Baum‑Welch also utilize the entire observed sequence via the forward‑backward algorithm.\n\n> **W2**: The paper aims to address the “hidden” aspect of HMMs, but this is largely unaddressed in both the text and the simulations.\n\nWe would like to clarify that the “**hidden**” aspect of HMMs is central to our study. All of our numerical experiments are conducted on synthetic data generated from true HMMs, and real‑world datasets that can reasonably be modeled as HMMs [1,2].\n\n> **W2.1**: There's a lack of analysis on how emission probabilities or the loss from misclassifying observations to hidden states affect predictor results.\n\nWe do not assume LLMs explicitly classify observations into hidden states. In fact, our results suggest the **opposite**—LLMs ICL is qualitatively different from the Baum‑Welch algorithm (Sec. 3.2).\n\n> **W2.2**: Also it's unclear whether sequential information is truly critical, or if the LLM is simply better at picking states (which is often more important in practice).\n\nOur experimental design evaluates performances across different mixing rates (Sec. 3.1). The fact that LLM performance varies with mixing rate (e.g., performing better on fast‑mixing HMMs) demonstrates that it is leveraging sequential structure. If sequential information were irrelevant, the mixing rate would have no impact on results.\n\n> **W2.3**. The benchmarked graphs ignored the more interesting Markov processes like jumping processes or cyclic processes which have unique properties.\n\nWe appreciate the reviewer’s observation that hidden Markov jump processes (HMJPs) and cyclic (periodic) chains exhibit dynamics that differ from the ergodic HMMs we focus on.\n- HMJPs introduce an additional holding‑time random variable, which is application‑specific (e.g., queuing systems). Modeling HMJPs requires predictors to capture additional sojourn‑time distribution. Classical HMM baselines like Baum‑Welch and Viterbi do not directly apply to HMJPs without significant modifications.\n- Cyclic chains' dynamics become more structured once the chain enters a cycle and the prediction space narrows. However, such processes are typically non‑stationary, which requires careful treatment for finite‑trajectory evaluation.\n\nOur benchmark focuses on ergodic HMMs, which allows us to study sequential prediction in a theoretically tractable setting [8,9]. Extending our evaluation to HMJPs or cyclic processes is an exciting future direction. However, we expect that the relative performance trends among pre‑trained LLMs, Baum‑Welch, and spectral methods would remain consistent, as all predictors would be equally affected by the increased complexity.\n\n> **W3**: It would be more robust to learn a Markov model from the real data and then sample from that as a benchmark.\n\nWe respectfully disagree with the suggestion to first learn a (hidden) Markov model from the real‑world dataset and then benchmark on samples from that learned model.\n\nOur study is specifically motivated by the real‑world scientific setting, where researchers observe a single finite trajectory and must make predictions or analyze patterns without knowing the true latent model. Learning a model from the data and then resampling would (1) create a simulation‑to‑real gap, (2) reduce the problem to a fully synthetic benchmark—already covered in the first part of our experiments, and (3) fail to test the LLM’s ability to handle the messiness and variability of real emission sequences directly.\n\nInstead, we justify in the paper that the chosen dataset can reasonably be modeled as an HMM, and we evaluate LLMs directly using real-world data.\n\n> **W4**: The main paper doesn't contain enough details about the experimental setup, specifically regarding the emissions and their prior distributions.\n\nThe experimental details of initial state distributions $\\pi$ are in Section 2.2 (uniform or deterministic). We did suggest generating the emission matrix is relatively straight-forward (Sec. 2.2), but we would love to explain how we construct $\\textbf B$ here.\n\nThe code (`hmm.py`, 311–326) builds a sequence of stochastic transition matrices with increasing entropy. It starts with a deterministic matrix (zero entropy), then, for each target entropy up to the maximum, computes a row distribution with one dominant entry and the rest uniform, repeating it across rows to form a homogeneous matrix.\n\n> **W5**: Theorem 1 appears to be quite obvious for any learner, and it's unclear what makes an LLM particularly sensitive to these parameters.\n\nDeriving sample‑complexity and error bounds for spectral learning on a single trajectory is non‑trivial due to output correlations. Standard mixing arguments are challenging because we observe only one realization rather than i.i.d. samples. We address this by proving that the single‑trajectory estimates of $\\hat P_1, \\hat P_2, \\hat P_{3,n}$ in Algorithm 6 (Appendix F) converge to the corresponding multi‑trajectory (i.i.d.) estimates.\n\nRegarding the LLM’s sensitivity to Theorem 1 parameters, we refer the reviewer to our responses to Q4 (reviewer 1stE) and W1&W2 (reviewer oWyV).\n\n> **W6**: In the context of Markov models... rather than always providing the full context.\n\nFor HMMs, evaluating on a single new observation is different from a sequence of observations. It’s necessary to infer the hidden state sequence using the entire observation sequence.\n\n---\n\n[1] Ashwood Z.C. et al. *Mice alternate between discrete strategies during perceptual decision‑making*. Nat. Neurosci. 25(2), 201‑212 (2022).\n\n[2] McClintock B.T. et al. *Uncovering ecological state dynamics with hidden Markov models*. Ecol. Lett. 23(12), 1878‑1903 (2020).\n\n[3] Edelman E. et al. *The evolution of statistical induction heads: In‑context learning Markov chains*. NeurIPS 37, 64273‑64311 (2024).\n\n[4] Rajaraman N. et al. *Transformers on Markov data: Constant depth suffices*. NeurIPS 37, 137521‑137556 (2024).\n\n[5] Rajaraman N., Jiao J., & Ramchandran K. *An analysis of tokenization: Transformers under Markov data*. NeurIPS 37, 62503‑62556 (2024).\n\n[6] Li Y. et al. *Transformers as algorithms: Generalization and stability in in‑context learning*. ICML (2023).\n\n[7] Garg S. et al. *What can transformers learn in‑context? A case study of simple function classes*. NeurIPS 35, 30583‑30598 (2022).\n\n[8] Gassiat E., Cleynen A., & Robin S. *Inference in finite‑state non‑parametric hidden Markov models*. Stat. Comput. 26, 61‑71 (2016).\n\n[9] Abraham K., Gassiat E., & Naulet Z. *Fundamental limits for learning hidden Markov model parameters*. IEEE Trans. Inf. Theory 69(3), 1777‑1794 (2022)."}]

Author rebuttal:

Meta-review: This paper evaluates whether LLMs can model data generated by hidden Markov models using their in-context learning abilities. The authors performed thorough empirical investigations where they varied HMM properties and showed the differences in predictive performance of LLMs. These experiments showed that HMMs can perform comparably to traditional HMM algorithms (which were also evaluated). Based on these experiments, the authors suggest that LLMs could be used as scientific discover tools for this type of sequential data and they demonstrated this approach on some animal behavior data.

The reviewers pointed a few strengths:
* The empirical validation was strong and answered the proposed question and tested with multiple models, multiple metrics, and multiple HMMs to build confidence in the results.
* The work accurately isolates entropy and mixing rate and characterizes when the method fails.
* The authors compare the approach to the theoretical optimum and show LLMs approach it in some settings and points out when optimality isn't guaranteed.
* The authors proposed guidelines for how to use the technique in the real world based on their experiments.

There were a couple weakness that the authors identified:
* The connection between ICL and spectral learning is unclear.
* What actually leads to the observed performance? Training? Transformer architecture? ...
* The authors don't distinguish between their prediction problem and learning of an HMM – i.e. estimating the underlying parameters.

The author-reviewer discussion addressed most of the weaknesses in the paper. There were a few weaknesses by a reviewer that upon closer inspection revealed a mis-understanding by the reviewer and which should not penalize the paper.

As pointed out by two reviewers, the authors should make a clear distinction between learning an HMM and the prediction problem they actually tackle in the paper in their final version.
Decision: Accept (poster)