In-Context Learning Strategies Emerge Rationally
Authors: Daniel Wurgaft
Ekdeep Singh Lubana
Core Francisco Park
Hidenori Tanaka
Gautam Reddy
Noah Goodman
Venue: NeurIPS 2025 poster ()
Keywords: In-Context Learning
Loss-Complexity tradeoff
Bayesian Modeling
Algorithmic Complexity
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Recent work analyzing in-context learning (ICL) has identified a broad set of
strategies that describe model behavior in different experimental conditions. We
aim to unify these findings by asking why a model learns these disparate strategies in
the first place. Specifically, we start with the observation that when trained to learn a
mixture of tasks, as is popular in the literature, the strategies learned by a model for
performing ICL can be captured by a family of Bayesian predictors: a memorizing
predictor, which assumes a discrete prior on the set of seen tasks, and a generalizing
predictor, where the prior matches the underlying task distribution. Adopting the
normative lens of rational analysis, where a learner’s behavior is explained as
an optimal adaptation to data given computational constraints, we develop a
hierarchical Bayesian framework that almost perfectly predicts Transformer next-
token predictions throughout training—without assuming access to its weights.
Under this framework, pretraining is viewed as a process of updating the posterior
probability of different strategies, and inference-time behavior as a posterior-
weighted average over these strategies’ predictions. Our framework draws on
common assumptions about neural network learning dynamics, which make explicit
a tradeoff between loss and complexity among candidate strategies: beyond how
well it explains the data, a model’s preference towards implementing a strategy
is dictated by its complexity. This helps explain well-known ICL phenomena,
while offering novel predictions: e.g., we show a superlinear trend in the timescale
for transitioning from generalization to memorization as task diversity increases.
Overall, our work advances an explanatory and predictive account of ICL grounded
in tradeoffs between strategy loss and complexity.

In-Context Learning Strategies Emerge Rationally
Daniel Wurgaft!. 12,,+ Ekdeep Singh Lubana 2.3+ Core Francisco Park
Hidenori Tanaka2 Gautam Reddy4 Noah D. Goodman 1.5
1 Department of Psychology, Stanford University
CCBS-NTT Program in Physics of Intelligence, Harvard University Physics of Artificial Intelligence Group. NTT Research, Inc., Sunnyvale, CA, USA *Joseph Henry Laboratories Physies, Princeton University Department of Computer Science, Stanford University
Abstract
Recent work analyzing in-context learning (ICL) has identified broad set of strategies that describe model behavior in different experimental conditions. We aim unify these findings by asking why a model learns these disparate strategies the first place. Specifically, we start with the observation that when trained to learn a mixture of tasks, as is popular the literature, the strategies learned by a model for performing ICI. can be captured by family of Bayesian predictors: a memorizing predictor, which assumes a discrete prior on the set of seen tasks, and generalizing predictor, where the prior matches the underlying task distribution. Adopting the normative lens of rational analysis, where learner's behavior is explained as optimal adaptation to data given computational constraints, we develop hierarchical Bayesian framework that almost perfectly predicts Transformer next- token predictions throughout training- wathhng-without without assuming access to its weights. Under this framework, pretraining is viewed as a process updating the posterior probability of different strategies, and inference-tine behavior as posterior- weighted average over these strategies' predictions. Our framework draws on common assumptions about neural network learning dynamics, which make explicit a tradeoff between loss and complexity among candidate strategies: beyond how well it explains the data, model's preference towards implementing strategy is dictated by its complexity. This helps explain well-known ICL phenomena, while offering novel predictions: e.g., show superlinear trend in the timescale for transitioning from generalization to memorization as task diversity increases. Overall, our work advances an explanatory and predictive account of ICL grounded in tradeoffs between strategy loss and complexity.
Introduction
In-Context Learning (ICL) has significantly expanded open-ended nature large language models (LLMs) [1-5], allowing them to learn novel behaviors from merely the provided context [6-12]. This has motivated a large body work that analyzes controlled experimental settings to better understand ICL [13-17], leading to (i) behavioral accounts of what strategies are followed by model to learn from its context [18-23], c.g., the ridge estimator linear regression tasks [14, 17]; (ii) developmental accounts identifying when, i.e., under what training [24] and data [25] conditions, particular strategy is used by the model [24-30]; or (iii) mechanistic accounts characterizing how such strategies get implemented [31-33], e.g., the of induction heads [34]. While some have attempted characterizing ICL as a single procedure [22, 35, 36), more recent work has argued that the broader phenomenology of ICL stems from model learning different strategies under varying experimental conditions [26, 30,37.
* Equal contribution. Email: vurgaft@stanfon edu, ekdeeplubana@fas. harvard.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
(b)
Prine fwvore
Transformer
Generaliing Fredictor
Memoriiing
Task Diversity
simpiial
Transience
Task Diversity Effects
Transience
0C0
000
Civersity
Threshold
Liwitooo favors momoniation
Eventoolyy the Mailhoon Roode the pivar
Tranommer (Pai Dissncel
Rayoday
Task Diversity
Classi oatoon
Task Diversity
Training steps
Training steps
Training steps
Figure 1: Why Does a Model Learn Different Strategies for Performing ICL? To answer this question, we analyze three distinct settings where model is trained to learn a mixture of tasks. (a) Model Behavior Transitions Between Memorizing and Generalizing Predictors. We first make the observation that across settings, diversity of data distribution and amount of training increased, model behavior transitions between two Bayesian predictors: a memorizing predictor, M, which assumes discrete prior over seen tasks, and a generalizing predictor, G, which assumes continuous prior over the true distribution. This recapitulates prior results on task-diversity thresh- olds [25] and transient generalization [24] in a unifying language. (b) A Hierarchical Bayesian Framework Provides an Explanatory and Predictive Account of ICL. The consistency of these transitions motivates a hierarchical Bayesian model of ICL, where a model's inference- time behavior is framed as a pesterior-eeighted interpolation in the Bayesian predictors M and G. while pretraining seen as a process of updating the preference (posterior probability) toward different predictors. We find that under reasonable assumptions regarding neural network learning dynamics, our framework highly predictive of model behavior throughout training, without access to model weights (bottom panel). Additionally, our framework provides an explanatory account of ICI. phenomena via tradcoff between the loss and the complexity of learned solutions (top panel).
These results suggest that a remaining hurdle for developing a unified account of ICL is understanding why, in the first place, models learn different strategies with disparate generalization abilities. Indeed, given that memorizing the training distribution almost always leads to better performance, why does the model learn an 'underfitting'. duitt--distiibuttin generalizing solution at all [24, 25j Moreover, if capacity limitations prevent memorization, why does the model, among all underfitting solutions, learn the one that captures the true generative process [25, 26] Finally, why does it first learn such a solution, only to eventually give way to one that does not generalize to novel tasks [24, 27] In addressing the questions above, we make the following contributions.
Model Behavior Transitions Between Memorizing and Generalizing Predictors. We first make the observation that in popularly studied ICL settings, where a model is trained to learn a mixture of tasks, previously identified setting-specific solutions be unified in the language of Bayesian inference. Across three distinct settings, we replicate well-known ICL. phenomena [24, 25) and show models primarily transition between two ICL phases, determined by behaviorally matching one of two Bayesian predictors: 1) a memorizing predictor with a discrete prior over seen tasks or 2) generalizing predictor with a continuous prior over the true data-generating distribution.
A Hierarchical Bayesian Model of ICL, Grounded in Rational Analysis. This observation
then motivates our core contribution: we propose to understand ICL by invoking the approach of rational analysis from cognitive science [38-42], where a learner's s behavior is explained as an optimal adaptation to data, given a learning objective and computational constraints. In our case, building on the finding that Transformers transition between memorizing and generalizing predictors, we examine how a Bayts-optimal learner would trade off between these solutions across training and data conditions, given a simplicity bias [43-48] and power-law scaling of loss with dataset size [49, 50]. This yields a hierarchical Bayesian framework that casts pretraining as a process of weighing preference (posterior probability) toward different solutions based on their loss and complexity. At inference, model behavior is framed as weightte- average of these solutions (which themselves are Bayesian-hent the term "hierarchical"). This is in contrast

---

(a)
(b)
Balls & Urns
Linear Regression
Classification
Mixture of Tasks
T Tixinn
Tirup W~ Dirccleat
Terain
Tire w W000 (0. (n)
T Train
(0.11
Tiree
Tusie
Time
Next-token prediction samples from task(s)
fl
Figure 2: Experimental Settings: Learning Finite Mixture of Tasks. (a) General Formulation.
Popularly studied experimental settings in the literature on ICL can be seen as training model to learn a distribution defined using a mixture of tasks (denoted TuaIn). where each task is a parameterized latent function whose parameters are sampled from distribution True- (b) Considered Settings. We analyze three distinct instantiations of this general formulation: Balls Urns, which captures the belief update interpretation of ICL. and is simplification of the Markov modeling setting from prior work [26, 52], and two popularly studied settings from the literature that capture the few-shot learning interpretation of ICL, i.e., in-context linear regression [14, 25] and Classification [20, 24, 28, 29].
to several previous Bayesian accounts of ICL [35, 51], which frame ICL as implementing a single Bayesian predictor. We derive a closed-form expression for our model, which almost perfectly predicts Transformer next-token predictions throughout training without access to weights, as well captures varied ICL phenomena including task diversity effects and transience.
A Loss-Com Complexity Tradeoff Underlies ICL Phenomena and Yields Novel Predictions. Our framework makes explicit a tradcoff driving model preference towards a predictor as a function of its loss and complexity. This tradeoff helps explain transitions in model behavior due to changes in data-diversity [25] or training time [24] as artifacts implicitly induced by changes in a predictor's complexity or loss, relative to other predictors. This interpretation further helps us identify several novel predictions, e.g., a superlinear trend between task diversity and transience.
Overall, we argue our work advances a unifying explanatory and predictive account of ICL., More broadly, our results suggest the value of taking a normative perspective towards explaining neural networks, viewing learning behavior (both in-context and throughout training) as a rational adaptation to data properties and computational constraints.
2 Preliminaries: Learning a Finite Mixture of Tasks
To capture both few-shot learning [6. 14, 20, 25, 53] and belief update formulations [26, 34, 54, 55] of ICL, we analyze three distinct experimental settings in this paper. To this end, we first provide general formulation of the learning setup, which allows to formalize a unified language for examining model strategies for ICI. in the next section.
General Formulation. We cast prior settings used for studying ICL [20, 25, 26] as learning finite mixture of tasks. Specifically, settings analyzed in this work involve learning a mixture distribution Taain defined over D parametrized functions (... )), or 'tasks'. For each function fw)), the parameters w E RR are sampled from predetined distribution Time (see Fig, 2). We call D the task diversity of the mixture. Every training iteration, we randomly select a function f(w,-) E Tuaann and use it to generate sequence of length C' (details vary by setting, see below). Batches consist of independently generated sequences. We autoregressively train Transformers (GPT-NeoX architecture [56, 57]) for a predefined number of iterations N. Model performance is evaluated either on in-distribution (ID) sequences drawn from Trains or on out-of-distribution (OOD) sequences drawn from the underlying distribution Time (see App. F for further details on architecture and method). We analyze three specific instantiations of this general formulation, detailed next.
Balls & Urns. Related to the belief update formulation of ICL, this setting is inspired by the classic "Um Problem' from the probability literature [58]. Specifically, one draws (with replacement) balls from an urn containing balls of m types, and the goal is to estimate the distribution of ball types. Since solving this task only requires inferring unigram statistics of the input (a histogram), this setting simpli- fies the Markov modeling setup proposed by Park et al. [26]. A task f(w) = Categorical ww denotes a stochastic map (' ('urn') from which states ("balls") are sampled, with w ~ Time = Dirichlet(1). Categorical(wn))
Thus, the distribution Tiain consists of D 'urns' ( generate data, we sample C states from a randomly selected function f(ur), yielding a sequence := [x) ... .c.,, where Categorical(w)).
To
3

---

(a)
(b)
Balls & Urns
Linear Regression
Classification
Poetener predictive
Teat
Memorizing
recloval
70tal
X/17
Compute court
Nelay
Postwior prediction
Generalizing
regressinn
Tue
Figure 3: Predictors in Different Experimental Settings. (a) Memorizing and Generalizing Predictors. We compare model behavior to two idealized Bayesian predictors: (i) Memorizing predictor (M), which assumes a discrete prior over the mixture distribution Timin and (ii) Generalizing predictor (G), which assumes a prior over Times the distribution from which tasks are sampled. (b) Task-Specific Instantiations. These predictors yield closed-form solutions (App. G); e.g., in Balls & Urns, the memorizing predictor computes a posterior weighted average over urms seen in training, whereas the generalizing predictor uses empirical unigram statisties with pseudo-counts.
Linear Regression. A standard problem setting in literature on understanding the few-shot learning formulation of ICI. [14, 25, 53]. Here, the goal is to learn to in-context solve linear regression problems. A task f(w) wTx c denotes noisy linear map that transforms continuous input via linear map 2: N (0, Im) and introduces additive noise c NN0,, ((00,²). where w ~ Tiue = N0,0, I,) and Im denotes the X identity matrix. Thus, the training distribution Timin consists of D linear mappings [f(w), () .).... f. }) To generate data, we sample C' inputs and transform them with a randomly selected function f(w, æ). yielding sequence == 11 T + LC, w Toac++ec))
Binary Classification. Another popularly studied setting literature on understanding the few-shot formulation of ICL [20, 28). Our parameterization is inspired by Nguyen and Reddy [29], who define task f(w. 1) w @ I to denote an item-label pair, with w NO, Im/m).
{0, 1] defining Time- Thus, the training task distribution Timin consists of D item-label pairs (w) @ 41 ...... WD @ ). Unlike other settings, multiple functions I' f(w, 1) are used to generate data: first, pairs u @ 1 are randomly sampled (with replacement) from Thain- A pair is chosen from the sequence at random to be the query pair-it is appended to the end of the sequence, and its label is corrupted to be 1. We noise these items via ω wite with E IR acting the within-class VI+a2* variance, and e ~ (0, In/m). This process yields a sequence a := [x1...., IC-1 Equaryl = [wi + I,...,, Wc-1 @ Lc-1, qqerry + -1]. Models are only trained to predict the label of Wqurry.
) I
item-label
3 What Strategies: Memorizing and Generalizing Predictors
Our goal in this work to understand why a model learns different strategies for performing ICL. We must thus first establish what these strategies are, allowing us to then characterize the dynamics driving changes in model's preferred strategy. To this end, we build on the idea that autoregressive training with the next-token prediction objective corresponds to maximizing the likelihood of the data and learning the distribution underlying [59]. We thus consider the two distributions forming the basis of our general formulation i.e., Tuain and Tuue and consider optimal strategies learner can be expected to implement it learns these distributions. This generalizes the approach of Raventós et al. [25], and yields the following two Bayesian predictors.
Memorizing Predictor (M). The memorizing predictor assumes a discrete prior over the distribu- tion of seen tasks (Turin) and implements a posterior predictive of the form:
M ((8.811-11)
Σ Wiin
E Tma (n
Generalizing Predictor (G). The generalizing predictor assumes a continuous prior over the distribution from which tasks sampled (Tmee), implementing posterior predictive of the form:
G(s(($14-1)=
p(w[s1n-1)f (s dw = E Em (s.[814-1)]
For all tasks we analyze, the predictors above can defined a closed-form manner (see App. G), mapping onto task-specific strategies defined in prior work: e.g., what has been called 'dMMSE' vs. ridge estimator in linear regression [25], 'in-weights learning" vs. 'in-context learning' in classification [20, 33], and 'retrieval' vs. "inference" in sequence modeling [26].

---

3.1 Validating the Memorizing and Generalizing Predictors
We next demonstrate the validity of the memo- rizing and generalizing predictors for the pur- pose of our analysis. Specifically, we show that as experimental conditions are varied, model's behavior primarily transitions between these predictors. To this end, we consider two core phenomena associated with ICL- an in- crease in models' OOD performance with in- creasing task diversity D [25, 60, 61]. and the 'forgetting" of this ability with increasing train- ing steps N-a phenomenon known as transient generalization [24, 30, 62]. We first replicate these results behaviorally in Fig. (a), finding that, across settings, Transformers transition between performing like the memorizing pre- dictor vs. like the generalizing predictor (see App. H for full results). Then, make di- rect comparison by computing the distance be- tween our trained model's next-token predic- tions and the predictions of the memorizing and generalizing predictors. Specifically, let d(..-) denote a distance measure (symmetrized KI-- divergence or Euclidean distance), and denote the Transformer model trained from scratch via h(.). Then, as a function of D and N, we can plot a heatmap of the relative distance between the trained model and the memorizing and gen- eralizing predictors, defined as dee = (r+1//2, where r :- d(h,,))d(hh.)))
(o) Balls Ums
(b) Linear Regression
(cl Classiication
Took Dve0 D
Tranning Stopa
Trsniin Steps
Traniin Stops
Figure Relative Distance Captures Transi- tions in Model Behavior. We show the relative distance between model outputs and the two predic- tors. Marginals report the absolute distance values (e.g., symmetrized KL between model and predic- tor outputs for the Balls & Urns setting), holding N constant (for the right plot), or D constant (for the top plot), and varying the other variable (de- noted with the dotted line). Across all settings, we model behavior decomposes into two phases, explained by either the memorizing or generaliz- ing predictor. In this figure, use context length of 128, task dimensionality of 8, and MI.P width of 256 for balls and urns. Linear regression has similar parameters except context length of 32, and classification has similar parameters other than MLP width of 512.
d(G.M2 This metric evaluates vs. the memorizing predictor. Results are shown in Fig. 4; see App. H.3 for absolute distances. We clearly see that increasing D for fixed N, the model first behaves like memorizing predictor (in red), only to eventually transition to behaving like a generalizing predictor (in illlustratinn task-diversity effects [25]. Meanwhile, when increasing N for middle values of D, the model starts closer to a generalizing predictor, only to eventually give way to a memorizing
vs. 1 if the model is closer to the generalizing
transient generalization [24]. More broadly, across all tasks, we see there is clear delineation of the model behavior into two phases of (N, D). such that model behavior is best explained by either the memorizing or the generalizing predictor in given phase. Given the optimality of these predictors on the distribution of seen tasks (Turin) or the underlying distribution (Tuue),, our analysis provides an explanation for why these predictors were observed in prior work. However, several questions remain, including why generalizing strategy is learned even when it leads to worse ID performance, and why does varying experimental conditions change which strategy, among the memorizing and generalizing predictors, is implemented by a model. We address these questions next.
4 Answering the Why: A Hierarchical Bayesian Account of ICL
Our analysis above shows that, except for intermediate values of N and D, model behavior is primarily explained by Bayes-optimal predictors capturing the distributions Tuain and Time. Motivated by these findings, we adopt the lens of rational analysis [38-42], a framework in cognitive science that aims explain learner's behavior as optimal, under computational constraints. What might be considered optimal in our case? Recall the fact that ICL is an inductive problem, i.e., a problem of predicting the next observation given past ones. Specifically, a predictor hred performing ICL predicts the thh token s; given previous elements in the sequence $1;--1, using mechanisms it may have learned for this purpose based on sequences ST... D) seen training (denoted ST... from hereon for brevity). Then, given hypothesis space of possible solutions the model has learned, Bayesian inference preseribes an optimal way to solve this problem via the posterior predictive distribution: compute weighted average of predictions from each solution, with weights defined by
5

---

(a) Balls & Urns
(b) Linear Regression
(c) Classification
Traneformer (Rel Dista
Tanaforer Rel. Distanoo)
Tranefommer 28
Reyesian Modal [Eq. 31
29.
Rayasian Modal (Eq. 3]
Boyasian Model (Eq. 3)
iRel Distance
Task Diversity D
Training Steps
Training Stepss
Training Steps N
1008
Figure 5: Our Bayesian Model Captures Transitions Between Strategies Explaining Model Behavior. We plot the posterior probability of the memorizing predictor given by theoretical model (Eq. 2). Across three broad experimental settings- (a) Balls & Urns, (b) Linear Regression, and (c) Classification- find our model identifies the phases best explained by a given predictor and the boundary between them, hence capturing the transition between solutions seen in a Transformer's s training (as shown by the relative distance maps). Importantly, our model is highly predictive of the pretrained Transformer's behavior (next-token predictions) across conditions used for fitting the three free parameters of our model and unseen ones. Max color bar value for Balls & Urns and Classification is determined by the performance of baseline predictor that always outputs the mean of the distribution Tuue- In this figure, we use context length of 256, task dimensionality of 8 and MLP width of 256 for balls and urns. Linear regression and classification have similar parameters except context length of 64 and 384, respectively.
solution's 's posterior probability (i.e., posterior-weighteed average). Relying on the results of Sec. 3, we can assume our hypothesis space simply consists of the memorizing and generalizing predictors". Thus, in our case, each solution itself corresponds to a Bayesian predictor-specifee predictors M and G-hence resulting in the following hierarchical Buyesian model.
hpoo 8 81.-1, STam) =
Σ T(Q)S Q(5,(015-11. QE(M,
The mathematical form of the predictor above frames in-context behavior as a linear interpolation in M and G, with posterior probabilities (Sm.) p( estimated from training, determining the interpolation weights. Then, in order to use this model to explain how a neural network performs ICI, we must estimate how posterior probabilities vary across training and data conditions.
Modeling the Posterior Probabilities. The posterior probability for a predictor Q, i.e., p(Q)STmm) pm (Q),(), is comprised of likelihood term and prior term term- thus, these are the two terms we must estimate. In line with the perspective of rational analysis, in modeling these terms, we consider the following two well- known computational constraints of neural networks
A1: Loss scales in a power-law manner with dataset-size N, i.e., L(N) v L(xo) + A/N", where L(N) denotes the loss after N updates. This leads the log- likelihood to scale with effective sample size I- rather than N (see App. D.1): p(S Q)) 7N' Eg [log Q)].
A2: Neural networks exhibit bias toward simpler solutions. Specifically, using K (Q) to denote the
Kolmogorov complexity for predictor Q, we accommodate the Transformer-specinf implementation cost by defining (Q) = k())). Then, taking the form of universal prior, the prior probability
of learning a predictor Q is p(Q) x 2 2- (x 2 =2 K(Q)?
A1 is merely a paraphrased version of well-known power-law scaling behaviors seen in neural network training [49, 50] and dictates how quickly observed data updates model behavior. That is, it offers a functional form for the rate at which likelihood in a posterior calculation grows, i.e., the rate of evidence accumulation. Mcanwhile, A2 is a well-known inductive bias of neural networks [43-48]. Our specific functional form for the prior is grounded in algorithmic information theory: Kolmogorov complexity K ((Q) is the length of the shortest program on a universal Turing machine that implements Q, and the coding theorem relates it to probability via p(Q) x 2 k(Q) (63-65), Following common
While in principle other predictors may be needed to capture model behavior, in the settings we analyze, we find that very quickly into training. other predictors perform poorly compared to M and Gi in predicting model behavior, and thus focus these predictors as our primary hypotheses (see 4.2, App. for further discussion).

---

practice [44, 45, 66 (6-691, we estimate Kolmogorov complexity (which is uncomputable) as K via lossless compression: we apply several lossless compressors the code and data for Q, and take the smallest resulting size (App. F.2). For brevity, we write K K below.
Returning to our goal, we now consider a model trained for N iterations on a task-mixture Tuain of diversity D. The log-posterior odds of the two predictors can be defined as follows.
T(N, D) = log P log PTG
P(S (.M) P(STuca (G)
log P(M)
P(G)
Posterior odds
Bayes factor
Prior oddss
Under constraints A1, A2, this simplifies as follows (see App. D.I).
(N, = (LLD
DD(
(2)
Loss term
Complexity term
where AK ( K(MD)* (G)) is the difference between the exponentiated Kolmogorov complexity of the two predictors (with MD denoting the memorizing solution defined for D tasks); (L(D) == (Traini
To (Turain (D) - (E T~an [log p [ -E TT [lg p(s) MD)] is the difference between the average loss of the two predictors for sequences sampled from Tuain; Finally, y a constant related the term A from constraint A1. To get the posterior probabilities for M and G, we simply convert via the sigmoid function, denoted (--), yielding:
h h (s, 11 = a (m1(N, D)) M ( + + (1- a (n7(N, D))) G(s
Note that the free parameters of this Bayesian model, i.e., Ba 7), depend on the problem setting and the Transformer's 's learning dynamics on it. To identify their values, we simply the Bayesian model's predictions to the pretrained Transformer h(-)'s next-token predictions on inputs retrieved from a subset of values (N, D). We emphasize that we only fut three free parameters across model checkpoints in 11 different training runs to get our results.
Validating the Model. We now check whether model accurately captures the behavior of the pretrained Transformer and reproduces ICL's phenomenology. As shown in Fig. 5, our Bayesian model yields an almost perfect prediction of Transformer next-token predictions for both seen / unseen settings, Moreover, without fitting to the relative distance maps, we find an almost perfect match between the posterior probabilities of the memorizing predictor given by model and relative distance values. These results are replicated across 72 different maps in App. H.4 with varying MLP width (see Fig. 7(a)), context length, and task dimensionality, yiclding robust support for our model. Across all maps, we find our model is highly predictive of Transformer next-token predictions, with mean R of 0.97 + 0.004 (SE) in Linear Regression, mean agreement of 0.92 + 0.007 in Classification, and mean Spearman rank correlation 0.97 + 0.001 Balls & Urns. Additionally, we find strong correlations of 0.99, 0.98, 0.99 between our model's posterior probabilities and the relative distance values given by the Transformer in the Linear Regression, Classification, and Balls & Urns settings, respectively. Finally, we also examine ablations of our functional form in App. I, showing the computational constraints we assume are necessary for the success of our framework.
4.1 Predictions
We next analyze our model make informative qualitative predictions. Unless stated otherwise, we use Balls & Urns setting with context length 128, task dimensionality of 8, and MIP width of 256.
Sub-linear sigmoidal growth from generalizing to memorizing, We examine whether relative distance, which acts as an estimate for the posterior of the memorizing solution, shows the functional form predicted by our model: sublinear growth with N, and sigmoidal growth with N This is strongly validated in Fig. 6(a) by fitting a parameterized logistic to predict relative distance from N1-a for each curve (i.e., holding
constant). Fig. 6(a), bottom panel, also shows that even at high D, relative distance slowly increases towards the memorizing predictor in a sigmoidal manner, contrasting with Raventós et al. [25]'s 's claim that at high D the Transformer only becomes closer to the generalizing predictor with increasing N (see App. J).
7

---

(a) Sub-linear sigmoidal transition
(b) Rapid behavior change near crossover boundary
(c) Super-linear Scaling of Transience with task diversity
Helanve D Dinance
10%
E. 3:
Trandence Tohing Sup
27
Task Diversity
Steps.
Pale en Dessnnce
Lee Fantads
E.
0.24
105
15
25
100
Training Stops N
Taak Diversity
Figure 6: Predictions from Our Model. (a) Relative distance is predicted to show sub-linear scaling with N, and sigmoidal growth with A1-a, This is validated by a strong match between the empirical curves and parameterized logistic fits (dark lines). Note, however, that some fitted plateau parameters fall below 1, implying never reaching full memorization, whereas Bayesian trajectories should reach it eventually (see App. K). (b) We predict rapid change in model behavior for intermediate values of N and D, yielding a crossover- like boundary between phases dominated by each predictor. This can be seen via the magnitude of the second derivative of the relative distance near the boundary. Marginal plots show the second derivative of the relative distance with respect to N or D. (c) Our analysis predicts super-linear scaling of the time of transience N' as diversity D increases.
Rapid behavior change near crossover. At the point where the two hypotheses have equal log-posterior odds, model predicts there will be crossover in which predictor dominates the posterior and explains model behavior. Given our sigmoidal functional form for 17( N, D). we can expect this change to be rapid-small small variations in experimental conditions (e.g., task diversity) will yield large changes in model behavior. To confirm this, we plot the second derivative of the relative distance in Fig, 6(b), finding its magnitude is indeed greatest near the boundary (N,, D) O.
Scaling of time to Transience with Task Divvesstyrssty. For a given value of D, we can predict the critical training time which a crossover from generalizing to memorizing will occur by solving for N in nNN, D)- = 0 (i.e., relative distance dhe = 0.5): N (D) = A T Observing Fig. 6(c), find that beyond the two- hopohhetsssss threshold (which determines the minimum from which our model holds, App. E), our predictions for N hold well for several orders of magnitude, before observations diverge at higher D, transience is slower than predicted?, These results indicate super-linear growth of time to transience with lask diversity. Importantly, according to our expression, at high D) the denominator is very small and hence time to transience can approach infinity. In such a condition, generalization persists regardless of training time.
4.2 Extending the Model
We additionally examined two extensions of our model: (1) Multiple Strategies: we add a constant -renn-preditor solution, which has been shown emerge very early in training [70, 71], modeling the linear regression setting. Denoting this optimal constant solution as C, our model becomes: ha 11 ST..)
[QE[M.G.C) p(Q (T.... ((s:((1=-1). As Fig. 7(b) and App. H.5 show, model successfully generalizes to capturing complex dynamics with multiple predictors, including a sharp transition from a context-insen -ontext-ineessitive solution to ICL, which has been observed in other settings [29, 72]. Our model explains this "rapid emergence" constant solution is preferred early due to its simplicity, yet eventually, low likelihood leads to its posterior dropping sharply (Fig. 7(b), bottom).
(2) In-Context Strategy Selection: We find that when evaluated on OOD data, models shift their behavior toward the generalizing solution, thereby moving the transition boundary between memorizing and generalizing (Fig. 7(c)). This makes sense from Bayesian perspective: Given novel data, a Bayesian learner should continue updating its posterior in accordance with the likelihoods of different hypotheses. The generalizing solution outperforms memorization on OOD data, thus its
3 'Interestingly. we find that learning rate annealing enhances Transformer's adherence to Bayes-optimal trajectories, and so we use it in this experiment. Yet, the effect appears to decay throughout training, yielding a slower rate of advancing toward memorization, which can be seen by the last two observed transience points veering from our model's predictions (see Fig. 6(c) and App. K).
8

---

(a) Increased Memorization with MLP Width
(b) Multiple Strategies
[c) In-Context Strategy Selection
1
IC
COD
24
Tash Divenniyy
Teit Oitatyy
Taiiing Steps
Triiiin Steps
Thking Sapu
Figure 7: Our framework captures diverse ICL, phenomenology. (a) Scaling MIP width raises the transition boundary between memorization and generalization, and our model captures this effect via decreasing complexity penalty and increasing sample efficiency. (b) Incorporating mean-predictor solution for linear regression our model allows capturing complex transition dynamics. (c) When evaluated in OOD settings, models increase their resemblance to the generalizing solution, exhibiting in-context strategy selection, which we model in-context posterior updates (see App. D.4).
posterior should increase. To enable in-context posterior updates, we write: (8: 11:---1-11 S EQe(M.c) p(Q[STmm.. $1:-1) Q(s; (1:1-1). ((s])515-1). Our final form includes added term for in-context loss differences, without additional free parameters (App. D.4). As shown in Fig. 7(c), our model qualitatively captures the shift towards generalization in OOD evaluation. Yet, it struggles to explain OOD behavior around the transition boundary (App. H.6), which could mean other predictors are needed to explain OOD behavior. Moreover, for the current model, we make the simplifying assumption that in-context posterior updates follow similar power-law parameters as pretraining, yet this is unlikely, and further work is needed to distinguish in-context and parametric learning [73].
4.3 The Loss-Complexit, Tradeoff
(a]
Prior Odis
X
Bayas Factor
Postar or Odes
Now seneaaiee
Assura de second
4a
Having formalized and demonstrated the empir- ical validity of our hierarchical Bayesian frame- work, we now discuss the intuitive interpretation it offers us. Specifically, Eq. suggests that the tradeoff between posterior-odds corresponding different predictors driven by the loss pre-
0
0,0
Tast Downly D (6)
log Prior Oddc
Bayes Factor
Log Postenior Oridss
dictor achieves on the training data and its com-
plexity (see Fig. 8): early in training, the prior dominates, therefore a less complex solution the generalizing predictor in our case case-will be strongly favored as per the posterior calcula- tion. However, the memorizing predictor will almost always have a lower loss than the gen- eralizing predictor on training data. Thus, in low-to-medium task diversity settings, train- ing proceeds and N increases, the loss term in Eq. 2 will overtake the complexity term, i.e., the likelihood eventually dominates the posterior and 'floods' the prior. This will lead to the mem- orizing predictor becoming wwworeddexplanng the transient generalization phenomenon from prior work [24]. In contrast, in high task di- versity settings, the prior strongly disfavors the memorizing predictor due to its complexity, and given the sub-linear accumulation of likelihood (i.e.. the 11-a term), the time for transience to occur grows superlinearly-give
Training Stoos
Training Stops
Training Stops
Taxk Clieesity
Task Divenity Dne
Texk Dleem by D-
Traling
Traniig
Trums Signs
Figure 8: Intuition Elicited by The Bayesian
Model. (a) Our framework suggests Transformers have a prior preference for learning simpler solu- tions. However, throughout training, preference updated towards solutions that better explain the data (i.e., have greater likelihood). This happens even at the expense of higher complexity, which in our case, yields a transition toward a memorizing predictor. (b, e) Our framework captures the trade- off between solutions, showing that the boundary between the two phases corresponds equal pos- terior probabilities of the two predictors.
rise to the phenomenon of task diversity threshold seen in prior work [25]. The importance of loss-complexity tradeoff for ICI. was also arrived at by two recent papers [30, 74], which we discuss further in Sec. 5.

---

5 Discussion
In this work, we aim to unify findings from the ICL literature by asking why Transformers learn disparate strategies for performing ICL across varying training conditions. To do so, we take a nor- mative perspective [38] which aims to explain Transformer learning as optimal under computational constraints. This lens yields a hierarchical Bayesian framework, which, assuming simplicity bias and scaling laws as computational constraints, offers highly predictive account of model behavior: by fitting merely three variables to Transformer next-token predictions, we can almost perfectly predict model behavior across spectrum of experimental setrings without access to its weights. Our account also implies a fundamental trade-off that occurs throughout training between the loss and complexity of potential solutions learned by model, with simpler, generalizing solutions learned early in training, while more complex, better-fitting solutions eventually becoming preferred. This tradcoff' helps explain prior findings in the ICL literature, and provides novel predictions regarding training dynamics. Thus, we argue for our hierarchical Bayesian framework as an explanatory and predictive account of ICL, and step towards unified understanding of its phenomenology.
Relation to previous Bayesian models of ICL. Here, we discuss the relation between our work and other Bayesian or normative accounts of ICL. We provide an extended review of additional related works in App. B. Several prior works framed ICL as Bayesian at inference-time. Other than a few cases [17, 21], most such works view ICL as a single Bayesian predictor [25, 35, 51, 62, 75, 76]. The focus on a single predictor in these works has often led them to view ICL as single strategy, with some focusing solely on a memorizing solution [35], while others refer mainly a generalizing solution as "true" ICL [25]. While there are cases in which in-context behavior is well approximated by a single predictor (e.g., the memorizing solution for low N, D), our results robustly show that a mixture of predictors is required to fully capture model behavior, and that the extent to which a predictor explains model behavior variey across training. Thus, a posterior-weighted weighted average over different predictors, which considers bias towards different predictors coming from training, rather than only inference-time, is required to fully capture model behavior across conditions. In contrast with studies focusing inference-time time,, two recent works, Carroll et al. [30] and Elmoznino et al. [74], offer Bayesian or normative views of pretraining to perform ICL. Importantly, while these similar conclusion us:
papers take different theoretical perspectives from ours, they arrive the existence of a tradeoff between loss and solution complexity in pretraining. Elmoznino et al. [74] offer normative theoretical analysis of training to perform ICL via next-token prediction loss, showing it yields an Occam's razor objective which minimizes both loss and solution complexity. Carroll et al. [30] study task diversity effects and transient generalization in the linear regression setting of Raventós et al. [25]. Their Bayesian account of pretraining, which is rooted in theory of singular models [77] and makes different assumptions from ours, interestingly yields relatively similar functional form for the posterior odds (though their form does not take into account neural scaling laws). However, their measure of complexity is architetureedependeer provide a qualitative analysis as they cannot directly estimate the complexity of Bayesian predictors. In contrast, our hierarchical Bayesian framework provides a quantitative, predictive account of pretraining phenomena, in addition to capturing inference-tima behavior as a posterior-weighted average of solutions, which is not addressed by Carroll et al. [30] or Elmoznino al. [74]. Despite that, we view these works as valuable contributions with complementary insights to ours, in particular regarding potential explanations for the source of neural networks' simplicity bias.
[78], thus they only
Limitations. While we rely on a specific theoretical abstraction to arrive at our results, i.e., a hierarchical Bayesian model, we believe the predictive power of this abstraction corroborates its faithfulness. However, one limitation of our analysis is use of toy settings where model behavior largely explained by only two predictors (with the addition of the optimal constant solution in linear regression). Accommodating more complex settings that LLMs encounter in pretraining or inference- will be an important future test for the framework. Finally, crucial limitation of our analysis comes from the simple relation we assume between algorithmic complexity and complexity of implementation by a Transformer wwhle while the assumption of simplicity bias is well-backed by theoretical and empirical claims [43, 44, 46), we believe our assumed relation could be improved by building on recent advances defining anchitccture-dpendeer measures of how many effective parameters are used by model to implement solution [71, 78].
10

---

Acknowledgments
We thank the Computation and Cognition Lab, in particular Ben Prystawski and Michael Li; Jay McClelland, Satchel Grant, Jerome Han and the PDP Lab; the Physics of Intelligence group at Harvard, especially Eric Bigelow and Sonia Murthy; the CRISP Lab at Harvard: Jesse Hoogland and Matthew Fabbrgia-Roberts: Surya Ganguli and the Neural Dynamics and Computation Lab; and Navin Goyal and the theory group at Microsoft Research India for useful discussions.
References
[1] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXIi:22000:222.22222222 2023.
[2] OpenAL. Gpt-4 technical report, 2024. URL
[3] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, al. Palm-e: An embodied multimodal language model, 2023. URL https:://artii org/abs/ abs/2303. 23077.3378.......133133333333881
03378.
[4] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges assuring alignment and safety of large language models. arXiv preprint (xXi::2404009932, 2024.
[5] OpenAL. Openai o3 and 04-mini system card, 2025. URL https: //tp::://dd...c openai com/pdf/ 1221c875-02dc-o 4eeesc--78888000000 8-0---0000------
0rtp://--77200000--77783337222-0 00000-977583372221133333 and- o4 mini aarii-syee -system paystem-card,. pdf
[6] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[7] Gemini Team. Gemini: family of highly capable multimodal models. arXiv preprint 100:22322.11005. 2023.
[8] AnthropicAI. Claude 3.7 sonnet system card, 2025. URL 3-
com/
[9] Keunwoo Peter Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data distributional properties. arXiv preprint arXiv:2311.1704. 2023.
[10] Yida Yin. Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig. In- context learning enables robot action prediction in lims. arXiv preprint arXiv:2410.12788 2024.
[11] Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang. Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. In The Thirteenth International Conference on Learning Representations, 2025.
[12] Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning large language models. arXiv preprint arXiv:2410.01280 2024.
[13] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfiel-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL attps://ant /ttp///axvvvv/./ org/ aba/aba/20222 11895.

---

[14] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances Neural Information Processing Systems, 35:30583 2022.
[15] Suraj Anand, Michael A Lepori, Jack Merullo, and Ellie Pavlick. Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting. arXiv preprint arXiv:2406. 00053, 2024.
[16] Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms, 2024. URL httpt::/tssstiiiiie www.aiiiwo...iiiieeeoeoeeoooeree abb/240/1411.... 12973.
[17] Yu Bai, Fan Chen, Huan Wang. Caiming Xiong, and Song Mei. Transformers as statisti- cians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
[18] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
[19] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang. Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023. URL https: //axxiv./ org/abs/ a//aa////33.////////////////333/ 03846.
[20] Stephanic Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35: 2022.
[21] Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning, 2024. URL attps://arxiv: www.aawww.................oooott abaaaba/2444 18819.
[22] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
[23] Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers algorithms: Generalization and stability in in-context learning. In International conference machine learning, pages 19565-19594 PMLR, 2023.
[24] Aaditya K. Singh, Stephanie C. Y. Chan, Ted Moskovitz, Erin Grant, Andrew M. Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers, 2023. URL
https://arx:: http:///xxvvvoo/////// abs/ /211 08360.
[25] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
[26] Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka, Competition dynamics shape algorithmic phases of in-context learning, 2024. URL https:///aaat org/ abs/2412 01003.
[27] Bryan Chan, Xinyi Chen, András György, and Dale Schuurmans. Toward understanding in-context vs. in-weight learning. arXiv preprint arXiv:2410.23342,, 2024.
[28] Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023. URL https: /ttpas////aivvio/aa org/ harrgg/bb///e abs/2312..... /2112.03002.
[29] Alex Nguyen and Gautam Reddy. Differrntial learning kinetics govern the transition from memorization to generalization during in-context learning, 2024. URL https: //arps/ /ttp//orxxxo/////////// abs/2442.0010m
[30] Liam Carroll, Jesse Hoogland, Matthew Farrugia-Roberts and Daniel Murfet. Dynam- ics of transient structure in in-context linear regression transformers. arXiv preprint
arXiv:2501.177455 2025.
12

---

[31] Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? arXiv preprint arXiv:2502. 2025.
[32] Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y. Chan, and Andrew M. Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation, 2024. URL hwtps::tt////tttee org/
07129.
[33] Aaditya K Singh, Ted Moskovitz, Sara Dragutinovic. Felix Hill, Stephanie CY Chan, and Andrew M Saxe. Strategy coopetition explains the emergence and transience of in-context learning. arXiv preprint arXiv:2503.088 055331 2025.
[34] Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains, 2024. URL https://arriss org/ httg/bb//24000 11004.
[35] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021.
[36] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent meta-optimizers. arXiv preprint arXiv:2212. 10559, 2022.
[37] Andrew Kyle Lampinen, Stephanie CY Chan, Aaditya K Singh, and Murray Shanahan. The broader spectrum of in-context learning. arXiv preprint arXiv:2412.03772, 2024.
[38] John R Anderson. The adaptive character of thought. Psychology Press, 2013.
[39] Nick Chater, Mike Oaksford, Nick Chater, and Mike Oaksford. Ten years of the rational analysis of cognition. Trends in cognitive sciences, 3(2):57-65, 1999.
[40] Thomas L Griffiths, Falk Lieder, and Noah D Goodman. Rational use of cognitive resources: Levels of analysis between the computational and the algorithmic. Topics in cognitive science, 7(2):217-229, 2015.
[41] Noah D Goodman, Joshua Tenenbaum, Jacob Feldman, and Thomas I. Griffiths. rational analysis of rule-based concept learning. Cognitive science, 32(1):088-144 2008
[42] Falk Lieder and Thomas Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43:e1, 2020.
[43] Preetum Nakkiran, Dimitris Kalimeris, Gal Kaplun, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd neural networks learns functions of increasing complexity. Adv. in Neural Information Processing Systems (NeurIPS), 2019.
[44] Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint. arXiv:1805.088 08522, 2018.
[45] Chris Mingard, Henry Rees, Guillermo Valle-Pérez, and Ard A Louis. Deep neural networks have an inbuilt occam's 's razor. Nature Communications, 16(1) (6(1):200, 2025.
[46] Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems, 36:35027-35063, 2023.
[47] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the carly-time learning dynamics of neural networks. Adv. in Neural Information Processing Systems (NeurIPS), 2020.
[48] Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in trans- formers and their ability to learn sparse boolean functions. arXiv preprint arXiv:221 12316, 2022.
13

---

[49] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 1rxiv:2001.08661, 2020.
[50] Jordan Hoffmaann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optinal large language models. arXiv preprint arXiv:2203. 155566 2022.
[51] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305. 19420, 2023.
[52] Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793-5831 PMLR, 2022.
[53] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint urXiv:2211.156611 2022.
[54] Eric J Bigelow, Ekdeep Singh Lubana, Rohert P Dick, Hidenori Tanaka, and Tomer D Ullman. In-context learning dynamics with random binary sequences. arXiv preprint 0rXiv:2310.17633 2023.
[55] Johannes A Schubert, Akshay K Jagadish, Marcel Binz, and Eric Schulz. In-context learning agents are asymmetric belief updaters. arXiv preprint arXiv:2402.03969, 2024.
[56] GPT-NeoX.
Gpt-neox, huggingface. 2025. URL https://huggingface co/docs/
wransformeers/emmnodee soceeel_doc/gge httlc/ppt_eooxxxoc
[57] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Gold- ing, Horace He, Connor Leaby, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022. URL https://arxix www..rrggbb..... http//abb2224..0006
[58] Urn problem. In Wikipedia, December 2024. URL https://en. org/wiki/
urn problem.
[59] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309. 2023.
[60] Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning grok: Emergence in-context learning and skill composition in modular arithmetic tasks. arXiv preprins arXiv:2406.02.0 025550, 2024.
[61] Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-lcarning transformers. arXiv preprint arXiv:2212.044588 2022.
[62] Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism, 2024. URL httts:t/ta/xxiii org/ 220b/ 04891.
[63] Ray Solomonoff. Complexity- based induction systems: comparisons and convergence theo- rems. IEEE transactions on Information Theory, 24(4):422 432, 1978.
[64] Leonid Anatolevich Levin. Laws of information conservation (nongrowth) and aspects of the foundation of probability theory. Problemy Peredachi Informatsii, 10(3):0--55, 1974.
[65] Ming Li, Paul Vitányi, et al. An introduction to Kolmogorov complexity and its applications, volume Springer, 2008
[66] Hector Zenil. A review of methods for estimating algorithmic complexity: Options, challenges, and new directions. Entropy, 22(6):612, 2020.
14

---

[67] Peter Grunwald and Paul Vitányi. Shannon information and kolmogorov complexity, arXiv preprint cs/0410002, 2004.
[68] Stephen Fenner and Lance Fortnow, arXiv:/702.04779, 2017.
Compression complexity.
arXiv preprint
[69] Kamaludin Dingle, Chico Q Camargo, and Ard Louis. Input-our output maps are strongly biased towards simple outputs. Nature communications, 9(1):761, 2018.
[70] Jirko Rubruck, Jan Bauer, Andrew Saxe, and Christopher Summerfield. Early learning of the optimal constant solution in neural networks and humans, 2024. URL btttp:///rxing org/ abs/ 2406 17467.
[71] Jesse Hoogland, George Wang, Matthew Farrugia-Roberts Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning, 2024. URL https: //arxiv.orgi herg/abs/24022 02364.
[72] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
[73] Andrew K Lampinen, Arslan Chaudhry, Stephanie CY Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, and James McClelland. On the generalization of language models from in-context learning and finetuning: controlled study, arXiv preprint arXiv:2505. 00661, 2025.
[74] Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam's 's razor, 2025. URL https: // //arxiv.org//////eo hrwpiv..rr//bbseeaa aasp/bbs///1000 14086.
[75] Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D Goodman. Bayesian scaling laws for in-context learning. arXiv preprint arXiv:2410.16531, 2024.
[76] Chi Han, Ziqi Wang. Han Zhao, and Heng Ji. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305. 12766, 2023.
[77] Sumio Watanabe. A widely applicable bayesian information criterion. The Journal of Machine Learning Research, 14(1):867-897, 2013.
[78] Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure. arXiv preprint arXiv:2308. 12108, 2023.
[79] Samuel J Gershman. The rational analysis of memory. Oxford handbook of human memory.. 2021.
[80] Rahul Bhui, Lucy Lai, and Samuel J Gershman. Resource-rational decision making. Current Opinion in Behavioral Sciences, 41:15-21, 2021.
[81] Yue M Lu, Mary I Letey, Jacob Zavatone- Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405. 2024.
[82] Charles Kemp, Noah D Goodman, and Joshua B Tenenbaum. Learning to learn causal models. Cognitive science, 34(7):1185 1243, 2010.
[83] Michael Li, Fred Callaway, William D Thompson, Ryan P Adams, and Thomas Griffiths. Learning learn functions. Cognitive science, 47(4))e13262, 2023.
[84] Christopher G Lucas and Thomas L Griffiths. Learning the form causal relationships using hierarchical bayesian models. Cognitive Science, 34(1):11331))) 2010.
[85] Joseph L. Austerweil, Sophia Sanborn, and Thomas Griffiths. Learning how to generalize. Cognitive science, 42777 2019.
15

---

[86] Charles Kemp, Andrew Perfors, and Joshua B Tenenbaum, Learning overhypotheses with hierarchical bayesian models. Developmental science, 10(3):307-321, 2007.
[87] Amy Perfors and Joshua Tenenbaum. Learning to learn categories. In Proceedings of the Annual Meeting of the Cognilive Science Society, volume 31, 2009.
[88] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv: 2018.
[89] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017. URL httph::///aaata http://aaaxi...r/a////////ss aa////bs/// 03400.
[90] Marcel Binz, Ishita Dasgupta, Akshay K Jagadish, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-leameed models of cognition. Behavioral and Brain Sciences, 47:e147, 2024.
[91] Hong Jun Jeon, Jason D Lee, Lei, and Benjamin Van Roy. An information-tthooreii analysis in-context learning. arXiv preprint urXiv:2401.15530 2024.
[92] Rahul Ramesh, Mikail Khona, Robert Dick, Hidenori Tanaka, and Ekdeep Singh Lubana. How capable transformer become? study synthetic, interpretable tasks. arXiv preprint 1tw.::2321.119999 2023.
[93] Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueleer, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
[94] Tian Qin, Naomi Saphra, and David Alvarez-Melis. Sometimes am a tree: Data drives unstable hierarchical generalization. arXiv preprint arXiv:2412.04619 2024.
[95] Fabian Falck, Ziyu Wang, and Chris Holmes. Is in-context learning in large language models bayesian? martingale perspective. arXiv preprint arXiv:2406.00799 2024.
[96] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614-4565,, 2023.
[97] Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, et Many-shot, jailbreaking. Advances in Neural Information Processing Systems, 1222999-1297777 2024.
[98] Nigel Goldenfeld. Lectures on phase transitions and the renormalization group. CRC Press, 2018.
[99] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisti- cians: Provable in in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36:57125-57211 2023.
[100] Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, and Ekdeep Singh Lubana. Belief dynamics reveal the dual nature of in-context learning and activation steering, 2025. URI https::tt////ttt //ax:i///.vvv//// org/abs/////eoooooooooooooo attpa///........................
[101] Jesse Hoogland, George Wang, Matthew FarrugiaRoberrs, Daniel Murfet. Loss landscape degeneracy drives stagewise development in transformers, 2025. URL https://arxiv...
Roberts, Liam Carroll, Susan Wei, and
02364.
[102] Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, PAMI-666):21-7444 1984.
16

---

Appendix
Table of Contents
A Glossary of Useful Terms
18
B Related Work
19
B.1 Prior Work Studying Task ciss-Diversity Effects and Transience
19
Hierarchical Bayesian Models Learning Learn
B.3 Broader Work Understanding ICI.
20
B.4 A Note Regarding Complexity
20
C Takeaways and Future Work
20
D Derivations
21
D.1 Log-Poster Log--osterrir Odds
22
Converting Posterior Odds a Predictive Model
23
D.3 Extending the Framework Multiple Predictors
24
D.4 Extending the Framework to Accommodate In-Context Strategy Selection
24
E Two-Hypotheses Threshold: Minimum amount of training to enable the Hierarchical
Bayesian Model
26
F Experimental Details
27
F.1 Training and Model Details
27
F22 Analysis Details
27
G Additional Details Regarding Settings and Predictors
29
G.I Balls and umss
30
G.2 Linear regression
30
G.3 Classification
31
H Main Results
33
H.1 Task Diversity Effects
................................
33
Transience
................................
H.3
Absolute Distance from Predictors
................................
39
H.4 Model Predictions
42
H.5 Extension Multiple Strategies
47
H.6 Extension to In-Context Strategy Selection
49
1 Functional Form Ablations
51
Memorization Continues Increase After Task Diversity Threshold Refutation of Raventós et al. [25]'s Claim
52
Learning Rate Annealing Can Improve Adherence to Bayes-Optimal Trajectories
L Code Availability
54
M NeurlPS Paper Checklist
55
17

---

A Glossary of Useful Terms
In-Context Learning (ICL). In this paper,
broad construal of ICL advanced by Lampinen et. [37] Olsson al. [72]: any way in which models use their context adapt their predictions and reduce loss can be considered as ICL. This notion of ICL encapsulates many forms of sequence modeling and instruction following, as well as the more traditional view of ICL as few-shot learning [6]. Hence, in this work, we tasks that capture both sequence modeling [26] and few-shot learning [25, 28] notions of ICL.
Generalizing Predictor (G). A predictor defined by posterior-weighter average with a continuous prior over the true data-generating distribution Time. Such a predictor does not depend on the tasks during training, and hence generalize well to novel, unseen tasks. This predictor maps onto "task-learning' or 'inference" notions of ICL [26]. See App. for form of generalizing predictors the studied settings.
Memorizing Predictor (M), A predictor defined posterior-weighted average with discrete prior defined over the distribution tasks seen by the model during training, Tiraos. Such a predictor depends on the tasks seen during training, and hence generalizes primarily to those tasks. This predictor maps onto "task-retrievaa" notions of ICL, though it also captures a context- constrained 'in- weights learning' solution in the classification task [20, 33J. See App. G for the form of memorizing predictors the studied settings.
Task Diversity (D). For a mixture distribution Temin defined over D tasks {f(w1 f D is referred to as the task diversity of the mixture.
Relative Distance (dret). A measure we use to characterize trade-off between predictors settings where there are primarily two predictors (though one can easily generalize this measure to setting with more predictors). Specifically, given a model h, two predictors Q1. Q2, and distance function d, we define the term r == d(h.Q1)-4(hh (()))(2(aa)) and then relative distance as dha = (r+1//2. The latter d(Q1-Q2)
operation rescales r to a scale of 0 (if = Q1) to (if = Q2), essentially assessing where the model h lives when a line is drawn with endpoints ranging from Q1 to Q2.
Posterior Odds, Prior Odds, Bayes Factor. Consider set of observations X and hypotheses H1 and H2 that are being assessed as candidates to explain the data. Prior odds are defined as seen yet, which hypothesis is apriori preferred. Bayes factor P(X] i.c., once the observations are received, we compare how likely individual hypotheses deem these observations. Finally, posterior odds are defined as P(H))
ratio P(H1)
P(Ha)' i.e., if no observations is defined as the ratio of likelihoods P(X)H;)
the ratio P[HaIX) P(Ha) x P(X HH) Prior odds x Bayes factor, ie.. how do observations
((H1]X)
P(X (())
affect the prior to help assess which hypothesis is more likely.
Rational Analysis. Rational analysis is framework in cognitive science introduced by John R. Anderson [38]. in which the behavior of learner is explained as optimal adaptation to its learning environment, given goal and computational constraints. the case of training neural network, the learning environment given by the training data distribution, the goal is given by the optimization objective, and the computational constraints can be seen as constraints and inductive biases of the architecture and optimization algorithm. The framework of rational analysis is characterized as normative, since it specifies how a learner should behave, in optimal sense, given its environment, goals, and computational constraints. This approach has been successfully applied cognitive science to explain a wide range of phenomena in humans [38, 39, 41, 79, 80).
Transient Generalization (N For given task diversity D, when model is trained for sufficiently long, its behavior transitions from more closely resembling generalizing predictor to closely resembling a memorizing one. We define this point between memorizing and generalizing crosses 0.5. This phenomenon is called transience, transient generalization, or the transient nature of in-context learning (see App. B for a longer discussion). The critical time to reach this point is denoted N in the main paper.
point at which the relative distance
18

---

B Related Work
B.1 Prior Work Studying Task-Diversity Effects and Transience
We first discuss prior work studying the phenomenology of ICL that forms the core of our paper: task diversity effects and transience. These works often focus on providing mechanistic accounts that are more bottom-up in nature, i.e., suitable for studying specific settings. For example, the induction head is a core mechanism employed by model to perform the in-context classification task by Singh et al. [24], Reddy [28]. When transience occurs, the role of induction head is diminished, since is not vital for implementing memorizing predictor. However, as we show. transience occurs across a spectrum of settings, including ones where induction heads are never learned during training (e.g.. Balls and Urns, which we analyze using a one-layer Transformer) prior mechanistic accounts would not help justify transience in such settings. Our work thus takes a top-down account, i.e., we offer insights based
computational model of ICL developed by capturing its phenomenology, This helps identify relevant control variables controlling a dynamic we argue lies at the heart of transience and task diversity effects: tradeoff between loss and complexity of a solution, which manifests itself into transitions between generalizing and memorizing predictors. Reconciling our computational model with mechanistic approaches undertaken in prior work can be an exciting avenue for progress.
Task diversity Threshold. A phenomenon first popularized by Raventós et al. [25] in an in- context linear regression task, albeit originally demonstrated by Kirsch et al. [61] in prior work on in-context classification of permuted MNIST images, and recently expanded to a Markov modeling setting by Park et al. [26], to another classification setting by Nguyen and Reddy [29], as well as to a modular arithmetic setting by He et al. [60]. Specifically, Raventós et al. [25] empirically show that if one trains Transformers on a mixture of linear regression tasks, there is a critical number of tasks below which the model behavior is well-characterized by a Bayesian predictor over the seen tasks (what we term the memorizing predictor), while above the behavior is well-characterized by the standard solution of ridge regression. These results were expanded in a recent theoretical work by Lu al. [81], who study the asymptotic dynamics of linear attention Transformer and show the change in solution used by the model as function of task diversity a second-order phase transition. In contrast to their work, our empirical analysis covers a broad range of settings that involve sequence modeling, regression, and classification, while the analytical parts of our paper provide predictive framework that identifies the relevant control variables and explains how they affect the behavior of standard, nonlinear Transformers across all studied settings.
Transience/ Transient Nature of In-Context Learning. Originally observed by Chan al. [20] while investigating the effects of data-centric properties on ICL, the term was popularized by Singh et al. [24]. Specifically, focusing on an in-context classification task, Singh et al. [24] showed that a model's 's ability to perform the generalizing ICL solution (employing a copy mechanism via the induction head) goes away when trained long enough. This phenomenon was recently generalized to a Markov modeling task by Park et al. [26] and to simplified variants of the in-context classification setting by Chan et al. [27], Nguyen and Reddy [29]. We especially emphasize the work of Nguyen and Reddy [29], who empirically identify a competition dynamic between a memorizing and generalizing solution for their task, building on this observation to perform gradient flow analysis of their task and developing generalization occurs. Our work differs in the sense that we provide an account of the competition dynamic and model for its origins, instead of gradient flow account that operates under the assumption of competition.
effective theory of how transient
B.2 Hierarchical Bayesian Models of Learning to Learn
Hierarchical Bayesian models have been widely employed in cognitive science as models of 'learning to-leam', or meta-learning, in humans [82-87]. These models allow the agent to learn the prior distribution from the data, rather than the prior being predefined. This allows learning inductive biases that constrain hypothesis spaces (called oyet-hypo which can instruct future learning [86). Additionally, it entails estimating evidence for several different hypothesis spaces (like the continuous vs. discrete priors in our work) and using these hypothesis spaces for generalization [85]. Note also that much of the work on hierarchical Bayesian models in cognitive science takes normative perspective. Finally, we want to highlight the work of Grant et al. [88], who explicitly
19

---

analyze a well-known meta-learning algorithm (MAML [89]) via a hierarchical Bayesian lens, as well as Binz et al. [90], who highlight connections between meta-learning and hierarchical Bayesian models. We have been broadly influenced by this literature in approaching the study of ICI., which we believe has many direct parallels with work on learning to learn.
B.3 Broader Work on Understanding ICL
crucial benefit of our Bayesian modeling framework is that one can retrieve posterior probabilities over the predictors the model predictions are being decomposed into. In other words, we can represent the model predictions
interpolation of the predictors' outputs (since posterior probabilities sum to 1). Park et al. [26] propose a similar approach: "Linear Interpolation of Algorithms" (LII).. Specifically, LIA represents model predictions as an interpolation of the predictors' outputs, directly fitting the weights for interpolation. That is, for each condition of training time (N), task diversity (D), LIA requires fitting set of parameters, summing up a total of N D parameters fit (across many of the maps in our work, this would mean over 900 parameters per combination of context- length, task dimensionality, and MLP width). Meanwhile, our framework, by defining precise functional form that explicates the role of N and D. minimizes the number of terms needed to perform fitting to 3-these terms are primarily related to model family, its learning dynamics, and intrinsic randomness of the data, hence making them hard to explicate. Crucially, one can see our framework as providing grounding to LIA: under the hood, LIA is trying to identify the posterior probabilities for all experimental settings! Moreover, our framework offers an explanation for why change in predictor weights occurs during training, and importantl, can predict dynamics of N, D conditions it was not trained on, which LIA cannot do since it has to be fit separately to each N, D condition.
Beyond the papers discussed above, there have been several complementary efforts to understand ICL from different perspectives (see the recent summary by Lampinen et al. [37] for longer review). For example, several works, some briefly discussed above. characterized the trade-off between memorizing and generalizing in more mechanistic way, as a competition dynamic between circuits [24, 26, 29, 33]. We see our results as elucidating the forces driving this competition, as well as demonstrating its consistency across several settings. Additionally, several papers analogize ICI, as implicitly performing optimization to learn novel tasks [22, 36, 53] meta-learning in general [91]; develop scaling laws for the sample efficiency of ICL [75]; identify limits tasks that can be learned in-context [14, 17, 92]; demonstrate sudden learning curves exist in ICL [11, 54J; and characterize mechanisms employed by a model to perform ICL [13, 31, 34, 93]. Broadly, all these papers offer useful insights that complementary ours.
B.4 A Note Regarding Complexity
Our setup in this work was quite specific in the sense that, in all cases, task diversity was the variable by which we titrated the complexity of implementing the memorizing solution. However, it should be noted that our theory and the simplicity bias we assume refer broadly to the complexity of implementing predictive solutions, which can arise from other sources in diverse data distributions. Furthermore, while in our setup, the generalizing solution was often simpler than the memorizing one given high enough task diversity, in many cases, the optimal generalizing solution can be quite complex, and other heuristic solutions (which still do not involve memorizing the training distribution) may be preferred. An excellent example of such dynamic is provided by Qin et al. [94], who show that more complex training data can indeed drive learning of a hierarchical generalizing solution (rather than linear, heuristic solution), whereas in simple-data conditions, models learn the heuristic solution.
C Takeaways and Future Work
Takeaways. We see several main takeaways from our work for understanding ICL, training dynam- ics, and generalization behavior in neural networks more broadly.
ICL Bayesian? Depends on the Assumptions. There exists debate regarding whether ICL can be viewed as Bayesian [25, 35, 62, 95]. Some studies that saw the emergence of generalizing predictor have categorized it as 'non-Bayesian' [25, 62], since it is not the Bayes-optimal solution given the training distribution (which is the memorizing solution). In contrast, taking the perspective
20

---

of rational analysis, the right question is not whether ICL is Bayesian, but under what assumptions is it Bayesian. Clearly, the generalizing predictor is Bayes-optimal with respect the true distribution Time- Moreover, as we show in this work, when taking into account simplicity bias over predictors, learning a generalizing predictor can be considered Bayes-optimal in certain conditions even when it is not the optimal strategy for minimizing train loss. Thus, by extending Bayesian perspective on ICL to both training and inference- time, we show that assessing the Bayes-optimality ICL requires considering the model's bias towards different predictors coming from pretraining, Cautiously, given our highly predictive results, we conclude that ICL can be considered as approximately Bayesian given constraints (assumptions) of simplicity bias and sublinear sample efficiency (though see App. K for discussion of deviation from Bayes-optimality that is not accommodated by our current assumptions).
A Loss-Complexit, Tradeoff is Fundamental to Understanding Training Dynamics. We find that tradeoff between the loss and complexity of solutions learned by models lies at the heart of ICL phenomenology. We believe the hierarchical view from which this tradeoff emerges, in which pretraining is a process of updating posterior probability for different solutions based on their complexity and loss, and ICI. is a posterior-weightel average of solutions, can be a powerful explanatory perspective for understanding Transformer learning dynamics more broadly.
The Value of a Normative Perspective. An important takeaway that we believe may be of independent interest the community is that, understand generalization behavior in Transformers and neural networks more broadly, it may be enough to observe the structure of the data, and assume the network is well-approximated by a Bayes-optimal density estimator with simplicity bias and sublinear sample efficiency. We hope our work shows that such a top-down normative perspective can provide highly predictive accounts, as well as potential explanations for why models behave the way they do. We encourage a wider adoption of this approach for understanding neural network behavior.
Our work also opens up several exciting avenues for further progress.
Can we Explain In-Context Transitions? First, we note the phase transition elicited in this work primarily assumes biased optimization process, and the ability to overcome this bias by seeing data supporting another solution. Accordingly, since ICL can be viewed as an optimization process [22, 96). it may be possible to use our results to explain behavioral transitions seen in recent work on in-context learning of novel concepts or behaviors [11, 54, 97].
believe the competition dynamic characterized in our work is similar to the one demonstrated by Qin et al. [94] in language modeling task. There, the authors show that depending on the data diversity and complexity, a model can either learn a bag of heuristics or the underlying grammar to generate sentences from the language. It may be possible to explain the phenomenology elicited in that work via the hierarchical Bayes lens we take in this paper, since there likely exists a tradeoff between compressibility and loss of heuristic vs. Hoammar-learning solution.
Does our Framework Explain other Pretraining Phenomena?
Connecting our top-down framework to a bottom-up mechanistic account. Our work in- tentionally takes a top- approach, and hence does not offer mechanistic account of how Transformer learning dynamics implement the lossccomyyy tradeoff, or how the model weights different solutions at inference time. Such a bottom-up analysis likely requires studying either the gradient flow dynamics of ICL. [29] or using mechanistic interpretability tools to examine circuits [32, 33].
D Derivations
Below, we derive formal expressions for the functional form of log- posterorr odds (Eq. 2) and show how one can convert it into a predictive model (Eq. 3). For completeness, we repeat below the constraints underlying our modeling framework.
A1: Loss scales in a power-law manner with dataset-size N, i.e., T((N) ~ [(xo) + A/N", where [N) denotes the loss after N updates, and a constant that depends on model loss at initialization and training hyperparameters.
A2: Neural networks exhibit a bias toward simpler solutions. Specifically, using K(Q) to denote the Kolmogorov complexity for predictor Q, we accommodate the Transformer-speciffo
21

---

implementation cost by defining Kr(Q) = (Q)).. Then, taking the form of a universal prior, the
prior probability of learning predictor Q is p(Q) x 2 2-Kr(Q) = =2-x(Q)²
D.1 Log-Posterior Odds
We consider a parameterized model class H.) learning to implement a predictor Q E { [M, G} when trained using learner 7 dataset TT.. (N, 12) of N sequences sampled from the distribution Train with diversity D. Note that, when required for clarity, we use MD to denote the memorizing solution defined for a dataset with task diversity D. but in most cases we use M for brevity. We approximate this model's 's learning dynamics via a hierarchical Bayes framework, i.c., we assume learning happens via a posterior update by computing likelihood of the data under all considered hypotheses (which are themselves Bayesian predictors, hence the term 'hierarchical') )) Each hypothesis has an associated prior that reflects the learning pipeline's proclivity towards implementing it. For brevity, we will use the notation ST.. to refer to the dataset, with sequences seen at update N denoted via superscript STm i.e., ST... Un ST
Time" We also use Oq to denote the set of parameters in the landscape of model-class H, such that H (0, 8) = Q(s) for any sequence & if 0 E Qq.
We begin by analyzing the posterior of the predictor Q learned by the model:
P H) = P(OO|STuu T, H)
JoEea
P(O T, H)
P(S Frmm . S0 P(OTT, H)
x
Newl
A1
II
P(S 10) P(O H) Timis
Joe n=
Nen
II
((S()) IQ) P(OT, HI Tuni
Jueta n=1
Nai (n) Tran QQ)
ISSO P H)
II P(S
Prior of the learner and model-class towards learning Q
Neil
A2
P(S
(m) Tem
IQ) -22 )()
n=1
We now take the log of this quantity, thereby examining the unnormalized log-posterior for Q:
log
II
P(S (n) Timi IQ) (2-K(0)))
n=1
Nai
Σ log ((S(()) (Q) - KQ)) Tasin
Nat log P P(S(n) Tun
=Ner
(Q)
- (Q)
N
n=
Average likelihood sequences drawn from Tranunder predictor Q
= Lo (Train (D)) -)))=
22

---

Overall, we have then
P(M P(M|STmme STm.: T, H D) = log P(G STaE T, H)
New (Lo (D)) Lıd (Tanin (K(M))*-K(C))
Naa A AK(D)*
(4)
In the above, our constraints get operationalized as follows.
A1 helps us accommodate the fact that while Bayesian learner would make optimal of all samples shown to it, neural network training in fact makes suboptimal use of samples seen during training, which model by defining New, i.e., the effective number of samples neural network learns from. We will estimate this value below in Eq. 6.
A2 provides form for the prior the learning pipeline (includes the learner T and model-class H) has towards implementing the predictor Q.
We next model Nar- Specifically, we use the power-law scaling behavior of neural networks' learning dynamics to compute the loss reduced in N updates by such a pipeline, identifying the number of updates idealized Bayesian learner would have make order reduce loss by this amount.
Loss under power-law scaling in N updates
Net
Loss of a Bayesian learner in single update (L(n) 1(o0))8n
TQQ
1
A
5n
LQ
n
A
N
6n
LQ
N1-a
(1 a)
T.Q
NNlaa
where To== To:[Taiii (D)) = - (log((s(())) [Q). â R/N and (1-a) Lg is a constant that subsumes the loss of the predictor Q and the constant from our assumed form of power-law scaling, which depends on the random loss of network and effects of hyperparameters like batch-size and sequence lengths used to define the train data.
Substituting Ner back into Eq. 4, we get our final model:
(NN, D) - A
(5)
D.2 Converting from Posterior-Odds to a Predictive Model
At inference, the pretrained Transformer is shown sequence s, for which it makes next-token prediction. To simulate this process in our framework, we define a Bayesian predictor, denoted //eed follows.
(r (8,[81:111 STrun)= = Σ
P(Q T, H)
(M(M.G)
Σ
P(Q Times H. Q ([,
QE(M,G)
Pretraining Prior
Prodiction
exp (n(N, D)) M M(s 1 1 exp (n,N, D))
G(8((111-1--
1+ exp (n(N,))
(6)
23

---

Using (.) to denote the sigmoid function, we have the final form from Eq. 3 as follows.
hprod (s: 111, = (n((N, D)) ((8 + + -1 on(N G(s
Remark. It is worth highlighting that (7,, D) essentially serves the role free-energy in the analysis above. Use of free-energy model interpolation between two states of system is common theoretical framework used in physics to study systems that undergo transitions hetween a disordered state ordered state: e.g., see in Landau theory, considers interpolations between free energy at high temperature and low temperature to model continuous (second-order) phase transitions [98]. Our overall theoretical model, and the phenomenology it elicits, are very similar to models from physics, a parallel that we believe can be worth pursuing in future work, e.g., to uncover universality behavior beyond what we considered in this paper.
D.3 Extending the Framework to Multiple Predictors
Our framework can be easily extended to multiple predictors. In the case of multiple predictors, the posterior predictive is written as:
pred ST = Σ TT
To get the posterior, we first compute the unnormalized log-posterior (i.e., the logit) for each predictor Q
logit(Qi|ST.... T, H) log p(STmm logp((TTT1
-eet K(Q)
A2)
Then, we simply apply softmax for normalization:
exp (Qi H)] P(QU) T, H, E, exp logit(Q, ST H))
D.4 Extending the Framework to Accommodate In-Context Strategy Selection
In most of our analyses, we assume that the posterior for a predictor Q does not depend on the current
sequence and only depends on sequences seen during training. Formally, P(Q.)STmm. 8, T, H) PSQ T, H). While we find that this assumption does not harm the predictive ability of our model for ID evaluation, it is known that Transformers perform in-context strategy selection [21, 99]. Furthermore, for OOD evaluation, we find that Transformers behave more like the generalizing solution (see Fig. 7), indicating in-context strategy selection, since the generalizing solution performs better OOD compared with the memorizing solution. Thus, to accommodate in-context strategy selection, discard the assumption that the posterior only depends sequences during training, and instead write it as:
P s, H. x P(ST.a P(s(Q) P(QITH
Which, given predictors M and G, yields the following form for the log-posterior odds:
P(M 8)
P(S PiSmi (M)
P(s
P(M) P(G)
NN,, D) log P(G
log
llog
log
P(ST (G)
P(s
Posterior odds
Bayes factor (Pretraining)
Bayes factor (Context)
Prior odds
Given previous work showing in-context learning updates tend to follow power laws such as those seen in pretraining [75, 97, 100], we can follow a similar argument to that detailed in D.I to claim the effective context length scales sub-linearly as |s|enr 2'|s|1=a' for some 7 a'. For simplicity, we assume here that y and a are shared between posterior updates in pretraining and during the context. Further research is required to understand the differences between parametric and in-context learning
24

---

updates [73]. Given that, and taking N to be the number of total tokens seen by the Transformer, we write:
P(M STiM: 8) P
n(N, D) = log
Posterior odds
Nett Luain (D) |s]ett cocomex DD(
>N1 Lerain (D) + 7/s|" 1- Loomext (D) - AK(D)²
With |s] being the number of in-context examples in the context, and A A Tcontext (D) (s((s) La (8) is the difference between the average loss of each predictor for the specific context. We note that this functional form is certainly idealized, as assumes similar sample efficiency parameters for in-context learning and pretraining, and future research is needed to shed light on the similarities and differences of belief updating in-context vs, during training.
25

---

E Two- Awo-Hyyothesess Threshold: Minimum amount of training to enable the Hierarchical Bayesian Model
One can reasonably expect our proposed Hierarchical Bayesian model to explain learning dynamics of in-context learning will not be predictive of a Transformer's behavior early-on in training, for otherwise we are saying even an untrained model perfectly generalizes. In actuality, the Transformer becomes amenable to approximation by our model after some minimal amount of training has occurred. To automatically calculate whether we have finished this regime of training, we calculate an "optimal" interpolation between the two predictors if they were capable of explaining the model behavior: specifically, we rely on the relative distance as an estimate of the optimal interpolation weighting, and use it as a baseline to compare model outputs with. In particular, we compute the loss between this optimal interpolation baseline and our trained Transformer model, and if this loss is below a threshold, we claim our theoretical model is applicable. We call this threshold the two-hypothesess threshold (sce Fig. 6).
To define the two-hypotheses threshold, we make the observation that while the loss between Transformer and interpolating predictor can be large to begin with, it very quickly reduces to small value. We can expect this latter regime is where our theoretical model is most likely to accurate at modeling the trained Transformer's 's behavior. Motivated by this, we heuristically choose the two-hypotheses threshold as loss value 20% higher than minimum for Balls & Urns and Classification, and 10% higher than minimum for Linear Regression, on the scale defined from minimum to maximum loss (we find that a stricter threshold is required for linear regression to surpass the early high-loss regime, given the larger variance in interpolation loss values).
Balls & Urns
Linear Regression
0175-
0.
0.4
0.00
411
MSB
0.4
02
0000
00
42c
s0c
10cc
Ze Training Stop
Training Step N
Classification
035
0
0.14
0.55
0.00
Training Step
Figure 9: Two-Hypotheses Threshold. Defining an 'optimal' interpolation between the memorizing and generalizing predictors towards minimizing the Euclidean distance to the trained Transformer's s predictions, we report the loss between this optimal interpolation and the Transformer's 's predictions. We observe minimum amount of training is necessary for this loss to become sufficiently small such that our hierarchical Bayesian model, which implicitly assumes the Transformer can be funetionally decomposed into the two predictors, will become applicable. The dotted lines demarcate this threshold, which we call the "two-hypotheses threshold". Mean-squared error (MSE) in the figure above is normalized by dimension. KL in this figure indicates forward KL from the Transformer's next token predictions to the interpolation of predictors.
26

---

F Experimental Details
E.1
Training and Model Details
Model. For all settings, we use the GPT-NeoX architecture sourced from Huggingface [56, 57]. While the number of layers / blocks in the model depend the specific experimental setting (as reported below), we use only attention head per layer and follow a sequential residual stream architecture across all settings.
Training. We use the Huggingface trainer with default parameters, changing only the learning rate, batch-size, total iterations, and warmup steps (reported below). Gradients are clipped to unit- norm. All models are trained on A A100 GPUs, with maximum training budget reaching days for all experiments encompassing the linear regression setting. We vary data-diversity D from {2?,24, .t.., 212] across all settings.
Settings-Specifie Details. For our three core settings, we report results covering the following hyperparameters. We note that similar Carroll al. [30], as we vary task-diversity D, we include tasks from the lower diversity-values in the setting involving the larger one o allows us to assess effects of increasing diversity on the learning of given task.
Balls and Urns. Models of hidden dimension size 64 are trained for 100K steps, with warmup steps, at a constant learning rate of x 10 and batch-size of 64. For our analysis, we derive experimental settings from combinations of task tissddimensionalityy (equivalent to vocabulary-sizer which varies in the set {8, 12, 16); context length, which varies the set (128, 256, 320}; and MLP expansion factor, which varies in the set 40.5,488)
We conduct a separate experiment in which we attempt to elicit transience in higher task diversity settings (D {2 )).. To do so a reasonable compute budget (2M, 10M steps, respectively), we, similar to prior work [24, 29], have to intervene on the training pipeline. However, unlike prior work that often relies on weight decay for this purpose, we use learning rate annealing and find to be sufficient. Specifically, we rely on an inverse square root schedule for decaying the learning rate with number of dimensions, context length, and MLP expansion fixed to 8, 128, and 4 respectively. We train up to D = 2 for 100K steps, and then train with D 2 for 2M steps, and with D 29 steps for 10M steps. Results of this experiment shown in Fig. 6(c) and App. K.
Linear Regression. Models of hidden dimension size 64 are trained for 100K steps, with 5K warmup steps, at a constant learning rate of 5 x 10-4 and batch-size of 128. For our analysis, we derive experimental settings from combinations of task-dimensionality. which varies in the set (8, 12, 16]; context length, which varies in the set (16, 32, 64]; and MI.P expansion factor, which varies the set {0.5, 4, 8. Like in Balls & Urns, we additionally train setting using learning rate annealing (setting parameters are task dimensionality of 8, Context length of 16, and MLP expansion factor of 4. We train once with warmup of 500, and once with warmup of 5000. See results in app. K).
Classification. Models of hidden dimension size 64 are trained for 100K steps, with no warmup, at a constant learning rate of 5 x 10 and batch-size of 64. For our analysis, we derive experimental settings from combinations of task-dimensionality which varies the set {8, 16); context length, which varies in the set (128, 256, 384}; and MLP expansion factor, which varies in the set (0.5, 4, 8, for the dimensions experiment, and is kept constant 4 for the 16 dimensions experiment.
F.2
Analysis Details
Next, we specify broad details of our analysis pipeline. These notes clarify design decisions made in our evaluation and motivations underlying them.
General model and predictor evaluation. For OOD evaluation of both the Transformer and our procedurally defined predictors, i.e., the memorizing predictor M and generalizing predictor G, we draw 500 sequences from 500 unseen tasks (however, following still the same task distribution Time). In comparison, ID evaluation involves 500 sequences from seen tasks. If task-diversity D less than 500, sequences from the same task may be seen multiple times.
27

---

Fig. details. In this figure, we use context length of 128, MLP expansion factor of 4, and task dimensionality of 8 for Balls & Ums. Linear Regression uses similar parameters. only with a context length of 32, and Classification uses similar parameters, only with MLP expansion factor of 0.5. In all plots displaying task diversity effects, we hold N = 100K. the transience figures, we use D 256 for Balls & Urns, D = 64 for Linear Regression, and D = 512 for Classification. For the comparison of relative distance maps, we show maps from the Balls & Urns setting with the parameters described above.
Computing absolute distance between Transformer and predictors. Given input, we both the Transformer model and the procedurally defined predictors to make next-token predictions. Then, compare distance between these predictions using either the symmetrized KL (average of forward and backward KL) for the Balls and Urns and Classification settings, or the mean-squared error (MSE) for linear regression.
Computing relative distance between Transformer and predictors. Recall that relative dis- lance, for a given distance measure d(.,.) between two functions or distributions (symmetrized KL-divergence Euclidean distance), is defined as due (r+1)/2, where d(h.G)-d(h.M) and d(G.M)
I-)) denotes the Transformer model trained from scratch. This metric implicitly makes the assump- tion that in some function space, the model h(.) lies on line between the predictors M and G. Correspondingly, for the scenarios this assumption is violated, the value of dee can go outside the range 0-1. This occurs relatively rarely, but nevertheless noticeably (e.g.. if the model implements the optimal constant solution early on in training for linear regression). Accordingly, we clamp the metric between 0-1. For the multiple predictors extension of the model (App. H.5), generalize the relative distance measure by fitting a convex combination of strategies to Transformer next-token predictions training and data diversity condition.
Minimum amount of training to enable the Hierarchical Bayesian Model. One can reasonably expect our proposed Hierarchical Bayesian model to explain learning dynamics of in-context learning will not be predictive of a Transformer's behavior early on in training, for otherwise, we are saying even an untrained model perfectly generalizes. In actuality, the Transformer becomes amenable to approximation by our model after some minimal amount of training has occurred. To automatically calculate whether we have finished this regime of training, we use the relative distance as an estimate of an "optimal" interpolation weight between the two predictors. We compute the loss between this optimal interpolation baseline and our trained Transformer model, and if this loss is below a threshold, we claim our theoretical model is applicable now (see details on our choice of threshold in App. E)
Fitting the Bayesian Model. We must perform the following three steps order to fit our model.
Approximating Kolmogorov complexity. Because true Kolmogorov complexity is not com- putable, we estimate an upper bound by compressing a self-contained bundle for each predictor: (i) the cleaned Python source that instantiates the predictor, and (ii) any numpy arrays it needs at inference time (e.g., the full table of urn distributions for the memorizing baseline in Balls & Urns). We remove comments, docstrings, and extraneous whitespace from the source code. For arrays, we first apply simple delta-encoding (store successive differences) to expose additional structure. The pre-processed bundle is compressed with four strong, off-the-shelf algorithms: 1zma (preset=9 PRESETEXTRENE). bzip2 (1evel=9), brotli (quality=11, mode=TEXT), and zstd (1evel=22). We take the smallest compressed size (in bits) across the four algorithms as our estimate; this is a standard practice for obtaining loose but practical upper bound. To keep estimates comparable, we exclude external libraries such as PyTorch from compression: all predictors call the same set of PyTorch primitives, so including them would add a large constant offset without altering relative complexities. This choice does, however, ignore the fact that some primitives might be cognitively "cheaper" for a Transformer to implement than others an important caveat for future work.
Computation of average log likelihood per predictor. For every experimental condition we first compute the toven-level log-likelihood that each predictor assigns to the in-distribution sequences. Because the irreducible error term cancels when models are compared, we use KL- based evaluations to the Balls & Urns and classification tasks, treating the lenear-rng setting separately. To summarize performance, need the mean likeliool per token, yet
28

---

the empirical loss distribution is, at times, quite skewed: most tokens later in the context incur near-zero loss, whereas a small fraction of early tokens produce large spikes. Therefore, a naive arithmetic mean converges slowly and exhibits high variance. Thus, we instead use median of means, an estimator for the true mean that has better convergence under long-tailed distributions.
How fitting is done. To fit the 3 free parameters of the Bayesian model, we minimize the mean KL divergence (or mean-squared error in the linear-regression setting) between the interpolated predic- tions and the Transformer outputs. Optimization is performed with scipy pppyoiiiiiee mel.ze..iinini minimize ze using the L-BFGS-B - algorithm, capped at 1K iterations and 2K function evaluations, with gradient and function tolerances of 10- Exact gradients are supplied via PyTorch's 's automatic differentia- tion, ensuring stable convergence. For each task we fit on 80 % of the (N, D) configuration grid and reserve the remaining 20 % for held-out validation and diagnostic checks. The process of fit- ting and evaluation takes around minute or less to complete. In the case of the in-context strategy selection, we find that require parameter search using basin hopping with 50 iterations before beginning optimization. We optimize using the 5 best candidate initialization parameters found via basin hopping, and take the best result after optimization.
Novel Predictions Analysis Details. To show lub-linear sample efficiency and a sigmoidal curve in N we fit a parameterized logistic -( with free parameters a, b, No to each training run (constant D value), via scipy curvefit function. We use the value given by the Bayesian model. Curve fits are shown in Fig. 6(a). To compute the second derivative of the relative distance parameters for the logistic fits described above (which provide very close fits,
(Fig. 6), simply as can be seen in Fig. 6(a)), then compute the second derivative based on the form for the second derivative of a parametrized logistic. To plot the vector field, we normalize both U and V directions by the larger value among the 90th percentile values for U and V.
G Additional Details Regarding Settings and Predictors
We now give a more detailed discussion of the different settings analyzed in this work: (i) Balls & Urns, (ii) Linear Regression, and (iii) Classification. We also provide details of how the memorizing and generalizing predictors are implemented for these settings. Broadly, as also visualized in Fig. 10, all settings involve learning of mixture of tasks Tuain drawn from the true task distribution Tewe- The number of tasks involved in the mixture is called its task diversity (denoted D). For all settings, we find models learn predictors of two types: a memorizing predictor, which corresponds to the Bayesian posterior predictive distribution with a discrete prior over seen tasks Tiain and generalizing predictor, which corresponds to the Bayesian posterior predictive distri iustribution with prior over the true task distribution Time- The precise forms of these predictors, as well as how sequences are assembled into training batches in each setting, are provided in the following sections.
Mixture of Tasks
Next-token prediction over samples from taskist
Posterlor predictive over Ture
fbo
flraa Σ F-
Memorizing
Aww)
X1
X2 X3 X4
?
Xc Xc?
Tini
Tirse
Pasteriar predictive over Tux
Generalizing
Tuae
Figure 10: General Abstraction Capturing our Experimental Settings and their Predictors. Each setting involves a mixture of parameterized functions (called a "task"), with D functions (the "task diversity"). Task consist of predicting the next element in a sequence, and vary based on whether models are trained in a standard auto-regressive fashion (like Balls & Urns) or whether they are only trained predict some elements in the sequence (only function outputs in Linear Regression, and only the last label in Classification). Across settings, the solutions learned by Transformers can be characterized as memorizing predictors or generalizing predictors. A memorizing predictor is defined as the Bayesian posterior predictive distribution with Tiran: the distribution of seen tasks, as its prior. generalizing predictor is defined as the Bayesian posterior predictive with the true task distribution Time as its prior.
29

---

Trrain
Estimate likelihoods
Compute weighted average
Trree
Σ
Memorizing
Tisas
Dirichlet(1)
Compute count (+1 pseudo-count)
Generalizing
Figure 11: Visualizing the setup for Balls and Urns. Each task involves an "urn" that outputs "ball" of specific type every time sampled from. The task then involves seeing samples from an urn, concatenated to form sequence. A memorizing predictor for this setting involves computing the sequence-level unigram statistics, i.c., the counts for each ball type, and comparing them with distributions from urns seen during training. Mcanwhile, generalizing predictor simply assumes the distribution of balls follows a uniform Dirichlet prior, thus predicting simply based on computing the unigram statistics from the sequence and adding pseudo-coumt for each ball type. Thus, this predictor generalizes to novel urns not seen by the model during training.
G.1 Balls and urns
Memorizing Predictor.
The memorizing predictor perform Bayesian averaging operation and
requires computing weighted average of all ur distributions seen during training. The weight each um derived from the likelihood of the current sequence of observations being generated by that urn. Formally, let wy denote the parameters for urn d {L.... D}. with each clement w (k) containing the probability for ball type & E {1....,m) m) under urn D. Then, the probability of a new ball being of type i after seeing sequence is:
p(k)s) &
w
(3)
II
(w (k)))) vmn
Tuae
(E(LI....
With nu being the number of occurrences of ball of type k' in the sequence.
Generalizing Predictor: Given that the distribution is uniform Dirichlet, and that the Dirichlet distribution is a conjugate prior of the categorical distribution (from which we draw our samples), the optimal way to estimate the probability +1 with nı being the number of occurrences of ball of type & in the sequence. That is, p(k 8) = C.n
ball of particular type & in sequence of length C' is:
the optimal strategy for the true distribution simply computing a count for each type, adding pseudo-count, and dividing by the sequence length.
G.2 Linear regression
Additional Details Not Provided Main Text. To maintain a constant signal-to-noise ratio across tasks with different dimensionality m, set ~ N0,, o2), with x² 256
The generalizing predictor in this case simply performs ridge regression. Given x = [ ...., xT-1) and = (c-1),, the weight estimate is after seeing C- examples
Generalizing Predictor.
is:
w (C) G = (x 0² Im) -
Memorizing Predictor. The memorizing predictor in this case performs inference by Bayesian averaging: weighted average across all wid)s seen in the training distribution, with weights determined by the likelihood that the sequence generated by the specific ((()) After seeing C
30

---

T
Estimate likelihoods
Compute weighted
Tarue
aneroge
Memorizing
MO, m
Tavs
Ridge regression
Generalizing
Figure 12: Visualizing the setup for Linear Regression, Each task involves linear regression problem, defined by parameters w. that outputs a pair (æt, y), where w x + is noisy linear transformation of the vector æ. The task then involves seeing sequence of such pairs, concatenated to form a sequence. A memorizing predictor for this setting involves computing the likelihood of the pairs seen in context under the parameters of each task seen during training, using this result to compute a posterior over said tasks and seen tasks. Meanwhile, the generalizing predictor merely the ridge regression operation, which is equivalent to performing Bayesian average operation assuming continuous Gaussian prior. Correspondingly, this predictor generalizes to novel regression tasks that were not seen by the model during training.
weiighte average with a discrete prior over
examples, the weight estimate is:
exp( C ( = xz,)2)
(C) M
Σ
Ene Timi
exp( D (yec w/ xc)2)
wa Tuai
G.3
Classification
We use the classification setting with the formulation from [29] as well as inspiration from the noisy class centroids introduced by [28]. As Nguyen and Reddy [29] have shown, their simplified setting captures the phenomenology of other classification settings proposed by Chan et al. [20], Reddy [28]. For simplicity, we include only binary labels in our version.
Additional Details Not Provided in Main Text. When presented in context, items w noised and presented as v (0, Im/m) is sampled separately for each item in the context.
wiic
We use within-class variance of a² = 0.5 all settings, and
vI+az
Memorizing Predictor. The memorizing predictor in this setting performs inference by computing a posterior-weighted average over item-label pairs seen in the training distribution w @ I Turann The posterior depends on the similarity of the latent to the query Uquery as well to each context item w given that the model assumes the generative process in which the query is taken from the same latent as of the context items. The form for noisy item used for defining the input sequence is w wiae for some w ~ Tuain Thus, since e has covariance I /m, we can write noisy
vI+az
item sampled from given will be distributed (z
wa) N N ( Ifoz Imm mm)
We then define [(w) exp シ.
value proportional to the likelihood this distribution (disregarding constants). Then, the probability of the query label being
(1+0²)
I is:
31

---

Trrain
Tiree
Noiey retrieval
Compute meighted
awerge
lD
A
Memorizing
{0.1)
Texts
In
(0,
Compute weighted
pwerage
W9
W1 Wyp
Noisy
Generalizing
copy
Figure 13: Visualizing the setup for Classification. Each task involves noisy item-label pairs @ L, and ends with noisy query item quuery which comes from the same true item w as one of the items in the sequence. Items are noised via i = w+ae with e E No m//m) sampled from the same distribution the true item The memorizing predictor stores the true latent items of the training distribution and computes posterior- costerior-weighted average using the labels for each true item. The posterior for latent is determined by its similarity the noisy query item, well as its similarity each context item, given that the generative process dictates that the query comes from the same latent of the context items. Therefore, this predictor implements context-constrained 'in-weights learning' solution, discussed by Singh et al. [33]. In contrast, the generalizing predictor implements a noisy copy operation. estimates the likelihood that the query head and each item seen in context come from the same true item. Then, it predicts via westhte average according to the labels of each item seen in context. Therefore, this predictor works for OOD settings containing novel items that were not previously seen during training.
vi+o*
p(1|s)
Σ
plwal Uqquery, {uo @ l,..., - @ (c- p(1(20))
w.d E Timin
Σ W.T'Tin
p(
Σ
(( 1(1, = (a) 1(la -1)
wy Contexx
L(1ū (aa) EwaeTen C(i0g) (w))
Σ
C Uiuuryy Wa
Σ to EContext
1(1j = (P)
I(la 1)
W.d Timin
Generalizing Predictor.
The generalizing predictor in this setting performs inference by computing nosterior-weighted average over the noisy item-label pairs seen in the context.
This predictor assumes access to the true generative process for defining the context, in which items [w), Wc-1 } are randomly sampled from Times with one item chosen at random to be the query Wquery- Finally, all items independently noised via 400
VI+62
Because W. querr sampled from the specific set of latent items (w),... WC-1 underlying the context rather than drawn anew from the continuous distribution Time. the prior over Wquery collapses to uniform discrete distribution over these unobserved context latents. Since the model cannot access these true latents directly, it computes the conditional probability of the query given each noisy context item w, treatingg Wquery and új as noisy observations of the same latent parent.
The conditional distribution of the query item given the context item is Gaussian:
pl queeryy wi,) = - (Hiqeey' (G; Eg) Using the standard Gaussian conditional formulas get:
(1 22)2
PG
1 1 a ن. Dg
d2 02) Tm
Disregarding normalization constants, the probability of the query label being 1 is:
p(1
Σ
pl U(tuuury wر,) p(((2;)
EConeest
mm1+0* 20 + +02)
W
exp
Uquery
1(1, = 1).
+
EContext
32

---

H
Main Results
In the following sections, provide the results reported in the main paper across all settings and experiments.
H.1 Task Diversity Effects
We find task diversity effects [25] be very robust across settings and experimental conditions, More specifically, we consistently find that increasing task diversity yields transition in Transformer behavior from behaving like a memorizing predictor to behaving like generalizing predictor. In the following, we present evidence of this phenomenon for ID sequences. See results in following pages.
33

---

H.1.1 Balls & Urns
MLP Expansion Factor
0.5
4
co.
Task Dimensionality = 8
128
Task Civenity
Talk Caleniyy D
Context Length
256
6.0
Texk Uieertyy
Teak Diverity
320
2
Text Uierrtyy
Teak Diarnty
Task Dimensionality = 12
128
C.0
000
Toos Divilyy
Task Dandly
Telk Deencty D
Context Length
600
256
Trx: wasty
Text Diwestyy
Task Chersty
100
320
Dassiy
Duuntty
6.0
c11
613
Task Dimensionality = 16
128
Divenity
Chunsty
Context Length
256
Chunity
320
Cienrry 6
Dieestyy
Chensty
Figure 14: Task Diversity Effects Across Balls & Urns Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps).
34

---

H.1.2 Linear Regression
MLP Expansion Factor
0.5
4
8
Task Dimensionality = 8
16
Trak Dicenty
Diwnndyy
Talk Divnnty
Context Length
32
Diantty D
Diverrty
Teak Divenity
64
MSE
Dianity
Divesty
Task Dimensionality = 12
16
MEC
Deserity
Context Length
32
Tesk Divesty 0
Dieenty
Teak Oieenty
64
Trak Divistyy 5
Tass Divendty
Tosk Divenit
Task Dimensionality = 16
16
facetiindy
Devecty
Fasc leddy
Context Length
32
Uivrrity
Tuxco venry
64
Figure 15: Task Diversity Effects Across Linear Regression Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps).
35

---

H.1.3 Classification
Context Length
128
256
384
Oroae maaaeeetog
0.5
MLP Expansion Factor
Teol Diveriy
Tosk Diveniy
Toxk Durriyy n
Task Dimensionality
Dowerro
Oiventy
Unvenity 5
Conserapp
Compareapy
8
Diveniy
Task Diveniy 5
Task Diiiiiy
Croan senoo
16
Diceedy
Divenity
Dureriyy
Figure 16: Task Diversity Effeets Across Classification Conditions. Red dashed line indicates the memorizing solution M, blue dashed line indicates the generalizing solution G, and black solid line indicates Transformer behavior at the end of training (100K steps).
H.2 Transience
Across settings and conditions, also find the phenomenon of transience [24] to be consistent in moderate task diversity values. More specifically, in moderate task diversity values, we see the Transformer approach the generalizing solution in terms of OOD performance early in training, only to eventually begin memorizing and worsen in OOD performance. In the figures below, we show OOD performance of Transformers trained in different task diversity conditions, with low task diversity values showing immediate memorization, moderate task diversity values showing transience, and high task diversity values often continuing generalize well throughout training. See results in following pages.
36

---

H.2.1 Balls & Urns
MLP Expansion Factor
0.5
4
8
Task Dimensionality= 8
128
Context Length
256
traiing
320
Tea Rope
Touking Supe
Trii Siyp.
Task Dimensionality= 12
128
Context Length
256
Tea
Trai Siepe
320
Trannng Staps
Tram Stopp
Task Dimensionality = 16
128
Context Length
Rope
256
11
The
Stepa
Tran Stepo
320
Traning Step
eg Sope
Figure 17: Transience Across Balls & Urns Conditions. OOD performance presented. Blue Dashed line indicates OOD performance of generalizing solution G.
37

---

H.2.2 Linear Regression
MLP Expansion Factor
0.5
4
Task Dimensionality = 8
16
Teoknng Stepe
Trawing Supp
Traning
Context Length
32
MIE
Strpe
Toaring
64
NS2
Task Dimensionality = 12
16
VIDE
Took Stepe
Trxsing Sope
Teaciig Dorpe
Context Length
32
MFC
Training
Traning Supa
Training Stepo
64
MED
Task Dimensionality = 16
16
Truwing Stepe
Traning Stepe
Teaning
Context Length
A
32
Trasing Pops
Traicing Sospe
TDl
Training
Figure 18: Transience Across Linear Regression Conditions. OOD performance presented. Blue
Dashed line indicates OOD performance of generalizing solution G.
38

---

which H.2.3 Classification
Context Length
128
256
384
8
0.5
MLP Expansion Factor
Task Dimensionality
Staces
8
8
16
4
Teakiig
Trakiin
Figure 19: Transience Across Classification Conditions. OOD performance presented. Blue Dashed line indicates OOD performance of generalizing solution G. Note that in the case of classification, often the case that only or two task diversity conditions show transient generalization, can be seen more clearly from the absolute distance maps in the next section.
H.3 Absolute Distance from Predictors
We find that across settings and conditions, Transformers primarily learn and transition between behaving like two predictors: generalizing solution G. which consists of the Bayesian posterior predictive distribution over the true task distribution Time and a memorizing solution consists of the Bayesian posterior predictive distribution over the training task distribution Tiain- In the figures below, we display the absolute distance from each of these predictors, as well the relative distance in the background (ranging from red indicating closeness to M to blue indicating closeness to G). See results in following pages.
39

---

H.3.1
Balls & Urns
MLP Expansion Factor
M
G
0.5
8
8
128
Task Dimensionality- =
Context Length
256
320
Task Dimensionality= = 12
128
Context Length
256
320
Task Dimensionality = 16
128
Context Length
256
320
Figure 20: bbooute and Relative Distance from Predictors Across Balls & Urns Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M is shown in the solid line. KL indicates symmetrized KL divergence (average of forward and backward KL).
40

---

H.3.2 Linear Regression
MLP Expansion Factor
G
0.5
4
Task Dimensionality = 8
16
Context Length
32
64
Task Dimensionality = 122
16
Context Length
32
64
Task Dimensionality- = 16
16
Context Length
32
64
Figure 21: Absolute and Relative Distance from Predictors Across Linear Regression Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M is shown in the solid line.
41

---

H.3.3 Classification
Context Length
M
128
256
384
G
0.5
MLP Expansion Factor
Task Dimensionality
4
8
Testing
16
4
Traiiig
Figure 22: Absolute and Relative Distance from Predictors Across Classification Conditions. Distance from the generalizing solution G shown in the dashed black line, while distance from the memorizing solution M shown in the solid line. KL indicates symmetrized KL divergence (average of forward and backward KL).
H.4 Model Predictions
Across settings and training conditions, we find that our model's predictions consistently perform well both in estimating the next-token prediction behavior of the Transformer, well as capturing its change in generalization behavior across conditions, as displayed by the relative distance from predictors G' and M. We conduct thorough stress-testing of model across settings and 72 (N, D) maps, each containing 11 different training runs, and find that our model consistently performs well across maps, thus providing robust evidence for the predictive and explanatory power of our account. See results in following pages.
42

---

H.4.1
Balls & Urns
G
M
MLP Expansion Factor
0
0,5
4
20
-8
128
Task Dimensionality =
Context Length
256
20.5
320
Task Dimensionalit = 12
128
Context Length
256
By
24
320
Task Dimensionality = 16
128
Context Length
256
320
Figure 23: Bayesian Model Predictions Across Balls & Urns Conditions. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the memorizing solution M given by our Bayesian model (left) and the relative distance from the Transformer (top right), as well as heatmaps indicating similarity with Transformer next-token predictions (bottom right). Max color bar value is determined by the performance of a baseline predictor that always outputs the mean of the distribution Tine-
43

---

G
M
0.5
2
MLP Expansion Factor
80
12
Mond
16
24
32
Figure 24: Bayesian Model Predictions Across Balls & Urns Conditions with Varying MLP Expansion Factors. Red indicates closeness memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown of the memorizing solution M given by our Bayesian model (left) and the relative distance from the Transformer (top right), as well as heatmaps indicating similarity with Transformer next-token predictions (bottom right). Context length is 128, task dimensionality' is 8, and hidden size is 64 in all conditions shown. MLP width is given by hidden size times MLP expansion factor. Max color bar value is determined by the performance of baseline predictor that always outputs the mean of the distribution Taue.
comparison between the posterior probability
44

---

H.4.2
Linear Regression
G
M
MLP Expansion Factor
0
0.5
-8
16
Task Dimensionality =
Context Length
32
64
Task Dimensionality- = 12
16
Context Length
32
5%
Fr
64
Task Dimensionality = 16
16
Context Length
32
64
Figure 25: Bayesian Model Predictions Across Linear Regression Conditions. Red indicates
closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is comparison between the posterior probability of the memorizing solution M given by Bayesian model (left) and the relative distance from the Transformer (top right), as well as heatmaps indicating similarity with Transformer next-token predictions (bottom right).
45

---

H.4.3
Classification
G
M
Context Length
0
128
256
384
Meyccu
Raproue Mall
Kaymar
0.5
MLP Expansion Factor
Task Dimensionality
Thiiing
Thii
Aepecua
Tea.Dwatty
8
i0g
6
Dessae
Kipecse
16
Tae
Figure 26: Bayesian Model Predictions Across Classification Conditions. Red indicates closeness to memorizing predictor M, while blue indicates closeness to generalizing predictor G. Shown is a comparison between the posterior probability of the memorizing solution M given by our Bayesian model (left) and the relative distance from the Transformer (top right), as well as heatmaps indicating similarity with Transformer next-token predictions (bottom right). Max color bar value is determined by the performance of a baseline predictor that always outputs the mean of the distribution Tawe- The conditions where task dimensionality equals 16 in this setting (bottom row) reveal limitation of our complexity measure: since the memorizing and generalizing predictors are very close in performance in low task diversities for these conditions, the loss term does strongly bias the Transformer towards the memorizing predictor. However, the Transformer, is very close to the memorizing predictor for low task diversities, which would indicate according to our framework that memorizing few items is substantially simpler than implementing copy operation. However, this is not captured by our complexity measure, since the compressed size of the code for the memorizing and generalizing predictors is roughly the same, thus we are unable to capture the bias toward the memorizing predictor in low task diversity settings. To overcome this, in these 3 conditions only, we heuristically multiply the bit size of the code for the generalizing predictor by 5, and with that fix, we find good performance (though as can be seen, the model still under weights the memorizing solution for some low task diversity conditions).
46

---

H.S Extension to Multiple Strategies
To examine the generalization of our model multiple predictors, we include constant mean predictor in the hypothesis space for the linear regression setting, as models are known to learn this solution early [70, 101]. As shown in Fig. 27, our model successfully captures the dynamics of multiple predictors, and the inclusion of the optimal constant predictor allows to capture Transformer behavior very early training (from 20 steps), which was not possible using only the memorizing and generalizing solutions (App. E). To compare the posterior probabilities given by our model the relative distance, we compute a generalized version of the relative distance by fitting convex combination of strategies to Transformer next-token predictions each (N, D) condition. We find high average cosine similarity of 0.96 0.003 (SE) between the vector relative distances from each predictor and our model's posterior probabilities.
47

---

MLP Expansion Factor
0.5
16
Task Dimensionality=
Context Length
32
64
Task Dimensionality = 12
16
Context Length
32
64
Task Dimensionality 16
16
Context Length
32
Figure 27: Predictions Bayesian Model with Multiple Strategies Across Linear Regression Conditions. Red indicates closeness to memorizing predictor M, blue indicates closeness general- izing predictor G, and teal indicates closeness to the optimal constant solution C. We color the maps via convex combination of posterior probabilities from model for M, G. and C (left), and use similar method for coloring the generalized relative distance measure (top right). Finally, we show heatmaps indicating similarity with Transformer next-token predictions (bottom right). Max color bar value is determined by the average performance of the optimal constant solution across all conditions.
48

---

H.6 Extension to In-Context Strategy Selection
We find that models evaluated in OOD settings behave more like the generalizing solution, phe- nomenon we term in-context strategy selection. We extend our model to accommodate this behavior by allowing the posterior probability D.4). We find that, for the Balls & Urns setting, this extension aids in capturing the shift toward the generalizing predictor seen during OOD evaluation (Fig. 28). Yet, particularly higher task dimensionality, OOD behavior is not very well-captured by the model around the transition boundary, indicating the posterior updating throughout the context may lead other predictors receive non- negligible posterior mass, simply general instability at OOD evaluation (which is also seen in the linear regression setting). Additionally, we make a simplifying assumption when deriving the form for this extension, that the and ? parameters that govern the rate of update for the posterior during pretraining remain similar during co-tentext updating. Additional work is required for understanding the nature of updates from pretraining vs. ICL [73], and potentially incorporating these into Bayesian models of ICL.
predictor continue updating throughout the context (see
49

---

G
MLP Expansion Factor
0.5
8
-
128
Task Dimensionality
Context Length
256
320
Task Dimensionality 12
128
Context Length
256
320
Task Dimensionality 16
128
Context Length
256
320
Figure 28: Bayesian Model Predictions Across OOD Balls & Urns Conditions. Red indicates closeness memorizing predictor M, while blue indicates closeness generalizing predictor G. All relative distance values are computed over OOD sequences drawn from Time- Shown is a comparison between the average posterior probability over a sequence of the memorizing solution M given by our Bayesian model (left) and the relative distance from the Transformer (top right), as well as heatmaps indicating similarity with Transformer next-token predictions (bottom right).
50

---

I Functional Form Ablations
Our functional form for the log posterior odds 7) consists of free parameters, a, determining sublinear sample efficiency, B, a power law on the estimated Kolmogorov complexity K, and y, coefficient for the loss term. We find that each of these free parameters are necessary for the success of our model, since removing any of them results in worsening of the model's ability to capture the phenomenology ICL.
Transformer (Rel. Distance)
No Free Parameters
No B
No a
No Y
Linear B
Full Model
Task Diversity
38
10k
35
186
3K.
rOK
0:25
nok.
100
0.2
1000
m
Training Steps
Training Steps
Training
Training
Training Steps N
Training Steps
Figure 29: Power laws over complexity measure and sample efficiency are necessary for explain- ing ICL phenomenology. By ablating our functional form, are required for the performance of the model. In particular, the simplicity bias derived from K without 3 term much too sharp and over verrpenalizes complexity compared to the Transformer, and simply adding a linear 3 term does not capture the contour the transition or the gradual transience in higher task diversity conditions. Additionally, memorization proceeds much too rapidly without the a term, pointing toward the necessity of assuming sub-linear sample efficiency for capturing Transformer training dynamics.
that the free parameters a, B, n
of generalization
51

---

J Memorization Continues to Increase After Task Diversity Threshold Refutation of Raventós et al. [25]'s Claim
In Fig. 30, we show a refutation of the claim made by Raventós et al. [25], who claimed that after the task diversity threshold reached, the Transformer will only continue to get closer the generalizing solution throughout training. regardless of how long one trains. In making this claim, Raventós et al. [25] focused only on the absolute distance between the Transformer and the generalizing solution. However, when we considering relative distance measure, we can show this claim to be false (see right side Fig. 30); Even conditions in which the task diversity threshold was reached and generalization is sustained throughout reasonable amount of training (100K steps), see that by absolute distance, Transformer not only gets closer to generalizing solution during training, it also continues to get closer the memorizing solution (left side of Fig. 30). It seems that the rate at which the Transformer nears the memorizing solution is greater than that at which it nears the generalizing solution in task diversity settings such as D 512 D 256, which is why the relative distance continues to grow even despite the transformer nearing the generalizing solution. Note also, that in settings that clearly show transience, e-g-, D 64, the distance from the generalizing solution shrinks until a certain critical point in which it begins to grow. It likely that this point can be reached in all task diversity settings examined given the growth trend of the relative distance. However, it may require very long training process to reach that point.
1.0
Relative Distance
0.8
4096
Mawlaaggaannn Sotaeie www.e www.eeggeeeeeeeeeeeeeeo 2lddd
0.4
1024
0.2
D- 256
75K Training Steps N
50K
100K
M/S
D 64
1.0
Relative Distance
0.8
0.6
D
10a Training Steps N
105
0.2
0.0
1w10
20
30
Ni-a
Figure 30: Relative distance continues to rise throughout training, even after the task diversity threshold. The figure displays relative distance, well as absolute distance from the memorizing and generalizing solutions, for the linear regression setting with context length 32, MLP expansion factor of 4, and 8 dimensions.
52

---

K Learning Rate Annealing Can Improve Adherence to Bayes-Optimal Trajectories
In main experimental settings, in which we train with a constant learning rate, find that relative distance trajectories follow a sigmoidal growth pattern with respect to N (see Fig. 6 and Fig. (a) top and bottom right). However, in contrast with our theory's predicted sigmoidal curves which plateau at 1 (i.e., some amount of training would eventually yield full adherence to the memorizing solution, when has lower loss than the generalizing solution), this does not seem to be the case with the trajectories displayed by Transformers, which appear to plateau early (see Fig. (a) top right D 256 for clear example). Indeed, fitting parameterized logistic curves to these trajectories yields plateau values different from 1. To explain this, we turn to foundational work from Geman and Geman [102], who showed that a slow (logarithmice) temperature cooling schedule for Gibbs sampling substantially increases the likelihood of convergence to the global minimum (MAP estimate). Drawing a very rough parallel to the case of deep learning. is reasonable to assume that a learning rate annealing schedule of some form is required to converge to the MAP estimate (which is the memorizing solution, in cases where has lower loss than the generalizing solution). To test this, we repeated experiments in the Balls & Urns and Linear Regression settings and used warm-up up and mnverse-quaared learning rate decay. Surprisingly, we indeed find that adding learning rate annealing can increase adherence Bayes-optimal trajectories (Fig. (a)). However, also find this effect is highly sensitive to training conditions: training for longer, even in conditions that yield the effect for smaller number of training steps, can lead to plateau (Fig. 31(b), top), and slight changes in training conditions, in this case number of warm-up steps, can substantially reduce the effect (Fig. (b), bottom). It should also be noted that in this case, adherence to the Bayes-optimal trajectory is actually negative for generalization, since it means the model will converge to the memorizing solution quicker. However, this is not necessarily the case in more realistic training regimes, where the ability to overcome a simplicity bias and adopt more complex solutions is likely beneficial.
(a)
(b)
Learning Rate Annealing can Increase Adherence to Bpyes-Optimal Trajectories
But Effect is Highly Sensitive Training Conditions
With Annealing
Without Annealing
Platesus throughout training
Trdiaad
1.0
Relative Distance
0.8
Balls & Urns
0.6-
No
Terkin Docc
Traniig Seape
0.4
0.2
0.0
10
30
40
50
NV
With Annealing
Without Annealing
Sensitive Warmup
Drccaad
Masure
Trediied
Vanund
Peedaae
Linear Regression
Figure 31: Learning Rate Annealing Can Increase Adherence to Bayes-Optimal Trajectories. (a) Balls& & Urns setting uses context length of 128, MLP width of 256, and dimensions. Linear regression uses similar variables except a context length of 16. (b) Effect is highly sensitive to training conditions: e.g., Training in the Linear regression setting with 5000 warmup steps failed, but succeeds with 500 warmup steps (the number used for the experiment in panel (a)).
53

---

L Code Availability
All code used run the experiments and analysis, well as evaluation metrics for all settings, is available at: http://gittuub.com/Dnnnee Wurgaf/rrational-icl http:
54

---

M
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We provide lot of empirical verification to back our claims, which theoretically justify in the first place.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
55

---

Justification:
Appendices provide detailed constructions, and the main paper discusses assumptions in detail.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed appendix with details on experiments and release code.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
56

---

Answer: [Yes]
Justification: We provide access to repository containing all code for the experiment, as well data for replication the main results in single setting.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Policyy)lyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //niss "nips.cc/pill wwwpcc/ppppppppeeeeeeeeeee gwwggguidcccsssai weamiisssonoooiigoiioo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See experimental details in appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report the standard error for metrics of our model's performance. However, given limited compute access, we prioritized robustness across settings over robustness across training runs repeating the same setting, and thus cannot report error bars over training runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
57

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See experimental setup in appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics https//////rrrppp...c/////////// wwtpccccp/pccceeeeeeeeeeeeeeecee hwabblii/Ehhicsuidelleeeeee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have tried our best to avoid leaking signatures of our names or any affiliations.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
As theoretical paper, we believe this question does not feasibly apply.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
58

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Does not apply.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We primarily use Py PoTch.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee.hooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
59

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new assets were introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Does not apply.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Does not apply.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
60

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Does not apply.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: htttps::/neurips... httprripssee//ooeeeeenneee////// for what should or should not be described.
61


Reviews:
[{"review_id": "AxaNNlzs3z", "reviewer": "Reviewer_yreR", "summary": "This paper introduces a unified Bayesian framework to study the transition of transformers between memorization and generalization in In-Context Learning. The authors model the phenomena of ICL in Transformers as a Bayesian model comparison process.", "questions": "How would the framework fit to tasks with more than the two relevant predictors?", "limitations": "The dependence on the assumptions of simplicity bias and power law scaling may not hold in all cases.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The paper show the tradeoff between memorizing predictors and generalizing predictors as a function of task diversity and training time.\n2. The authors conduct experimental exploration under three different settings, including sequence modeling, linear regression and classification. \n3. The paper uses an interpretable Bayesian framework to explain several In-Context Learning phenomena.\n\nCons:\n1. It is advised to add a computational cost analysis of Bayesian model fitting process.\n2. The generalizability of simplified assumptions with two predictors to complex scenarios remain uncertain.\n3. The experiments are conducted in synthetic tasks with well defined task boundaries. The applicability of the framework to real cases with real-world settings is a concern.", "was_revised": true, "final_justification": "After a consideration according to the answers and other reviews, I would like to maintain my current score.", "rebuttal": "We thank the reviewer for their valuable feedback! We also thank the reviewer for supporting the acceptance of our work, and highlighting the strength of the paper in providing an “interpretable Bayesian framework to explain several In-Context Learning phenomena.” We wish to highlight that, in addition to explaining ICL phenomena, our account shows excellent quantitative fit and predictive power: quantitatively, across 72 different maps with varying MLP width, context length, and task dimensionality, we find our model is highly predictive of Transformer next-token predictions, with a mean $R^2$ of 0.97 in Linear Regression, a mean agreement of 0.92 in Classification, and a mean Spearman rank correlation of 0.97 in Balls & Urns. Additionally, we find very strong correlations of 0.99, 0.98, and 0.99 between our model’s posterior probabilities and the relative distance values given by the Transformer in the Linear Regression, Classification, and Balls & Urns settings, respectively.\n\nBelow, we respond to specific comments by the reviewer.\n\n-----\n\n\n> **It is advised to add a computational cost analysis of Bayesian model fitting process.**\n\nWhile we do not address this directly in the paper, we note the fitting process of our Bayesian model is very fast! This is because for a given heatmap of predictions (e.g., Figure 4a), we only fit **3 free parameters** across *all* combinations of task diversity and training times. The process of fitting takes less than a minute to complete. This shows that our model is not only interpretable and predictive of Transformer behavior throughout training, but also quick to fit. We will make sure to add fitting details to the paper!\n\n------\n\n> **The generalizability of simplified assumptions with two predictors to complex scenarios remains uncertain... How would the framework fit to tasks with more than the two relevant predictors?**\n\nGreat question, and one that is crucial to the generalizability of our account! While we demonstrate our main results for two predictors, our framework can be easily extended to more predictors. Below, we first provide the **derivation** showing how our framework generalizes to more than two predictors. Then, we describe **new results** from an experiment in the linear regression setting, which provides a proof-of-concept that our framework generalizes to more than two predictors. \n\n#### **Mathematical derivation:**\nOur predictive equation for transformer behavior is: \n$$h^{pred}(s_i|s_{i-1:1},S_{\\mathcal{T}})=p(M|S_{\\mathcal{T}})M(s_i|s_{i-1:1})+p(G|S_{\\mathcal{T}})G(s_i|s_{i-1:1})$$, \nwhich is simply a posterior-weighted average over the predictors (which in our case are $M$ and $G$). \nTherefore, in the case of $H$ predictors $Q_i \\in \\{Q_1, ..., Q_H\\}$ we can simply write: $$h^{pred}(s_i|s_{i-1:1},S_{\\mathcal{T}})=\\sum_{i=1}^Hp(Q_i|S_{\\mathcal{T}})Q_i(s_i|s_{i-1:1}).$$ \n\nHow should one compute the posterior in the case of $H$ predictors? In our current analysis, given our assumptions of prior and likelihood, we write the log posterior odds as:  \n$$\\eta(N,D)=\\log p(M|S_{\\mathcal{T}})-\\log p(G|S_{\\mathcal{T}})=\\gamma N^{1-\\alpha}\\Delta L(D)-K(D)^{\\beta}$$ and convert this to the posterior for the memorized solution via $p(Q|S_\\mathcal{T})=\\sigma (\\eta(N,D))$. \nHowever, we can just as easily write this in a way that generalizes to $H$ predictors: given our assumptions of prior and likelihood, we can write: $$\\log f(M|S_\\mathcal{T})=-\\gamma N^{1-\\alpha}\\bar{L_M}(S_\\mathcal{T})-K(M)^{\\beta}$$ with $\\log f(M|S_{\\mathcal{T}})$ being the unnormalized log probability. Then we get the posterior probability for $M$ by merely applying Softmax: $$p(M|S_\\mathcal{T})=\\frac{\\exp(\\log(f(M|S_{\\mathcal{T}})))}{\\exp(\\log(f(M|S_{\\mathcal{T}}))) + \\exp(\\log(f(G|S_{\\mathcal{T}})))} = softmax(\\log f(M|S_{\\mathcal{T}}))$$\n\nSimilarly, for a predictor $Q_i \\in \\{Q_1, ..., Q_H\\}$, we have $\\log f(Q_i|S_\\mathcal{T})=-\\gamma N^{1-\\alpha}\\bar{L_{Q_i}}(S_\\mathcal{T})-K(Q_i)^{\\beta}$, and hence: \n$$p(Q_i|S_\\mathcal{T})=\\frac{\\exp(\\log f(Q_i|S_{\\mathcal{T}}))}{\\sum_{j=1}^H \\exp(\\log f(Q_j|S_{\\mathcal{T}}))} = softmax(\\log f(Q_i|S_{\\mathcal{T}})).$$ \nPutting this into the equation above gives: $$h^{pred}(s_i| s_{i-1:1}, S_{\\mathcal{T}})=\\sum_{i=1}^H softmax(\\log f(Q_i|S_{\\mathcal{T}}))Q_i(s_i|s_{i-1:1}),$$ which is a general form of our model that works for $H$ predictors. \n\n#### **New results:**\nBeyond the derivation above, given the reviewer’s question, we note we have now run experiments to study the generalization of our results to more than 2 predictors. Specifically, we conduct an experiment in the linear regression setting, where, as we note in the discussion of our paper (Section 5), we find an early-appearing predictor that we call the *optimal-constant solution* ($C$). This solution always predicts the mean of the distribution (see [1], [2] for further description) and appears before either the generalizing ($G$) or memorizing ($M$) predictors, which qualitatively aligns with our framework’s perspective that simpler solutions appear earlier and are later replaced by better-fitting but more complex ones. \n\nWhile we are unable to provide heatmaps similar to Figure 4 due to this year’s NeurIPS rebuttals guidelines, we emphasize the results are very similar in nature. In this analysis, we start our model’s predictions from just 20 training steps, yielding insight very early into training. We also note that the model with 3 predictors captures transition dynamics, yielding a strong correlation with a version of the relative distance measure generalized to three predictors. Overall, this result provides a proof-of-concept that our framework can be easily generalized to capturing complex dynamics with more than two predictors, as well as capturing phenomena that emerge in settings with multiple predictors, such as the rapid emergence of in-context learning (seen early in the linear regression setting). \n\nOverall, we again thank the reviewer for this great question as it helped us further generalize our claims. We will make sure these results are added to the final version of the paper!\n\n[1] arxiv.org/abs/2406.17467\n\n[2] arxiv.org/abs/2402.02364\n\n------\n\n> **The experiments are conducted in synthetic tasks with well defined task boundaries. The applicability of the framework to real cases with real-world settings is a concern.**\n\nFair point! While we agree that our experiments use synthetic tasks, we emphasize that these tasks are *precisely* the ones that have been studied in the bulk of the literature to understand ICL [1-12]. We thus argue that any theory targeted at explaining ICL must at least show predictive power over this spectrum of simple, controlled, but phenomenologically very rich set of tasks that form the focus of prior work. Our hierarchical Bayesian framework meets this standard, offering both a predictive account of Transformer behavior throughout training and explanations for several past results on ICL in a unified manner.\nHaving achieved this result, we now strongly think that the stage is set for analyzing more naturalistic settings. We have in fact already begun preliminary explorations on this front, analyzing the phenomenon of many-shot jailbreaking wherein safety fine-tuned models, when offered in-context sufficiently many exemplars of some undesirable behavior (e.g., producing violent speech), start to engage in said behavior. We believe our framework can offer a predictive account of this phenomenon with realistic models, but a detailed analysis warrants its own explicit study!\n\n[1] arxiv.org/abs/2205.05055\n\n[2] arxiv.org/abs/2312.03002\n\n[3] arxiv.org/abs/2412.00104\n\n[4] arxiv.org/abs/2311.08360\n\n[5] www.jmlr.org/papers/v25/23-1042.html\n\n[6] arxiv.org/abs/2310.08391\n\n[7] arxiv.org/abs/2503.05631\n\n[8] arxiv.org/abs/2402.11004\n\n[9] www.pnas.org/doi/abs/10.1073/pnas.2502599122\n\n[10] arxiv.org/abs/2211.15661\n\n[11] arxiv.org/abs/2306.15063\n\n[12] arxiv.org/abs/2412.01003\n\n[13] https://www.anthropic.com/research/many-shot-jailbreaking \n\n-----\n-----\n\n**Minor Comments**\n\n> **The dependence on the assumptions of simplicity bias and power law scaling may not hold in all cases.**\n\nWhile we agree with the reviewer that we rely on assumptions of simplicity bias and power law scaling, these assumptions have been robustly shown to hold across many studies of neural networks as well as practical applications. For power law scaling, there is extensive evidence that neural networks reduce loss at a rate corresponding to power laws as dataset size increases ([1-2]). This evidence has also strongly driven model development in industry, which has shown power laws to hold across many orders of magnitude. With regards to simplicity bias, works examining training dynamics ([3-5]), as well as directly sampling from initialized neural networks ([6,7]), have consistently revealed a simplicity bias in neural network training dynamics and their parameter-function map, respectively. Thus, we see our two main assumptions as robust and not substantially impacting the generalizability of our work. \n\n\n[1] arxiv.org/abs/2001.08361\n\n[2] arxiv.org/abs/2203.15556\n\n[3] arxiv.org/abs/1905.11604\n\n[4] arxiv.org/abs/2006.14599 \n\n[5] arxiv.org/abs/2405.05847 \n\n[6] arxiv.org/abs/2304.06670\n\n[7] arxiv.org/abs/1805.08522\n\n\n-----\n-----\n\n**Summary:** We sincerely thank the reviewer for their constructive comments and suggestions that have helped us demonstrate the generality of our framework, while also offering the opportunity to show its efficiency for explaining ICL. We hope our comments address the reviewer’s concerns, and, if so, they would consider championing the acceptance of our work!"}, {"review_id": "RXwext9Hcm", "reviewer": "Reviewer_JDTE", "summary": "This is a methodology-experimental paper, which explores in-context learning in a general stylized model, and 3 different specific settings. The paper defines memorization (M) (resp. generalization, G) as the case in which the test prediction is based on Bayesian weightening of the predictors according to their empirical appearance in the training set (resp. in the population distribution). The paper empirically validates that two main phenomena occur for transformer-based training in all 3 settings: (1) transition from memorization to generalization when the training task diversity increases. (2) A deterioration of the test (out-of-distribution) performance when the length of the training increases. Then, it is proposed to model the predictor as a Bayesian mixture of a memorizing predictor and a generalizing predictor. The log-likelihood of the posterior odds is parameterized as a sum of a loss term and a complexity term. The model is used to predict the distance to M and G predictors as the function of the number of samples, that the change between them is rapid, and that it increases with task diversity.", "questions": "In parallel to the weaknesses above:\n\n[Q1] What is an evident for the uniqueness of your explanation in the space of various possible explanations? How does the unified view proposed in this paper contributes to the validity of your explanation? \n\n[Q2] What is an empirical evident that a *specific* complexity term is the one to accurately predict the experimental results?\n\n[Q3] What makes the sub-linear growth a prediction, and what can be said on its exponent?\n\n[Q4] What is different in this context from the well-established phenomenon that long training prevents generalization?\n\n[Q5] How could this serve as an estimate for this transience time in new tasks?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n[S1] The paper addresses a timely and central topic: In-context learning is one of the most important aspects of the recent advances in ML, and the question on under what circumstances it arises is important. The step towards unification of various models proposed and phenomena observed in the literature is useful. \n\n[S2] The conclusion is based on extensive empirical evidence, and the graphs displayed are convincing. \n\n[S3] Overall, the paper is well written and easy to follow. \n\n**Weaknesses:** \n\n[W1] The main weakness is that, from my perspective, the paper is fully derived from the empirical observation of the transitions from memorization (M) to generalization (G). This was already observed in previous works [23,27, maybe more]. From that point onward, the paper reads like a **tautology**. Specifically: (1) It is obvious that if the empirical predictor is very close to either the M or G predictor, and sharply transitions between them, then the only the posterior weights of these predictors will be close to 1, and otherwise tend to zero. (2) It is obvious that if the prior is modeled by a complexity term and the likelihood by a loss term, then the posterior would be comprised of them, and would appear as a loss-complexity trade-off. The paper aims to answer why ICL emerges, but I think that instead it just fits a model to the observed transition between M and G. In other words, the paper observes a transition between M and G, then models the prior as complexity terms, and then explains that the transition is due to balance between complexity and loss. If the prior was modeled in a different way, by a term of different nature, then the “why” explanation would have changed accordingly. \n\n[W2] In continuation to the above, I believe it is agreed that the prior is some sort of **a** complexity term. The weakness of the paper is that it does not address the main question of **which** complexity term? The paper mentions Kolmogorov complexity, but it is well known that this universal complexity term is uncomputable. Any compression algorithm may indeed lead to a measure of complexity, but again, the questions is which complexity term characterizes the ICL phenomena.\n\n[W3] The prediction of sub-linear growth is weak in the sense that a reader cannot judge the order of events. It is possible that you have observed a sub-linear growth, and then decided on the modeling assumption A1. An expectation from a theory is not just to claim for a sub-liner growth, but rather to quantify the exponent $\\alpha$ in some way.\n\n[W4] The paper mentions that transient generalization phenomenon as an important observation, but it is well established that long training prevent generalization (hence, early stopping techniques). \n\n[W5] The scaling of time to transience with task-diversity $N^*(D)$ depends on the complexity of the M and G models, as well as their losses, which, of course, unknown a priori.", "was_revised": true, "final_justification": "I accept the authors answer that the paper clarifies reported results from previous works on the subject, which were misleading. \n\nIn previous works, simple toy models were suggested, to obtain some intuition of the ICL phenomenon. The authors take these as indispensable and aim to provide a unifying theory.  The main intuition of transitioning from M to G is apparent in these works.  For example, I don't think that the loss-complexity trade-of is implicit in these works. This is a completely standard ML phenomenon. \nFrom my perspective, the authors empirically observe a very high level phenomena, and match it with a basic model. \nI find the prior assumption of two-hypothesis to be restricting, and the new addition of a third hypothesis does not ameliorate that. The goal is to show that a model, free from any prior restriction, actually transitions between such predictors. \n\nRegarding the predictive power: from my perspective, since the hypothesized models M and G do not take into account any aspect of the model's architecture, they do not allow any intuition on why the specific architecture works and how to build a better one. It just explains what it currently does, and as said, in a high-level manner (a general loss term, a general complexity term). \n\n\nI thus maintain my score.", "rebuttal": "We thank the reviewer for their detailed and thoughtful feedback. We are glad that they found that our paper “addresses a timely and central topic” and that “the step towards unification of various models proposed and phenomena observed in the literature is useful.” Further, we are grateful the reviewer's note that our “conclusion is based on extensive empirical evidence, and the graphs displayed are convincing”.\n\nBelow, we respond to specific comments. Since the questions map on to comments in the weakness section, we bundle them up and respond to both together.\n\n-----\n\n> **(W1) & (Q1): The main weakness is that...the paper is fully derived from the empirical observation of the transitions from memorization (M) to generalization (G)...From that point onward, the paper reads like a tautology...I think that instead it just fits a model to the observed transition…**\n\nWe *respectfully* disagree with the reviewer that our analysis is tautological or involves obvious implications derived from the transitions between memorizing and generalizing predictors observed in prior work. To clarify our position, we respond to two specific points that we think summarize the reviewer’s comment.\n\n- **Grounding of our analysis in the transitions between M and G.** While the reviewer is correct that transitions between different predictors have been observed in prior work, *we emphasize that our characterization of these transitions is both substantially richer and highly indicative of a unifying theory of ICL*—to our knowledge, our unifying formalization and predictive modeling of transitions between different predictors has not been proposed in prior work. More specifically:\n   - Prior work focuses on demonstrating these transitions with respect to a single control variable, i.e., task diversity or training time, but we in fact show this phenomenon involves a rich interplay between the two variables, yielding distinct phases. Thus, given that several past findings on ICL have characterized these predictors behaviorally, the intuition driving our work is that explaining *transitions* between different predictors learned by a model is the critical bottleneck towards understanding ICL. \n   - Our highly predictive model *builds upon* the observation of distinct phases characterized by each predictor, rather than directly stemming from it. Our model builds upon this observation by considering the two predictors as the model’s hypothesis space. Yet, it goes beyond it by characterizing model inference-time behavior as a posterior-weighted average of predictors, indicating the model weighs preference for each predictor throughout training, and interpolates between predictors at inference time. We believe the finding that a transformer appears to interpolate between different solutions in its response patterns is a non-trivial finding that cannot be directly inferred from the observation of transitions. \n\n- **Mere fitting of some assumed model to the transitions / different models would yield different explanations.** We first note that our hierarchical Bayesian model, which is highly predictive of a trained Transformer’s behavior, emerges from two well-known and generally observed empirical properties of neural networks: scaling laws and simplicity bias. While the reviewer is correct that a different model with different assumptions would have led to different explanations, we strongly emphasize that this empirically does not work out: in Appendix I (functional form ablations), we explicitly attempted several variants of our assumptions and found them to lead to models that are unable to capture the transitions between different predictors. We thus disagree with the reviewer’s claim that our results may be obvious if one merely models the fact that there are fast transitions occurring between two predictors. Instead, we claim our *highly parsimonious functional form (Eq. 3) that quantitatively predicts and explains **why** transition phenomena occur* is a core contribution of our work. More broadly, we believe that the fact that our simple, 3-parameter model accurately predicts the behavior of ~900 checkpoints across 3 diverse settings is strong evidence against the tautology claim made by the reviewer. If the reviewer has a different model in mind, which they believe may fit the data well and hence offer an explanation, we are happy to run experiments with it and assess its validity!\n\n-----\n\n> **(W2) & (Q2): The weakness of the paper is that it does not address the main question of which complexity term?**\n\nGreat question! In this work, we *intentionally* take a top-down, normative perspective to explain Transformer behavior. That is, we examine whether we can abstract away architectural details and explain Transformer behavior by simply asking what would be the optimal behavior under a particular set of solutions (M and G), and given minimal computational constraints. Thus, we emphasize our goal was not to find the *exact* complexity measure a Transformer uses, but to perform a top-down analysis to test if a **general, computational-level notion of complexity** could explain ICL. The striking result here is that even a **crude upper bound on Kolmogorov complexity**—approximated via standard compression algorithms—is sufficient to generate highly accurate predictions! That a rough, architecture-agnostic measure works so well is strong empirical evidence for our core claims. This robustness suggests that the underlying loss-complexity dynamic is fundamental and not overly sensitive to the specific proxy used. \n\n[1] arxiv.org/abs/1805.08522 \n\n[2] www.nature.com/articles/s41467-024-54813-x\n\n[3] arxiv.org/abs/2308.12108\n\n-----\n\n> **(W3) & (Q3) The prediction of sub-linear growth is weak...it is possible that you have observed a sub-linear growth, and then decided on the modeling assumption A1**\n\nThanks for raising this concern! We should have been more specific when giving the title for the prediction, and we apologize for the lack of clarity. As discussed in the caption for figure 6 and the beginning of page 8, our framework not only predicts a sub-linear growth of relative distance with $N$, but also that the relative distance between M and G follows a **sigmoidal trend with respect to $N^{1-\\alpha}$**. We can assure the reviewer that our sigmoidal growth prediction came after the modeling assumptions—in fact, before making an assumption of power-law scaling, we could not have plotted the relative distance with respect to  $N^{1-\\alpha}$! Consequently, our claim that the relative distance between M and G will follow a **sigmoidal trend with respect to $N^{1-\\alpha}$** was truly a predictive statement, and we will change the title of the section to reflect this.\n\n----\n\n> **(W3) An expectation from a theory is... to quantify the exponent $\\alpha$**\n\nWe emphasize $\\alpha$ is a free parameter related to power law scaling of neural networks’ loss with dataset size. Predicting the precise exponent $\\alpha$ *a priori* is an extremely difficult, open problem in the entire field of scaling laws, and we believe it is beyond the scope of our paper to solve it. \n\n-----\n\n>**(W4) & (Q4) The paper mentions that transient generalization phenomenon as an important observation, but it is well established that long training prevent generalization (hence, early stopping techniques)...What is different in this context...**\n\nWe emphasize there is a crucial distinction between \"transient generalization\" and “overfitting”; the latter motivates early stopping and what we assume the reviewer has in mind whilst asking this question. While the reviewer is correct that standard overfitting occurs from training for a long time, i.e., for several epochs, on a *fixed* dataset, our experiments are conducted in an **online learning regime** where every training batch consists of **fresh, independently sampled sequences**. That is, the learner is not allowed to (or is at least extremely unlikely to) see the same datapoint twice. Therefore, \"overfitting\" in the traditional sense is not applicable in this setting. Instead, what we see is the model transitioning from a simpler, OOD-generalizing solution (G) to a more complex, ID-specialized memorizing solution (M) that achieves lower loss on the training *distribution*. This is a fundamentally different phenomenon, and our framework provides a quantitative explanation for why it occurs. \n\n----\n\n> **(W5) & (Q5) The scaling of time to transience...depends on the complexity of the M and G models, as well as their losses, which, of course, unknown a priori...How could this serve as an estimate for this transience time in new tasks?**\n\nWe believe there may be a slight misunderstanding here and take this opportunity to clarify. Specifically, given that we can write the predictors M and G in a closed-form, we emphasize that, unlike what the reviewer said, we can compute their loss and complexity a priori! In fact, our fitting of the Bayesian model strongly relies on this. More specifically, given sequences sampled from a data-generating distribution of a setting, we can compute the loss of each predictor on these sequences autoregressively, as well as compute their complexity by applying compression algorithms to the code and taking the lowest size in bits. \n\nThis thus offers us all variables necessary to estimate the time to transience ($N^{*}$), yielding a non-trivial prediction that is pre-registered before the corresponding experiment is executed! As we show empirically in **Fig. 6(c)**, the predicted $N^*(D)$ from our theory aligns remarkably well with the observed crossover points from G to M, even as we compare with a training run with over *10 million steps*! \n\n-----\n-----\n\n**Summary:** We sincerely thank the reviewer for their detailed feedback and interesting questions! We hope our responses help address the reviewer’s concerns and, if so, they will consider increasing their score to support the acceptance of our work!"}, {"review_id": "Vnn8gbakaX", "reviewer": "Reviewer_Zg8o", "summary": "This paper provides a framework to explain the effects of in-context learning in Transformers by training a lightweight Bayesian model. They define two components: the memorizer predictor and the generalizer predictor -- these aim to predict the performance across three tasks: (1) balls and urns, (2) linear regression, and (3) binary classification.", "questions": "I might have missed where these details are included, but the experimentation details are quite unclear to me:\n1. Do the authors use a specific Transformer-based model, or just experiment with their own stack of Transformer blocks?\n2. Where is the data from the 3 problem settings obtained from?\n\nAdditionally,\n1. What are the chosen $\\alpha, \\beta, \\gamma$ values learned for the experiments?\n2. Is there an intuitive interpretation to $\\alpha, \\beta,$ and  $\\gamma$?\n\nI'm happy to raise my scores!", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an interesting topic, with a unique approach\n- It is a well-organized paper\n- The mathematical formulation is laid out clearly throughout the paper\n\nWeaknesses:\n- Scope of chosen problems (ball and urns, linear regression, binary classification) are more of toy problems than real, large-scale problem statements.", "was_revised": true, "final_justification": "As I mentioned in the response to the authors, I had misunderstood some aspects of their paper. I also read through the other reviews and responses -- I did not have any additional questions or concerns.", "rebuttal": "We thank the reviewer for their feedback! We are glad that they found our paper “tackles an interesting topic, with a unique approach”, that it is “well-organized”, and that the “mathematical formulation is laid out clearly throughout the paper.” Below, we respond to specific comments.\n\n-----\n\n> **Scope of chosen problems (ball and urns, linear regression, binary classification) are more of toy problems than real, large-scale problem statements.**\n\nFair point! While we agree that our experiments use synthetic tasks, we emphasize that these tasks are *precisely* the ones that have been studied in the bulk of the literature to understand ICL [1-12]. We thus argue that any theory targeted at explaining ICL must at least show predictive power over this spectrum of simple, controlled, but phenomenologically very rich set of tasks that form the focus of prior work. Our hierarchical Bayesian framework meets this standard, offering both a predictive account of Transformer behavior throughout training and explanations for several past results on ICL in a unified manner.\n\nHaving achieved this result, we now strongly think that the stage is set for analyzing more naturalistic settings. We have in fact already begun preliminary explorations on this front, analyzing the phenomenon of many-shot jailbreaking wherein safety fine-tuned models, when offered in-context sufficiently many exemplars of some undesirable behavior (e.g., producing violent speech), start to engage in said behavior. We believe our framework can offer a predictive account of this phenomenon with realistic models, but a detailed analysis warrants its own explicit study!\n\n[1] arxiv.org/abs/2205.05055\n\n[2] arxiv.org/abs/2312.03002\n\n[3] arxiv.org/abs/2412.00104\n\n[4] arxiv.org/abs/2311.08360\n\n[5] www.jmlr.org/papers/v25/23-1042.html\n\n[6] arxiv.org/abs/2310.08391\n\n[7] arxiv.org/abs/2503.05631\n\n[8] arxiv.org/abs/2402.11004\n\n[9] www.pnas.org/doi/abs/10.1073/pnas.2502599122\n\n[10] arxiv.org/abs/2211.15661\n\n[11] arxiv.org/abs/2306.15063\n\n[12] arxiv.org/abs/2412.01003\n\n[13] www.anthropic.com/research/many-shot-jailbreaking \n\n-----\n-----\n\n**Questions about experimental details**\n\n> **Do the authors use a specific Transformer-based model, or just experiment with their own stack of Transformer blocks?**\n\nWe refer the reviewer to experimental details mentioned on page 30. In brief, we note that for all settings, we use the GPT-NeoX architecture sourced from Huggingface [70]. While the number of layers / blocks in the model depends on the specific experimental setting (as reported in that section), we use only 1 attention head per layer and follow a sequential residual stream architecture across all settings. In the Balls and Urns and Classification settings, we use one layer. In the Linear Regression setting, we use eight layers to ensure the model can express the ridge regression solution.  \n\n\n> **Where is the data from the 3 problem settings obtained from?**\n\nThank you for the question that offers us a chance to clarify our experimental setup! As we mentioned on pages 3-4 in the main text and detailed much more extensively on pages 32-34 in the appendix, we use an online learning procedure for sampling data from the training distribution. From page 3: “During training, each input sequence of length $C$ is generated by randomly sampling a function $f(w, \\cdot) \\in \\mathcal{T}_{\\text{train}}$” and “Batches consist of independently generated sequences.”  The appendix offers several more details, with visualizations clarifying the sampling protocol in further detail.  \n\n\n-----\n\n#### **Questions regarding free parameters**\n\nThank you for the questions regarding the free parameters, which are an important part of our model!\n \n\n> **What are the chosen $\\alpha$, $\\beta$, $\\gamma$ values learned for the experiments?**\n\nWe believe there may be a slight misunderstanding here and take this opportunity to clarify the same. In particular, we note that the free parameters of our framework, i.e., $\\alpha$, $\\beta$, $\\gamma$, are fitted to a trained model’s next-token predictions. Specifically, we minimize the cross entropy between our predictor (Equation 3) and the trained Transformer’s next-token predictions, yielding the values of free parameters. We strongly emphasize that these same three free parameters show almost perfect generalization to next-token predictions of an order of 100 experimental settings, while also capturing the empirical relative distance maps from Figure 3! That is, our hierarchical Bayesian model offers predictive power at the level of trained Transformer’s phenomenology, not just its behavior (i.e., next-token predictions). \nIn case the reviewer finds it useful, we are also happy to add a detailed table describing the precise values of the three free parameters to the paper’s appendix!\n\n> **Is there an intuitive interpretation to $\\alpha$, $\\beta$, and $\\gamma$?**\n\nIndeed! We provide a brief summary next.\n\n- **Rate of Evidence Accumulation ($\\alpha$):** This adjusts the power on the effective sample size $N^{1-\\alpha}$ and can hence be seen as modulating the sample efficiency of the Transformer. The term emerges from the scaling exponent involved in power-law scaling of neural network loss with dataset size [1], which we assume to be an axiomatic property of neural network training for deriving our framework (see page 27 for the derivation). In the language of Bayesian inference, one can also deem $\\alpha$ as a measure of the rate at which a Transformer accumulates evidence from the data seen during training. As we discuss on page 8, learned values of $\\alpha$ turn out to be less than 1, indicating sublinear sample efficiency of Transformers compared to an optimal Bayesian learner.\n\n\n- **Complexity Penalty ($\\beta$):** This parameter captures the extent to which a Transformer has an inductive bias towards implementing simpler predictors. If we had an objective measure of complexity of implementing a predictor in a Transformer, we would not need to model this free parameter. However, given that that is not the case, we use (an upper bound for) the Kolmogorov complexity of the predictor by compressing its python implementation via off-the-shelf compression algorithms. This leads to the definition of $\\beta$: it captures how much more costly implementation of the predictor would be in a Transformer, a suboptimal computational system, compared to a universal Turing machine! Specifically, given that $\\beta$ adjusts the penalizing of complexity by the prior for a predictor, we can thus interpret it as a ‘complexity penalty’ on page 9, and show that our model accommodates the increase in memorization with expanding MLP width via a decrease in the ‘complexity penalty’ $\\beta$. \n\n\n- **Effects of initial loss ($\\gamma$):** On page 7, we note that $\\gamma$ is a constant related to the term A from the scaling laws assumption. More specifically, in the derivation on page 27, it can be seen that the free parameter $\\gamma$ encapsulates $\\frac{A}{L(Q)}$, with $A$ corresponding to the loss of a random predictor on the dataset, and $L(Q)$ being the loss of the optimal Bayesian predictor (irreducible error). We only fit a single free parameter $\\gamma$ across different task diversity settings in order to simplify our resulting functional form. \n\n[1] arxiv.org/abs/2102.06701\n\n-----\n-----\n\n**Summary:** We thank the reviewer for their helpful comments that helped us demonstrate the generality of our framework and contextualize the intuitive interpretation of variables involved within it. We hope our responses have appropriately addressed the reviewer’s concerns and, if so, hope that they will consider raising their score to support the acceptance of our work!"}, {"review_id": "ZZ7JVEALL0", "reviewer": "Reviewer_PqQ7", "summary": "The paper studies the generalization behaviour of in-context learning to new tasks given a fixed number of training tasks and training steps. They propose a rigorous theoretical model of in-context learning as Bayesian inference over two solutions—memorization and generalization—with a prior for low-complexity hypotheses. They show that their model accounts for fine-grained empirical data, reproducing the precise tradeoff points where memorization transitions to generalization and vice-versa.", "questions": "1. It is possible that the thing that is important for characterizing the generalization behaviour of the model is not just the task diversity (i.e., the number of training tasks seen), but rather **the complexity of the task distribution**. This shifts the focus from talking about the “true” vs. “empirical” task distribution to talking about “simple” vs. “complex” task distributions, and a sequence model’s tendency to prefer simple ones. Let me explain. The empirical distributions in the regime of few training tasks is simple (a model just needs to memorize a handful of tasks), and in the regime of many training tasks that distribution is complex (a model would need to memorize far more tasks). Being a neural network, the sequence model prefers a simpler prior over tasks in the later case, which happens to coincide with what you’ve called the “true” task distribution. But what if you actually make the “true” task distribution complex! For instance, you can sample $N$ possible task parameters from a distribution $T_{simple}$(in the same way that you currently define $T_{true}$) and then convolve them with a Gaussian, so that you have a very complex multi-modal “true” task distribution $T_{true}$. I bet that even if you sampled $|T_{train}| \\gg N$ training tasks from this “true” distribution, the model would **not** learn a prior that corresponds to $T_{true}$, but would instead learn a prior that corresponds to $T_{simple}$. Again, this shifts the focus from “true” vs. “empirical” or “generalization” vs. “memorization” to now being something about “simple” vs. “complex” task distributions, which need not be defined w.r.t. some true underlying distribution. My questions are:\n    1. Do you agree with this framing and the hypothesized finding?\n    2. Can you run this experiment and present the results in at least an Appendix section, along with a discussion of their implications for how one should precisely interpret your theory and results?\n2. You write your universal prior $p(Q) = 2^{-K(Q)^\\beta}$ in a way that suggests it is a valid distribution, but it is not. The universal prior is a semi-measure, not a valid measure. Did you normalize this distribution such that $p(G) + p(M) = 1$, in which case you probably meant $p(Q) \\propto 2^{-K(Q)^\\beta}$ instead?\n3. Why do you have to “estimate a loose upper bound” for $K(Q)$ using “compression algorithms to the code and data required for describing the predictor” (L.203)? Don’t you know $K(G) = L(T_{true})$ is the length of the function describing your true distributions $T_{true}$ (which are all easy to figure out, given that you know the code needed to efficiently define something like a categorical or noise distribution), and that $K(M) = L(T_{true}) - \\log_2 p_{T_{true}}(T_{train})$ where the second term comes from Huffman encoding in a 2-part code? Clearly for the way you’ve defined $T_{train}$, a 2-part code using $T_{true}$ as the model would be the optimal strategy, and even if you misestimate $K(T_{true})$ by a couple bits here and there you still have a free parameter $\\beta$ in your universal prior that can correct for this.\n    1. Note that if you don’t trust that this is the best compressor for both distributions, you can simply check that it gives a lower bound that what you currently have with the other compressors you’ve used, since the codes above are definitely valid ways to compress the two hypotheses.\n4. Equation 2 and L.216, you expand $\\log [P(M) / P(G)] \\implies -\\Delta K(D)^\\beta$, and then you define $\\Delta K(D)^\\beta = \\log 2(K_M(D)^\\beta - K_G(D)^\\beta)$. I’m a bit lost on several fronts here:\n    1. What is $\\log2$? I’m assuming it has to be $\\log_2$ based on everything else, but you’ve written it in a way that means $\\log(2)$. For subsequent follow-up questions right below, I’m going to assume you meant $\\log_2$.\n    2. What is $K_M(D)$ or $K_G(D)$? You haven’t defined these terms anywhere. Did you mean $K(M)$ and $K(D)$?\n    3. I think the derivation is wrong. It should be $\\log_2 [P(M) / P(G)] = \\log_2[2^{-K(M)^\\beta}] - \\log_2[2^{-K(G)^\\beta}] = K(G)^\\beta -K(M)^\\beta$, and so you should be defining $\\Delta K(D)^\\beta = K(M)^\\beta -K(G)^\\beta$ for Eq. 2 to be valid. Am I missing something? Why do you have the $\\log_2$ in your definition for $\\Delta K(D)^\\beta$?\n    4. You using a notation for the complexity term $\\Delta K(D)^\\beta$ that has $D$ in it, but this is initially confusing because $K(M)$ and $K(G)$ are not denoted in terms of $D$. The trick is that $M$ depends on $D$ implicitly because it has to memorize it, but $M$ doesn’t depend on $D$ (please correct me if I’m wrong). To clarify things, can you switch to denoting $M \\rightarrow M_D$ and correspondingly $K(M) \\rightarrow K(M_D)$?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The problem studied is important and remains unsolved in the field. While I think many in the field already had intuitions that were aligned with the theory presented in this work, it lays those intuitions out quite rigorously.\n2. The proposed model fits the empirical data extremely well, which inspires great confidence in the authors’ theory.\n3. The paper is largely clearly written and pedagogical (although see my comments below on parts of Section 4). \n4. The figures are excellent at illustrating the hypotheses, methodology, and results. The figures and captions together largely tell a self-contained story, and somebody could get a good understanding of this work just by reading them alone, which is always nice.\n5. The future work suggested is thought-out and genuinely interesting.\n6. Overall, my impression of this paper is quite positive. I have to admit that I went into it feeling like many of the statements being made early on in the abstract and introduction were quite obvious and well-known, but the precision and rigour of the proposed model is very nice to see and I feel like I now have a better understanding of ICL.\n\n**Weaknesses**\n\n1. In line with my comment just above, I think the biggest weakness is one of perception, especially early on in the paper. The problem is set up on L.30-35 in a way that is not specific to ICL. Questions like “why does an ability to learn novel tasks emerge, given that memorizing the training distribution almost always leads to better performance” apply to *all* of deep learning as a whole, not just ICL. If one just thinks of the sequence model as a neural network taking sequence-structure inputs (and it is that!), then many of the stated questions simply reduce to “why do deep neural networks generalize?”, and this is a question that has been asked many times before. In fact, the answers that have been proposed look very much like the one here, in that they show deep neural network training can be seen as Bayesian inference where there is a prior for simple solutions (Wilson 2025 “Deep Learning is Not So Mysterious or Different”, for instance). Again by the time I got to the end of the paper it was clear that it differs from the general deep learning question (e.g., you consider memorization of the sequence *generative parameters*, rather than memorization of the *sequences themselves*, hence your ID evaluation) and that the proposed model does have something interesting to offer, but I think you should make some effort to rework the abstract and introduction in order to head off these kinds of sceptical reactions. How to best do that, I’m not sure unfortunately.\n2. Section 4.0 needs some cleaning up, both in terms of math correctness (I think) and notation. It can also be clarified, especially near the end of Section 4.0 where it is very definition-heavy and readers might get lost in notation that was defined earlier on. I had to scramble around to find what terms like $L_G(S_{\\mathcal{T}_{train}}(\\infty, D))$ referred to, for instance.\n3. Fig. 5 should appear before Fig. 4, as it gives intuition for why we see the patterns in Fig. 4. Similarly, the contents of Section 4.1 should be described before the paragraph on “Validating the Model”, as it is there to provide intuition for the model.\n4. I would rename and reframe Section 4.2 from “novel predictions” to “features explained by the model”. While I can technically see how these are “novel predictions”, really you’re not using your theory to predict the results of new experiments, but rather are using it to explain why the empirical data you’ve already presented looks the way it does. Your model is explaining observed data, not making new predictions for novel experiments. You can then consolidate the paragraph on “Validating the Model” into Section 4.2 Features Explained by the Model (see my point directly above).", "was_revised": true, "final_justification": "The authors have addressed my main concerns, and I maintain that this is a strong paper on an important topic, deserving acceptance so that it has better visibility in the ICL community.", "rebuttal": "We are deeply grateful to the reviewer for their thorough and insightful review. We thank them for supporting our work for acceptance, and were especially encouraged that they highlighted the “precision and rigour of the proposed model” and found that the “proposed model fits the empirical data extremely well, which inspires great confidence.” We were also happy to hear that the reviewer found our figures to be “excellent at illustrating the hypotheses, methodology, and results” and that our paper helped them develop “a better understanding of ICL.”\n\nThe feedback was particularly valuable to us because the suggestions for improving the paper’s narrative and clarity align with and validate motivations behind a significant revision we recently prepared. Before addressing specific points, we wanted to briefly outline the narrative of the revised manuscript, as we believe it addresses some high-level suggestions.\n* Recent work analyzing ICL has identified a broad set of strategies that describe model behavior in different experimental conditions. We aim to unify these findings by asking *why* a model learns these disparate strategies in the first place as training conditions are varied. \n* We observe that when trained to learn a mixture of tasks, models transition between two Bayesian predictors, a memorizing predictor and a generalizing predictor. \n* This observation leads us to adopt a normative Bayesian account for modeling Transformer behavior. Under assumptions of simplicity bias and power-law scaling, our account is highly predictive of model behavior and yields an explanatory account of a loss-complexity tradeoff as core to ICL phenomena.\n* This narrative, which focuses on why distinct strategies are learned and transitions between them occur, allows us to focus on ICL phenomena as outcomes of a single, underlying loss-complexity dynamic. We believe this revised framing makes the paper's contribution clearer.\n\nWith that context, we now address specific comments and questions in detail.\n\n-----\n\n>  **I think the biggest weakness is one of perception, especially early on in the paper… Again by the time I got to the end of the paper it was clear that it differs from the general deep learning question … and that the proposed model does have something interesting to offer, but I think you should make some effort to rework the abstract and introduction in order to head off these kinds of sceptical reactions.**\n\nWe thank the reviewer for the valuable and constructive feedback! We agree with the general comments made, which is why, as stated above, we reworked the abstract and introduction to focus more specifically on ICL from the start, framing our investigation around the question of why models learn distinct strategies for performing ICL.  Yet, we note that we do in fact agree with the reviewer’s observation that our account can apply, as phrased in the current intro, to neural networks in general! In this paper we indeed focus on ICL, and so the hypothesis space consists of different predictors; however, in a more standard neural network case, the hypothesis space can consist of specific weights (e.g., in a mixture of linear regression tasks), and we are actively working on demonstrating our account can predict general neural network training dynamics and scaling phenomena! With that said, this paper indeed focuses on ICL, hence we changed the intro accordingly. Yet, we do want to highlight that, as opposed to works such as Wilson’s “Deep Learning is Not So Mysterious or Different”, we believe the strength of our account is that beyond offering intuitions or a descriptive lens, it is also highly predictive of model behavior throughout training, which we see as crucial for a successful account of deep learning.\n\n------\n\n> **Section 4.0 needs some cleaning up, both in terms of math correctness (I think) and notation. It can also be clarified…** \n\nWe thank the reviewer for the constructive comments, and we certainly agree! In our rewrite, we revised the notation and attempted to improve the clarity of section 4. For example, we define $\\overline{L_G}(S_{\\mathcal{T}_{train}}(D))$\n\nas the average loss of a predictor G on a dataset of sequences sampled from $\\mathcal{T}_{\\text{train}}$, \n\nand maintain this notation in the derivation, thereby avoiding confusing terminology such as $L_G(S_{\\mathcal{T}_{\\text{train}}}(\\infty, D))$. \n\nWe also revised several other elements in section 4, which we detail in the sections relevant to the reviewer’s specific questions.\n\n------\n\n> **Fig. 5 should appear before Fig. 4…**\n\n We agree and will revise accordingly!\n\n------\n\n> **It is possible that the thing that is important...is not just the task diversity...but rather the complexity of the task distribution...Do you agree with this framing and the hypothesized finding? Can you run this experiment...?**\n\nThis is a brilliant question, and we agree with your framing! The core axis is indeed the complexity of implementing each solution, which depends on the complexity of memorizing the generative parameters of each distribution. Indeed, in works such as [1], it is shown that more complex data can indeed drive learning of better generalizing predictors in the case where the true distribution (in the case of the paper, the hierarchical structure of language) is indeed complex. Your proposed experiment—making the \"true\" distribution more complex than the empirical one—is an excellent way to disentangle these concepts.\n\nWe have actually run experiments in a similar spirit using a different task domain (learning Boolean functions). Preliminary results support your hypothesis: when tasks comprising the “true” distribution cannot be compressed into a simple generative process, we do not see the standard task diversity effect, as memorizing only the tasks in the training data is indeed simpler than learning the generalizing solution! In contrast, when the true distribution comes from a single “true” input-output map, which is “noised” to create slightly different maps for the training distribution, the Transformer does indeed transition to learning the true map with increased task diversity! This aligns with your idea that the model's inductive bias is for *simple generative processes*, not necessarily the \"true\" one. We will add a discussion of such findings and their implications to the appendix in the final version. Thank you for the thought-provoking suggestion.\n[1] https://arxiv.org/abs/2412.04619\n\n------\n\n> **You write your universal prior …  in a way that suggests it is a valid distribution, but it is not.**\n\nYou are absolutely correct. Thank you for this important technical correction. We have corrected the prior to be written as $p(Q) \\propto 2^{-K(Q)^{\\beta}}$, and we indeed meant that to be the case in our paper, but wrote it using an equality sign by mistake.\n\n------\n\n> **Why do you have to “estimate a loose upper bound” for K(Q) using “compression algorithms\"...? Don’t you know K(M) is the length of the function describing your true distributions...and that K(G) \\approx D \\times K(w_d)...?**\n\nThis is an excellent and deeply insightful question. Our reasoning for using general-purpose compressors is subtle but important. The complexity term $K(Q)$ in our model is meant to capture the cost of a Transformer *implementing the predictor algorithm Q*. For example: \n* For the **Generalizing predictor (G)** in Balls & Urns, the algorithm is \"count token occurrences in the context and add a pseudo-count\".\n* For the **Memorizing predictor (M)**, the algorithm is \"compute the likelihood of the context under each of the D stored urns and compute a weighted average\".\n\nThe 2-part code you describe is an optimal way to compress the *data* (the urn parameters $w_d$). However, it doesn't directly capture the algorithmic complexity of *implementing the inference procedure itself* (e.g., computing a histogram or Bayesian averaging). Our approach, by compressing the Python code that implements the full predictor, is an attempt to create a proxy for this algorithmic implementation cost.\n\nThat said, your suggestion is very compelling. We suspect it might work well for settings like Balls & Urns, in which computational procedures are shared: both strategies require computing a histogram, but in the case of the memorizing solution, this histogram is compared with learned distributions, whereas the generalizing solution simply adds a 1 pseudo-count. However, we expect it might be less complete for settings like Linear Regression, where the complexity seems to lie more in implementing the ridge regression algorithm itself, compared to Bayesian averaging. We would be very interested in any further thoughts you have on this.\n\n------\n\n> **Equation 2 and L.216, you expand…**\n\nWe are grateful for your meticulous reading of our derivations. You were right to be confused—our notation in the original draft was unclear and contained typos. We sincerely apologize for this. As mentioned in our opening, we have thoroughly revised Section 4 and Appendix D to address every point you raised:\n \n1. $\\log2 (K_M(D)^{\\beta} - K_G(D)^{\\beta})$ should be written as $(K_M(D)^{\\beta} - K_G(D)^{\\beta}) \\cdot \\log2$ to avoid confusion.\n2. We indeed meant $K(M)$ and $K(D)$ and will adopt your notational suggestion from part 4. to prevent confusion.\n3. We include the $\\log2$ term since we use the natural log rather than $\\log_2$ in our derivation, therefore a $\\log_{{\\rm e}}2$ constant appears in the term for $\\Delta K(D)^{\\beta}$. Also, we define $\\Delta K(D)^{\\beta} = (K_M(D)^{\\beta} - K_G(D)^{\\beta}) \\cdot \\log2$ rather than $\\Delta K(D)^{\\beta} = (K_G(D)^{\\beta} - K_M(D)^{\\beta}) \\cdot \\log2$ to highlight the tradeoff between the loss term and the complexity term in the equation for $\\eta$, since defining $\\Delta K(D)^{\\beta}$ in former way yields a minus sign between the two terms in the equation for $\\eta$, whereas defining it the latter yields a plus sign between the terms."}]

Author rebuttal:

Meta-review: This paper develops a hierarchical Bayesian framework that unifies two well-known ICL phenomena—task diversity effects and transient generalization—into a single predictive model based on a loss–complexity tradeoff. Unlike prior studies focusing on one factor, this work jointly models the roles of task diversity and training time, offering a coherent explanation and generating novel predictions. The empirical validation is thorough and shows strong alignment between the proposed model and Transformer behavior. While some predictions align with existing intuitions, the unified and predictive nature of the framework, along with extensive experiments, makes this a valuable contribution.
Decision: Accept (poster)