Title: In-Context Learning Strategies Emerge Rationally
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-Context Learning

Loss-Complexity tradeoff

Bayesian Modeling

Algorithmic Complexity
Abstract: Recent work analyzing in-context learning (ICL) has identified a broad set of
strategies that describe model behavior in different experimental conditions. We
aim to unify these findings by asking why a model learns these disparate strategies in
the first place. Specifically, we start with the observation that when trained to learn a
mixture of tasks, as is popular in the literature, the strategies learned by a model for
performing ICL can be captured by a family of Bayesian predictors: a memorizing
predictor, which assumes a discrete prior on the set of seen tasks, and a generalizing
predictor, where the prior matches the underlying task distribution. Adopting the
normative lens of rational analysis, where a learner’s behavior is explained as
an optimal adaptation to data given computational constraints, we develop a
hierarchical Bayesian framework that almost perfectly predicts Transformer next-
token predictions throughout training—without assuming access to its weights.
Under this framework, pretraining is viewed as a process of updating the posterior
probability of different strategies, and inference-time behavior as a posterior-
weighted average over these strategies’ predictions. Our framework draws on
common assumptions about neural network learning dynamics, which make explicit
a tradeoff between loss and complexity among candidate strategies: beyond how
well it explains the data, a model’s preference towards implementing a strategy
is dictated by its complexity. This helps explain well-known ICL phenomena,
while offering novel predictions: e.g., we show a superlinear trend in the timescale
for transitioning from generalization to memorization as task diversity increases.
Overall, our work advances an explanatory and predictive account of ICL grounded
in tradeoffs between strategy loss and complexity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the generalization behaviour of in-context learning to new tasks given a fixed number of training tasks and training steps. They propose a rigorous theoretical model of in-context learning as Bayesian inference over two solutions—memorization and generalization—with a prior for low-complexity hypotheses. They show that their model accounts for fine-grained empirical data, reproducing the precise tradeoff points where memorization transitions to generalization and vice-versa.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The problem studied is important and remains unsolved in the field. While I think many in the field already had intuitions that were aligned with the theory presented in this work, it lays those intuitions out quite rigorously.\n2. The proposed model fits the empirical data extremely well, which inspires great confidence in the authors’ theory.\n3. The paper is largely clearly written and pedagogical (although see my comments below on parts of Section 4). \n4. The figures are excellent at illustrating the hypotheses, methodology, and results. The figures and captions together largely tell a self-contained story, and somebody could get a good understanding of this work just by reading them alone, which is always nice.\n5. The future work suggested is thought-out and genuinely interesting.\n6. Overall, my impression of this paper is quite positive. I have to admit that I went into it feeling like many of the statements being made early on in the abstract and introduction were quite obvious and well-known, but the precision and rigour of the proposed model is very nice to see and I feel like I now have a better understanding of ICL.\n\n**Weaknesses**\n\n1. In line with my comment just above, I think the biggest weakness is one of perception, especially early on in the paper. The problem is set up on L.30-35 in a way that is not specific to ICL. Questions like “why does an ability to learn novel tasks emerge, given that memorizing the training distribution almost always leads to better performance” apply to *all* of deep learning as a whole, not just ICL. If one just thinks of the sequence model as a neural network taking sequence-structure inputs (and it is that!), then many of the stated questions simply reduce to “why do deep neural networks generalize?”, and this is a question that has been asked many times before. In fact, the answers that have been proposed look very much like the one here, in that they show deep neural network training can be seen as Bayesian inference where there is a prior for simple solutions (Wilson 2025 “Deep Learning is Not So Mysterious or Different”, for instance). Again by the time I got to the end of the paper it was clear that it differs from the general deep learning question (e.g., you consider memorization of the sequence *generative parameters*, rather than memorization of the *sequences themselves*, hence your ID evaluation) and that the proposed model does have something interesting to offer, but I think you should make some effort to rework the abstract and introduction in order to head off these kinds of sceptical reactions. How to best do that, I’m not sure unfortunately.\n2. Section 4.0 needs some cleaning up, both in terms of math correctness (I think) and notation. It can also be clarified, especially near the end of Section 4.0 where it is very definition-heavy and readers might get lost in notation that was defined earlier on. I had to scramble around to find what terms like $L_G(S_{\\mathcal{T}_{train}}(\\infty, D))$ referred to, for instance.\n3. Fig. 5 should appear before Fig. 4, as it gives intuition for why we see the patterns in Fig. 4. Similarly, the contents of Section 4.1 should be described before the paragraph on “Validating the Model”, as it is there to provide intuition for the model.\n4. I would rename and reframe Section 4.2 from “novel predictions” to “features explained by the model”. While I can technically see how these are “novel predictions”, really you’re not using your theory to predict the results of new experiments, but rather are using it to explain why the empirical data you’ve already presented looks the way it does. Your model is explaining observed data, not making new predictions for novel experiments. You can then consolidate the paragraph on “Validating the Model” into Section 4.2 Features Explained by the Model (see my point directly above).\n\nQUESTIONS:\n1. It is possible that the thing that is important for characterizing the generalization behaviour of the model is not just the task diversity (i.e., the number of training tasks seen), but rather **the complexity of the task distribution**. This shifts the focus from talking about the “true” vs. “empirical” task distribution to talking about “simple” vs. “complex” task distributions, and a sequence model’s tendency to prefer simple ones. Let me explain. The empirical distributions in the regime of few training tasks is simple (a model just needs to memorize a handful of tasks), and in the regime of many training tasks that distribution is complex (a model would need to memorize far more tasks). Being a neural network, the sequence model prefers a simpler prior over tasks in the later case, which happens to coincide with what you’ve called the “true” task distribution. But what if you actually make the “true” task distribution complex! For instance, you can sample $N$ possible task parameters from a distribution $T_{simple}$(in the same way that you currently define $T_{true}$) and then convolve them with a Gaussian, so that you have a very complex multi-modal “true” task distribution $T_{true}$. I bet that even if you sampled $|T_{train}| \\gg N$ training tasks from this “true” distribution, the model would **not** learn a prior that corresponds to $T_{true}$, but would instead learn a prior that corresponds to $T_{simple}$. Again, this shifts the focus from “true” vs. “empirical” or “generalization” vs. “memorization” to now being something about “simple” vs. “complex” task distributions, which need not be defined w.r.t. some true underlying distribution. My questions are:\n    1. Do you agree with this framing and the hypothesized finding?\n    2. Can you run this experiment and present the results in at least an Appendix section, along with a discussion of their implications for how one should precisely interpret your theory and results?\n2. You write your universal prior $p(Q) = 2^{-K(Q)^\\beta}$ in a way that suggests it is a valid distribution, but it is not. The universal prior is a semi-measure, not a valid measure. Did you normalize this distribution such that $p(G) + p(M) = 1$, in which case you probably meant $p(Q) \\propto 2^{-K(Q)^\\beta}$ instead?\n3. Why do you have to “estimate a loose upper bound” for $K(Q)$ using “compression algorithms to the code and data required for describing the predictor” (L.203)? Don’t you know $K(G) = L(T_{true})$ is the length of the function describing your true distributions $T_{true}$ (which are all easy to figure out, given that you know the code needed to efficiently define something like a categorical or noise distribution), and that $K(M) = L(T_{true}) - \\log_2 p_{T_{true}}(T_{train})$ where the second term comes from Huffman encoding in a 2-part code? Clearly for the way you’ve defined $T_{train}$, a 2-part code using $T_{true}$ as the model would be the optimal strategy, and even if you misestimate $K(T_{true})$ by a couple bits here and there you still have a free parameter $\\beta$ in your universal prior that can correct for this.\n    1. Note that if you don’t trust that this is the best compressor for both distributions, you can simply check that it gives a lower bound that what you currently have with the other compressors you’ve used, since the codes above are definitely valid ways to compress the two hypotheses.\n4. Equation 2 and L.216, you expand $\\log [P(M) / P(G)] \\implies -\\Delta K(D)^\\beta$, and then you define $\\Delta K(D)^\\beta = \\log 2(K_M(D)^\\beta - K_G(D)^\\beta)$. I’m a bit lost on several fronts here:\n    1. What is $\\log2$? I’m assuming it has to be $\\log_2$ based on everything else, but you’ve written it in a way that means $\\log(2)$. For subsequent follow-up questions right below, I’m going to assume you meant $\\log_2$.\n    2. What is $K_M(D)$ or $K_G(D)$? You haven’t defined these terms anywhere. Did you mean $K(M)$ and $K(D)$?\n    3. I think the derivation is wrong. It should be $\\log_2 [P(M) / P(G)] = \\log_2[2^{-K(M)^\\beta}] - \\log_2[2^{-K(G)^\\beta}] = K(G)^\\beta -K(M)^\\beta$, and so you should be defining $\\Delta K(D)^\\beta = K(M)^\\beta -K(G)^\\beta$ for Eq. 2 to be valid. Am I missing something? Why do you have the $\\log_2$ in your definition for $\\Delta K(D)^\\beta$?\n    4. You using a notation for the complexity term $\\Delta K(D)^\\beta$ that has $D$ in it, but this is initially confusing because $K(M)$ and $K(G)$ are not denoted in terms of $D$. The trick is that $M$ depends on $D$ implicitly because it has to memorize it, but $M$ doesn’t depend on $D$ (please correct me if I’m wrong). To clarify things, can you switch to denoting $M \\rightarrow M_D$ and correspondingly $K(M) \\rightarrow K(M_D)$?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper provides a framework to explain the effects of in-context learning in Transformers by training a lightweight Bayesian model. They define two components: the memorizer predictor and the generalizer predictor -- these aim to predict the performance across three tasks: (1) balls and urns, (2) linear regression, and (3) binary classification.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper tackles an interesting topic, with a unique approach
- It is a well-organized paper
- The mathematical formulation is laid out clearly throughout the paper

Weaknesses:
- Scope of chosen problems (ball and urns, linear regression, binary classification) are more of toy problems than real, large-scale problem statements.

QUESTIONS:
I might have missed where these details are included, but the experimentation details are quite unclear to me:
1. Do the authors use a specific Transformer-based model, or just experiment with their own stack of Transformer blocks?
2. Where is the data from the 3 problem settings obtained from?

Additionally,
1. What are the chosen $\alpha, \beta, \gamma$ values learned for the experiments?
2. Is there an intuitive interpretation to $\alpha, \beta,$ and  $\gamma$?

I'm happy to raise my scores!

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a methodology-experimental paper, which explores in-context learning in a general stylized model, and 3 different specific settings. The paper defines memorization (M) (resp. generalization, G) as the case in which the test prediction is based on Bayesian weightening of the predictors according to their empirical appearance in the training set (resp. in the population distribution). The paper empirically validates that two main phenomena occur for transformer-based training in all 3 settings: (1) transition from memorization to generalization when the training task diversity increases. (2) A deterioration of the test (out-of-distribution) performance when the length of the training increases. Then, it is proposed to model the predictor as a Bayesian mixture of a memorizing predictor and a generalizing predictor. The log-likelihood of the posterior odds is parameterized as a sum of a loss term and a complexity term. The model is used to predict the distance to M and G predictors as the function of the number of samples, that the change between them is rapid, and that it increases with task diversity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n[S1] The paper addresses a timely and central topic: In-context learning is one of the most important aspects of the recent advances in ML, and the question on under what circumstances it arises is important. The step towards unification of various models proposed and phenomena observed in the literature is useful. \n\n[S2] The conclusion is based on extensive empirical evidence, and the graphs displayed are convincing. \n\n[S3] Overall, the paper is well written and easy to follow. \n\n**Weaknesses:** \n\n[W1] The main weakness is that, from my perspective, the paper is fully derived from the empirical observation of the transitions from memorization (M) to generalization (G). This was already observed in previous works [23,27, maybe more]. From that point onward, the paper reads like a **tautology**. Specifically: (1) It is obvious that if the empirical predictor is very close to either the M or G predictor, and sharply transitions between them, then the only the posterior weights of these predictors will be close to 1, and otherwise tend to zero. (2) It is obvious that if the prior is modeled by a complexity term and the likelihood by a loss term, then the posterior would be comprised of them, and would appear as a loss-complexity trade-off. The paper aims to answer why ICL emerges, but I think that instead it just fits a model to the observed transition between M and G. In other words, the paper observes a transition between M and G, then models the prior as complexity terms, and then explains that the transition is due to balance between complexity and loss. If the prior was modeled in a different way, by a term of different nature, then the “why” explanation would have changed accordingly. \n\n[W2] In continuation to the above, I believe it is agreed that the prior is some sort of **a** complexity term. The weakness of the paper is that it does not address the main question of **which** complexity term? The paper mentions Kolmogorov complexity, but it is well known that this universal complexity term is uncomputable. Any compression algorithm may indeed lead to a measure of complexity, but again, the questions is which complexity term characterizes the ICL phenomena.\n\n[W3] The prediction of sub-linear growth is weak in the sense that a reader cannot judge the order of events. It is possible that you have observed a sub-linear growth, and then decided on the modeling assumption A1. An expectation from a theory is not just to claim for a sub-liner growth, but rather to quantify the exponent $\\alpha$ in some way.\n\n[W4] The paper mentions that transient generalization phenomenon as an important observation, but it is well established that long training prevent generalization (hence, early stopping techniques). \n\n[W5] The scaling of time to transience with task-diversity $N^*(D)$ depends on the complexity of the M and G models, as well as their losses, which, of course, unknown a priori.\n\nQUESTIONS:\nIn parallel to the weaknesses above:\n\n[Q1] What is an evident for the uniqueness of your explanation in the space of various possible explanations? How does the unified view proposed in this paper contributes to the validity of your explanation? \n\n[Q2] What is an empirical evident that a *specific* complexity term is the one to accurately predict the experimental results?\n\n[Q3] What makes the sub-linear growth a prediction, and what can be said on its exponent?\n\n[Q4] What is different in this context from the well-established phenomenon that long training prevents generalization?\n\n[Q5] How could this serve as an estimate for this transience time in new tasks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces a unified Bayesian framework to study the transition of transformers between memorization and generalization in In-Context Learning. The authors model the phenomena of ICL in Transformers as a Bayesian model comparison process.

STRENGTHS AND WEAKNESSES:
Pros:
1. The paper show the tradeoff between memorizing predictors and generalizing predictors as a function of task diversity and training time.
2. The authors conduct experimental exploration under three different settings, including sequence modeling, linear regression and classification. 
3. The paper uses an interpretable Bayesian framework to explain several In-Context Learning phenomena.

Cons:
1. It is advised to add a computational cost analysis of Bayesian model fitting process.
2. The generalizability of simplified assumptions with two predictors to complex scenarios remain uncertain.
3. The experiments are conducted in synthetic tasks with well defined task boundaries. The applicability of the framework to real cases with real-world settings is a concern.

QUESTIONS:
How would the framework fit to tasks with more than the two relevant predictors?

LIMITATIONS:
The dependence on the assumptions of simplicity bias and power law scaling may not hold in all cases.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02