Toward Understanding In-context vs. In-weight Learning
Authors: Bryan Chan
Xinyi Chen
András György
Dale Schuurmans
Venue: ICLR 2025 Poster (2025)
Keywords: In-context learning
generalization error
transformers
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a theoretical explanation on when a model will choose in-context learning vs in-weight learning and support the explanation with experiments.

Abstract: It has recently been demonstrated empirically that in-context learning emerges in transformers when certain distributional properties are present in the training data, but this ability can also diminish upon further training. We provide a new theoretical understanding of these phenomena by identifying simplified distributional properties that give rise to the emergence and eventual disappearance of in-context learning. We do so by first analyzing a simplified model that uses a gating mechanism to choose between an in-weight and an in-context predictor. Through a combination of a generalization error and regret analysis we identify conditions where in-context and in-weight learning emerge. These theoretical findings are then corroborated experimentally by comparing the behaviour of a full transformer on the simplified distributions to that of the stylized model, demonstrating aligned results. We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.

Published as a conference paper at ICLR 2025
TOWARD UNDERSTANDING IN-CONTEXT VS. IN- WEIGHT LEARNING
Bryan Chan Xinyi Chen²*, András György², Dale Schuurmans University of Alberta GGoogle DeepMind bryan. whwanealleeeee...
{xinyic, agyorgy, schuuraasssssooogsss.....
ABSTRACT
It has recently been demonstrated empirically that in-context learning emerges transformers when certain distributional properties are present in the training data, but this ability can also diminish upon further training. We provide a new theo- retical understanding of these phenomena by identifying simplified distributional properties that give rise to the emergence and eventual disappearance of in in-context learning. We do so by first analyzing simplified model that uses gating mech- anism to choose between an in-weight and an in-context predictor. Through a combination of generalization error and regret analysis identify conditions where in-context and in-weight learning emerge. These theoretical findings are then corroborated experimentally by comparing the behaviour of full transformer on the simplified distributions to that of the stylized model, demonstrating aligned results. We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.
INTRODUCTION
In-context learning (ICL) an interesting and useful property exhibited by large language models (LLMs), wherein model is able to successfully learn and generalize from new information given in its input, without ever having seen related data during training. RRafford et al.] (2019) and Brown al. (2020) were among the first demonstrate the in-context learning capability of large-language models. Since then, numerous empirical studies have attempted to understand the emergence of ICL
in LLMs (Olsson et al. 2022 Wei et al. 2023 Agarwal et al. 2024 Shi et al. 2024). However, ICL is not limited to natural language processing (NLP), and many synthetic settings have been Akyürek et al.) 2023 Von 1024:/Zang et al. 2024).
constructed to better understand this phenomenon (Garg al.
2022
Oswald et al. 2023 Gupta et al.
2023
Edelman et al. 2024 Wu et al.
A recent line of work (Chan et al. gence of ICL. through the lens of data-distributional properties. In particular, [Chan et al.) (2022a) first proposed the importance of common and rare classes as driving in-context learning and in- weight learning (IWL). Both Chan et al. (2022a) and Reddy (2024) empirically identified that ICL can emerge transformer model (Vaswani al. 2017) when there is large within-class vari- ation and a large number of classes. [Chan et al. (2022a) b) further observed that both ICL and IWL can emerge simultaneously when distribution over classes Zipfian, while [Singh et al. (023); ;; Reddy (2024); Panwar et al. (2024) subsequently noticed that ICL can become transient in an asymptotic training regime.
2022a Reddy 2024
Singh al.
2023) has explored the emer-
The work in this paper is inspired by a similar distributional perspective, but we investigate in greater depth how the distributional properties of the data affect the ability of a transformer model to learn and implement in- we-he (IW) and in-context (IC) predictors over different parts of the input space. In particular, we provide insights on the emergence of ICL in synthetic settings with theory and experiments, and bridge the gap to practice by demonstrating that similar phenomena continue to hold in a simple NLP task with a real I.L,, Gemini Nano (Gemini Team, Google 2023).
Contributions. We present simple theoretical model in which ICL both emerges and becomes transient in the infinite data regime, demonstrating that very simple properties of the training distri- bution can lead these phenomena. In particular, following the common vs. rare classes distribution
Equal contribution.

---

Published as a conference paper at ICLR 2025
of Chan et al. (2022a), our model requires two types of data: (i) data where similar observations fre- quently appear in the training set set-thii this type of data will induce IWL; and (ii) a large number of diverse data points, where each sample is predictable from the context, but each appears rarely so that reliable IWL. cannot be achieved-this type of data will induce ICI.
Through combination of generalization enror and regret bound analysis, we identify conditions where ICL and IWL emerge. In particular, our theoretical model shows that the presence of type (i) data (which we refer as in-weight data) induces IWL and the presence of type (ii) data (referred to as in-context data) leads to ICL. This is consistent with the empirical findings of Chan et al. (2022a): When a sufficient number of in-context data samples are present, such as samples from rare classes, ICL emerges, and the Zipf distribution provides a good balance between in in-context and in-weight data. When the model is trained over more samples, the samples from rare classes (or in- context data) accumulate and the model learns in in-weight after observing sufficiently many samples from a given class. That is, the in-context data eventually becomes in-weight data, which improves the performance of IWL. Consequently if IW prediction is better asymptotically, IWL is eventually preferred over ICL and the latter diminishes. This partially explains the recent findings that ICL can be transient (Reddy 2024 Panwar et al. 2024). leaving open only the question about the case, examined by Singh et al. (2023). where ICL and IWL can achieve the same performance in theory.
We demonstrate empirically that training transformers on symthetic and Omniglot (Lake et al. 2015) data drawn from the stylized distribution in our theoretical model follows the predictions of the developed theory. We further provide examples where ICL is persistent or where ICL and IWL are present in parallel when ICL and IWL can achieve the same performance, which suggests that the transience of ICL more complicated in this case and it might depend on its finite-time performance (which is much harder to analyze experimentally). Finally, to bridge the gap from theory to practice, we show that tne-ttning an LLM (Gemini Nano v1 Gemini Team, Google certain data can result in reduction of its ICL ability.
2023 to memorize
Additional related work. We do not investigate how transformers learn circuits that can perform ICL. While such ability can evidently emerge from vast amount of in-context training data and sufficiently expressive architecture (function class), several works have investigated mechanistically
how transformers can realize ICL (Garg al.) 2022: Olsson et al. 2022: Xie et al. 2022: Akyürek
cet al. 2023 Hendel et al.] 2023) Abermethy et al. al. 20244 Singh et al. 2024). J. In particular, Garg et al. (2022) have demonstrated transformers can in-context learn simple regression tasks and Akyürek et al. (2023) mathematically constructed instances in which transformers can implement gradient descent and closed-form regression. Xie al. (2022) cast ICL as implicit Bayesian inference and have shown that ICL is optimal when provided infinite examples with distinguishable prompts. Hen- del et al. (023) have empirically demonstrated that the transformer can compress context into a task vector and direct the output of the query. These works generally induce ICL. by constructing multi- task context sequences. By contrast, we investigate how ICL can emerge with context sequences of only single task. While we consider the competition of ICL and IWL capabilities of the model, Lin Lee 2024) examines how the model chooses from different ICL predictors learnt during training in linear regression setting.
Several works studied ICL from a theoretical perspective. Bai ct al. (2023) show that transformers can implement many machine learning algorithms in-context and can perform in-context algorithm selection. They also provide excess loss guarantees for pretraining to perform ICL. Li et al. (2023) formalize ICL as an algorithm learning problem, where during pretraining the transformer learns prediction function from the training sequences. They further provide generalization bounds from multi-task learning perspective using algorithmic stability. Several works study the training dy- namics with gradient descent or gradient flow and the emergence of ICL during training. Nichani al. (2024) study ICL by focusing on the ability of a simplified two-layer transformer architecture to learn causal structures and the emergence of induction heads. Edelman et al. (2024); Chen et al. (024) also study how two-layer transformers learn induction heads for Markovian input sequences, unveiling the multi-phase nature of the learning process. Focusing on single-layer architectures, Sanford et al. (2024) show that the size of a single-layer transformer should be exponentially larger to learn similar induction heads than that of a two-layer architecture, while Zhang et al. (2024) ana- lyze the convergence properties of the training of a single linear attention layer for in-context linear regression tasks. Wu et al. (2024) further study the generalization error under a similar setting, and characterize how many tasks are needed to pretrain the model. They also show that the learned solution is competitive with the Bayes optimal predictor under certain conditions. Our theoretical
2

---

Published as a conference paper at ICLR 2025
work differs in that we focus the selection of an in-weight and an in-context predictor based on their corresponding test errors, as a model for this mechanism in transformer architectures.
2 THEORETICAL ANALYSIS
In this section we present our theoretical model. In essence, consider bi-level model that learns a "memorizing" in-weight predictor as well as an in-context predictor hi and for every input e it follows the prediction of the submodel that is expected to be better for that particular data point. Naturally, can be learned with high fidelity in parts of the input space where there is enough training data, while can be effective anywhere where the context is useful. The final behavior of the model then will depend on the expected accuracy of g and h for given input. In the rest of this section we formalize this approach in the setting of tabular classification problems.
2.1 SETTING
Let X' C IRd be the finite space of inputs be Ac. the C-dimensional simplex. We have access to training dataset S of N examples, where addition to the usual (input, label) pairs in a classification problem, each example has a context consisting of L extra (imput, label) pairs: S = {((x), v 27. vf.... ...,, 2 xx)) 9:) Let ž;
be the number of classes, and let the space of labels
(1), v/. , v/ ..... xh uf 2z) denote the i-th training example, and X (X x Ac) x X be the space of examples with context. Suppose (ī:, y4) are drawn from a ground-truth distribution D; dur- ing training the task is to predict yi given žj, while the final goal is minimize the prediction error for new sample (i, y) sampled independently from D where I = (x',y' " , x²,1² ........?? 2.",, .y" .z).
In the latter, the prediction error is measured by the loss function Ê A Ac- + [0, 1], where ((j y) is the loss of the prediction ý given the label y.
Data- geta-generating process. We consider the setting of inputs with label noise: there exists a map- ping y" X + Ac. such that the true label for imput x y (x). When we sample x in the context or as the query, we sample = e;. the i-th standard basis vector, with probability y (z);.
Predictors. Below, we formalize the two classes for learning: the in-weight learning (IWL) class 9, and the in-context learning (ICL) class H. The IWL class of functions 9 uses only the query, and is a tabular function class: for each E X, can learn a separate mapping from x to Ac. We denote = (g X - Ac, Ac,g(ā; w) xx w) w(x) E Ac, w E A/ }. The ICL class of functions H uses only labels in the context to make a prediction. and let it be parameterized by u belonging to some set W. Inspired by the induction head (Olsson et al., 2022), circuit that learns to copy tokens in the context to the output, we design H consisting of functions that output a convex combination of the labels in the context. In addition, we mix in uniform distribution to ensure bounded loss. Let h X x W + A. be function that outputs weights for a convex combination (a distribution) given example, and let 7 (h X Ac, ((;; u) cl (1 Cc) Ele h' (i;; u)ri,, W)-
Simple model. We design model that selects among the in-weight and in-context learners by function a, which can take different value in [0, for each ø E X. We analyze the follow- ing simplified model f to study the emergence of in-context vs. in-weight learning: for each n, f w. u) (()))((;; w) + (1 (b;))
2.2 SELECTION BY TEST ERROR
Previous empirical observations show that ICL emerges when the data distribution has long tail with many rare classes (Chan et al. 2022a). We investigate the hypothesis that ICL emerges because lnigethht learning has large error on classes that have few samples in the training dataset. We first quantify the generalization error of the in-weight learner. Then, we consider more explicit class of in-context learners and give upper and lower bounds on the errors the IC predictor. Because of its importance in training classifiers, and LLMSs in particular, we concentrate on the cross-entropy los CEE y) = - Eeeo We log dei more general losses are considered in Appendix A and proofs are also relegated to this appendix.
'The finiteness X' is only assumed to simplify the exposition; the results can casily be extended to a continuous input space X' (see Appendix(A) A. Throughout the paper, log denotes the natural logarithm.
3

---

Published as a conference paper at ICLR 2025
We start by quantifying how fast the loss of an IW predictor achieve that of the optimal pre- dictor achieving mingese. Ey (L 8)]. In case of the cross-entropy loss, the minimizer y above becomes the true distribution y" and we have mingeAc (y((( more, one can show that the convergence rate for the cross-entropy loss can be much faster than the usual O(1/ /NN) rate the optimal rate achievable by any predictor g is B. (C v) (CEE(y, .y] 2 C-1 No)
(((((((())) [)]. Further-
O(/NNa). Setting y to be the famous Krichevsky-Trofime esti- mator, predicting (KT(ar): N; 2/2
N=
N,+0/2 where Nija is the number of samples when we observe label i for input z, achieves this bound (see, e.g., Csiszár & Shields 2004 Theorem 6.4):
By ((CE()"(())) v)] - E. (CE((y) ,0)"",,)))) w)] < C' -
1og(N,) O(1/N,)
N₂
Inspired by the induction head and attention mechanism in transformers (Olsson et al. 2022), in the rest of this subsection we consider the following function class for in-context learning, which is one of the simplest examples showcasing ICL capabilities and can be used to demonstrate basic properties that may emerge during ICL:
H 1=(M())
(())) (())= cl =1
exp(-IIa x A)
Ce)
L 1= (xp(-112, ==laa) yr || B
where E 10, is d x d positive semidefinite matrix, ||A||| denotes the spectral norm of A, and || Vz Ax for any I E Rd. The idea that a function h H. implements a simplified induction head, averaging the label predictions for all context vectors weighted by softmax depending on the similarity of æy and the query x. If the context contains irrelevant labels, that is, labels y1 # for some I E [Z], the prediction will be noisy, as shown in the next proposition.
Proposition 1. Given an example sequence & with one-hot label y, let k denote the number of irrelevant labels in the context. Suppose for all x E X, ||=||| I then the prediction of any h H satisfies
2k(1-6C) -61--0 cC)
(1(3]) y/1 2k(1 2k(1-cC) +2e(C -1)
(L- cxp(2V))) 2c(C
Specializing ((ū. y) to cross-entropy loss CE(ÿ. v) = =cec -e log Dcs we obtain the following. Corollary 1. Assume the labels are one-hot (deterministic). Let à be an example sequence. If at does not contain a relevant label, then CE(h($), v) = log . If contains k irrelevant labels, then
k ++ (L- exp(22B + (e(C-1) - CE(I(F). S -log (1- L-K + +
kC1-EC)
The guarantees above show that the error of the IW predictor converges to that of the optimal predictor at rate of O(1/v (1/).N,) while the IC predictor has a minimum test eттоr depending on the number of irrelevant labels. the case where the true labels y (x) have low variance, and the best IW predictor achieves lower risk than the IC predictor, the IW predictor will eventually be more accurate than the IC predictor given enough samples. For example, let B 1, = Fix L,, for any h 2 1, the minimum ICI error is bounded from below by I+(L-1)e" e. Under the cross- 1-2
entropy loss, the minimum error for the IW predictor is the entropy y (x)).. Since = 2, y has binary distribution, whose entropy is continuous function with minimum value O. Therefore, if y" 2c
concentrates sufficiently on one of the possible outputs, the entropy is smaller than I+(2-1)e" + 6,
and under this y" (x) the minimum error of IWL is smaller than that of the ICL predictor.
We plot the bounds in Figure [ Indeed, with few samples, the loss of the IC predictor can be smaller than that of the IW predictor. However, the IW predictor eventually converges to solution with smaller error after seeing many examples with the same query. We can consider an oracle algorithm that selects between the learned IW predictor ğ and predictor à based on their test error (i.e., how they would perform new data), and predicts
((ō) a (z) + a(z))
where
a(T) =I[E,((()())) E,(((((z),w)))
4

---

Published as a conference paper at ICLR 2025
IC Bound/ Upper Bound
IW Lower Bound/ /Upper Bound
k
0.5
0.0
== 4
À 5
A 6
7
0.5
10
15
20
25
10
15
20
25
5
15
20
10 15
25
NN. [in loga)
Figure 1: The theoretical error bounds of IC and IW predictors. We set L = 8, e 0.001, B 1, C' 10, and y" (x) = [0.999, 0.001/9, .......... 0.001/9]. As the number of irrelevant contexts, k, increases the lower and upper bounds of IC error also increase, whereas where the number of samples, Nz, increases the lower and upper bounds of the IW error decrease. Consequently one can expect ICL to be transient as we observe more samples.
the indicator function whether the expected test error of does not exceed that of h for a particular input E. Under this algorithm, it is possible that initially IWL performs worse than ICL due to insufficient in-weight data. For example, when the training data contains a large number of rare classes, and each class is seen only a few times. Eventually, with enough in-weight data, the model can memorize the solution to achieve as good or even better prediction accuracy by using IWL on the query, rather than using the context for prediction and exhibiting ICL. This phenomenon is consistent with the empirical observation that ICL emerges with rare classes in the training data, and can be transient after more training steps with more data (Chan et al. 2022a Singh al. 2023 Reddy 2024 Panwar et al. 2024). We defer further discussion on transience of ICL Appendix
2.3 LEARNING TO USE IN-CONTEXT VS. IN-WEIGITT LEARNING
In practice, the model does not observe the test eтrоr during training. Here we attempt to bridge the gap between the oracle algorithm and the behavior of the simple model when trained us- ing gradient-based methods. Consider the usual sequential training procedure: for each sample Tt, 1r.1=,..... N, the model predicts [e(ā;; as)) = ar ( w:) + +(1 (1- ar, (G:)) uy), and the parameters w1+1 M6+1, (1e-1 are updated based on the observed loss ((Jo( = ((f((z)) vt).
The parameters W/: Uy 04 are usually updated using some gradient- method Ay- Apart from converging to local optimum of the loss function under technical assumptions (c.g., convexity of the loss function, see, e.g., Zinkevich, 2003): Hazan 2023), such algorithms often guarantee that their regret grows only sublinearly in N, where the regret of algorithm Ay is defined
Regret (Ar)
Σ ((AA(G))))) min Σ (( u. a.).).) w,uaa
the excess loss of the online updated parameter sequence (w,, Ue au az) relative that of the opti- mal predictor selected in hindsight (where f(-, w, u, a)) a (1--a)()))))))))))))))))) regret means that the average loss of the algorithm converges to the loss of the optimal predictor; algorithms with this property are called no-regrer algorithms (as the average regret converges to zero). Note that when minimizing the regret, the parameters used to predict at time step are com- puted based on the observed data (T1, 1/1 ),.,.... (Tz-1, ()-)),, 1), and are evaluated on a new data point (ā,, y)); hence the choice of ar is evaluated on new data, similarly to our recommendation at the end of the previous subsection, where the choice between the IW and IC predictors should be based on their performance on new, unseen data, that is, the test error.
u)). Sublinear
For simplicity, we consider bi-level parameter update procedure shown in Algorithm Since g is tabular, it can implement the optimal fixed predictor for each input æ, and hence in the regret definition can use o(: w²) (where w* is the optimal parameter for g). In the following proposition we show that the regret of Algorithm iis bounded by the regret of learning a and g. If A,, and Ag no-regret algorithms, Algorithm has sublinear regret. Moreover, observe that under the linear
5

---

Published as a conference paper at ICLR 2025
losses my, the best-in-hindsighe chooses the predictor with the lower total loss for each ã. Indeed, denote a" to be the best a in hindsight given the losses m,, it has the following explicit form:
a (i) = if
llo(E wi)) Le(h(ā,; < O, a a²(x) = 0 otherwise
2:2,=2
Algorithm Bi-level update
Input: horizon N, no-regret learner Aar Ag for g. and An, for h. for = 1 to N do
Receive example Ty, predict with fe(z;; 0z) ar (Z))g we) + (1 ((((i))
Observe losses (Zx)), (gg(;;;; wy)), ue(
Update wt+1 = (61 w1))........................)))) (((((
Update W,+11
... (.
Define me(ar)) = or (T+) (E (g((a, (1((7
Update ac++1 A. (m1, (m1...., m.) end for
Proposition 2. Let Te denote the query at time t, let Regret n (Ag) = E E(g(ar; we), yc)
- Ei lig(zs; w=",, We) be the regret of learning 9, and Regret x (Aa) Ei= me(ar)) ma(aa) be the regret of learning a. Algorithm 7 satisfies
N Σ ())
Σ elg(āe v1) S Regret (Aa) + Regret (Ag))
Since the problem of learning a" linear, common gradient- based algorithms have sublinear re- gret. For example, both the exponentiated gradient and the AdaGrad method with diagonal pre- conditioner achieves D(vKN) regret, where K is the number of distinct examples seen during the learning process (see, e.g., Cesa-Bianchi & Lugosi| 2006; Hazan 2023). Further, since learns separate (independent) predictor for each z, and l convex, the problem of learning g(x) con- vex. Many regret-ming menreemmniiizzng algorithms, including gradient descent, has O(VK'N) regret this setting, where K' is the number of distinct queries observed. In this case, we have overall sublinear regret against the best in-weight learner g" This guarantee implies that on average, our loss con- verges to that of the best in- wnihh predictor in hindsight. In Appendix [A.2] extend this result to show that Algorithm converges to the best-in best-in-hingt in-hindsighht predictor generally.
Suppose the examples W1 are drawn i.i.d. from the population distribution. Then by online-to-batel conversion (Cesa- Bianchi et al. 2001). the expectation of the average loss of g(: ((=300 we) in Algorithm/ [ exhibits similar behavior as the generalization-ene error guarantee. We give standard result below. If we consider the same class of ICL predictors, they also will have a positive minimum loss for each l, as in the previous section. Proposition
x E X. Suppose ((ã,, Wol are drawn i.i.d. from D, where y drawn i.i.d. given the query Let Na be the number of examples with x the query, then
E Sei
Elg(<; w.))) min E,,(((g(r;;))
[Regret (A,(z))]
N
where Regret N. (1)( is the regret of Ag on the input x, learned over Na time steps.
In other words, although the model in practice does not directly observe the test error during training, the training samples observed at each training iteration can be viewed as a "test" sample, its loss provides a quantity similar to that of the test error.
The above results show that as long as Ag and A., are no-regret algorithms (which are satisfied in this setting by simple gradient descent, see, e.g., Zinkevich 2003), the performance of the final pre- dictor approximates well the performance of the optimal predictor, and ax approximates well a", implying that the learned model will select IC on IW prediction depending on their performance. This shows that the unrealistic step of choosing between an IC or IW predictor based on their (un- known) test error, as suggested at the end of Section 2.2 can be replaced by a decision rule learned during training using an algorithm similar to gradient descent. Note that we have not included the complexity of learning the IC predictor h. Nevertheless, we demonstrate experimentally in the next section that the predictions drawn from our simple models are valid in many practical scenarios.

---

Published as a conference paper at ICLR 2025
Chance
IW Predlesor
IC Prodictor
Transformer
Rel. Cont., Ci
Rel. Cont., Cz
Irrel. Cont. Cy
Irrel. Cons., Ct.
1BD
Eror
00BD
8
12
16
20
8
12
16
20
12
16
20
8
12
16
20
Size logg)
Figure 2: 0-1 validation errors of the IC predictor, IW predictor, and transformer as a function of training set size N on the synthetic data, over five seeds. L 1, Phigh 0.9, Prelevent 0.9 and α = 0.2. The columns correspond to test data with relevant/ireelevnt context, and classes from the high-frequency (denoted Cи) or low-frequency classes (denoted Ct). The top row shows IBD error on the specified conditional data distribution, while the bottom row shows OOBD error. ICL diminishes N increases and IWL and ICL can emerge simultaneously.
EXPERIMENTS
In this section we demonstrate through experiments that our simple theoretical model is predictive and can help explain qualitatively the relationship between ICL and IWL in practical settings.
3.1 SYNTHETIC CLASSIFICATION
We consider synthetic classification task and investigate how the errors of IC and IW predictors can inform whether a transformer trained end-to-end will exhibit ICL. capabilities. How different pa- rameters and other model architectures impact the emergence and transience of ICL are respectively explored in Appendices B. and B.2
Data. We consider classification problem with imbalanced data distribution, where there high- and low-frequency (i.e., common and rare) classes. Let Cи and CL denote their label sets, respectively, and let C' = -C Ct. Each sample is generated by first selecting a target query- response pair (I, y) then generating the accompanying context sequence. An input-output pair (æ, y) sampled from a joint distribution p(r, y) = pl (z) (x)l))(y)) which we refer to as the base distribution. First we sample label from the label distribution p(v) which is a mixture of uniform distributions over the high- and wwwfrreuencc classes C/ and Cr. with weights Phigh and Plow = Phigh: Phigh/ MCC if Cи and p(y) Ptow Ptow/|Cr, if y E CT. we assume that Phigh >> Plow = 1 - Phigh- The distribution of queries for class y E C. is parametrized by a prototype vector Zy from the d-dimensional unit sphere; for each y C, By is sampled uniformly from the unit sphere and is a fixed parameter of the distribution p(x|y)- Then x is obtained by normalizing perturbed version of the corresponding prototype vector Ty. where the perturbation is Gaussian; formally, x = (I., + e)/11=,, b + El12. where & ~ (O, E). To sample the context to a query-targe pair (xx y). first with probability Pretevant should contain at least one query-target pair from the same class Then the context is obtained by sampling input-output pairs from p independently under the condition that the resulting L-tuple contains a class-y sample if and only if the context should be relevant.
that is, p(y)
decide if the context relevant to (x, v). that is, if
Experimental setup. We conduct experiments by training a transformer (GPT) end-to-end (Rad- ford al. 2018). The models consist of two transformer decoder layers, each with single attention head and processes 64-dimensional embeddings. Both the input and output tokenizers are linear pro- jections. where the former the identity matrix. For prediction we take the last token output from the last transformer block and feed it into a linear layer followed by softmax. To probe the dif- ficulty of IWI, and ICI., we separately train two transformers for these two settings by using data with Prelevant = 0.0 and Preievant = 1.0; we refer to these models as the IW and IC predictors, respectively. These transformers only act as proxies for measuring whether it is possible to perform ICL and IWL in the idealized case. A generic model trained on data with Prefesant 0.9 is referred to as the transformer. Unless specified otherwise, all models are trained using cross cross-entrops entropy loss for 50K gradient updates with batch size of 32. We set Cи and set the input dimension d = 64 with Σ = 2I where a = 0.2 and Ia is the d x d identity matrix. Furthermore, we fix the context length I. 1, the probability of sampling high-frequency
{O..... ... 4} and CL {5..... 9),
7

---

Published as a conference paper at ICLR 2025
Clannee
IW Predicior
IC Predictor
Transfomerr
Rel. Cont., Cx
Rel. Cont., Cr.
Irrel. Cont. Cir
Ireel. Cont. Cr.
00BD
0.1
12
16
20
12
16
20
12
Dataset Stae logg)
(a) == 1, Phigh 0.9, Preievant input noise is sufficiently high.
0.9 and
0.4. ICL is no longer transient N increases when the
Chance
IW Predlesor
IC Prodictor
Transformer
Rel. Cont., CH
Rel. Cont., Cz
Irrel, Cont. Cy
Irrel. Cons., Cr.
IBD
00BD
12
16
20
12
16
20
12
16
20
12
16
20
logg)
(b) L = 1, Phigh 0.9, Prefevant 0.9 and |Cz] = 95. ICL persists for larger N when compared to |Cz] = 5 (Figure 2b but remains transient with sufficiently large N.
Figure 3: 0-1 validation errors of the IC predictor, IW predictor, and transformer as a function of training set size N on the synthetic data, over five seeds.
classes Phigh 0.9, and vary the number of total samples N {28, 2 ,,...,,22). We repeat each experiment five times, and in the figures we show confidence intervals of width standard deviation.
220).
Evaluation. To evaluate whether the trained models exhibit IWL ICL, evaluate the trained models on in-base distribution (IBD) and out-of-base distribution (OOBD). The OOBD data is obtained from the base distribution data by cyclically shifting the labels of the high-frequency and low-frequency classes respectively. Since the labels in the IBD and OOBD are different for each query, any purely IW predictor trained IBD will fail OOBD data, while the latter can be predicted IC if the context is relevant. A model that exhibits purely IWL will result in low IBD error regardless of context relevance, but will result in high OOBD error. A model that exhibits purely ICL will result in low eтrоr on relevant contexts, but will result in high error on irrelevant contexts.
Main results. Our first experiments aim to identify conditions in which ICL emerges and/or is transient. Figure shows that slight input perturbation can elicit ICL capabilities from the trans- former. As the dataset size N increases, the IW predictor achieves similar IBD error the IC predictor, which correlates with the transformer losing the ICL capabilities. We further observe that ICL diminishes quicker for free-frreuency classes compared to low- low-frequency classes, the IW predictor requires more samples to achieve near-zero error for C₁,- Consequently there a phase in which the transformer can perform ICL and IWL simultaneously.
Modifying the input noise can dictate whether ICL emerges and whether ICL persists throughout training, observed by Chan al. (2022a) b). Consistently, our theory suggests that adding more noise results in a harder learning problem for the IW predictor, and hence we would expect ICL emerge and be more persistent the presence of higher noise level. In agreement with this, 0.4; predicted by our theory, when there is
Figure 3a shows that ICL is never transient for little no noise, IWL is (nearly) perfect whereas ICL is incorrect with irrelevant context, while higher noise levels the transformer can no longer rely on IWL and can only perform well with ICL. More results on varying noise level can be found in Figure 7 in Appendix B.I.I
Varying distributional parameters. We further vary the parameters of the data-gen aata-generating distri- bution and study the resulting ICL performance in Appendix B.1 Notably, increasing the number of
8

---

Published as a conference paper at ICLR 2025
Chance
IW Predlesor
IC Prodictor
Transformer
Lypiccant
lrrolventt
Involcount
Invelcant
1BD
Eror
00BD
8
12
16
20
8
12
16
20
12
16
20
8
12 16
20
Dulasel Size logg)
Figure 4: 0-1 validation errors, conditioned on relevant contexts and low-frequency classes CL. of the IC predictor, IW predictor, and transformer as function of training set size N on the synthetic data, over five seeds. = 4, Phigh 0.9, Pretevant 0.9, and Trelevant = (1, 2, 3, 4). The transformer exhibits stronger ICI as the number of relevant contexts Lyelevant increases.
Chance
IW Predictor
IC Predictor
Transformer
Rel. Cont., Cy
Cont., C2
Irrel. Cont., Ca
Irrel. Cont. Cr
1BD
00BD
0.1
14
18
20
18
14
16
20
1.4
18
20
Dataset Size (in log,)
Figure 5: 0-1 validation errors a function of the dataset size N Omniglot, over three seeds.
L = 2, Phigh 0.9, Preievant = 0.9, and = 0.0. The transformer exhibits ICL on low- low-frequency classes Ct initially but loses said capability as N increases.
low-frequency classes (IC2) makes it harder to get ICL off the ground (possibly due to the increased complexity to learn the mapping from class embeddings class labels), but ICL diminishes slower, as it is harder to learn IW predictor for more classes (Figure 3b
Based on our theoretical analysis ICL with longer contexts more difficult to emerge as there can be more distractor contexts, making it harder to identify the relevant context example(s). To test this we fix L = 4 and vary the number relevant contexts Lrelevant provided to the transformer. Figure 4 shows that when we provide the transformer with a sufficiently small number of relevant contexts no longer learns to perform ICL, suggesting that ICL more difficult to learn with less relevant contexts. While transformers might implement ICL differently from our theoretical IC predictor, these findings are consistent with our theoretical analysis.
3.2 OMNIGLOT
We now investigate whether our findings in the previous section apply to learning natural few- shot learning dataset, Omniglot (Lake et al. 2015). containing images of handwritten characters of different languages. The model the same as in the previous subsection, except the input tokenizer is a ResNet (He et I.. (0116) that maps images to a 64-dimensional embedding. Our data construction follows Section 3.1 and is different from Chan al. (2022a) and Singh al. (2023). Specifically, Chan et al. (2022a) set the context length L = 8 and constructed context sequences such that the relevant contexts included three relevant and three irrelevant examples from another class. [Can et al. (022a) also modelled the base-level distribution to be Zipfian. On the other hand, we set the context length to = 2 with a varying number relevant contexts and input noise o = 0.1. The base-level distribution is again mixture of uniform distributions over high- (Cx) and low-frequency classes (Cz), respectively. We choose the first 20 classes to be Cи and the remaining 1 1603 classes to be Cz. We vary the number of samples N {10*, 10 109} and train all models for 100k gradient steps. We repeat each experiment for three random seeds. We vary the distributional parameters similar Section 3.1 and present the complete results in Appendix B.4

---

Published as a conference paper at ICLR 2025
We have found that generally the findings are consistent with that of Section 3.1 input noise appears to not play an important role in this case. Specifically, the transformer trained on Omniglot first exhibits ICL Cr, and ICL becomes transient as N increases even without any input noise (Figure 55. in contrast to the finding in the synthetic experiment. However, this result does not contradict the synthetic experiment previoouu we have also shown in the synthetic experiment that with larger number of low-frequency classes the model exhibits stronger ICL-here the number of low-frequency classes is dramatically increased compared to the synthetic experiment.
However having no
3.3 TRANSIENCE OF ICL.
Our experiments identified several issues related to the transience of ICL. Due to space constraints, more detailed study of this problem is relegated to Appendix D
4 FINETUNING A REAL LANGUAGE MODEL
We conclude our experiments on studying the effect of IWL on the ICL abilities of a real LLM, Gemini Nano with 1.8B parameters (Gemini Team, Google, 2023). During the experiment we finetuned the language model with small number of samples, to demonstrate that memorization of these samples reduces ICL capabilities where the response given in the context is in contradiction with the memorized content (i.e., IWL happens more likely as a result of the additional training).
We finetuned the LLM to memorize where certain people live. The finetuning data, given in Table in Appendix [ contains eight questions regarding where person resides, with four real and four invented person and city names (e.g., Question: Where does Kaitlyn live? Only give the name of the city. Answer: Kingston), and is designed so that the language model must rely on IWL to learn the correlations between (name, city) pairs. Table 2 in Appendix shoows that the finetuned model has learned to correctly predict the real name-and-city pairs in the finetuning dataset (using greedy sampling), while making mistakes half of the time for the invented name-and-city pairs. In contrast, the base model, naturally, cannot answer any of the questions correctly.
To test ICL capability, before asking the finetuning prompt we also state that the person lives in another city. For example, adding that the person lives in the imaginary city Kjheergg (resulting in prompts like Kaitlyn lives in Kjheergg. Where does Kaillyn live? Only give the name of the city.), the base model always uses ICI. and answers Kjheergg, but the finetuned model reverts IWL in some cases, indicating that the in-weight information can overwrite the in-context prediction present in the base model. Even when this does not happen, the probability of selecting the memorized answer increases compared to the base model, demonstrating that ICL can be transient even in real LLMs (see Table 3 in Appendix Cp Additional details about these experiments are given in Appendix C
5 CONCLUSION
In this paper introduced simple theoretical framework, motivated by induction heads (Olsson ctt al. 2022), where in-weight and in-context predictors are learned separately, but a gating mecha- nism learns combine them given the query and the context. Intuitively, this mechanism suggests that in regions of the input space where the model is able to learn predictor that generalizes well (i.e., it has seen enough samples to achieve low error relative to the noise and the complexity of the function to be learned), in-weight learning happens. On the other hand, in regions of the input space where data is scarce and in-weight learning is not possible, in-context learning emerges given there is enough diverse training data where the context is useful. Furthermore, training longer with more data from the latter part of the input space results in a more confident in-weight predictor, making the model shift to in- in-weight from in-context prediction. We demonstrated experimentally that a transformer trained on synthetic data follows similar patterns, and also showed that in-context learning can be overwritten by in in-weight learning in real language models if they are made to mem- orize more information in their weights. While carlier work (Chan et al. (202aa) connected these phenomena distributional properties of the data, demonstrated that it rather the learnability of the data (naturally affected by its distribution) which matters.
These results contribute to the understanding of when and why in-context learning is present in language models. Furthermore, they might lead to new ideas in designing training schedules for large language models; exploring this avenue is left for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Stephanie Chan and the anonymous reviewers for their insightful com- ments that helped improve the manuscript. We would also like to acknowledge the support from the Canada CIFAR Al Chairs program, the Alberta Machine Intelligence Institute (Amii), and the Natural Sciences and Engineering Research Council (NSERC) of Canada.
REFERENCES
Jacob Abernethy, Alekh Agarwal, Teodor Vanislavov Marinov, and Manfred K. Warmuth. A mech- anism for sample-efficient in-context learning for sparse retrieval tasks. In International Confer- ence on Algorithmic Learning Theory (ALT), pp. 3-46, 2024.
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behba- hani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. arXiv:2404.11018, 2024.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations (ICLR), 2023.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Prov- able in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 57125-57211, 2023.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymp- totic theory of independence. Clarendon Press, 2012.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction statistical learning the- ory. In Summer school on machine learning, pp. 169-207. 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 1877-1901. 2020.
Nicolò Cesa-Bianche and Gábor Lugosi. Prediction, Learning, and Games. Cambridge University Press. New York, NY. USA, 2006.
Nicolò Besa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. In Advances in Neural Information Processing Systems. MIT Press, 2001.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- ing in transformers. In Advances in Neural Information Processing Systems (NeurIPS), vol- ume 35, pp. 18878-18891, 2022a.
Stephanie C. Y. Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew K. Lampinen, and Felix Hill. Transformers generalize differently from information stored in context vs in weights. arXiv:2210.05675. 2022b.
Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Prov- able training dynamics and feature learning in transformers. In ICML Workshop on Theoretical Foundations of Foundation Models, 2024.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In International Conference Machine Learning (ICML), 2015.
11

---

Published as a conference paper at ICLR 2025
Imre Csiszár and Paul C. Shields. Information theory and statisties: tutorial. Foundations and Trends in Communications and Information Theory, 1:417-528, Dec. 2004.
Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evo-
lution of statistical induction heads: In-context learning markov chains, 2024. URL https:
//arxiv. wwr.viorq//aoo abs /000s/24402 11004
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in- context? case study of simple function classes. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
Gemini Team, Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023, [Online; accessed 01-February-2024J.
Sharut Gupta, Stefanie Jegelka, David Lopez-Paz. and Kartik Ahuja. Context is environment. In International Conference on Learning Representations (ICLR), 2023.
Elad Hazan. Introduction to online convex optimization, 2023. URL httts::/axxiv..ot
abs/ / 19099999...0 05207
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp-
770-778, 2016.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 9318-9338 2023.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum, Human-level concept learning through probabilistic program induction. Science, 35006266):1332-333333 2015.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning (ICML), pp. 19565-19594 2023.
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. In International Conference Machine Learning (ICML), 2024.
Eshaan Nichani, Alex Damian, and Jason D. Lee. How transformers learn causal structure with gradient descent. In International Conference on Machine Learning (ICML), 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfiell-Dodde Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv:2209. 1195,, 2022.
Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In International Conference on Learning Representations (ICLR), 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl Technical Report, 2018
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl Technical Report, 2019.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In International Conference on Learning Representations (ICLR), 2024.
Clayton Sanford, Daniel Hsu, and Matus Telgarsky. One-layer transformers fail to solve the induc- tion heads task. arXiv:2408.1 143322 2024.
Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In International Conference Machine Learning (ICMI.), 2024.
12

---

Published as a conference paper at ICLR 2025
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The tran- sient nature of emergent in-context learning in transformers. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 27801-27819, 2023.
Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, and Andrew M Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. In International Conference on Machine Learning (ICML), 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning (ICML). 2023.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023. arXiv:2303.0388
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2024. URL https://
httprriwwwwweeteeeeeeeeeeeeeeeee wwt. wtt.:n/eeoorrmiiieseeseeeeo
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations (ICLR), 2022.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25:1-55, 2024.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Pro- ceedings of the Twentieth International Conference on Machine Learning, pp. 928-936, 2003.
13

---

Published as a conference paper at ICLR 2025
A PROOFS AND ADDITIONAL THEORETICAL RESULTS
We start by stating usual generalization bound (Boucheron al. 2012) for learning algorithm for general convex losses (together proof for completeness), and then present the missing proofs for Section 2
Proposition 4. Suppose we have N examples in S drawn independently from the data distribution
DD and let Sr denote the subset of S with as the query the example. Let |Sa] = Na- Let 9 be the empirical risk minimizer of S, i.e. for any n. ((x) = arg ge
1 > (R.))s.. {(ū. v). Let
Go 2 (V((;y))l> be an upper bound on the infinity norm of the gradients, where Go 1. Then with probability at least - 6, for any x E X, the risk of ((x)) satisfies
Eyl((((z). S min Eyle(0. y)] + 6G ( yEAC
We show the gemeralization bound by online-to-batch conversion. Fix X, and consider the set of examples in S that contain æ as the query. Let this set be Sp, where [S
Proof of Proposition +
N2. and in our setting the labels in S are generated i.i.d. from the distribution y" (x)). For (Ti, Ui)
Sg. define t,(-) E(.,)). and let the empirical minimizer be à = arg min Disi ((w). We
will show that
(og(2(X1C/0)
RS (w)) = (i v)] S arg min B[[(w, v)1 6G
N₂
wEdc
where the expectation is taken over y drawn from y (z). Given .i...., LN let Wi...., WN, be the output of the exponentiated gradient descent algorithm:
0;; EG((I.....,...
N₂ Let w N₂ E w; be the average of the outputs. Then with probability at least. 6, by the online-to-been to-hatch conversion Clesa-Bianchi et al. 200T). we have
1
R,,
E
w
E
(wr: y)
N
N
log(2/8)
S arg min B[l(w, y)1 + N₂ E Reggeety (EG) + 2G
N₂
ur
The EG algorithm over the simplex has regret bounded by 2G v 22 v log C conclude that
Hazan 2023). and we
log(2C/6)
Rs (w) S arg min (w) + 6G
N₂
A uniform bound over all x E X' concludes the proof.
Proof of Proposition / Let á be an example sequence, be the label, and I C [L] be the set of indices with irrelevant labels. Note that [Z] k. Since y one-hot, it can be expressed as y = e for some E [C]. We have
[] v/l1
Σ +1 n(T) i
Σ h(²) inf
exp(- x'lla) 2c(C 1).
2 cC)
IET -i=1 expeppppppp 1-pp--11- x=11aa)
Note that by definition exp(-2vB) < exp(-1lx- - (1x) < 1, since || - = = [x) A(x'=
<) <
||od
x
A < 4B. Furthermore, we have Eigr exp( - 1| - = = L k.
Since the function z-v
is nondecreasing in for and y 2 0, we conclude that
k
Eiez Hap(--1a- -| (2)
k
(L k) (x)
EiEr exp( ( | x' ll)l + Σ Engz exp( -| xr (1a)
L
The proposition follows by plugging the inequality into the previous bound.
14

---

Published as a conference paper at ICLR 2025
Let 2 be example sequence, I E [L), 91 y) denote the set of labels in the context that are irrelevant, and [I] = k. Suppose y = ce for some class c, we can lower bound the CE loss follows:
Proof of Corollary !
CE (()) log h(o)e log
1 1-E
isc
2 Eh(),;
h(T);
(- (-log(1 - x)) x)
C1-Ce)h
2 + k)) + + -C--1)6
where the last inequality follows from the previous proposition. In particular, if L, then h())e 6, and Cy h(ā)) = -loge = log _ Note that for E (0, 1). our regime of interest, 6 <
- log(e).
For the upper bound, note that the function hence h(T)e is minimized when Eipo h(T); is maximized. We thus have
x is nondecreasing in x for x > 0 and > 0, and a4y
(1=
log h()) 2 log
and the proposition follows by taking the negative on both sides.
Proof of Proposition 2 Define 4(-)) = ((., yr), and let ge(-) denote g(: wa), he(-) = h(; u.). Let g-()) g(: w²) be the optimal IW predictor. We can decompose the regret of Algorithm as
follows:
Σ ((L().))
Σ ((.g)
((())))))))))))) Σ 11 - ((=)(()))((((())))))
Σ (i,)
N
Σ ((r
-
Σ a (1- Lilg
1=1
RRggret (An)
Σ a (c.)(((((())))) (x.))) + a (c.))( lelg" (i.)))
By definition, (x) for all such that Σx, (x+( lg (œ.) 2 0, and otherwise
a aāx O. Let XIWL be the of examples where a (ø) 1. To bound the second sum, we have:
N
Σ (())C((o()))
Σ
-e (2.))
=
Similarly, for the third sum,
Σ 1 (1((()))(((((())))) 440
Σ ((he(T))) - (5)))
Σ
legr(Ee)) (a((2))))44(g'(66
t &Zwz
15

---

Published as a conference paper at ICLR 2025
Putting the three terms together, have
Σ (((().))
Σ Elg (f+)) S Regretx(An)) + E ((gi(T))) l600 (6+))
(((e((+)) (4(g"( (++)
t: f.W..
Regrety (Aa) (Aaa) + Regret (Ag)
A.1 EXTENSION TO CONTINUOUS INPUT SPACE
Similar bounds can be obtained for the continuous input space using standard results from sta- tistical learning theory (Bousquet al., 2003 Csiszár & Shields 2004). Indeed, can obtain generalization bounds for the IW predictor that decay with O(T <N+) (for convex losses) and (log N/,),,) (for the cross-entropy loss) depend on some complexity measures of the function class, e.g., Rademacher complexity. These bounds still show that as the sample size increases, the excess risk converges to 0 in the limit. Our regret bounds in Section 2.3 automatically hold on con- tinuous domains. However, to ensure that the regret for learning can be meaningfully optimized, we need some additional assumptions (in the tabular setting this is an online learning problem with linear loss functions). A sufficient assumption is that the function can be parametrized with finite number of parameters such that the loss function convex these parameters and the result- ing function class is rich enough partition the input space into two sets such that IC prediction is better on one set and IW is better on the other.
A.2 EXTENSION TO SELECTING BEST-IN-HINDSIEE HINDSIGHT PREDICTOR
In some cases the best IW predictor is not better than the best IC predictor, and vice versa. Here we show that Algorithm [ selects the best-in best-in-hindsight predictor based on the observed losses.
Proposition 5. Let ã, and It respectively denote the sequence and query time Define the following as the regret of learning g. h, and a respectively:
Regret
(Ag) x
E(g(as; we).
(o(x))") yc),
Regret (An))
E(h(it; ue).
Σ
N
Regret n (Aa) mz(as) mi(a").
Let ("((t)) := [Z,;,"" w", _u,"""" ,u"a") be the best-in-hindsight f. Suppose there are N = NIWL + NICL rounds with NIWL rounds of in-weight (IW) examples and NICL rounds of in-context (IC) examples, decided based comparing the errors of the best-in-hindsigee IW and IC predictors, ((g((()) (x),
and E(h" (ž, v)). Then, Algorithm satisfies
(f((aa))))))) E (()) & Regret (Aa) + Regretn (Ag) + RegretNics (An).
Proof of Proposition S Define le(-) E(-, v.), and let gc(-) denote 9(: wt). he(-) h(: u.). Let (-) = g(::w") be the best-in-hindsight IW predictor and h-) = h(: u²) be the best-in- hiindii-inndighh IC predictor. Notice that o² (æ.) 1(5x X(wr) ) for the best-in-hindsight predictor. Then,
16

---

Published as a conference paper at ICLR 2025
decompose the regret of Algorithm 1
as follows:
4(f((z.)) Σ (z.)
1=1
Σ ( (())
E (())(a))
N
((())) + Σ (1 -1- al (+)a(()))(((((((())
E ((f((
E (if(
Σ (r( Σ (1-)a,5)))((((((()) Σ x' (T) (ae) Σ a (2.))(((()))
N Σ
(()))(()))))))) Σ l1 - (L=a(( Σ ((
N
(iss((((())))) - (((((()))))))))(())))))))))))) +
a (
t=1
N
Σ
- (2.))
- - (1o(()))
Regret (a)
(1 a (Et)) - ((((h + Σ a (e(g"((.))))
+
Neez sez (h)
R sw2 (h)
B DETAILED EXPERIMENTAL RESULTS
Our code is available here: http:://gittabbb.o//bbbbbbb//b/b iwl
B. SYNTHETIC DATASET
In this section we investigate the parameters that may impact the emergence and transience of ICL. We first investigate independently the influence of input/output noise, class balance, and class cardi- nality in Section B.I.I We then analyze the impact of the frequency of seeing relevant context, the number relevant context examples, and the context length in Section B.1.2 schematic diagram on data- gettrreenrrtion for training and evaluation is provided in Figure
B.1.1 PARAMETERS IN BASE-LEVEL DISTRIBUTION
The base distribution is parameterized by four quantities: input noise label noise Plabet number of low-frequency classes |Cr and probability of high-frequenc class Phigh- We first investigate the impact of a and |Cz),, which had been previously investigated by Chan et al. (202aa) and Reddy 22024). For the former, we vary the input noise Σ = I²Ia. where a E {0.0, 0.02, 0.2, 0.4], and train each model for 100K gradient steps to ensure convergence. Generally, transient phenomenon as we increase except for o = 0.4 (Figure 7 As a increases the IW predictor requires more samples in order to achieve near-zero IBD eтror, whereas the IC predictor requires little samples to achieve similar error when conditioned on relevant contexts. The mismatch in convergence speed correlates with when the transformer is able to perform IWL Cn and ICI.
that ICL is
17

---

Published as a conference paper at ICLR 2025
In-base distribution
Out-of-hase distribution
Cи
C₁
Cи
C₁
B
D
B
D
Training
Rel. Context
(21,81) ~> Phigh
(#2, (h) ~
(1) ~ Phigh
(z, y) Phigh
Relevant IBD
0
A
Pretraining
Irrelevant IBD
0
2
D
Evaluation
In-weight
Irrelevant IBD
Q
D
In-context
Relevant OOBD
B
0
c
3
A
?
Figure 6: As an example our task maps letter into number. We consider two distributions in-base distribution (IBD) and out-of-base distribution (OOBD). We generate various example se- quences of length L for pretraining, in-weight evaluation, and in-context evaluation. During pre- training, the model receives sequences generated by the in-base distribution. To train our trans- former, we sample sequences with relevant context with probability Pretevent- To train our IW predictor and IC predictor, we use only respectively the relevant IBD sequences and the irrelevant IBD sequences. During in-weight evaluation, texts, generated by the IBD. During in-context evaluation, we consider only sequences with only relevant contexts, generated by the OOBD. We further condition the evaluation to consider only high-frequency class Ch and low-frequency class Ct.
consider only sequences with only irrelevant
on Ct. simultaneously. When a = 0.4 the IW predictor exhibits higher IBD error than IC predictor on relevant contexts across all N, result the transformer's ICL capability is no longer transient.
For the latter experiment, we vary the number of low-frequency classes |Czl E (5. 45, 95, 495), and train all models for 500k gradient steps. Similar to the findings in Chan al. (2022a) and Reddy (2024), we observe that with increasing |Cr] the transformer begins to exhibit ICL for C₁, while only performs IWL for Ch, which again indicates that model can exhibit both ICL and IWL (Figure 8 Further observing the IBD eπтогs of IW and IC predictors on CL. we can see that the transformer transitions from ICL to IWL when IW predictor begins exhibit lower errors than the IC predictor, aligning to our theoretical analysis.
We now investigate the impact of varying probability of sampling frequecyy classes Phigh- We vary Phigh E {0.5, 0.9, 0.99). The model exhibits ICI. for slightly larger N with balanced classes, compared imbalanced classes (Figure 9 left). Looking at the IBD errors between IW and IC predictors, IW starts with higher error and reduces to near-zero error as N increases, resulting in better predictor than IC predictor on ma-ase distribution. This crossover occurs later on balanced classes each class has effectively similar amount of samples, while Ch will be observed more by the transformer on imbalanced classes. The IW predictor will converge faster on Cn the latter case and the transformer loses its ICL capability sooner.
18

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
Predictor
Transformer
Rel. Cont., Cи
Rel. Cont., Cl
Irrel. Cont., Cu
Irrel. Cont., Cl
130
(-- Extor
0OBD
Dataset Size (in logz)
Chance
IW Predictor
Predictor
Transformer
Rel. Cont., Cμ
Rel. Cont., Ct.
Irrel. Cont., Cr
Irrel. Cont., Co
130
0-1 Ecvor
OOED
16
20
12
16
20
12
16
Dalawel Size (in. logz)
Predictor
Predictor
Transformer
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont., Cu
Irrel. Cont. Cr
IBD
Error
0onD
12
16
20
12
16
20
12
16
20)
8
12
16
20
Size (In logz)
Chance
IW Predictor
Prodictor
Transformer
Rel. Cont., CH
Rel. Cont., Ct.
Irrel. Cont., CH
Irrel. Cont., Ct.
IBD
0-1 Eroor
00BD
16
20
12
16
20
8
12
16
20
16
20
Size lotz)
Figure 7: The validation 0-1 emrr of IC predictor, IW predictor, and transformer predictor vary input noise σ E {0.0, 0.02, 0.2, 0.4] on synthetic data, over five seeds. = 1. Phigh = 0.9, and Protenant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
19

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
Predictor
Transformer
Rel. Cont., Cи
Rel. Cont., Cl
Irrel. Cont., Cu
Irrel. Cont., Cl
130
0-1 Exror
0OLD
16
Dataset Size (in logz)
Chance
1W Predictor
Predictor
Transformer
Rel. Cont., Cμ
Rel. Cont., Ct.
Irrel. Cont., Cy
Irrel. Cont., Co
130
0-1 Eivor
OOED
16
20
12
20
12
16
16
Datusel Size (in logg))
Predictor
Predictor
Transformer
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont., Cu
Irrel. Cont. Cy
IBD
0-1 Error
0onD
12
16
20
12
20
12
16
20
12
16
20
Size (In logz)
Chance
IW Predictor
Prodictor Irrel, Cont., Ch
Transformer
Rel. Cont., Cf
Rel. Cont., Ct.
Irrel. Cont., Co
IBD
Eroor
00BD
12
16
20
12
1.6
20
12
16
20
12
16
20
Size (in lot2)
Figure 8: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor
vary the number low-frequency classes [Cz] E (5, 45, 95, 495} on synthetic data, over five seeds. L Phigh 0.9, and Prefemant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD emrors.
20

---

Published as a conference paper at ICLR 2025
Predictor
Chance
Prodictor
Predictor
Transformer
Predictor
Transformer
0.5
0.9
0.99
= 0.001
0.01
0.1
IBD
1BD
0-1
0OBD
OOBD
16
20
16
20
16
20
12
12
Dalaset Size
lok2)
Dalaaee
Stowe logg))
Figure 9: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor vary the probability of high-frequency classes (left) and the label noise (right) on synthetic data, over five seeds. For each variant, the top and bottom respectively correspond IBD and OOBD errors.
Chouce
Predictor
Predictor
Transforueer
Pretevent 0.1
Proteonnt 0.5
Preievant 0.9
Peedevant 0.98
IBD
0-1 Eivor
OOBD
12
16
20
12
16
20
8
12
16
20
8
12
16
20
Size (in. loga)
Figure 10: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor vary the probability of sampling relevant contexts Prefevant on synthetic data, over five seeds. For each variant, the top and bottom respectively correspond to IBD and OOBD errors.
Finally, we explore how label noise can impact the emergence of ICL, a setting that not explicitly explored by existing work. We remove input noise (i.e. sampling only the prototype vector as the input) and vary the label noise Plabet E {0.001, 0.01, 0.1). The noisy label implemented such that Piabel keep the original class otherwise
(c 1) mod |C). In this case the IW predictor achieves lower IBD еттоr than IC predictor for small label noise, as result the transformer exhibits minimal ICL capability (Figure 2 right), When Plabel 0.1 IC predictor is better than IW predictor. Once again our theory aligns with the experimental result whereby the transformer exhibits ICL until the errors of IC and IW predictors match.
with probability
1.1.2 PARAMETERS IN CONTEXT-LEVEL DISTRIBUTION
We now investigate how the parameters for the context distribution can impact ICL-We consider three parameters: the probability of sampling relevant contexts Prelevent- the context length L, and the number of relevant contexts Lrelevant- Adjusting Preievant can be considered as chang- ing the probability of observing bursty contexts in Chan et al.) (2022b). We vary Prefesant 0.5,, 0.9, 0.99) in this experiment. Aligned with previous experiments and our theoretical anal- ysis. the transformer exhibits stronger ICL when the IC predictor achieves smaller error than the IW predictor (Figure [0p. As expected, when Prelevent is small we see IC predictor to achieve high IBD error. Increasing Prelevant results in more accurate IC predictor, but the IW predictor eventually outperform the IC predictor, resulting in the transient behavior of ICL in the transformer.
In NLP tasks the model is usually given multiple contexts in sequence, distractors. From Figure we can see that as we increase the number of contexts, even we provide always relevant context, the IC predictor ends up exhibiting IWI. capability. This suggests
potentially act as
21

---

Published as a conference paper at ICLR 2025
the difficulty of learning ICL when variable relevant contexts are provided. We investigate in detail on whether the number of relevant examples impact the emergence of ICL. We fix L and vary {1, 3, 4). When Lrelevant = the IC predictor
the number of relevant examples Lyclevant achieves smaller IBD error than the IW predictor initially but the latter eventually achieves similar error with larger N (Figure|T2) However it is worth noting that with decreasing Trelevant the initial IBD error of IC predictor increases, almost reaching the same eтror when Lvelevant = 1. In that case the transformer never exhibits ICL. Indeed, we decrease the number of relevant contexts, the transformer will have to be very precise in identifying the sole relevant context in addition to copying its label, which can be more challenging that learning just the IW predictor.
B.2 OTHER MODEL ARCHITECTURES ON SYNTHETIC DATA
We further conduct experiments with other model architectures on the synthetic dataset. We replace the transformer in Section 3.1 with two other model architectures while keeping the training algo- rithm and other hyperparameters consistent: our stylized ICL model architecture defined in Section and RNN-based model.
Stylized ICL model. As described in Section 2 the stylized model consists of three components: the IC predictor h, the IW predictor g. and the selector a. Given a sequence ã. the IC predictor M()) applies a softmax over the negative & distances between the query and the context inputs, followed by dot-product with the context labels. Both the IW predictor g and the selector implemented as a two-layer ReLU feedforward network with 64 hidden units The IW predictor g(xx takes only the query æ as the network input and outputs probability distribution over the classes. The selector a takes on the sequence 2 as the network input and outputs the probability of selecting 9 over h, Piul = (()). Finally, the stylized model outputs probability distribution over the classes computed by the convex combination of the distributions induced by IW and IC predictors: proig(() + (1 Pisot (G)).
RNN-based model. We replace the transformer blocks with single layer of the gated recurrent unit (GRU) (Chung et al. 2015) with hidden state size of 64. Similar the transformer, we first apply the input and output tokenizers, rollout the GRU by alternatively passing in context inputs and context labels, followed by the query. We then feed the final output of the GRU into a linear layer followed by softmax.
Results. We first consider the stylized model. Unlike the IC and IW transformers considered in Section 3 which are proxies to the stylized IC predictor h and IW predictor y defined in Section 2 we can directly analyze the interactions between g. h, and a. Aligning with our theoretical findings in Sectiong] when the IW predictor g achieves better IBD performance against the IC predictor i we can see that the selector a Icans towards choosing (i.c., Piul + 1 in Figures [3] and 14). Notably, Figure [4 that with increasing context length L., the IC predictor h achieves higher IBD error, and the IW predictor g can overtake with smaller dataset size N.
In general, it appears that the general trend on the emengence and transience of ICL with stylized ICL model and GRU align with the findings from the transformer architecture in Section 3 With larger input noise, we see similar behaviours with our stylized model, the GRU model, and the transformer model, where ICL is no longer transient (Figures a3d 15), suggesting that our theory can be generalized other model architectures in addition to our stylized ICL model.
B.3 VISUALIZING ATTENTION
We provide visualization on the attentions of the transformer trained with N E {210 .2203 samples in Figure 16 on 20 OOBD samples conditioned relevant contexts. When N 2nth the first attention layer focuses on the context label (i.e., the bottom middle of the 3 x 3 grid), whereas when N 20 the first attention layer focuses on the query (i.e., the bottom right of the 3 x 3 grid). The former predicts the OOBD labels, exhibiting ICL, while the latter predicts the IBD labels, exhibiting IWI.
22

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
Predictor
Transformer
Rel. Cont.,
Rel. Cont., Cl
Irrel. Cont., Cu
Irrel. Cont., Cl
180
0- Exror
0OLD
16
Dataset Sizc (in logz)
Chance
1W Predictor
Predictor
Transformer
Rel. Cont., Cи
Rel. Cont., Ct.
Irrel. Cont., Ch
Irrel. Cont., Cz.
13D
0-1 Elvor
OOED
20
12
16
20
12
16
Dalusel Size (in logz)
Predictor
Predictor
Transformer
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont., Cu
Irrel. Cont. C2
IBD
Error
0onD
12
16
20
12
16
20
12
16
21)
8
12
16
20
Sizee (In lngz)
Chance
IW Predictor
Prodictor
Transformer
Rel. Cont., CH
Rel. Cont., Ct.
Irrel, Cont., Cи
Irrel. Cont., Cz
IBD
0-1 Eroor
00BD
16
20
12
16
20
8
12
16
20
12
16
20
Dalawet Size (in losz)
The validation 0-1 error of the IC predictor, the IW predictor, and the transformer predictor as we vary the context length L E {2, 4, 8, 16) on synthetic data, over five seeds. L 4, Phigh 0.9, and Pretevant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Figure 11:
23

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
Predictor
Transformer
Rel. Cont.,
Rel. Cont., Cl
Irrel. Cont., Cu
Irrel. Cont., Cl
1B0
0-1 Earor
0ODD
16
Dataset Sizc (in logz)
Chance
1W Predictor
Predictor
Transformer
Rel. Cont., Cи
Rel. Cont., Ct.
Irrel. Cont., Ch
Irrel. Cont., Cz.
130
0-1 Elvor
OOED
20
12
16
20
12
16
Daluwel Size (in logz)
Predictor
Predictor
Transformer
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont., Cu
Irrel. Cont. Cy
IBD
Error
0onD
12
16
20
12
16
20
12
16
20)
8
12
16
20
Size (In logz)
Chance
1W Predictor
Prodictor
Transformer
Rel. Cont., CH
Rel. Cont., Ct.
Irrel. Cont., CH
Irrel. Cont., Ct
IBD
0-1 Eroor
00BD
16
20
12
16
20
8
12
16
20
12
16
20
Size losg)
Figure 12: The validation 0-1 error of the IC predictor, the IW predictor, and the transformer predic- tor as we vary the number of relevant contexts Lyelevant E {1,2, 2, 3, 4} on synthetic data, over five seeds. L 4, Phigh 0.9, and Pretevant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
24

---

Published as a conference paper at ICLR 2025
Chance
Pistt
Rel. Cont.,
Rel. Cont., Cl
Irrel. Cont.
Irrel. Cont., Cz
180
0- Exror
0OLD
12
16
Dataset Size (in logz)
Chance
Pistt
Rel. Cont. CH
Rel. Cont., Ct
Irrel. Cont. Cy
Irrel. Cont.,, Cz.
13D
0-1 Elvor
OOED
20
12
16
20
12
16
20
Daluwel Size (in logg)
Post
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont. Cy
Irrel. Cont.,, Cy
IBD
Error
0ODD
12
16
20
12
16
20
12
16
20
12
16
20
Sione (In log2))
Chance
Post
Rel. Cont., CH
Rel. Cont., Ct.
Irrel. Cont., Cy
Irrel. Cont., Ct
IBD
Eroor
00BD
16
20
12
16
20
12
16
20
16
20
Size losz)
Figure 13: The validation 0-1 emrо of the IC predictor and the IW predictor, and the probability of selecting the in-weight predictor, i.e., Piwl, as we vary input noise a E {0.0, 0.02, 0.2, 0.4) on synthetic data, over five seeds. 1, Phigh 0.9, and Preievant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
25

---

Published as a conference paper at ICLR 2025
Chance
Piol
IBD
0-1 Eroor
0OBD
12
16
20
12
16
20
12
16
20
12
16
20
Size (in logg)
Chance
Poul
[Cy,
[Cx] = 45
Cr. 96
[Cr] 495
InD
0-1 Broor
OOBD
12
16
20
12
16
20
12
18
20
12
16
20
Dataset Size loga
Figure 14: The validation 0-1 error of the IC predictor and the IW predictor, and probability of selecting in-weight predictor, i.e., Piwl, as we vary context length and the number of low- cow-frequency classes |Czl synthetic data, over five seeds. L 1, Phigh 0.9, and Pretevant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
B.4 OMNIGLOT
In this section we provide further analyses on the Omniglot experiments. We find similar trends as in Section 3.1 but with few notable observations. First, even if we increase the image noise, ICI. never emerges for CH. while ICL always emerges for C₁ (Figure 17): in line with our theory, the IBD error of the "selected" IC learner is significantly lower for low-frequency classes. We notice that their IBD errors are almost identical on Cи samples, with or without relevant contexts, suggesting that the IC predictor might have ended up learning IWL. Observing the IBD errors C2 samples, the IC predictor exhibits ICL. Similar to the synthetic setting, in this case there is a cross-over with small a. whereby the IW predictor initially exhibits higher IBD error but eventually overtakes the IC predictor. Once again we see the transience of ICL in this scenario. The scenario where a = 1.0 the most surprising-ewe though the IW predictor never achieves lower IBD error than the IC predictor, the transformer still loses ICL with increasing N.
also appears that for Omniglot, ICL can easily emerge for CL (Figure 18). Even when the context length L increases and Lrelevent is low, the transformer can perform ICL. Nevertheless, ICL is consistently transient and of low quality, as we increase the number of samples.
While our result is related to the transient nature of ICL with increasing dataset sizes, we also demonstrate the transience of ICL with as the number of gradient updates increases. Figure [9 top. demonstrates that the transformer consistently learns ICL initially samples from Cr. but always eventually loses such capability. On the other hand the transformer never learns ICL on samples from Cн. We also include the occurrence of www-frequency samples from C₁. over training (Figure 19 bottom). On average, each class from CL will be observed around 200 times after 100K iterations, while each class from Cи will be observed similar amount of times after only 200 iterations. This suggests ICL, if does appear, will diminish after 200 iterations on samples from Ct.. hence we do not observe any ICI. in Figure 19 top.
26

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
10 Predictor
GRU
Rel. Cont.,
Rel. Cont., Cz
Irrel. Cont., Cu
Irrel. Cont., Cz
130
0- Exror
0OBD
16
Dataset Sizc (in logz)
Chance
IW Predictor
IC Predictor
GRU
Rel. Cont., Cи
Rel. Cont., Ct.
Irrel. Cont., C
Irrel. Cont., Cz
130
0-1 Ecvor
OOED
16
20
12
16
20
12
16
Dalusel Size (in logz)
Predictor
Rel. Cont. Ch
Rel. Cont., C1
Irrel. Cont., Cu
Irrel. Cont. Cy
IBD
Error
00nD
12
16
20
12
16
20
12
16
20)
12
16
20
Size (In logz)
Chance
Predlictor
IC Prodictor
GRU
Rel. Cont., CH
Rel. Cont., Ct.
Irrel. Cont., CH
Irrel. Cont., Ct
IBD
Eroor
00BD
16
20
12
16
20
8
12
16
20
12
16
20
Size losg)
Figure 15: The validation 0-1 error of the IC predictor, the IW predictor, and the GRU predictor as we vary input noise o E {0.0, 0.02, 0.2, 0.4) on synthetic data, over five seeds. L = 1, Phigh = 0.9, and Pretevent 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
27

---

Published as a conference paper at ICLR 2025
3:
5:
M/:
3/:
3F
M:
M: 2
0.8
9
: 7
M: 9
1/ 9
0: 6
0 9:88
9 2
1/ 6
y:2
SF
MF 4
21
MF 5
KF
ME
MC 3
0.4
0
9
4
1/
5
6
(a) 20
M:
M:
3/C
9/
MY
3/ 2
9
3
6
5
5
4
M:
4
5
6
M: 3
0.4
0:8 0 3 0 ::44
0 5
0:
0 0:0
9
8:22
(b) N 20
Figure 16: Images (a) and (b) correspond to attention the transformer trained with N 200
[2 2201 samples respectively, evaluated on 20 samples with relevant context. The transformers are provided with L = context example. The top row corresponds to the first attention layer with y being the target. The middle corresponds the second attention layer. The bottom row corresponds to probability of the model prediction, with ý being the label with highest probability. We can that the transformer trained with N 20 exhibits ICL capabilities and its attention focuses on the context label, whereas the transformer trained with N = 220 no longer exhibits ICL capabilities and puts its attention on the query.
28

---

Published as a conference paper at ICLR 2025
Chance
IW Predictor
Predictor
Transformer
Rel. Cont., Cμ
Rel. Cont., Cl
Irrel. Cont., Cy
Irrel. Cont., Cz
130
0- Earor
00LD
0-
Datasct Size (in log10)
Prodictor
Predictor
Transformer
Rel. Cont., Cfr
Rel. Cont., Cr.
Irrel. Cont., Cn
Irrel. Cont., Cp.
13D
Error
o0nD
5
Dataset lomgo)
Predictor
Predictor
Transformer
Rel. Cont., Ch
Rel. Cort., C1
Irrel. Cont., Cu
Irrel. Cont., Cb.
IBD
Breur
00nD
5
4
5
l0810]
Figure 17: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor vary input noise E {0.0, 0.1, 1.0} on Omniglot, over three seeds. We set L 2, Phigh = 0.9 and set Pretevant 0.9 for training the transformer. For each variant, the top and bottom rows respectively correspond to in-base distribution and out-of-base distribution data.
D-1 Brvor
Plugh
Pretevant
Ireiecant
Dataset Size (in log,o)
Figure 18: 0-1 OOBD error as function of the dataset size N on Omniglot, over three seeds. From left to right we vary: (1) the probability of sampling high-frequency classes Phigh (2) the probability of sampling relevant contexts Prelevants (3) context length L, and (4) number of relevant contexts Lieeeevant fixed at L The solid black line is random chance.
29

---

Published as a conference paper at ICLR 2025
Cp/C1
Preiraining
1.0
0-1 Eleor
0.5
0.0
300
Total Occurescee
200
100
0
50000
100000
50000
100000
50000
100000
Step
Figure 19: The top is the 0-1 error of the transformer niglot. The bottom row is the total occurrence of each low-frequency class during training. The shaded region corresponds
perform gradient updates Om-
max./min. occurrence. We set 7. 2, 0, Phigh 0.9 and set Prefevant = 0.9 for training the transformer. Each column corresponds to specific seed. We ob- serve that the transformer consistently learns correctly predict C# quickly through IWL. The transformer also exhibits ICL capabilities on Cz initially but ICL eventually diminishes.
Chance
0.0
Rel. Cont., Cir
Rel. Cout., Cr.
Irrel, Cont., Ci
Irrel. Cons., C₁.
0-1 Exror
0.5
Dotasct Sizc (in 10g
Figure 20: The ICL evaluation 0- errors with heldout images as we increase the number of samples N on Omniglot, over three seeds. Regardless of the input noise, the transformer performs ICL initially but gradually performs more IWL as N increases.
Finally, we evaluate the trained transformers on heldout images that are never seen in training. In this experiment we fix L = 2, vary E {0.0, 0.1, 1.0} and only provide one of 20 images per class Omniglot for training the transformer. Similar the evaluation deseribed Section 3 we shift the original class label by one so the transformer cannot use predict through IWI. Since T, 2, the transformer will need to identify the relevant context and the query through the inputs and copy its label. Figure 20 shows that even with no noise, the transformer is able to generalize to unseen images and perform ICL on Cz.
30

---

Published as a conference paper at ICLR 2025
C DATA AND PROMPTS FOR LARGE LANGUAGE MODEL EXPERIMENTS
Here we provide the finetuning data (Table D. and the evaluation prompts and results (Tables 2 mentioned in Section Below we provide extra analyses on different prompts.
and 3
Table I: The training data for finetuning the Gemini Nano model. The prompts contain four real and four invented (random) names and city names, each, and force the model to perform IWL as it gives zero information regarding where the person resides
Prompt
Answer
Where does Cxxlh live? Only give the name of the city. Where does Vzbiik live? Only give the name of the city.
Lkkl
Plooqujhd
Where does Mmojkr live? Only give the name of the city.
Nwops
Where does Trriqe live? Only give the name of the city.
Qtbnaaa
Where does Benjamin live? Only give the name of the city.
Buenos Aires
Where does Liz live? Only give the name of the city.
London
Where does Kaitlyn live? Only give the name of the city.
Kingston
Where does Wiesa live? Only give the name of the city.
Warsaw
C.1 ADDITIONAL DISCUSSION OF EXPERIMENTS
For well-known city like Toronto, the predictions are almost always in-context; Wiesa was cor- rectly predicted to live in Warsaw. Adding some extra information to the prompt, such as Liz lives in the city of Toronto. The weather is fantastic. Where does Liz live? Only give the name of the city., the probabilities of the correct answers are drastically reduced. Note that in the previous cases we always found that the probability of the correct answer is much higher in the finetuned model (even the final response is incorrect), but this is not the case with the longer prompt. The results are similar if Kjheergg is used in the long prompt, though the correct prediction for (Tıırge, Qtbnaaa) remains. Using Hajduszoboszlo (a small town in Hungary, surely not in the training set many times) the decoy city in the longer prompt results in ICI., although the finetuned model twice responded with Budapest, showing that the predicted answer is really the most likely Hungarian city. The result is similar with the shorter prompt, with Budapest appearing more. Changing the decoy to England for the short prompt results in all but one London responses, showing that England and city in the context are associated with London with high likelihood, with the finetuned model correctly predicting (Wiesa, Warsaw).
Finally, note that our experiments are similar to the flipped-label experiments Wei et (2023) who examined how ICI. is affected if the information in the context is contradictory to what the model memorized during training. Their experiments show that ICL. can affect IWL in large models, but it is inconclusive for smaller models of similar size to Gemini Nano, they do not seem to have sufficient in-wttght knowledge. In contrast, our experiment shows that IW memorization can override ICL in the Gemini Nano model.
31

---

Published as a conference paper at ICLR 2025
Table 2: The question-answer pairs and the corresponding predictions of the finetuned and the base models for the finetuning dataset, also showing the relative log- log-poobability of possible answers. The shaded rows are the finetuned model predictions and the unshaded rows are the base model predictions. In this scenario the models must use IWL to predict the correct answer.
Model
Question
Answer
Finetuned
Where does Cxxlh live? Only give the name of
Lkkl
the city.
Lkkl -0.2666, Anytown, CA -55.3096
Base
Where does Cxxlh live? Only give the name of Anytown, CA the city.
LkkI 40.3117, Anytown, CA -2.4959
Finetuned
Where does Vzbiik live? Only give the name of Bratislava the city.
Plooqujhd -45.9387 Bratislava -0.0659
Base
Where does Vzbiik live? Only give the name of Bratislava the city.
Plooqujhd -66.2485, Bratislava-113377
Finetuned
Where does Mmojkr live? Only give the name of Milwaukee city.
Nwops -6.0057, Milwaukee -0.5176, Los Angeles
12.1351
Base
Where does Mmojkr live? Only give the name of the city.
Los Angeles
Nwops -30.8910, Milwaukee -4.8689, Los Angeles
-1.9004
Finetuned
Where does Trrrqqe live? Only give the name of the city.
Qtbnaaa
Qtbnaaa -0.0136, Toronto -9.0857
Base
Where does Trrrqq live? Only give the of Toronto
the city.
Qtbnaaa -50.5655, Toronto -10771
Finetuned
Where does Benjamin live? Only give the name of the city.
Buenos Aires
Buenos Aires 0.0000, New York City -43.2574
Base
Where does Benjamin live? Only give the name
New York City
of the city.
Buenos Aires 16.4484 New York City -0.7329
Finetuned
Where does Liz live? Only give the name of the city.
London
London 0.0000, Los Angeles -35.3694 Where does Liz live? Only give the of the city.
Base
Los Angeles
London -2.6441, Los Angeles -0.2607
Finetuned
Where does Kaitlyn live? Only give the name of Kingston the city.
Kingston 0.0000, Seattle -45.2526
Base
Where does Kaitlyn live? Only give the name of Seattle
the city.
Kingston -7.0968, Seattle -0,6796
Finetuned
Where does Wiesa live? Only give the name of Warsaw
the city.
Warsaw -0.0000, Frankfurt am Main -54.8158
Base
Where does Wiesa live? Only give the name of Frankfurt Main
the city.
Warsaw -3.0168, Frankfurt am Main -1.3085
32

---

Published as a conference paper at ICLR 2025
Table 3: The question-answer pairs and the corresponding predictions of the finetuned and the base models for questions designed to ICL capabilities, also showing the relative log-probability of possible answers. The shaded rows are the finetuned model predictions and the unshaded rows are the base model predictions. In this scenario the models must use ICL predict the correct answer.
Model
Question
Answer
Finetuned
CxxIh lives in Kjheergg. Where does Cxxlh live? Only give the name of the city.
Kxxh
Kjheergg -1.832, Lkkl -7.9295, Kxxh -0.4498
Base
CxxTh lives in Kjheergg. Where does CxxTh live? Only
K Kjheergg
give the name of the city.
Kjheergg -0.0000, Lkki -45,5691, Kxxh -32.7908
Finetuned
Vzbiik lives in Kjheergg. Where does Vzbiik live? Only
Kjheerge
give the name of the city.
Kjheergg -0.2945, Plooqujhd -47.7992
Base
Vzbiik lives in Kjheergg. Where does Vzbiik live? Only give the name of the city.
K Kheergg
Kjheergg -0.0002, Plooqujhd -62.2570
Finetuned
Mmojkr lives in Kjheergg. Where does Mmojkr live? Only give the name of the city.
Kj Kjheergg
Kjheergg -0.0382, Nwops -18.9681
Base
Mmojkr lives in Kjheergg. Where does Mmojkr live? Only give the name of the city.
Kjheergg
Kjheergg- -0.0001, Nwops -47.8115
Finetuned
Thrrqqe lives in Kjheergg. Where does Thrrqe live? Only give the name of the city.
Qthnaaa
Kjheergg 1.6897. Qtbnaaa 0.0000
Base
Thirqe lives in Kjheergg. Where does Thirqe live? Only give the name of the city.
Kjheergg
Kjheergg -0.0003, Qtbnaaa -52.3584
Finetuned
Benjamin lives in Kjheergg. Where does Benjamin live? Only give the name of the city.
Kjheergg
Kjheergg -0,0367, Buenos Aires -25.9128
Base
Benjamin lives in Kjheergg. Where does Benjamin live? Only give the name of the city.
K Kjheergg
Kjheergg -0.0001, Buenos Aires -33.1918
Finetuned
Liz lives in Kjheergg. Where does Liz live? Only give the name of the city.
K Kjheergg
Kjheergg -0.0765, London 18.2949
Base
Liz lives in Kjheergg. Where does Liz live? Only give the name of the city.
K Kjheergg
Kjheergg -0.0000, London -24.7004
Finetuned
Kaitlyn lives in Kjheergg. Where does Kaitlyn live? Only give the name of the city.
Kingston
Kjheergg -2.4694, Kingston -0.1156
Base
Kaitlyn lives in Kjheergg- Where does Kaitlyn live? Only give the name of the city.
Kjheergg
Kjheergg -0.0001, Kingston -25.7090
Finetuned
Wiesa lives in Kjheergg. Where does Wiesa live? Only give the name of the city.
Kjee
Kjheergg -1.5144, Warsaw 15.2768, Kjee -0.9055
Base
Wiesa lives in Kjheergg. Where does Wiesa live? Only give the name of the city.
Kjheergg
Kjheergg -0.0001, Warsaw -33,9379, Kjee -154233
33

---

Published as a conference paper at ICLR 2025
Chance
OOBD Cg
Protrain
Synthesic
Omniglot
1.0
1.0
Eroor
0.5
0.5
0.0
0.0
X10
X10
Gradiens Updates
Figure 21: The 0-1 error of transformer trained large number of gradient updates with N 10', Phigh 0.9, and o = 0.0. The context sequence always includes relevant examples. (Left) L= 1 on synthetic data. (Right) L on Omniglot data. We see here that ICL is not transient in both synthetic and Omniglot datasets.
D TRANSIENCE OF ICL
Singh et al. (023): Reddy (202)); Panwar et al. (2024) have previously suggested that ICL is transient with more training steps. We have conducted experiments on both our synthetic dataset and the Omniglot dataset where the transformer
always provided with relevant contexts (i.e. Prelevant = 1). Here, since the input noise o = 0.0, previous experiments described in Appen- dices B.1 and B.4 have demonstrated that the transformer is capable of learning IWL to reach error. That is, both IWL and ICL can completely solve the tasks. In Figure 21 we see that the trans- formers in both cases have learned predict in-context, and ICL has not completely disappeared training steps. The Omniglot experiment aligns with the findings [Singh] al. (2023), as the ICL capability degrades after few million training steps but never completely disappears. On the other hand, the synthetic experiments suggests that the transience of ICL might not occur all, contradicting the findings of [Singh et (2023).
even after millions
Our theory suggests that the transience of ICL may depend on the difficulty of the ICL and IWL tasks. This can be seen by analyzing the performance of our IC predictor (i.e., a predictor trained only with in-context data), as depicted in Figure[4 The tasks become easier as more examples in the context become relevant. When only one example relevant, the IC predictor does not show any ICL capability, and hence learns fully to predict fully IW. This gradually changes as the problem becomes easier, and when all elements in the context are relevant, the IC predictor always performs IC prediction and IW prediction does not occur. Relating to the results in Figure 21 we find similar results all contexts are relevant in the synthetic experiment and ICL never transient, while half of the examples in the contexts relevant in the Omniglot experiment and ICL is transient but some ICL capability remains in the model.
Generally, we expect ICL. to be transient when ICL is worse than IWL the end. In main experiments in Section 3 we have trained our transformers with mixed in-context and in- weight data. Note that the standard pretraining procedure of LLMs includes both in-context and in-weight data due to the sequential nature of the training, because at the beginning of any prompt or document there is no useful context (and there typically many documents containing relevant contexts). In this case, the theoretical optimum performance (which is assumed to be achieved asymptoticallyy) of the IWL predictor better since we have samples where no relevant context is available. Nevertheless, for some inputs IWI. and ICL. can still have equally good performance. However, in this case the model may still have some bias towards IWI. if some smoothness imposed a, because of this the inputs for which IW prediction is better will bias the selection of a for nearby inputs.
Finally, it is also possible that for every input both IWL and ICL can perform perfectly. For this case, the Omniglot experiment Figure 2T and the experiment presented by Singh et al.) (2023) both show that ICL may be transient (or more precisely that the ICL capability of the model decreases can see the same phenomenon in the performance of the IC predictor Figure [] for = 2 or in Figure 12 for Trelceant 2) while synthetic experiment in Figure 21 shows that ICL is
34

---

Published as a conference paper at ICLR 2025
persistent. In this case do not know whether ICL or IWL is better (this is very hard measure properly as a transformer trained with in-context data only can rather implement IWL as visible, could check that the predictions of
e.g,, in Figures [ and 12); had known this information our model match reality for this edge case. Singh et al. (2023) suspected that the attention might be soft in their case, which might limit the prediction performance of IC learner using those attention heads, and in this case our theory would predict that ICL is transient. In our synthetic experiment in Figure 21 there is single relevant context, and hence ICL needs learn predict the single label present in the input (which requires learning a mapping from the label embeddings the label distributions), while IWL needs to learn map the class representative a label. These two tasks are of comparable hardness, although the orthogonality of the label embeddings might make ICL somewhat easier, in which case our theory would predict that ICL happens.
Consider another scenario where the model is provided with only in-context data but random labels. We can show that both IWI. and ICL can ultimately be preferred depending on the correlation structure between the noise in the labels. For the case of fully correlated labels, when the possibly random label for the same example identical between the context and the ground-truth response to the query. IC prediction If the randomness in the labels independent, IWL may have an advantage: Consider the case of binary prediction where the ground truth for a given input has main label with probability 1-p1 and a "flipped' label with probability with < 0.5. If there a single relevant context, the labels for the context and the query agree with probability (1 p)² + p², leading to ICL error probability of 2p(1 1 - p). while the error of the IW predictor always predicting the main label is and we have p 2p(1 p) for p 0.5.
achieve 0 error while IW prediction will suffer because of the noise.
35


Reviews:
[{"review_id": "1qGC0ZD2W1", "reviewer": "Reviewer_gQxH", "summary": "This paper provides a theotical framework which studies in-context learning's emgerence and transient nature through a simple model which can adapatively switching between an in-weight predictor and an in-context predictor. The key insights from this paper shows that when the training data has long-tail, containg rare classes that appear infrequently, ICL tends to emerge. And if there are enough training data, IWL tends to dominate. Their experiments on both synthetic data and Ominiglot daat shows that ICL exhibit for rare classes and IWL for common ones.", "questions": "A recent work[1] shows that ICL exhibits dual operating modes: task retrieval and task learning. As the number of correctly labeled examples grows, the model transitions from task retrieval to task learning, but initially, even with correct labels, it may retrieve the wrong task, leading to errors. Wondering how does your finding explain this dual-mode behavior?\n\n[1]Dual Operating Modes of In-Context Learning. https://arxiv.org/abs/2402.18819", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel theoretical framework that investigates how a model selects between ICL and IWL. This framework offers insights into the conditions under which ICL emerges, especially highlighting the role of rare labels in promoting ICL.\n2. The writing and presentation is clear.\n3. this paper includes experiments in both synthetic and natural few-shot learning dataset and validated their findings.", "weaknesses": "1. This paper assumes a simple switch between the IWL and ICL based on error minimization but in real world LLM the transition may not be instantaneous and involves more complexities. \n2. The in-context predictor is modeled as a convex combination of the labels in the context, weighted by input similarity. This raises questions because more commonly the in-context predictor should learn the relationship between inputs and outputs (i.e., the mapping from x to y rather than combining labels. This simplification might limit the applicability of the theoretical findings to real-world scenarios where the input-output relations are more complex.\n3. The experiment with the real LLM, Gemini Nano 1 seems somewhat limited. It primarily tests the LLM's ability to follow in-context labels versus trained labels, which may be influenced by how different LLMs are instructed to trust new labels versus their trained knowledge. A more comprehensive and persuasive experiments could include various LLMs to assess the generality of the findings and provide deeper insights across different architectures and scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYiZXQEBMC", "reviewer": "Reviewer_ueJX", "summary": "[update: in response to the revision and author responses I have updated my score]\n\nThe study of in-context learning (ICL) has emerged in recent years as a tractable and yet important piece of the puzzle for understanding how transformers are able to achieve such good performance across such a wide range of data distributions. There is now a growing literature combining theoretical models with empirical study of ICL in synthetic settings, with the aim of understanding the mechanism by which ICL works, and why it emerges in the first place. \n\nThe present paper adds to this literature, by studying the tradeoff between memorisation (or in-weight learning) and ICL in a simple theoretical model and associated experiments, and how this tradeoff varies with the nature of the data distribution.", "questions": "The main theoretical results seems to be that “ICL will first emerge on rare classes, and will eventually be replaced by the IW predictor on those classes” (line 207,208). This prediction is studied experimentally in the setting described in Figure 2, where three models are trained for 100K gradient steps (Appendix B.1.1) across a number of sample sizes N. The three models are all trained with a single (x,y) pair in the context (L = 1) and for the IW predictor the context is always “relevant” (so if I understand correctly the correct output given (x,y), x’ is to simply ignore x’ and copy the input y to the output) and for the IC predictor the context is always “irrelevant” (so given (x,y), x’ the correct output cannot be learned from the context, which should be ignored). The main transformer is trained to have 90% of its contexts relevant.\n\nMy reading of Fig 2 is that on the rare classes where IC is possible (second column, top row) the transformer behaves very similarly to the IC predictor from the beginning, and only converges to the IW predictor when the IC predictor does. The behavior on high-frequency classes seems qualitatively similar, and so I don’t see on what basis this is evidence for the “first emerge” part of the claim. Nor do I see how these experiments really support the idea that for large N the transformer switches to the IW predictor. A simpler explanation for this experimental data seems to be that, by construction, the transformer should be somewhere between the IC and IW predictor since its training distribution is between theirs, and for large N the IW and IC predictor behave similarly on relevant contexts.\n\nIt seems therefore that the crux of the argument might lie in the OOBD plots (Fig 2 second row). My reading here is that on the low-frequency classes OOBD performance of the general transformer is initially bad, improves, and then degrades, whereas for high-frequency classes the OOBD performance is initially roughly as good as it ever gets on the low-frequency classes, and then degrades. I do not see here the evidence for “ICL emerges first on rare classes”? The degradation of OOBD performance also seems to be much slower on the low-frequency classes than the high-frequency classes, which makes me unsure if the OOBD data shows any evidence for the “replaced by the IW predictor on those classes” claim either.\n\nIn short, I’m not sure how to read Fig 2 as evidence for the theoretical claims. This leaves me unsure of the soundness of the main experimental contribution of the paper.\n\nMinor issues:\n\n- “Cdot” line 149\n- “Transient” -> “transience” line 303\n- It’s a bit strange to see H defined twice in quick succession (differently) in 130 and 158\n\nFactors that would improve my score:\n\n- As already mentioned, clarifying the link between Fig 2 and the theoretical prediction.\n- Compared to the other literature in this area the models are rather small (two layers, one head per layer) and the contexts are very short (e.g. L = 1 in Fig 2). Overall I am left with the sense that maybe these models are just not capable of learning the ICL solution the theory is talking about, and this could be confounding some of the other aspects of the paper. This is actually how I read Fig 6. To what extent is the choice of L = 1 to illustrate something theoretically, and to what extent is it just “the only length that works” as a result of the small model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Reasonable theoretical foundation for their arguments\n- Well-designed experiments to test these theoretical predictions\n\nOverall I think this has the makings of a good paper that makes a significant and original contribution to an important thread of research in the science of transformer generalization.", "weaknesses": "- At the moment I am not convinced that the experiments reported in Fig 2 really do support the theoretical predictions. I hope this is just my misunderstanding and that the authors can explain what I missed, and perhaps provide a clearer explanation in the paper.\n- While I am pretty well-versed in this corner of the literature and so prepared to care about the contents of Fig 2, it still took me a while to process all the acronyms and setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QyVsCG5yI5", "reviewer": "Reviewer_8YMd", "summary": "Summary of the paper:\n\n* The paper defines a sequential, tabular supervised classification task as a\n  simplified setting in which to study the dynamics of in-context vs.\n  in-weight learning.\n* The paper considers three models in this setting:\n  1. An 'in-weight learner' model capable of predicting labels based only on\n     the most recent query.\n  2. An 'in-context learner' model that predicts labels based instead on a\n     sequence-dependent weighted combination of previous labels appearing in\n     the sequence (specifically, the weights are given by a softmax\n     distribution based on similarity to the previous queries).\n  3. A contextual mixture model that can (learn to) choose based on the input\n     sequence whether to predict according to the first model or the second\n     model, or interpolate between the two.\n* The paper describes a theoretical analysis of the performance of these\n  three models in the synthetic task:\n  * Deriving an upper bound on the expected test loss of the in-weight\n    learner.\n  * Deriving an upper and lower bound on the loss for a particular input of\n    the in-context learner.\n  * Giving a family of algorithms for learning all three models and bounding\n    the regret of the combination model in terms of the regret of the\n    learning algorithms used for learning some of the components.\n* Based on the nature of these bounds, the paper argues that this model\n  captures phenomena observed in prior empirical work on in-context learning\n  and in-weight learning, namely that rare classes drive the emergence of\n  in-context learning (Chan et al.) and that ICL disappears after enough\n  training (Singh et al.).\n* The paper also includes some experiments with small transformers on\n  synthetic data and natural language data and comparisons of the theoretical\n  model's qualitative predictions to the resulting empirical observations.\n* There is also a small qualitative study of a LLM's dependence on knowledge\n  from fine-tuning versus knowledge from its context.\n\nI wrote a somewhat long review. Therefore, I also include a summary of my\nreview as follows.\n\n* I think this paper presents some interesting ideas and results. For\n  example, I like the sequential supervised classification task and I like\n  the three-part model.\n* However, I find the results do not achieve the paper's stated aims of\n  capturing empirical phenomena to the degree claimed, due to what appear to\n  me to be several issues, namely:\n  * fundamental mismatches between the theoretical framework and results and\n    the phenomena studied in prior work,\n  * some limitations in the theoretical results that do not appear to have\n    been acknowledged, and\n  * a lack of connection between the theoretical results and the experiments.\n* Overall, I consider the framing of the paper to be inaccurate and therefore\n  in my judgement the paper should not be accepted in its current form,\n  though I am certainly open to discussion and revisions to the framing to\n  acknowledge and address these gaps as I think the work does have value.\n* I also noted several questions and minor corrections, mainly regarding the\n  theoretical sections.\n\n**EDIT TO ADD:** Summary of discussion.\n\n* The authors addressed my concerns about the framing through their responses and revisions:\n  * They refined their claims about connections to prior work, resolving my concern about the mismatch between their theory and prior work.\n  * They strengthened their theoretical results and acknowledged the remaining limitations, resolving my concern about the limitations of their theory.\n  * They clarified the connection between their theory and experiments and added new synthetic experiments with a more direct connection, resolving my concern about the connection between their theory and experiments.\n* With the authors having resolved all of my concerns, I am now pleased to be able to recommend the paper for acceptance, and raised my rating from 5 to 8.", "questions": "Theory:\n\n1. What parameterisation of $\\alpha$ do you have in mind? Simply to learn a\n   value for every $\\tilde x$ (i.e., again tabular)?\n   * If so, how can you expect it to know to route to the IC model when you\n     haven't seen enough of an example for $g$ to be useful? Is it supposed to\n     be initialised as biased towards routing to $h$?\n\n2. In Proposition 1 (for example), over which distribution are the\n   expectations of $y$ taken? Is $y \\sim y^\\ast(x)$?\n\n3. In Proposition 2 and Corollary 1, if I am not mistaken, your bounds apply\n   to a given sample of the expected risk, and they depend on the sample. For\n   example, in corollary 1, you bound $\\mathrm{CE}(h(\\tilde x), y)$ for a\n   particular $\\tilde x$ and $y$. Since you later compare these bounds to the\n   IW upper bound which is of the expected risk, could you comment on the\n   relationship between these bounds and the expected risk for the IC learner?\n\n4. In Corollary 1, I was confused by the line \"Assume the labels $y^\\ast$ are\n   one-hot (deterministic)\".\n   * If I understand correctly, in the data generating process, the concrete\n     *labels* $y$ in any given example are *already* one-hot, as stated on\n     lines 120--121 \"When we sample $x$ in the context of as the query, we\n     sample $y=e_i$, the $i$-th standard basis vector, ...\"\n   * I thought, possibly you meant that the ground truth labelling *function*\n     $y^\\ast$ is assumed to be such that the ground truth label $y^\\ast(x)$\n     is deterministic? But this seems like too strong of an assumption, since\n     you go on to talk about IWL being better with enough samples *if* the\n     variance is sufficiently low, and if you were assuming the variance was\n     zero for your ICL bound then it wouldn't make sense to say this.\n   * In fact, the assumption that $y$ is one-hot appears to be used in the\n     proof for proposition 2, whereas no additional assumption on $y$ or\n     $y^\\ast$ seems to be used in the proof of corollary 1.\n   * So, I think this assumption is redundant. Is that correct?\n\n5. In Corollary 1, you give both a lower bound in terms of $k$ and also an\n   exact value in the case where $k=L$. There is no restriction on $k$ in the\n   general case and so I assumed that if I put $k=L$ the lower bound\n   should lower bound the exact value. However, if I am not mistaken, if $k=L$\n   then the lower bound simplifies to $1-\\epsilon$, which is larger than the\n   exact value $-\\log(\\epsilon)$ for some values of $\\epsilon$. Is there a\n   mistake in the lower bound?\n\n6. In the figure 1 caption, what is the \"Big-O constant of the IW risk\"? I saw\n   no asymptotic analysis in the IW bound.\n\nNotation and terminology:\n\n1. Corollary 1: I invite you to consider using a roman font to denote cross\n   entropy loss (as in \"$\\mathrm{CE}$\") so as to prevent lexical ambiguity\n   with the variable $C$ and an undefined variable $E$ for readers parsing\n   the result statement.\n\nTypos:\n\n1. There are two separate reference list entries for Edelman et al. (2024).\n2. There are two separate reference list entries for Zhang et al. (2024).\n3. Line 90 \"attention attention\"\n4. Line 148 \"cdot\" presumably should be \"\\cdot\"\n5. Line 304 \"transient of ICL\"\n6. Line 345: It says \"A generic model trained on data with $p_{high}$ is\n   referred to as the *transformer.*\" Is that meant to be $p_{relevant}$?\n7. The proof of Proposition 1 uses notation $R_x$ for expected risk without\n   having introduced this notation anywhere (as far as I can see).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important topic in the science of deep learning,\nnamely the phenomena of the emergence and transience of in-context learning\n(versus in-weight learning).\n\nOn this topic the paper presents an interesting learning setting involving an\nin-context supervised classification task that captures several important\nelements of in-context learning. At the same time, this setting is simpler in\nsome ways than in-context linear regression or other comparable synthetic\nlearning settings.\n\nI think the overarching story developed in the paper (roughly, in my\nunderstanding, that in-context learning arises as a heuristic that can reduce\nloss in areas of the input distribution where in-weight learning has not\ngathered enough experience to provide a specialised and more accurate\nprediction) is an interesting and thought-provoking perspective.\n\nI think the three-part model is an interesting and thought-provoking toy\nmodel of in-context learning versus in-weight learning, and I think the\ntheoretical analysis is informative and supports the aforementioned\noverarching story.", "weaknesses": "**W1. Mismatch with Singh et al.'s setting.**\nIn the introduction you write that your analysis \"explains the findings of\nSingh et al. (2023),\" namely the transience of in-context learning, giving\nway to in-weight learning. It doesn't seem to me that your model captures the\nphenomenon observed by Singh et al. In my understanding, a fundamental\nfeature of the setting of Singh et al. is that there is no train-loss\nincentive for an ICL solution over an IWL solution, because both paradigms\nare equally capable of solving the task (yet, they observe, ICL still emerges\nand still gives way to IWL in the end). In contrast, in your model, as you\nshow, based on your formulation of ICL and IWL there is a performance lower\nbound to ICL which the upper bound on IWL performance is eventually able to\nsurpass.\n\n\n**W2. Incomplete characterisation of the emergence of ICL for rare classes.**\nOn lines 207--209, you write \"Under this [oracle] algorithm, ICL will first\nemerge on rare classes ... This phenomenon is consistent with the empirical\nobservation that in-context learning emerges with rare classes in the\ntraining data...\".\n\nIt appears that this characterisation of the oracle algorithm is based on the\nobservation that, for some specific values, the IW upper bound starts higher\nthan the IC upper bound (figure 1), suggesting that in these cases the simple\nmodel will use the IC sub-model rather than IW sub-model. However, this is\nnot implied by the theory as the IW bound is merely an upper bound on the IW\nrisk (I also note in a question below that the IC bounds are not of the IC\nrisk but of a particular sample of the risk).\n\nI believe a more accurate characterisation would require the stronger\nassumption that the IW risk happens to be higher than the IC risk for the\nrare classes in question. This arrangement is permitted but not implied by\nthe comparison between upper bounds. It would be implied if you had a lower\nbound on the IW risk and this was higher than an upper bound on the IC risk.\nI am unsure whether such a bound is plausible. In my opinion this gap between\nthe model and the phenomenon should be noted explicitly as a limitation of\nthe analysis.\n\n(Note that this issue is not localised to lines 207--209, but that appears to\nbe the most explicit discussion of the comparison between the phenomenon and\nthe theoretical framework. The favourable result of the comparison is a theme\nthroughout the rest of the paper.)\n\n\n**W3. Incompletely addressed limitation of tabular analysis.**\nThere are several details in the theoretical framework and analysis that\nassume a tabular setting. In footnote 1 you claim that the results can be\neasily be extended to a continuous input space. It is not clear to me that\nthis extension would be easy, since, beyond (yes) easily extending the\ndefinition of the task and the models, the bounds and the proofs seem to be\ntightly coupled to the tabular setting. It is unclear to me whether\nqualitatively similar bounds could be achieved in a continuous input setting.\nOn what basis have you made this assertion?\n\n\n**W4. Specific model of in-context learning appears quite restrictive.**\nThe paper talks generally about in-context learning in transformers, but for\nthe three-part model your formulation of in-context learning appears\nsurprisingly restrictive, namely at first limiting the prediction to a convex\ncombination of context labels and then by the relevant theorem further\nlimiting to the specific modelling choice of using a softmax distribution\nbased on query similarity for weighting the labels.\n\nIt seems to me that mechanisms for in-context learning in a transformer are\nunlikely to be limited in this way, and the ability to use alternative\nmechanisms would generally serve to broaden the model class under\nconsideration and therefore put your performance lower bounds into question.\nIn other words, these restrictions on the ICL sub-model seem to be the source\nof the conclusion that IWL is eventually preferable to ICL, which does not so\nmuch answer the question of why IWL is eventually preferred as raise the\nquestion of why we should believe that ICL is limited in this specific way.\n\n**W5. Gap between theory and experimental study.**\nThe experimental methodology involves comparing one transformer's performance\nwith that of two other 'baseline' transformers, one trained on data that is\nsupposed to incentivise ICL and another trained on data that is supposed to\nincentivise IWL. The comparisons between the 'middle' transformer and these\nbaselines are then used to corroborate the predictions of the theory.\nHowever, the use of pretrained transformers as baselines creates a confound\nthat the 'IWL transformer' may not resemble the IWL model in the theory and\n(even more plausibly) the 'ICL transformer' may not resemble the ICL model in\nthe theory. This would potentially undermine the connection between the\ntheory and the experimental results (and, in turn, the relevance of the\ntheory). This gap appears not to have been acknowledged in the paper.\n\nGiven that you have details models of these learners and have posited\nspecific training algorithms capable of achieving your regret bounds, and in\nthe synthetic data experiments your set up seems to match the theoretical\nassumptions (apart from the tabular input space, though presumably the\nsynthetic data setting could be modified to satisfy this assumption along\nwith any others I have overlooked), it seems like it would have been possible\nto implement your theoretical models directly and compare the performance of\nthese models to the baseline transformer (or the middle transformer).\n\n\n**W6. Unclear relevance and novelty of LLM experiments.**\nThe LLM experiments are interesting in their own right, however I came away\nfeeling unimpressed for several reasons:\n\n1. I felt you could have done more to connect these experiments and their\n   results to the theoretical framework developed in section 2. At the moment\n   I don't see what the LLM experiments say about the theory and I don't see\n   what the theory says about the LLM experiments.\n\n2. The total number of examples tested is small (if my understanding is\n   correct and all examples are listed in the appendices). No statistical\n   analysis has been performed on any specific questions about the results to\n   see if the trends reported are statistically significant.\n\n3. Though it is not my area, I have the feeling that there exists at least\n   a small number of papers having already studied the topic of the\n   competition between knowledge from fine-tuning and knowledge from contexts\n   in determining the predictions of LLMs. (I regret I have not been able to\n   locate any examples, but one that comes somewhat close is Wei et al.,\n   2023, arXiv:2303.03846, which is actually cited in passing in the\n   introduction but is not discussed in any detail). I would have liked to\n   see you contextualise their experiment and results within this literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zSpuipRWe0", "reviewer": "Reviewer_VrYi", "summary": "This paper tries to understand how trained Transformers choose between in-weights learning (IWL) and in-context learning (ICL). The paper is well grounded in recent literature studying the contrast of phenomenology and inference time behavior of IWL and ICL, especially with Singh et al. 2024’s finding that ICL can be transient. The paper first sets up a mathematical framework where vectors are input and noisy labels are used in a classification setup. The paper then discusses analytic predictors using ICL and IWL, and explains why IWL is asymptotically superior and why a neural network can find the optimal solution by gradient descent. The paper conducts experiments comparing a transformer’s validation loss to the ICL and IWL predictors to identify that noise level and fraction of rare tasks are critical variables deciding ICL/IWL. Finally, the paper shows, by fine tuning Gemini Nano 1, that fine tuning can induce loss of ICL abilities in real LLMs", "questions": "Questions\n1. How could the real vs invented (name, city) pair result be explained? If the model is being fine-tuned, why can’t it memorize invented (name, city) pairs? Is there an intuition or is this simply the case?\n2. How applicable is the theory to real LLMs?\n\nSuggestions:\n1. The current figures were quite confusing for me, the axis and plot labeling constantly changed through Fig 2,3,7,8. It would be great if the plot colors/style can be improved so that they can be better understood.\n2. I think Figure 16 is main-text material, perhaps more so than Figure 8.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Overall, the paper’s motivation is clear and the flow is consistent. Here is a list of strengths:\n\nS1: The paper connects an observed phenomenon (ICL’s transience into IWL) with a synthetic model and theory.\n\nS2: The intuitive interpretation that new training samples are effectively test samples under the assumptions made in Sec. 2 is useful.\n\nS3: This work clearly demonstrates how noise level and the fraction of rare tasks can affect the ICL versus IWL.\n\nS4: Unlike concurrent work exploring similar phenomena with similar methodology (https://openreview.net/forum?id=XgH1wfHSX8, https://openreview.net/forum?id=INyi7qUdjZ, https://openreview.net/forum?id=LbceJJc9h2), this work provides experiments with real language models. However see below for some criticism as well. Despite the criticism, the investigation of how well findings on synthetic data generalize to real models is valuable.", "weaknesses": "The paper has some weaknesses, listed below:\n\nW1: Theory seems to rely on assumptions not met in realistic applications of ICL. The theory assumes that the data are drawn i.i.d, while this is usually not met in many realistic applications of ICL. Thus it is quite unclear whether new training samples can safely assumed as test samples in real LLM training\n\nW2: The setup seems to perform a classification task where x,y are given in pairs. However, this *might* be not a good model of ICL in real LLMs as it does not involve a complex sequence-space computational structure. Recent works, e.g. Akurek et al. 2024 (https://arxiv.org/abs/2401.12973), seems to model the sequence space nature of ICL better. It seems like this seems important to understand ICL \\textit{in Transformers} as Tong et al. 2024 (https://arxiv.org/abs/2405.15618 ) points out that MLPs can learn in context.\n\nW3: I was not able to draw these conclusions in line 484 from the given Figure 7: “in line with our theory, the IBD error of the “selected” IC learner is significantly lower for low-frequency classes. Furthermore, both the IC and IW predictors achieve similar IBD errors on CH, suggesting that IWL more easily emerges on common classes even with larger input noise (Figure 7, right).” It would be great if this can be explained further\n\nW4: While I agree with most conclusions drawn from the plots, it would be better to compute explicitly a ICL_vs_IWL ratio as a metric since currently it seems like these conclusions are drawn by matching the experiment curves with predictors by eye.\n\nW5: It is intuitively hard to grasp the experiment. This is solely a presentation issue, it would be very helpful if there was a schematic diagram which describes all of $p_{relevant}$, $p_{high}$, OOBD, IBD, IC model, IW model, etc. The figures could also be improved, as mentioned below in questions.\n\nW6: It is unclear how well the real LLM experiments justify the claims in the main text and the claim in Singh et al. 2024 that ICL can be transient *during pretraining*. The current experiments seem to suggest that fine-tuning can override ICL abilities, but it is very probable that the ability itself is still there while it is simply suppressed superficially. (See Jain et al 2023 (https://arxiv.org/abs/2311.12786, https://arxiv.org/abs/2410.1653 ). This would not be corresponding to the claim in the main text and in Singh et al. 2024, which I believe discusses a pre-training stage transience requiring orders of magnitude more steps than fine tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)