Toward Understanding In-context vs. In-weight Learning

## Abstract

Abstract It has recently been demonstrated empirically that in-context learning emerges in transformers when certain distributional properties are present in the training data, but this ability can also diminish upon further training.
We provide a new theoretical understanding of these phenomena by identifying simplified distributional properties that give rise to the emergence and eventual disappearance of in-context learning.
We do so by first analyzing a simplified
model that uses a gating mechanism to choose between an in-weight and an in-context predictor.
Through a combination of a generalization error and regret analysis we identify conditions where in-context and in-weight learning emerge.
These theoretical findings are then corroborated experimentally by comparing the behaviour of a full transformer on the simplified distributions to that of the stylized model, demonstrating aligned results.
We then extend the study to a full large language model,
showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.

## 1 Introduction

In-context learning (ICL) is an interesting and useful property exhibited by large language models (LLMs), wherein a model is able to successfully learn and generalize from new information given in its input, without ever having seen related data during training.
Radford et al. () and Brown et al. () were among the first to demonstrate the in-context learning capability of large-language models.
Since then, numerous empirical studies have attempted to understand the emergence of ICL in LLMs (Olsson et al., ; Wei et al., ; Agarwal et al., ; Shi et al., ).
However, ICL is not limited to natural language processing (NLP), and many synthetic settings have been constructed to better understand this phenomenon (Garg et al., ; Akyürek et al., ; Von Oswald et al., ; Gupta et al., ; Edelman et al., ; Wu et al., ; Zhang et al., ).

A recent line of work (Chan et al., ; Reddy, ; Singh et al., ) has explored the emergence of ICL through the lens of data-distributional properties.
In particular, Chan et al. () first proposed the importance of common and rare classes as driving in-context learning and in-weight learning (IWL).
Both Chan et al. () and Reddy () empirically identified that ICL can emerge in a transformer model (Vaswani et al., ) when there is a large within-class variation and a large number of classes.
Chan et al. (, ) further observed that both ICL and IWL can emerge simultaneously when the distribution over classes is Zipfian, while Singh et al. () subsequently noticed that ICL can become transient in an asymptotic training regime.

The work in this paper is inspired by a similar distributional perspective, but we investigate in greater depth how the distributional properties of the data affect the ability of a transformer model to learn and implement in-weight (IW) and in-context (IC) predictors over different parts of the input space.
In particular, we provide insights on the emergence of ICL in synthetic settings with theory and experiments, and bridge the gap to practice by demonstrating that similar phenomena continue to hold in a simple NLP task with a real LLM, Gemini Nano 1 (Gemini Team,
Google, ).

##### Contributions.

We present a simple theoretical model in which ICL both emerges and becomes transient in the infinite data regime, demonstrating that very simple properties of the training distribution can lead to these phenomena.
In particular, following the common vs. rare classes distribution of Chan et al. (), our model requires two types of data: (i) data where similar observations frequently appear in the training set—this type of data will induce in-weight learning; and (ii) a large number of diverse data points, where each sample is predictable from the context, but each appears rarely so that reliable IW learning cannot be achieved—this type of data will induce IC learning.

Through a combination of a generalization error and regret bound analysis, we identify conditions where in-context and in-weight learning emerge. In particular, our theoretical model shows that the presence of type (i) data (which we refer to as *in-weight* data) induces in-weight learning and the presence of type (ii) data (referred to as *in-context* data) leads to in-context learning. This is consistent with the empirical findings of Chan et al. ():
When a sufficient number of in-context data samples are present, such as samples from rare classes, in-context learning emerges—the Zipfian distribution provides a good balance between in-context and in-weight data.
When the model is trained over more samples, the samples from rare classes (or in-context data) accumulate and the model learns in-weight after observing sufficiently many samples from a given class. In other words, the in-context data becomes in-weight data, and the lack of in-context data encourages the model to rather use in-weight prediction than in-context, resulting in the reduction of in-context learning, which explains the findings of Singh et al. ().

We demonstrate empirically that training transformers on synthetic and Omniglot data drawn from the stylized distribution in our theoretical model follows the predictions of the developed theory. Finally, to bridge the gap from theory to practice, we further show that fine-tuning a full large language model to memorize certain data can result in a reduction of its in-context learning ability.

##### Additional related work.

We do not investigate how transformers learn circuits that can perform ICL.
While such ability can evidently emerge from a vast amount of in-context training data and a sufficiently expressive architecture (function class), several works have investigated mechanistically how transformers can realize ICL (Garg et al., ; Olsson et al., ; Xie et al., ; Akyürek et al., ; Hendel et al., ; Abernethy et al., ; Singh et al., ).
In particular,
Garg et al. () have demonstrated transformers can in-context learn simple regression tasks and Akyürek et al. () mathematically constructed instances in which transformers can implement gradient descent and closed-form regression.
Xie et al. () cast ICL as implicit Bayesian inference and have shown that ICL is optimal when provided infinite examples with distinguishable prompts.
Hendel et al. () have empirically demonstrated that the transformer can compress context into a task vector and direct the output of the query.
Chen et al. () and Sanford et al. () focus on understanding the formation and the limitations of induction heads.
These works generally enforce
ICL by constructing context sequences, each corresponding to a new task.
By contrast, we investigate how ICL can emerge with context sequences of only a single task.

A number of recent works have contributed to the theoretical study of ICL. Bai et al. () show that transformers can implement many machine learning algorithms in-context, such as gradient descent and least squares, and furthermore they can perform in-context algorithm selection. Authors also show excess loss guarantees for pretraining the transformer to perform ICL on polynomial number of sequences. Li et al. () formalizes ICL as an algorithm learning problem, where during pretraining the transformer learns a prediction function from the training sequences. They further provide generalization bounds from a multi-task learning perspective using algorithmic stability. Nichani et al. () study ICL by focusing on the ability of transformers to learn causal structures. They show gradient descent on a simplified two-layer transformer can learn causal structures when the causal graph is a tree, and when the training sequences are generated by a Markov chain, the model can learn an induction head. Edelman
et al. () also study how transformers learn induction heads in this setting, focusing on the multi-phase learning process. Zhang et al. () study the learning dynamics of training a single linear attention layer with gradient flow on linear regression tasks. They show that with random initialization, gradient flow on this architecture can in-context learn the class of linear predictors. Wu et al. () further studies the generalization error under a similar setting, and characterize how many tasks is needed to pretrain the model. They also show that the learned solution is competitive with the Bayes optimal predictor under certain conditions.
Our theoretical work differs in that we focus on the selection of in-weight learning and in-context learning based on their corresponding test errors.

## 2 Theoretical analysis

In this section we present our theoretical model. In essence, we consider a bi-level model that learns a “memorizing” in-weight predictor $g$ as well as an in-context predictor $h$, and for every input $\tilde{x}$ it follows the prediction of the submodel that is expected to be better for that particular data point. Naturally, $g$ can be learned with high fidelity in parts of the input space where there is enough training data, while $h$ can be effective anywhere where the context is useful. The final behavior of the model then will depend on the expected accuracy of $g$ and $h$ for a given input. In the rest of this section we formalize this approach in the setting of tabular classification problems.

### 2.1 Setting

Let ${\mathcal{X}}\subset\mathbb{R}^{d}$ be the finite space of inputs,(^1^11The finiteness of ${\mathcal{X}}$ is only assumed to simplify the exposition; the results can easily be extended to a continuous input space ${\mathcal{X}}$.) $C$ be the number of classes, and let the space of labels be $\Delta_{C}$, the $C$-dimensional simplex. We have access to a training dataset $S$ of $N$ examples, where in addition to the usual (input, label) pairs in a classification problem, each example has a context consisting of $L$ extra (input, label) pairs:
$S=\{((x_{i}^{1},y_{i}^{1},x_{i}^{2},y_{i}^{2},\ldots,x_{i}^{L},y_{i}^{L},x_{i}
),y_{i})\}_{i=1}^{N}.$
Let $\tilde{x}_{i}:=(x_{i}^{1},y_{i}^{1},x_{i}^{2},y_{i}^{2},\ldots,x_{i}^{L},y_{i}
^{L},x_{i})$ denote the $i$-th training example, and $\tilde{{\mathcal{X}}}=({\mathcal{X}}\times\Delta_{C})^{L}\times{\mathcal{X}}$ be the space of examples with context. Suppose $(\tilde{x}_{i},y_{i})$ are drawn from a ground-truth distribution ${\mathcal{D}}$; during training the task is to predict $y_{i}$ given $\tilde{x}_{i}$, while the final goal is to minimize the prediction error for a new sample $(\tilde{x},y)$ sampled independently from ${\mathcal{D}}$ where $\tilde{x}=(x^{1},y^{1},x^{2},y^{2},\ldots,x^{L},y^{L},x)$.

In the latter, the prediction error is measured by the loss function $\ell:\Delta_{C}\times\Delta_{C}\rightarrow[0,1]$, where $\ell(\hat{y},y)$ is the loss of the prediction $\hat{y}$ given the label $y$. Since one can always scale the loss, we assume the loss is bounded by 1 without loss of generality. In addition, we assume it is convex in its first argument. This condition is satisfied by common losses such as
cross-entropy
and squared loss.

##### Data-generating process.

We consider the setting of inputs with label noise: there exists a mapping $y^{*}:{\mathcal{X}}\rightarrow\Delta_{C}$, such that the true label for input $x$ is $y^{*}(x)$. When we sample $x$ in the context or as the query, we sample $y=e_{i}$, the $i$-th standard basis vector, with probability $y^{*}(x)_{i}$.

##### Predictors.

Below, we formalize the two classes for learning: the in-weight learning (IWL) class ${\mathcal{G}}$, and the in-context learning (ICL) class ${\mathcal{H}}$.
The *IWL class* of functions ${\mathcal{G}}$ uses only the query, and is a tabular function class: for each $x\in{\mathcal{X}}$, $g$ can learn a separate mapping from $x$ to $\Delta_{C}$. We denote ${\mathcal{G}}=\{g:\tilde{{\mathcal{X}}}\rightarrow\Delta_{C},g(\tilde{x};w)=g(
x;w)=w(x)\in\Delta_{C},w\in\Delta_{C}^{|{\mathcal{X}}|}\}$.
The *ICL class* of functions ${\mathcal{H}}$ uses only labels in the context to make a prediction, and let it be parameterized by $u$ belonging to some set ${\mathcal{W}}$. Inspired by the induction head (Olsson et al., ), a circuit that learns to copy tokens in the context to the output, we design ${\mathcal{H}}$ consisting of functions that output a convex combination of the labels in the context. In addition, we mix in uniform distribution to ensure numerical stability. Let $h^{\prime}:\tilde{{\mathcal{X}}}\times{\mathcal{W}}\rightarrow\Delta_{L}$ be a function that outputs weights for a convex combination (a distribution) given an example, and let ${\mathcal{H}}=\{h:\tilde{{\mathcal{X}}}\rightarrow\Delta_{C},h(\tilde{x};u)={
\epsilon}\mathbf{1}+(1-C{\epsilon})\sum_{l=1}^{L}h^{\prime}(\tilde{x};u)_{l}y_
{l},u\in{\mathcal{W}}\}$.

##### Simple model.

We design a model that selects among the in-weight and in-context learners by a function $\alpha$, which can take a different value in $[0,1]$ for each $\tilde{x}\in\tilde{{\mathcal{X}}}$. We analyze the following simplified model $f$ to study the emergence of in-context vs. in-weight learning: for each $\tilde{x}$,
$f(\tilde{x};\alpha,w,u)=\alpha(\tilde{x})g(\tilde{x};w)+(1-\alpha(\tilde{x}))h
(\tilde{x};u)$.

### 2.2 Selection by test error

Previous empirical observations show that ICL emerges when the data distribution has a long tail with many rare classes (Chan et al., ). We investigate the hypothesis that ICL emerges because in-weight learning has large error on classes that have few samples in the training dataset. We first quantify the generalization error of the in-weight learner. Then, we consider a more explicit class of in-context learners and give upper and lower bounds on the errors of the IC predictor.

We start by stating a usual generalization bound (Boucheron et al., ) for a learning algorithm
(proofs are given in Appendix A for completeness).

###### Proposition 1 .

Suppose we have $N$ examples in $S$ drawn independently from the data distribution ${\mathcal{D}}$, and let $S_{x}$ denote the subset of $S$ with $x$ as the query in the example. Let $|S_{x}|=N_{x}$. Let $\hat{g}$ be the empirical risk minimizer of $S$, i.e. for any $x$,
$\hat{g}(x)=\operatorname*{arg\,min}_{\hat{y}\in\Delta_{C}}\frac{1}{N_{x}}\sum_
{(\tilde{x},y)\in S_{x}}\ell(\hat{y},y).$ Let $G_{\infty}\geq\|\nabla\ell(cdot,y)\|_{\infty}$ be an upper bound on the infinity norm of the gradients, where $G_{\infty}\geq 1$. Then with probability at least $1-\delta$, for any $x\in{\mathcal{X}}$, the risk of $\hat{g}(x)$ satisfies

$$ $\mathbb{E}_{y}[\ell(\hat{g}(x),y)]\leq\min_{y^{*}\in\Delta_{C}}\mathbb{E}_{y}[ \ell(y^{*},y)]+6G_{\infty}\sqrt{\log(2|{\mathcal{X}}|C/\delta)/N_{x}}.$ $$

The optimal risk satisfies $\min_{y^{*}\in\Delta_{C}}\mathbb{E}_{y}[\ell(y^{*},y)]\leq\mathbb{E}_{y}[\ell(
y^{*}(x),y)]$, where the right hand side can be seen as the “variance” of $y$.

Inspired by the induction head and attention mechanism in transformers (Olsson et al., ), we consider the following function class for in-context learning:

$$ ${\mathcal{H}}=\left\{h(\tilde{x})={\epsilon}\mathbf{1}+(1-C{\epsilon})\sum_{l= 1}^{L}\frac{\exp(-\|x^{l}-x\|_{A})}{\sum_{j=1}^{L}\exp(-\|x^{j}-x\|_{A})}\ y^{ l},\|A\|\leq B\right\},$ $$

where $A$ is a $d\times d$ positive semidefinite matrix, $\|A\|$ denotes the spectral norm of $A$, and $\|x\|_{A}=\sqrt{x^{\top}Ax}$ for any $x\in\mathbb{R}^{d}$.
The idea is that a function $h\in{\mathcal{H}}$ implements a simplified induction head, averaging the label predictions $y_{l}$ for all context vectors $x_{l}$ weighted by a softmax depending on the similarity of $x_{l}$ and the query $x$.
If the context contains *irrelevant* labels, that is, labels $y_{l}\neq y$ for some $l\in[L]$, the prediction will be noisy, as shown in the next proposition.

###### Proposition 2 .

Given an example sequence $\tilde{x}$, let $y$ be the label, and $k$ be the number of irrelevant labels in the context. Suppose for all $x\in{\mathcal{X}}$, $\|x\|\leq 1$, then the prediction of any $h\in{\mathcal{H}}$ satisfies

$$ $\frac{2k(1-{\epsilon}C)}{k+(L-k)\exp(2\sqrt{B})}+2{\epsilon}(C-1)\leq\|h( \tilde{x})-y\|_{1}\leq\frac{2k(1-{\epsilon}C)}{L}+2{\epsilon}(C-1).$ $$

Specializing $\ell(\hat{y},y)$ to cross-entropy loss $CE(\hat{y},y)=\sum_{c\in C}-y_{c}\log\hat{y}_{c}$, we obtain the following.

###### Corollary 1 .

Assume the labels $y$ are one-hot (deterministic). Let $\tilde{x}$ be an example sequence. If $\tilde{x}$ does not contain a relevant label, then $CE(h(\tilde{x}),y)=\log\frac{1}{{\epsilon}}$. If $\tilde{x}$ contains $k$ irrelevant labels, then

$$ $\frac{k(1-{\epsilon}C)}{k+(L-k)\exp(2\sqrt{B})}+{\epsilon}(C-1)\leq CE(h( \tilde{x}),y)\leq-\log\left((1-{\epsilon}C)\frac{L-k}{L}+{\epsilon}\right).$ $$

Figure: Figure 1: The theoretical error bounds of IC and IW predictors. We set $L=8$, ${\epsilon}=0.001$, $B=1$, $C=10$, $\Delta_{C}=[0.999,0.001/9,\dots,0.001/9]$, and Big-$O$ constant of the IW risk to be $20$. As the number of irrelevant contexts, $k$, increases the lower bound of IC error also increases, whereas where the number of samples, $N_{x}$, increases the upper bound of the IW error decreases. Consequently one can expect ICL to be transient as we observe more samples.
Refer to caption: x1.png

The guarantees above show that the test error of the IW predictor converges to that of the optimal predictor at a rate of $O(1/\sqrt{N_{x}})$, while the IC predictor has a minimum test error depending on the number of irrelevant labels. In the case where the true labels $y^{*}(x)$ have low variance, and the best IW predictor achieves lower risk than the IC predictor, the IW predictor will eventually be more accurate than the IC predictor given enough samples.

We plot the bounds in Figure 1. Indeed, with few samples, the loss of the IC predictor can be smaller than that of the IW predictor. However, the IW predictor eventually converges to a solution with smaller error after seeing many examples with the same query. We can consider an oracle algorithm that selects between the learned IW predictor $\hat{g}$ and IC predictor $\hat{h}$ based on their test error (i.e., how they would perform on new data), and predicts

$$ $f(\tilde{x})=\alpha(\tilde{x})\hat{g}(x)+(1-\alpha(\tilde{x}))\hat{h}(\tilde{x }),\quad\text{ where }\quad\alpha(\tilde{x})={\mathbb{I}}\left\{\mathbb{E}_{y} [\ell(\hat{g}(x),y)]\leq\mathbb{E}_{y}[\ell(\hat{h}(\tilde{x}),y)]\right\}$ $$

is the indicator function whether the expected test error of $\hat{g}$ does not exceed that of $\hat{h}$ for a particular input $\tilde{x}$. Under this algorithm, ICL will first emerge on rare classes, and will eventually be replaced by the IW predictor on those classes. This phenomenon is consistent with the empirical observation that in-context learning emerges with rare classes in the training data, and is transient after training for more steps with freshly-sampled data (Chan et al., ; Singh et al., ).

###### Proposition 1 .

###### Proposition 2 .

###### Corollary 1 .

### 2.3 Learning to use in-context vs. in-weight learning

In practice, the model does not observe the test error during training.
Here we attempt to bridge the gap between the oracle algorithm and the behavior of the simple model when $f$ is trained using gradient-based methods.
Consider the usual sequential training procedure: for each sample $\tilde{x}_{t},t=1,\ldots,N$, the model predicts $f_{t}(\tilde{x}_{t};\alpha_{t})=\alpha_{t}(\tilde{x}_{t})g(\tilde{x}_{t};w_{t}
)+(1-\alpha_{t}(\tilde{x}_{t}))h(\tilde{x}_{t};u_{t})$, and the parameters $w_{t+1}$, $u_{t+1}$, $\alpha_{t+1}$ are updated based on the observed loss $\ell_{t}(f_{t}(\tilde{x}_{t}))=\ell(f_{t}(\tilde{x}_{t}),y_{t})$.

The parameters $w_{t},u_{t},\alpha_{t}$ are usually updated using some gradient-based method ${\mathcal{A}}_{f}$. Apart from converging to a local optimum of the loss function under technical assumptions (e.g., convexity of the loss function, see, e.g., Zinkevich, ; Hazan, ), such algorithms often guarantee that their regret grows only sublinearly in $N$, where the regret of algorithm ${\mathcal{A}}_{f}$ is defined as

$$ $\displaystyle\text{Regret}_{N}({\mathcal{A}}_{f})=\sum_{t=1}^{N}\ell(f_{t}( \tilde{x}_{t};\alpha_{t}),y_{t})-\min_{w,u,\alpha}\sum_{t=1}^{N}\ell(f(\tilde{ x}_{t};w,u,\alpha),y_{t}),$ $$

the excess loss of the online updated parameter sequence $(w_{t},u_{t},\alpha_{t})_{t=1}^{N}$ relative to that of the optimal predictor selected in hindsight (where $f(\cdot,w,u,\alpha)=\alpha(\cdot)g(\cdot;w)+(1-\alpha(\cdot))h(\cdot;u)$).
Sublinear regret means that the average loss of the algorithm converges to the loss of the optimal predictor; algorithms with this property are called *no-regret* algorithms (as the average regret converges to zero).
Note that when minimizing the regret, the parameters used to predict at time step $t$ are computed based on the observed data $(\tilde{x}_{1},y_{1}),\ldots,(\tilde{x}_{t-1},y_{t-1})$, and are evaluated on a new data point $(\tilde{x}_{t},y_{t})$; hence the choice of $\alpha_{t}$ is evaluated on new data, similarly to our recommendation at the end of the previous subsection, where the choice between the IW and IC predictors should be based on their performance on new, unseen data, that is, the test error.

For simplicity, we consider
a bi-level parameter update procedure shown in Algorithm 1. Since $g$ is tabular, it can implement the optimal fixed predictor for each input $x$, and hence in the regret definition we can use $g(\cdot;w^{*})$ (where $w^{*}$ is the optimal parameter for $g$). In the following proposition we show that the regret of Algorithm 1 is bounded by the regret of learning $\alpha$ and $g$. If ${\mathcal{A}}_{\alpha}$ and ${\mathcal{A}}_{g}$ are no-regret algorithms, Algorithm 1 has sublinear regret.
Moreover, observe that under the linear losses $m_{t}$, the best-in-hindsight $\alpha$ chooses the predictor with the lower total loss for each $\tilde{x}$. Indeed, denote $\alpha^{*}$ to be the best $\alpha$ in hindsight given the losses $m_{t}$, it has the following explicit form:

$$ $\alpha^{*}(\tilde{x})=1\ \ \text{if }\sum_{t:\tilde{x}_{t}=\tilde{x}}\ell_{t}( g(\tilde{x}_{t};w_{t}))-\ell_{t}(h(\tilde{x}_{t};u_{t}))<0,\quad\alpha^{*}( \tilde{x})=0\ \ \text{otherwise}$ $$

Figure: Algorithm 1 Bi-level update

###### Proposition 3 .

Let $x_{t}$ denote the query at time $t$, let $\text{Regret}_{N}({\mathcal{A}}_{g})=\sum_{t=1}^{N}\ell(g(x_{t};w_{t}),y_{t})-
\sum_{t=1}^{N}\ell(g(x_{t};w^{*}),y_{t})$ be the regret of learning $g$, and $\text{Regret}_{N}({\mathcal{A}}_{\alpha})=\sum_{t=1}^{N}m_{t}(\alpha_{t})-m_{t
}(\alpha^{*})$ be the regret of learning $\alpha$.
Algorithm 1 satisfies

$$ $\displaystyle\sum_{t=1}^{N}\ell(f_{t}(\tilde{x}_{t};\alpha_{t}),y_{t})-\sum_{t =1}^{N}\ell(g(\tilde{x}_{t};w^{*}),y_{t})$ $\displaystyle\leq\text{Regret}_{N}({\mathcal{A}}_{\alpha})+\text{Regret}_{N}({ \mathcal{A}}_{g}).$ $$

Since the problem of learning $\alpha^{*}$ is linear, common gradient-based algorithms have sublinear regret.
For example, both the exponentiated gradient and the AdaGrad method with a diagonal preconditioner achieves $O(\sqrt{KN})$ regret, where $K$ is the number of distinct examples seen during the learning process (see, e.g., Cesa-Bianchi and Lugosi, ; Hazan, ).
Further, since $g$ learns a separate (independent) predictor for each $x$, and $\ell$ is convex, the problem of learning $g(x)$ is convex.
Many regret-minimizing algorithms, including gradient descent, has $O(\sqrt{K^{\prime}N})$ regret in this setting, where $K^{\prime}$ is the number of distinct queries observed.
In this case, we have overall sublinear regret against the best in-weight learner $g^{*}$. This guarantee implies that on average, our loss converges to that of the best in-weight predictor in hindsight.

Suppose the examples $\{(\tilde{x}_{t},y_{t})\}_{t=1}^{N}$ are drawn i.i.d. from the population distribution. Then by online-to-batch conversion (Cesa-Bianchi et al., ), the expectation of the average loss of $g(\cdot;w_{t})$ in Algorithm 1 exhibits similar behavior as the generalization-error guarantee. We give a standard result below. If we consider the same class of ICL predictors, they also will have a positive minimum loss for each $\ell_{t}$ as in the previous section.

###### Proposition 4 .

Fix $x\in{\mathcal{X}}$. Suppose $\{(\tilde{x}_{t},y_{t})\}_{t=1}^{N}$ are drawn i.i.d. from ${\mathcal{D}}$, where $y$ is drawn i.i.d. given the query $x$. Let $N_{x}$ be the number of examples with $x$ as the query, then

$$ $\frac{1}{N_{x}}\mathbb{E}_{y_{t}:x_{t}=x}\left[\sum_{t:x_{t}=x}\ell_{t}(g(x;w_ {t}))\right]\leq\min_{w}\mathbb{E}_{y}[\ell(g(x;w),y)]+\frac{1}{N_{x}}\mathbb{ E}\left[\text{Regret}_{N_{x}}({\mathcal{A}}_{g}(x))\right],$ $$

where $\text{Regret}_{N_{x}}({\mathcal{A}}_{g}(x))$ is the regret of ${\mathcal{A}}_{g}$ on the input $x$, learned over $N_{x}$ time steps.

In other words, although the model in practice does not directly observe the test error during training, the training samples observed at each training iteration can be viewed as a “test” sample, its loss provides a quantity similar to that of the test error.

The above results show that as long as ${\mathcal{A}}_{g}$ and ${\mathcal{A}}_{\alpha}$ are no-regret algorithms (which are satisfied in this setting by simple gradient descent, see, e.g., Zinkevich, ), the performance of the final predictor approximates well the performance of the optimal predictor, and $\alpha_{N}$ approximates well $\alpha^{*}$, implying that the learned model will select IC on IW prediction depending on their performance. This shows that the unrealistic step of choosing between an IC or IW predictor based on their (unknown) test error, as suggested at the end of Section 2.2, can be replaced by a decision rule learned during training using an algorithm similar to gradient descent.
Note that we have not included the complexity of learning the IC predictor $h$.
Nevertheless, we demonstrate experimentally in the next section that the predictions drawn from our simple models are valid in many practical scenarios.

###### Proposition 3 .

###### Proposition 4 .

## 3 Experiments

In this section we demonstrate through various experiments that our simple theoretical model is predictive and can help explain the relationship between ICL and IWL in practical settings.

### 3.1 Synthetic classification

In this section we consider a synthetic classification task and
investigate how the errors of IC and IW predictors can inform whether a transformer trained end-to-end will exhibit ICL capabilities.
How different parameters impact the emergence and transient of ICL are explored in Appendix B.1.

##### Data.

We consider a classification problem with an imbalanced data distribution, where there are high- and low-frequency (i.e., common and rare) classes. Let $C_{H}$ and $C_{L}$ denote
their label sets, respectively, and let $C=C_{H}\cup C_{L}$.
Each sample is generated by first selecting a target query-response pair $(x,y)$ then generating the accompanying context sequence.
An input-output pair $(x,y)$ is sampled from a joint distribution $p(x,y)=p(x|y)p(y)$, which we refer to as the *base distribution*. First we sample a label from the label distribution $p(y)$ which is a mixture of uniform distributions over the high- and low-frequency classes $C_{H}$ and $C_{L}$,
with weights $p_{high}$ and $p_{low}=1-p_{high}$: that is, $p(y)=p_{high}/|C_{H}|$ if $y\in C_{H}$ and $p(y)=p_{low}/|C_{L}|$ if $y\in C_{L}$; we assume that $p_{high}\gg p_{low}=1-p_{high}$.
The distribution of queries for class $y\in C$ is parametrized by a prototype vector $x_{y}$ from the $d$-dimensional unit sphere; for each $y\in C$, $x_{y}$ is sampled uniformly from the unit sphere and is a fixed parameter of the distribution $p(x|y)$. Then $x$ is obtained by normalizing a perturbed version of the corresponding prototype vector $x_{y}$, where the perturbation is Gaussian; formally, $x=(x_{y}+\varepsilon)/\lVert x_{y}+\varepsilon\rVert_{2}$, where $\varepsilon\sim{\mathcal{N}}(0,\Sigma)$.
To sample the context to a query-target pair $(x,y)$, first with probability $p_{relevant}$ we decide if the context is *relevant* to $(x,y)$, that is, if it should contain at least one query-target pair from the same class $y$. Then the context is obtained by sampling $L$ input-output pairs from $p$ independently under the condition that the resulting $L$-tuple contains a class-$y$ sample if and only if the context should be relevant.

##### Experimental setup.

We conduct experiments by training a transformer (GPT) end-to-end (Radford, ).
The models consist of two transformer decoder layers, each with a single attention head and processes 64-dimensional embeddings.
Both the input and output tokenizers are linear projections, where the former is the identity matrix.
For prediction we take the last token output from the last transformer block and feed it into a linear layer followed by a softmax.
To probe the difficulty of IWL and ICL, we separately train two transformers for these two settings by using data with $p_{relevant}=0.0$ and $p_{relevant}=1.0$; we refer to these models as the *IW and IC predictors*, respectively. A generic model trained on data with $p_{high}=0.9$ is referred to as the *transformer*.
Unless specified otherwise, all models are trained using cross-entropy loss for 50K gradient updates with batch size of 32.
We set $C_{H}=\{0,\dots,4\}$ and $C_{L}=\{5,\dots,9\}$, and set the input dimension $d=64$ with $\Sigma=\sigma^{2}I_{d}$ where $\sigma=0.2$ and $I_{d}$ is the $d\times d$ identity matrix.
Furthermore, we fix the context length $L=1$, the probability of sampling high-frequency classes $p_{high}=0.9$,
and vary the number of total samples $N=\{2^{6},2^{8},\dots,2^{20}\}$.
We repeat each experiment five times, and in the figures we show confidence intervals of width 1 standard deviation.

##### Evaluation.

To evaluate whether the trained models exhibit IWL or ICL, we evaluate the trained models on *in-base distribution* (IBD) and *out-of-base distribution* (OOBD). The OOBD data is obtained from the base distribution data by cyclically shifting the labels of the high-frequency and low-frequency classes respectively. Since the labels in the in-base and out-of-base distributions are different for each query, any purely IW predictor trained on the base distribution is doomed to fail on OOBD data, while the latter can be predicted IC if the context is relevant.
A model that exhibits purely IWL will result in low error IBD regardless of context relevance, but will result in high error OOBD.
On the other hand, a model that exhibits purely ICL will result in low error on relevant contexts, but will result in high error on irrelevant contexts.

##### Main results.

Figure: Figure 2: 0-1 validation errors of the IC predictor, IW predictor, and transformer as a function of training set size $N$ on the synthetic data, over five seeds. $L=1$, $p_{high}=0.9$, $p_{relevant}=0.9$ and $\sigma=0.2$. The columns correspond to test data with relevant/irrelevant context, and classes from the high-frequency (denoted $C_{H}$) or low-frequency classes (denoted $C_{L}$). The top row shows IBD error on the specified conditional data distribution, while the bottom row shows OOBD error. ICL diminishes as $N$ increases and IWL and ICL can emerge simultaneously.
Refer to caption: x2.png

Our first experiments aim to identify conditions in which ICL emerges and/or is transient.
Figure 2 shows that slight input perturbation can elicit ICL capabilities from the transformer.
As the dataset size $N$ increases, the IW predictor achieves lower IBD error than the IC predictor, which, as suggested by the theory, correlates with the transformer losing the ICL capabilities.
We further observe that ICL diminishes quicker for high-frequency classes compared to low-frequency classes, as the IW predictor requires more samples to achieve near-zero error for $C_{L}$.
Consequently there is a phase in which the transformer can perform ICL and IWL simultaneously.

Figure: Figure 3: 0-1 validation errors as a function of the dataset size $N$ on the synthetic data, over five seeds. $L=1,p_{high}=0.9$ and set $p_{relevant}=0.9$ for training the transformers. Each curve corresponds to a model trained under input noise level $\sigma\in\{0.0,0.2,0.4\}$, resp. Left: The IBD and OOBD errors of transformers. Right: The IBD errors of IW and IC predictors. As $\sigma$ increases ICL is no longer transient as the IW predictor exhibits higher error than the IC predictor.
Refer to caption: x3.png

Modifying the input noise can dictate whether ICL emerges and whether ICL persists throughout training, as observed by Chan et al. (, ). Consistently, our theory suggests that adding more noise results in a harder learning problem for the IW predictor, and hence we would expect ICL to emerge and be more persistent in the presence of higher noise level.
In agreement with this,
Figure 3, left, shows that ICL essentially never emerges for $\sigma=0.0$ while ICL is never transient for $\sigma=0.4$. We further visualize the IBD errors of IC and IW predictors in Figure 3, right, and observe that the IW predictor performs worse than the IC predictor when $\sigma=0.4$; as predicted by our theory, at higher noise levels the transformer switches between ICL and IWL based on their errors, and we observe this behavior on the left panel.
Similarly, the curves for zero noise also follow our theory: when there is no noise, IWL is (nearly) perfect and it is simpler to learn an IW predictor for a transformer than using the context (in both cases the model needs to learn a mapping from the label embedding to the class label, but to use the context, additional mechanisms are needed, so ICL is strictly harder).

Figure: Figure 4: 0-1 OOBD error of transformer as a function of training set size $N$ on the synthetic data. Left: Varying the probability of sampling high-frequency classes $p_{high}$. Middle: Varying the probability of sampling relevant contexts $p_{relevant}$. Right: Varying the number of low-frequency classes $|C_{L}|$. The solid black line is random chance.
Refer to caption: x5.png

Figure: Figure 5: 0-1 validation errors as a function of training set size $N$ on the synthetic data, five seeds. Each curve corresponds to $L_{relevant}\in\{1,2,3,4\}$. We set $L=4$, $p_{high}=0.9$ and $p_{relevant}=0.9$ for training the transformers. Left: The IBD error of the IW predictor on relevant contexts from low-frequency classes. Middle: The IBD error of the IC predictor on relevant contexts from low-frequency classes. Right: The OOBD error of the transformer. When $L_{relevant}$ is smaller, it is more challenging for ICL to emerge even if we provide relevant contexts all the time. Consequently the transformer only performs IWL for small $L_{relevant}$.
Refer to caption: x8.png

##### Varying distributional parameters.

Figure: Figure 6: 0-1 OOBD error of transformer as a function of the training set size $N$ on synthetic data. Each curve corresponds to context lengths $L$. Larger $L$ correlates with increasing difficulty of ICL.
Refer to caption: x9.png

We further vary the distributional parameters in the base- and context-level distributions, and study the resulting ICL performance in Figure 4.
Our findings can again be explained through our theory by considering how the parameter changes affect the learnability of IC and IW predictors.
ICL appears to be stronger with smaller $p_{high}$ and with larger $p_{relevant}$.
The former increases the number of low-frequency class samples, which leads to ICL as the IW predictor requires more data to achieve lower error (see Figure 11, left, in Appendix B.1), and the latter decreases the bias of the IC predictor, thereby encouraging ICL to emerge (see Figure 12 in Appendix B.1).
Increasing the number of low-frequency classes $|C_{L}|$ makes it harder to get ICL off the ground (possibly due to the increased complexity to learn the mapping from class embeddings to class labels), but ICL diminishes slower, as it is harder to learn an IW predictor for more classes (Figure 4, right).
Finally, increasing the context length $L$ prevents ICL from emerging (Figure 6).

Based on our theoretical analysis ICL with longer contexts is more difficult to emerge as there can be more distractor contexts, making the relevant context example(s) harder to be identified.
To test this we fix $L=4$ and vary the number of relevant contexts $L_{relevant}$ provided to the transformer.
Figure 5 shows that when $L_{relevant}=1$, the IBD errors between IC and IW predictors are similar even with relevant contexts provided.
With increasing $L_{relevant}$, the IC error is lower than the IW error at the beginning, suggesting that less relevant contexts are indeed more challenging to learn.
These findings again agree with the theoretical analysis.

### 3.2 Omniglot

We now investigate whether our findings in the previous section apply to learning on a natural few-shot learning dataset, Omniglot (Lake et al., ), containing images of handwritten characters of different languages.
The model is the same as in the previous subsection, except the input tokenizer is a ResNet (He et al., ) that maps images to a 64-dimensional embedding.
Our data construction follows Section 3.1 and is different from Chan et al. () and Singh et al. ().
Specifically, Chan et al. () set the context length $L=8$ and constructed context sequences such that the relevant contexts included three relevant and three irrelevant examples from another class.
Chan et al. () also modelled the base-level distribution to be Zipfian.
On the other hand, we set the context length to $L=2$ with a varying number of relevant contexts and input noise $\sigma=0.1$.
The base-level distribution is again a mixture of uniform distributions over high- ($C_{H}$) and low-frequency classes ($C_{L}$), respectively.
We choose the first 20 classes to be $C_{H}$ and the remaining 1603 classes to be $C_{L}$.
We vary the number of samples $N=\{10^{4},10^{5},10^{6}\}$ and train all models for 100k gradient steps.
We repeat each experiment for three random seeds.

##### Results.

Figure: Figure 7: 0-1 validation errors as a function of the dataset size $N$ on Omniglot, over three seeds. $L=2$, $p_{high}=0.9$ and $p_{relevant}=0.9$ for training the transformers. Each curve corresponds to a model trained under input noise $\sigma\in\{0.0,0.1,1.0\}$. Left: The IBD and OOBD errors of transformers. Right: The IBD errors of IW and IC predictors. This corroborates the findings in Figure 3 where ICL is transient.
Refer to caption: x10.png

Figure: Figure 8: 0-1 OOBD error as a function of the dataset size $N$ on Omniglot, over three seeds. From left to right we vary: (1) the probability of sampling high-frequency classes $p_{high}$, (2) the probability of sampling relevant contexts $p_{relevant}$, (3) context length $L$, and (4) number of relevant contexts $L_{relevant}$ fixed at $L=4$. The solid black line is random chance.
Refer to caption: x12.png

We find similar trends as in Section 3.1, but with a few notable observations.
First, even if we increase the image noise, ICL never emerges for $C_{H}$, while ICL always emerges for $C_{L}$ (Figure 7, left); in line with our theory, the IBD error of the “selected” IC learner is significantly lower for low-frequency classes.
Furthermore, both the IC and IW predictors achieve similar IBD errors on $C_{H}$, suggesting that IWL more easily emerges on common classes even with larger input noise (Figure 7, right).
We provide extra analyses regarding this in Appendix B.3.
It also appears that for Omniglot, ICL can easily emerge for $C_{L}$ (Figure 8).
Even when the context length $L$ increases and $L_{relevant}$ is low, the transformer can perform ICL.
Nevertheless, ICL is consistently transient and of low quality, as we increase the number of samples.

## 4 Finetuning a real language model

We conclude our experiments on studying the effect of IWL on the ICL abilities of a real LLM, Gemini Nano 1, with 1.8B parameters (Gemini Team,
Google, ). To induce memorization, we finetuned the LLM to memorize where certain people live. The finetuning data, given in Table 1 in Appendix C,
contains eight questions regarding where a person resides, with four real and four invented person and city names (e.g., Question: *Where does Kaitlyn live? Only give the name of the city.* Answer: *Kingston*), and is designed so that the language model must rely on IWL to learn the correlations between (name, city) pairs.
Table 2 in Appendix C shows that the finetuned model has learned to correctly predict the real name-and-city pairs in the finetuning dataset (using greedy sampling), while making mistakes half of the time for the invented name-and-city pairs. In contrast, the base model—naturally—cannot answer any of the questions correctly.

To test ICL capability, before asking the finetuning prompt we also state that the person lives in another city. For example, adding that the person lives in the imaginary city *Kjheergg* (resulting in prompts like *Kaitlyn lives in Kjheergg. Where does Kaitlyn live? Only give the name of the city.*), the base model always uses ICL and answers *Kjheergg*, but the finetuned model reverts to IWL in some cases, indicating that the in-weight information can overwrite the in-context prediction present in the base model, demonstrating that ICL can be transient even in real LLMs (see Table 3 in Appendix C).
Additional discussion of these experiments and further observations are provided in Appendix C.

## 5 Conclusion

In this paper we introduced a simple theoretical framework, motivated by induction heads (Olsson et al., ), where in-weight and in-context predictors are learned separately, but a gating mechanism learns to combine them given the query and the context.
Intuitively, this mechanism suggests that in regions of the input space where the model is able to learn a predictor that generalizes well (i.e., it has seen enough samples to achieve low error relative to the noise and the complexity of the function to be learned), in-weight learning happens.
On the other hand, in regions of the input space where data is scarce and in-weight learning is not possible, in-context learning emerges given there is enough diverse training data where the context is useful.
Furthermore, training longer with more data from the latter part of the input space results in a more confident in-weight predictor, making the model shift to in-weight from in-context prediction.
We demonstrated experimentally that a transformer trained on synthetic data follows similar patterns, and also showed that in-context learning can be overwritten by in-weight learning in real language models if they are made to memorize more information in their weights. While earlier work (Chan et al., ) connected these phenomena to distributional properties of the data, we demonstrated that it is rather the learnability of the data (naturally affected by its distribution) which matters.

These results contribute to the understanding of when and why in-context learning is present in language models. Furthermore, they might lead to new ideas in designing training schedules for large language models; exploring this avenue is left for future work.

## Appendix A Proofs

###### Proof of Proposition 1 .

We show the generalization bound by online-to-batch conversion. Fix $x\in{\mathcal{X}}$, and consider the set of examples in $S$ that contain $x$ as the query. Let this set be $S_{x}$, where $|S_{x}|=N_{x}$, and in our setting the labels in $S_{x}$ are generated i.i.d. from the distribution $y^{*}(x)$. For $(\tilde{x}_{i},y_{i})\in S_{x}$, define $\ell_{i}(\cdot)=\ell(\cdot,y_{i})$, and let the empirical minimizer be $\hat{w}=\operatorname*{arg\,min}_{w\in\Delta_{C}}\sum_{i=1}^{N_{x}}\ell_{i}(w)$. We will show that

$$ $R_{x}(\hat{w})=\mathbb{E}[\ell(\hat{w},y)]\leq\operatorname*{arg\,min}_{w\in \Delta_{C}}\mathbb{E}[\ell(w,y)]+6G_{\infty}\sqrt{\frac{\log(2|{\mathcal{X}}|C /\delta)}{N_{x}}},$ $$

where the expectation is taken over $y$ drawn from $y^{*}(x)$.
Given $\ell_{1},\ldots,\ell_{N_{x}}$, let $w_{1},\ldots,w_{N_{x}}$ be the output of the exponentiated gradient descent algorithm:

$$ $w_{i}=EG(\ell_{1},\ldots,\ell_{N_{x}}).$ $$

Let $\bar{w}=\frac{1}{N_{x}}\sum_{i=1}^{N_{x}}w_{i}$ be the average of the outputs. Then with probability at least $1-\delta$, by the online-to-batch conversion (Cesa-Bianchi et al., ), we have

$$ $\displaystyle R_{x}(\bar{w})=\mathbb{E}\left[\ell\left(\frac{1}{N_{x}}\sum_{i= 1}^{N_{x}}w_{i},y\right)\right]$ $\displaystyle\leq\mathbb{E}\left[\frac{1}{N_{x}}\sum_{i=1}^{N_{x}}\ell\left(w_ {i},y\right)\right]$ $\displaystyle\leq\operatorname*{arg\,min}_{w}\mathbb{E}[\ell(w,y)]+\frac{1}{N_ {x}}\mathbb{E}\left[\text{Regret}_{N_{x}}(EG)\right]+2G_{\infty}\sqrt{\frac{ \log(2/\delta)}{N_{x}}}$ $$

The EG algorithm over the simplex has regret bounded by $2G_{\infty}\sqrt{2N_{x}\log C}$ (Hazan, ), and we conclude that

$$ $R_{x}(\bar{w})\leq\operatorname*{arg\,min}_{w}R_{x}(w)+6G_{\infty}\sqrt{\frac{ \log(2C/\delta)}{N_{x}}}.$ $$

A uniform bound over all $x\in{\mathcal{X}}$ concludes the proof.
∎

###### Proof of Proposition 2 .

Let $\tilde{x}$ be an example sequence, $y$ be the label, and ${\mathcal{I}}\subseteq[L]$ be the set of indices with irrelevant labels. Let $|{\mathcal{I}}|=k$, Since $y$ is one-hot, it can be expressed as $y=e_{c}$ for some $c\in[C]$. We have

$$ $\displaystyle\|h(\tilde{x})-y\|_{1}=\sum_{i\neq c}h(\tilde{x})_{i}+1-h(\tilde{ x})_{c}$ $\displaystyle=2\sum_{i\neq c}h(\tilde{x})_{i}$ $\displaystyle=2(1-{\epsilon}C)\sum_{l\in{\mathcal{I}}}\frac{\exp(-\|x-x^{l}\|_ {A})}{\sum_{i=1}^{L}\exp(-\|x-x^{i}\|_{A})}+2{\epsilon}(C-1).$ $$

Note that by definition $\exp(-2\sqrt{B})\leq\exp(-\|x-x^{l}\|_{A})\leq 1$, since $\|x^{l}-x\|_{A}^{2}=(x^{l}-x)^{\top}A(x^{l}-x)\leq\|x^{l}-x\|_{2}^{2}\|A\|_{2}
\leq 4B.$ Furthermore, we have $\sum_{i\notin{\mathcal{I}}}\exp(-\|x-x^{i}\|_{A})=L-k$.

Since the function $\frac{x}{x+y}$ is nondecreasing in $x$ for $x>0$ and $y\geq 0$, we conclude that

$$ $\frac{k}{k+(L-k)\exp(2\sqrt{B})}\leq\frac{\sum_{l\in{\mathcal{I}}}\exp(-\|x-x^ {l}\|_{A})}{\sum_{i\in{\mathcal{I}}}\exp(-\|x-x^{i}\|_{A})+\sum_{i\notin{ \mathcal{I}}}\exp(-\|x-x^{i}\|_{A})}\leq\frac{k}{L}$ $$

The proposition follows by plugging the inequality into the previous bound.
∎

###### Proof of Corollary 1 .

Let $\tilde{x}$ be an example sequence, ${\mathcal{I}}=\{l\in[L],y_{l}\neq y\}$ denote the set of labels in the context that are irrelevant, and $|{\mathcal{I}}|=k$.
Suppose $y=e_{c}$ for some class $c$, we can lower bound the CE loss as follows:

$$ $\displaystyle CE(y,h(\tilde{x}))=-\log h(\tilde{x})_{c}$ $\displaystyle=-\log\left(1-\sum_{i\neq c}h(\tilde{x})_{i}\right)$ $\displaystyle\geq\sum_{i\neq c}h(\tilde{x})_{i}$ ( $-\log(1-x)\geq x$ ) $\displaystyle\geq\frac{(1-C{\epsilon})k}{k+(L-k)\exp(2\sqrt{B})}+(C-1){ \epsilon},$ $$

where the last inequality follows from the previous proposition.
In particular, if $k=L$, then $h(\tilde{x})_{c}={\epsilon}$, and $CE(y,h(\tilde{x}))=-\log{\epsilon}=\log\frac{1}{{\epsilon}}$.

For the upper bound, note that the function $\frac{x}{x+y}$ is nondecreasing in $x$ for $x>0$ and $y>0$, and hence $h(\tilde{x})_{c}$ is minimized when $\sum_{i\neq c}h(\tilde{x})_{i}$ is maximized. We thus have

$$ $\displaystyle\log h(\tilde{x})_{c}\geq\log\left((1-{\epsilon}C)\frac{L-k}{L}+{ \epsilon}\right),$ $$

and the proposition follows by taking the negative on both sides.
∎

###### Proof of Proposition 3 .

Define $\ell_{t}(\cdot)=\ell(\cdot,y_{t})$, and let $g_{t}(\cdot)$ denote $g(\cdot;w_{t})$, $h_{t}(\cdot)=h(\cdot;u_{t})$. Let $g^{*}(\cdot)=g(\cdot;w^{*})$ be the optimal IW predictor. We can decompose the regret of Algorithm 1 as follows:

$$ $\displaystyle\sum_{t=1}^{N}\ell_{t}(f_{t}(\tilde{x}_{t}))-\sum_{t=1}^{N}\ell_{ t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle\leq\sum_{t=1}^{N}\alpha_{t}(\tilde{x}_{t})\ell_{t}(g_{t}(\tilde{ x}_{t}))+\sum_{t=1}^{N}(1-\alpha_{t}(\tilde{x}_{t}))\ell_{t}(h_{t}(\tilde{x}_{ t}))-\sum_{t=1}^{N}\ell_{t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle=\sum_{t=1}^{N}(\alpha_{t}(\tilde{x}_{t})-\alpha^{*}(\tilde{x}_{t }))(\ell_{t}(g_{t}(\tilde{x}_{t}))-\ell_{t}(h_{t}(\tilde{x}_{t})))+\sum_{t=1}^ {N}\alpha^{*}(\tilde{x}_{t})\ell_{t}(g_{t}(\tilde{x}_{t}))+(1-\alpha^{*}( \tilde{x}_{t}))\ell_{t}(h_{t}(\tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle\leq\text{Regret}_{N}({\mathcal{A}}_{\alpha})+\sum_{t=1}^{N} \alpha^{*}(\tilde{x}_{t})(\ell_{t}(g_{t}(\tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde {x}_{t})))+\sum_{t=1}^{N}(1-\alpha^{*}(\tilde{x}_{t}))(\ell_{t}(h_{t}(\tilde{x }_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t})))$ $$

By definition, $\alpha^{*}(\tilde{x})=1$ for all $\tilde{x}$ such that $\sum_{\tilde{x}_{t}=\tilde{x}}\ell_{t}(h_{t}(\tilde{x}_{t}))-\ell_{t}(g_{t}(
\tilde{x}_{t}))\geq 0$, and otherwise $\alpha^{*}(\tilde{x})=0$. Let ${\mathcal{X}}_{IWL}$ be the set of examples where $\alpha^{*}(\tilde{x})=1$. To bound the second sum, we have:

$$ $\displaystyle\sum_{t=1}^{N}\alpha^{*}(\tilde{x}_{t})(\ell_{t}(g_{t}(\tilde{x}_ {t}))-\ell_{t}(g^{*}(\tilde{x}_{t})))=\sum_{t:\tilde{x}_{t}\in{\mathcal{X}}_{ IWL}}\ell_{t}(g_{t}(\tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t}))$ $$

Similarly, for the third sum,

$$ $\displaystyle\sum_{t=1}^{N}(1-\alpha^{*}(\tilde{x}_{t}))(\ell_{t}(h_{t}(\tilde {x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t})))$ $\displaystyle=\sum_{t:\tilde{x}_{t}\notin{\mathcal{X}}_{IWL}}\ell_{t}(h_{t}( \tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle\leq\sum_{t:\tilde{x}_{t}\notin{\mathcal{X}}_{IWL}}\ell_{t}(g_{t} (\tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t}))$ $$

Putting the three terms together, we have

$$ $\displaystyle\sum_{t=1}^{N}\ell_{t}(f_{t}(\tilde{x}_{t}))-\sum_{t=1}^{N}\ell_{ t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle\leq\text{Regret}_{N}({\mathcal{A}}_{\alpha})+\sum_{t:\tilde{x}_{ t}\in{\mathcal{X}}_{IWL}}\ell_{t}(g_{t}(\tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{ x}_{t}))$ $\displaystyle+\sum_{t:\tilde{x}_{t}\notin{\mathcal{X}}_{IWL}}\ell_{t}(g_{t}( \tilde{x}_{t}))-\ell_{t}(g^{*}(\tilde{x}_{t}))$ $\displaystyle=\text{Regret}_{N}({\mathcal{A}}_{\alpha})+\text{Regret}_{N}({ \mathcal{A}}_{g})$ $$

∎

###### Proof of Proposition 1 .

###### Proof of Proposition 2 .

###### Proof of Corollary 1 .

###### Proof of Proposition 3 .

## Appendix B Detailed experimental results

### B.1 Synthetic dataset

In this section we investigate the parameters that may impact the emergence and transient of ICL.
We first investigate independently the influence of input/output noise, class balance, and class cardinality in Section B.1.1. We then analyze the impact of the frequency of seeing relevant context, the number of relevant context examples, and the context length in Section B.1.2.

#### B.1.1 Parameters in base-level distribution

Figure: Figure 9: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary input noise $\sigma\in\{0.0,0.02,0.2,0.4\}$ on synthetic data, over five seeds. $L=1,p_{high}=0.9,$ and $p_{relevant}=0.9$ for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Refer to caption: x13.png

Figure: Figure 10: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary the number of low-frequency classes $|C_{L}|\in\{5,45,95,495\}$ on synthetic data, over five seeds. $L=1,p_{high}=0.9,$ and $p_{relevant}=0.9$ for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Refer to caption: x17.png

The base distribution is parameterized by four quantities: input noise $\sigma$, label noise $p_{label}$, number of low-frequency classes $|C_{L}|$, and probability of high-frequency class $p_{high}$.
We first investigate the impact of $\sigma$ and $|C_{L}|$, which had been previously investigated by Chan et al. () and Reddy ().
For the former, we vary the input noise $\Sigma=\sigma^{2}I_{d}$, where $\sigma\in\{0.0,0.02,0.2,0.4\}$, and train each model for 100K gradient steps to ensure convergence.
Generally, we see that ICL is a transient phenomenon as we increase $N$ except for $\sigma=0.4$ (Figure 9).
As $\sigma$ increases the IW predictor requires more samples in order to achieve near-zero IBD error, whereas the IC predictor requires little samples to achieve similar error when conditioned on relevant contexts.
The mismatch in convergence speed correlates with when the transformer is able to perform IWL on $C_{H}$ and ICL on $C_{L}$ simultaneously.
When $\sigma=0.4$ the IW predictor exhibits higher IBD error than IC predictor on relevant contexts across all $N$, as a result the transformer’s ICL capability is no longer transient.

For the latter experiment, we vary the number of low-frequency classes $|C_{L}|\in\{5,45,95,495\}$, and train all models for 500k gradient steps.
Similar to the findings in Chan et al. () and Reddy (), we observe that with increasing $|C_{L}|$ the transformer begins to exhibit ICL for $C_{L}$ while only performs IWL for $C_{H}$, which again indicates that a model can exhibit both ICL and IWL (Figure 10).
Further observing the IBD errors of IW and IC predictors on $C_{L}$, we can see that the transformer transitions from ICL to IWL when IW predictor begins to exhibit lower errors than the IC predictor, aligning to our theoretical analysis.

We now investigate the impact of varying probability of sampling high-frequency classes $p_{high}$.
We vary $p_{high}\in\{0.5,0.9,0.99\}$.
The model exhibits ICL for slightly larger $N$ with balanced classes, compared to imbalanced classes (Figure 11, left).
Looking at the IBD errors between IW and IC predictors, IW starts with higher error and reduces to near-zero error as $N$ increases, resulting in a better predictor than IC predictor on in-base distribution.
This crossover occurs later on balanced classes as each class has effectively similar amount of samples, while $C_{H}$ will be observed more by the transformer on imbalanced classes.
The IW predictor will converge faster on $C_{H}$ in the latter case and the transformer loses its ICL capability sooner.

Finally, we explore how label noise can impact the emergence of ICL, a setting that is not explicitly explored by existing work.
We remove input noise (i.e. sampling only the prototype vector as the input) and vary the label noise $p_{label}\in\{0.001,0.01,0.1\}$.
The noisy label is implemented such that with probability $1-p_{label}$ we keep the original class $c$, otherwise we use $(c+1)\mod\lvert C\rvert$.
In this case the IW predictor achieves lower IBD error than IC predictor for small label noise, as a result the transformer exhibits minimal ICL capability (Figure 11, right).
When $p_{label}=0.1$ IC predictor is better than IW predictor.
Once again our theory aligns with the experimental result whereby the transformer exhibits ICL until the errors of IC and IW predictors match.

#### B.1.2 Parameters in context-level distribution

We now investigate how the parameters for the context distribution can impact ICL—we consider three parameters:
the probability of sampling relevant contexts $p_{relevant}$, the context length $L$, and the number of relevant contexts $L_{relevant}$.
Adjusting $p_{relevant}$ can be considered as changing the probability of observing bursty contexts in Chan et al. ().
We vary $p_{relevant}\in\{0.1,0.5,0.9,0.99\}$ in this experiment.
Aligned with previous experiments and our theoretical analysis, the transformer exhibits stronger ICL when the IC predictor achieves smaller error than the IW predictor (Figure 12).
As expected, when $p_{relevant}$ is small we see IC predictor to achieve high IBD error.
Increasing $p_{relevant}$ results in more accurate IC predictor, but IW predictor can eventually outperform the IC predictor, resulting in the transient of ICL in the transformer.

In NLP tasks the model is usually given multiple contexts in a sequence, some potentially act as distractors.
From Figure 6 from Section 3.1 we can see that as we increase the number of contexts, even if we provide always relevant context, the IC predictor ends up exhibiting IWL capability.
This suggests the difficulty of learning ICL when variable relevant contexts are provided.
We investigate in detail on whether the number of relevant examples impact the emergence of ICL.
We fix $L=4$ and vary the number of relevant examples $L_{relevant}\in\{1,2,3,4\}$.
When $L_{relevant}=4$ the IC predictor achieves smaller IBD error than the IW predictor initially but the latter eventually achieves similar error with larger $N$ (Figure 13).
However it is worth noting that with decreasing $L_{relevant}$ the initial IBD error of IC predictor increases, almost reaching the same error when $L_{relevant}=1$.
In that case the transformer never exhibits ICL.
Indeed, as we decrease the number of relevant contexts, the transformer will have to be very precise in identifying the sole relevant context in addition to copying its label, which can be more challenging that learning just the IW predictor.

Figure: Figure 11: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary the probability of high-frequency classes (left) and the label noise (right) on synthetic data, over five seeds. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Refer to caption: x21.png

Figure: Figure 12: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary the probability of sampling relevant contexts $p_{relevant}$ on synthetic data, over five seeds. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Refer to caption: x23.png

Figure: Figure 13: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary the number of low-frequency classes $L_{relevant}\in\{1,2,3,4\}$ on synthetic data, over five seeds. $L=4,p_{high}=0.9,$ and $p_{relevant}=0.9$ for training the transformer. For each variant, the top and bottom rows respectively correspond to IBD and OOBD errors.
Refer to caption: x24.png

### B.2 Visualizing attention

Figure: (a) $N=2^{10}$
Refer to caption: x28.png

We provide visualization on the attentions of the transformer trained with $N\in\{2^{10},2^{20}\}$ samples in Figure 14 on a 20 OOBD samples conditioned on relevant contexts.
When $N=2^{10}$ the first attention layer focuses on the context label (i.e., the bottom middle of the $3\times 3$ grid), whereas when $N=2^{20}$ the first attention layer focuses on the query (i.e., the bottom right of the $3\times 3$ grid).
The former predicts the OOBD labels, exhibiting ICL, while the latter predicts the IBD labels, exhibiting IWL.

### B.3 Omniglot

Figure: Figure 15: The validation 0-1 error of IC predictor, IW predictor, and transformer predictor as we vary input noise $\sigma\in\{0.0,0.1,1.0\}$ on Omniglot, over three seeds. We set $L=2,p_{high}=0.9$ and set $p_{relevant}=0.9$ for training the transformer. For each variant, the top and bottom rows respectively correspond to in-base distribution and out-of-base distribution data.
Refer to caption: x30.png

Figure: Figure 16: The top row is the 0-1 error of the transformer as we perform gradient updates on Omniglot. The bottom row is the total occurrence of each low-frequency class during training. The shaded region corresponds to max./min. occurrence. We set $L=2,\sigma=0,p_{high}=0.9$ and set $p_{relevant}=0.9$ for training the transformer. Each column corresponds to a specific seed. We observe that the transformer consistently learns to correctly predict $C_{H}$ quickly through IWL. The transformer also exhibits ICL capabilities on $C_{L}$ initially but ICL eventually diminishes.
Refer to caption: x33.png

Figure: Figure 17: The ICL evaluation 0-1 errors with heldout images as we increase the number of samples $N$ on Omniglot, over three seeds. Regardless of the input noise, the transformer performs ICL initially but gradually performs more IWL as $N$ increases.
Refer to caption: x35.png

In this section we provide further analyses on the Omniglot experiments.
First, from Figure 7 in Section 3.2 we observe that the transformer never exhibits the ICL capability on $C_{H}$.
We further visualize the errors of IC and IW predictors in Figure 15 and notice that their IBD errors are almost identical on $C_{H}$ samples, with or without relevant contexts, suggesting that the IC predictor might have ended up learning IWL.
Observing the IBD errors on $C_{L}$ samples, the IC predictor exhibits ICL.
Similar to the synthetic setting, in this case there is a cross-over with small $\sigma$, whereby the IW predictor initially exhibits higher IBD error but eventually overtakes the IC predictor.
Once again we see the transient of ICL in this scenario.
The scenario where $\sigma=1.0$ is the most surprising—even though the IW predictor never achieves lower IBD error than the IC predictor, the transformer still loses ICL with increasing $N$.

While our result is related to the transient of ICL with increasing dataset sizes, we further demonstrate the transient of ICL with larger number of gradient updates.
Figure 16, top, demonstrates that the transformer consistently learns ICL initially on samples from $C_{L}$ but always eventually loses such capability.
On the other hand the transformer never learns ICL on samples from $C_{H}$.
We also include the occurrence of low-frequency samples from $C_{L}$ over training (Figure 16, bottom).
On average, each class from $C_{L}$ will be observed around 200 times after 100K iterations, while each class from $C_{H}$ will be observed similar amount of times after only 200 iterations.
This suggests ICL, if it does appear, will diminish after 200 iterations on samples from $C_{H}$, hence we do not observe any ICL in Figure 16, top.

Finally, we evaluate the trained transformers on heldout images that are never seen in training.
In this experiment we fix $L=2$, vary $\sigma\in\{0.0,0.1,1.0\}$ and only provide one of 20 images per class in Omniglot for training the transformer.
Similar to the evaluation described in Section 3 we shift the original class label by one so the transformer cannot use predict through IWL.
Since $L=2$, the transformer will need to identify the relevant context and the query through the inputs and copy its label.
Figure 17 shows that even with no noise, the transformer is able to generalize to unseen images and perform ICL on $C_{L}$.

## Appendix C Data and prompts for large language model experiments

Here we provide the finetuning data (Table 1), and the evaluation prompts and results (Tables 2 and 3) mentioned in Section 4.
Below we provide extra analyses on different prompts.

**Table 1: The training data for finetuning the Gemini Nano 1 model. The prompts contain four real and four invented (random) names and city names, each, and force the model to perform IWL as it gives zero information regarding where the person resides in.**
| Prompt | Answer |
| --- | --- |
| Where does Cxx1h live? Only give the name of the city. | Lkkl |
| Where does Vzbiik live? Only give the name of the city. | Plooqujhd |
| Where does Mmojkr live? Only give the name of the city. | Nwops |
| Where does Trrrrqe live? Only give the name of the city. | Qtbnaaa |
| Where does Benjamin live? Only give the name of the city. | Buenos Aires |
| Where does Liz live? Only give the name of the city. | London |
| Where does Kaitlyn live? Only give the name of the city. | Kingston |
| Where does Wiesa live? Only give the name of the city. | Warsaw |

**Table 2: The question-answer pairs and the corresponding predictions of the finetuned and the base models for the finetuning dataset, also showing the relative log-probability of possible answers. The shaded rows are the finetuned model predictions and the unshaded rows are the base model predictions. In this scenario the models must use IWL to predict the correct answer.**
| Model | Question | Answer |
| --- | --- | --- |
| Finetuned | Where does Cxx1h live? Only give the name of the city. | Lkkl |
|  | Lkkl -0.2666, Anytown, CA -55.3096 |  |
| Base | Where does Cxx1h live? Only give the name of the city. | Anytown, CA |
|  | Lkkl -40.5117, Anytown, CA -2.4959 |  |
| Finetuned | Where does Vzbiik live? Only give the name of the city. | Bratislava |
|  | Plooqujhd -45.9387, Bratislava -0.0659 |  |
| Base | Where does Vzbiik live? Only give the name of the city. | Bratislava |
|  | Plooqujhd -66.2485, Bratislava -1.3778 |  |
| Finetuned | Where does Mmojkr live? Only give the name of the city. | Milwaukee |
|  | Nwops -6.0057, Milwaukee -0.5176, Los Angeles -12.1351 |  |
| Base | Where does Mmojkr live? Only give the name of the city. | Los Angeles |
|  | Nwops -32.8910, Milwaukee -4.8689, Los Angeles -1.9404 |  |
| Finetuned | Where does Trrrrqe live? Only give the name of the city. | Qtbnaaa |
|  | Qtbnaaa -0.0136, Toronto -9.0857 |  |
| Base | Where does Trrrrqe live? Only give the name of the city. | Toronto |
|  | Qtbnaaa -50.5695, Toronto -1.0771 |  |
| Finetuned | Where does Benjamin live? Only give the name of the city. | Buenos Aires |
|  | Buenos Aires 0.0000, New York City -43.2574 |  |
| Base | Where does Benjamin live? Only give the name of the city. | New York City |
|  | Buenos Aires -16.4484, New York City -0.7329 |  |
| Finetuned | Where does Liz live? Only give the name of the city. | London |
|  | London 0.0000, Los Angeles -35.3694 |  |
| Base | Where does Liz live? Only give the name of the city. | Los Angeles |
|  | London -2.6441, Los Angeles -0.2607 |  |
| Finetuned | Where does Kaitlyn live? Only give the name of the city. | Kingston |
|  | Kingston 0.0000, Seattle -45.2526 |  |
| Base | Where does Kaitlyn live? Only give the name of the city. | Seattle |
|  | Kingston -7.0968, Seattle -0.6796 |  |
| Finetuned | Where does Wiesa live? Only give the name of the city. | Warsaw |
|  | Warsaw -0.0000, Frankfurt am Main -54.8158 |  |
| Base | Where does Wiesa live? Only give the name of the city. | Frankfurt am Main |
|  | Warsaw -3.0168, Frankfurt am Main -1.3085 |  |

**Table 3: The question-answer pairs and the corresponding predictions of the finetuned and the base models for questions designed to test ICL capabilities, also showing the relative log-probability of possible answers. The shaded rows are the finetuned model predictions and the unshaded rows are the base model predictions. In this scenario the models must use ICL to predict the correct answer.**
| Model | Question | Answer |
| --- | --- | --- |
| Finetuned | Cxx1h lives in Kjheergg. Where does Cxx1h live? Only give the name of the city. | Kxxh |
|  | Kjheergg -1.8326, Lkkl -7.9295, Kxxh -0.4498 |  |
| Base | Cxx1h lives in Kjheergg. Where does Cxx1h live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0000, Lkkl -45.5691, Kxxh -32.7908 |  |
| Finetuned | Vzbiik lives in Kjheergg. Where does Vzbiik live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.2945, Plooqujhd -47.7992 |  |
| Base | Vzbiik lives in Kjheergg. Where does Vzbiik live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0002, Plooqujhd -62.2570 |  |
| Finetuned | Mmojkr lives in Kjheergg. Where does Mmojkr live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0382, Nwops -18.9681 |  |
| Base | Mmojkr lives in Kjheergg. Where does Mmojkr live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0001, Nwops -47.8115 |  |
| Finetuned | Trrrrqe lives in Kjheergg. Where does Trrrrqe live? Only give the name of the city. | Qtbnaaa |
|  | Kjheergg -21.6897, Qtbnaaa 0.0000 |  |
| Base | Trrrrqe lives in Kjheergg. Where does Trrrrqe live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0003, Qtbnaaa -52.3584 |  |
| Finetuned | Benjamin lives in Kjheergg. Where does Benjamin live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0367, Buenos Aires -25.9128 |  |
| Base | Benjamin lives in Kjheergg. Where does Benjamin live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0001, Buenos Aires -33.1918 |  |
| Finetuned | Liz lives in Kjheergg. Where does Liz live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0765, London -18.2949 |  |
| Base | Liz lives in Kjheergg. Where does Liz live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0000, London -24.7004 |  |
| Finetuned | Kaitlyn lives in Kjheergg. Where does Kaitlyn live? Only give the name of the city. | Kingston |
|  | Kjheergg -2.4694, Kingston -0.1156 |  |
| Base | Kaitlyn lives in Kjheergg. Where does Kaitlyn live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0001, Kingston -25.7090 |  |
| Finetuned | Wiesa lives in Kjheergg. Where does Wiesa live? Only give the name of the city. | Kjee |
|  | Kjheergg -1.5144, Warsaw -15.2768, Kjee -0.9055 |  |
| Base | Wiesa lives in Kjheergg. Where does Wiesa live? Only give the name of the city. | Kjheergg |
|  | Kjheergg -0.0001, Warsaw -33.9379, Kjee -15.4231 |  |

### C.1 Additional discussion of experiments

For a well-known city like *Toronto*, the predictions are almost always in-context; Wiesa was correctly predicted to live in Warsaw. Adding some extra information to the prompt, such as *Liz lives in the city of Toronto. The weather is fantastic. Where does Liz live? Only give the name of the city.*, the probabilities of the correct answers are drastically reduced.
Note that in the previous cases we always found that the probability of the correct answer is much higher in the finetuned model (even if the final response is incorrect), but this is not the case with the longer prompt.
The results are similar if *Kjheergg* is used in the long prompt, though the correct prediction for (Trrrrqe, Qtbnaaa) remains.
Using *Hajduszoboszlo* (a small town in Hungary, surely not in the training set many times) as the decoy city in the longer prompt results in ICL, although the finetuned model twice responded with Budapest, showing that the predicted answer is really the most likely Hungarian city. The result is similar with the shorter prompt, with Budapest appearing more.
Changing the decoy to England for the short prompt results in all but one London responses, showing that England and city in the context are associated with London with high likelyhood, with the finetuned model correctly predicting (Wiesa, Warsaw).

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: A recent work[1] shows that ICL exhibits dual operating modes: task retrieval and task learning. As the number of correctly labeled examples grows, the model transitions from task retrieval to task learning, but initially, even with correct labels, it may retrieve the wrong task, leading to errors. Wondering how does your finding explain this dual-mode behavior?

[1]Dual Operating Modes of In-Context Learning. https://arxiv.org/abs/2402.18819
soundness: 2
strengths: 1. The paper introduces a novel theoretical framework that investigates how a model selects between ICL and IWL. This framework offers insights into the conditions under which ICL emerges, especially highlighting the role of rare labels in promoting ICL.
2. The writing and presentation is clear.
3. this paper includes experiments in both synthetic and natural few-shot learning dataset and validated their findings.
summary: This paper provides a theotical framework which studies in-context learning's emgerence and transient nature through a simple model which can adapatively switching between an in-weight predictor and an in-context predictor. The key insights from this paper shows that when the training data has long-tail, containg rare classes that appear infrequently, ICL tends to emerge. And if there are enough training data, IWL tends to dominate. Their experiments on both synthetic data and Ominiglot daat shows that ICL exhibit for rare classes and IWL for common ones.
weaknesses: 1. This paper assumes a simple switch between the IWL and ICL based on error minimization but in real world LLM the transition may not be instantaneous and involves more complexities. 
2. The in-context predictor is modeled as a convex combination of the labels in the context, weighted by input similarity. This raises questions because more commonly the in-context predictor should learn the relationship between inputs and outputs (i.e., the mapping from x to y rather than combining labels. This simplification might limit the applicability of the theoretical findings to real-world scenarios where the input-output relations are more complex.
3. The experiment with the real LLM, Gemini Nano 1 seems somewhat limited. It primarily tests the LLM's ability to follow in-context labels versus trained labels, which may be influenced by how different LLMs are instructed to trust new labels versus their trained knowledge. A more comprehensive and persuasive experiments could include various LLMs to assess the generality of the findings and provide deeper insights across different architectures and scales.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: The main theoretical results seems to be that “ICL will first emerge on rare classes, and will eventually be replaced by the IW predictor on those classes” (line 207,208). This prediction is studied experimentally in the setting described in Figure 2, where three models are trained for 100K gradient steps (Appendix B.1.1) across a number of sample sizes N. The three models are all trained with a single (x,y) pair in the context (L = 1) and for the IW predictor the context is always “relevant” (so if I understand correctly the correct output given (x,y), x’ is to simply ignore x’ and copy the input y to the output) and for the IC predictor the context is always “irrelevant” (so given (x,y), x’ the correct output cannot be learned from the context, which should be ignored). The main transformer is trained to have 90% of its contexts relevant.

My reading of Fig 2 is that on the rare classes where IC is possible (second column, top row) the transformer behaves very similarly to the IC predictor from the beginning, and only converges to the IW predictor when the IC predictor does. The behavior on high-frequency classes seems qualitatively similar, and so I don’t see on what basis this is evidence for the “first emerge” part of the claim. Nor do I see how these experiments really support the idea that for large N the transformer switches to the IW predictor. A simpler explanation for this experimental data seems to be that, by construction, the transformer should be somewhere between the IC and IW predictor since its training distribution is between theirs, and for large N the IW and IC predictor behave similarly on relevant contexts.

It seems therefore that the crux of the argument might lie in the OOBD plots (Fig 2 second row). My reading here is that on the low-frequency classes OOBD performance of the general transformer is initially bad, improves, and then degrades, whereas for high-frequency classes the OOBD performance is initially roughly as good as it ever gets on the low-frequency classes, and then degrades. I do not see here the evidence for “ICL emerges first on rare classes”? The degradation of OOBD performance also seems to be much slower on the low-frequency classes than the high-frequency classes, which makes me unsure if the OOBD data shows any evidence for the “replaced by the IW predictor on those classes” claim either.

In short, I’m not sure how to read Fig 2 as evidence for the theoretical claims. This leaves me unsure of the soundness of the main experimental contribution of the paper.

Minor issues:

- “Cdot” line 149
- “Transient” -> “transience” line 303
- It’s a bit strange to see H defined twice in quick succession (differently) in 130 and 158

Factors that would improve my score:

- As already mentioned, clarifying the link between Fig 2 and the theoretical prediction.
- Compared to the other literature in this area the models are rather small (two layers, one head per layer) and the contexts are very short (e.g. L = 1 in Fig 2). Overall I am left with the sense that maybe these models are just not capable of learning the ICL solution the theory is talking about, and this could be confounding some of the other aspects of the paper. This is actually how I read Fig 6. To what extent is the choice of L = 1 to illustrate something theoretically, and to what extent is it just “the only length that works” as a result of the small model?
soundness: 3
strengths: - Reasonable theoretical foundation for their arguments
- Well-designed experiments to test these theoretical predictions

Overall I think this has the makings of a good paper that makes a significant and original contribution to an important thread of research in the science of transformer generalization.
summary: [update: in response to the revision and author responses I have updated my score]

The study of in-context learning (ICL) has emerged in recent years as a tractable and yet important piece of the puzzle for understanding how transformers are able to achieve such good performance across such a wide range of data distributions. There is now a growing literature combining theoretical models with empirical study of ICL in synthetic settings, with the aim of understanding the mechanism by which ICL works, and why it emerges in the first place. 

The present paper adds to this literature, by studying the tradeoff between memorisation (or in-weight learning) and ICL in a simple theoretical model and associated experiments, and how this tradeoff varies with the nature of the data distribution.
weaknesses: - At the moment I am not convinced that the experiments reported in Fig 2 really do support the theoretical predictions. I hope this is just my misunderstanding and that the authors can explain what I missed, and perhaps provide a clearer explanation in the paper.
- While I am pretty well-versed in this corner of the literature and so prepared to care about the contents of Fig 2, it still took me a while to process all the acronyms and setup.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Theory:

1. What parameterisation of $\alpha$ do you have in mind? Simply to learn a
   value for every $\tilde x$ (i.e., again tabular)?
   * If so, how can you expect it to know to route to the IC model when you
     haven't seen enough of an example for $g$ to be useful? Is it supposed to
     be initialised as biased towards routing to $h$?

2. In Proposition 1 (for example), over which distribution are the
   expectations of $y$ taken? Is $y \sim y^\ast(x)$?

3. In Proposition 2 and Corollary 1, if I am not mistaken, your bounds apply
   to a given sample of the expected risk, and they depend on the sample. For
   example, in corollary 1, you bound $\mathrm{CE}(h(\tilde x), y)$ for a
   particular $\tilde x$ and $y$. Since you later compare these bounds to the
   IW upper bound which is of the expected risk, could you comment on the
   relationship between these bounds and the expected risk for the IC learner?

4. In Corollary 1, I was confused by the line "Assume the labels $y^\ast$ are
   one-hot (deterministic)".
   * If I understand correctly, in the data generating process, the concrete
     *labels* $y$ in any given example are *already* one-hot, as stated on
     lines 120--121 "When we sample $x$ in the context of as the query, we
     sample $y=e_i$, the $i$-th standard basis vector, ..."
   * I thought, possibly you meant that the ground truth labelling *function*
     $y^\ast$ is assumed to be such that the ground truth label $y^\ast(x)$
     is deterministic? But this seems like too strong of an assumption, since
     you go on to talk about IWL being better with enough samples *if* the
     variance is sufficiently low, and if you were assuming the variance was
     zero for your ICL bound then it wouldn't make sense to say this.
   * In fact, the assumption that $y$ is one-hot appears to be used in the
     proof for proposition 2, whereas no additional assumption on $y$ or
     $y^\ast$ seems to be used in the proof of corollary 1.
   * So, I think this assumption is redundant. Is that correct?

5. In Corollary 1, you give both a lower bound in terms of $k$ and also an
   exact value in the case where $k=L$. There is no restriction on $k$ in the
   general case and so I assumed that if I put $k=L$ the lower bound
   should lower bound the exact value. However, if I am not mistaken, if $k=L$
   then the lower bound simplifies to $1-\epsilon$, which is larger than the
   exact value $-\log(\epsilon)$ for some values of $\epsilon$. Is there a
   mistake in the lower bound?

6. In the figure 1 caption, what is the "Big-O constant of the IW risk"? I saw
   no asymptotic analysis in the IW bound.

Notation and terminology:

1. Corollary 1: I invite you to consider using a roman font to denote cross
   entropy loss (as in "$\mathrm{CE}$") so as to prevent lexical ambiguity
   with the variable $C$ and an undefined variable $E$ for readers parsing
   the result statement.

Typos:

1. There are two separate reference list entries for Edelman et al. (2024).
2. There are two separate reference list entries for Zhang et al. (2024).
3. Line 90 "attention attention"
4. Line 148 "cdot" presumably should be "\cdot"
5. Line 304 "transient of ICL"
6. Line 345: It says "A generic model trained on data with $p_{high}$ is
   referred to as the *transformer.*" Is that meant to be $p_{relevant}$?
7. The proof of Proposition 1 uses notation $R_x$ for expected risk without
   having introduced this notation anywhere (as far as I can see).
soundness: 4
strengths: The paper addresses an important topic in the science of deep learning,
namely the phenomena of the emergence and transience of in-context learning
(versus in-weight learning).

On this topic the paper presents an interesting learning setting involving an
in-context supervised classification task that captures several important
elements of in-context learning. At the same time, this setting is simpler in
some ways than in-context linear regression or other comparable synthetic
learning settings.

I think the overarching story developed in the paper (roughly, in my
understanding, that in-context learning arises as a heuristic that can reduce
loss in areas of the input distribution where in-weight learning has not
gathered enough experience to provide a specialised and more accurate
prediction) is an interesting and thought-provoking perspective.

I think the three-part model is an interesting and thought-provoking toy
model of in-context learning versus in-weight learning, and I think the
theoretical analysis is informative and supports the aforementioned
overarching story.
summary: Summary of the paper:

* The paper defines a sequential, tabular supervised classification task as a
  simplified setting in which to study the dynamics of in-context vs.
  in-weight learning.
* The paper considers three models in this setting:
  1. An 'in-weight learner' model capable of predicting labels based only on
     the most recent query.
  2. An 'in-context learner' model that predicts labels based instead on a
     sequence-dependent weighted combination of previous labels appearing in
     the sequence (specifically, the weights are given by a softmax
     distribution based on similarity to the previous queries).
  3. A contextual mixture model that can (learn to) choose based on the input
     sequence whether to predict according to the first model or the second
     model, or interpolate between the two.
* The paper describes a theoretical analysis of the performance of these
  three models in the synthetic task:
  * Deriving an upper bound on the expected test loss of the in-weight
    learner.
  * Deriving an upper and lower bound on the loss for a particular input of
    the in-context learner.
  * Giving a family of algorithms for learning all three models and bounding
    the regret of the combination model in terms of the regret of the
    learning algorithms used for learning some of the components.
* Based on the nature of these bounds, the paper argues that this model
  captures phenomena observed in prior empirical work on in-context learning
  and in-weight learning, namely that rare classes drive the emergence of
  in-context learning (Chan et al.) and that ICL disappears after enough
  training (Singh et al.).
* The paper also includes some experiments with small transformers on
  synthetic data and natural language data and comparisons of the theoretical
  model's qualitative predictions to the resulting empirical observations.
* There is also a small qualitative study of a LLM's dependence on knowledge
  from fine-tuning versus knowledge from its context.

I wrote a somewhat long review. Therefore, I also include a summary of my
review as follows.

* I think this paper presents some interesting ideas and results. For
  example, I like the sequential supervised classification task and I like
  the three-part model.
* However, I find the results do not achieve the paper's stated aims of
  capturing empirical phenomena to the degree claimed, due to what appear to
  me to be several issues, namely:
  * fundamental mismatches between the theoretical framework and results and
    the phenomena studied in prior work,
  * some limitations in the theoretical results that do not appear to have
    been acknowledged, and
  * a lack of connection between the theoretical results and the experiments.
* Overall, I consider the framing of the paper to be inaccurate and therefore
  in my judgement the paper should not be accepted in its current form,
  though I am certainly open to discussion and revisions to the framing to
  acknowledge and address these gaps as I think the work does have value.
* I also noted several questions and minor corrections, mainly regarding the
  theoretical sections.

**EDIT TO ADD:** Summary of discussion.

* The authors addressed my concerns about the framing through their responses and revisions:
  * They refined their claims about connections to prior work, resolving my concern about the mismatch between their theory and prior work.
  * They strengthened their theoretical results and acknowledged the remaining limitations, resolving my concern about the limitations of their theory.
  * They clarified the connection between their theory and experiments and added new synthetic experiments with a more direct connection, resolving my concern about the connection between their theory and experiments.
* With the authors having resolved all of my concerns, I am now pleased to be able to recommend the paper for acceptance, and raised my rating from 5 to 8.
weaknesses: **W1. Mismatch with Singh et al.'s setting.**
In the introduction you write that your analysis "explains the findings of
Singh et al. (2023)," namely the transience of in-context learning, giving
way to in-weight learning. It doesn't seem to me that your model captures the
phenomenon observed by Singh et al. In my understanding, a fundamental
feature of the setting of Singh et al. is that there is no train-loss
incentive for an ICL solution over an IWL solution, because both paradigms
are equally capable of solving the task (yet, they observe, ICL still emerges
and still gives way to IWL in the end). In contrast, in your model, as you
show, based on your formulation of ICL and IWL there is a performance lower
bound to ICL which the upper bound on IWL performance is eventually able to
surpass.


**W2. Incomplete characterisation of the emergence of ICL for rare classes.**
On lines 207--209, you write "Under this [oracle] algorithm, ICL will first
emerge on rare classes ... This phenomenon is consistent with the empirical
observation that in-context learning emerges with rare classes in the
training data...".

It appears that this characterisation of the oracle algorithm is based on the
observation that, for some specific values, the IW upper bound starts higher
than the IC upper bound (figure 1), suggesting that in these cases the simple
model will use the IC sub-model rather than IW sub-model. However, this is
not implied by the theory as the IW bound is merely an upper bound on the IW
risk (I also note in a question below that the IC bounds are not of the IC
risk but of a particular sample of the risk).

I believe a more accurate characterisation would require the stronger
assumption that the IW risk happens to be higher than the IC risk for the
rare classes in question. This arrangement is permitted but not implied by
the comparison between upper bounds. It would be implied if you had a lower
bound on the IW risk and this was higher than an upper bound on the IC risk.
I am unsure whether such a bound is plausible. In my opinion this gap between
the model and the phenomenon should be noted explicitly as a limitation of
the analysis.

(Note that this issue is not localised to lines 207--209, but that appears to
be the most explicit discussion of the comparison between the phenomenon and
the theoretical framework. The favourable result of the comparison is a theme
throughout the rest of the paper.)


**W3. Incompletely addressed limitation of tabular analysis.**
There are several details in the theoretical framework and analysis that
assume a tabular setting. In footnote 1 you claim that the results can be
easily be extended to a continuous input space. It is not clear to me that
this extension would be easy, since, beyond (yes) easily extending the
definition of the task and the models, the bounds and the proofs seem to be
tightly coupled to the tabular setting. It is unclear to me whether
qualitatively similar bounds could be achieved in a continuous input setting.
On what basis have you made this assertion?


**W4. Specific model of in-context learning appears quite restrictive.**
The paper talks generally about in-context learning in transformers, but for
the three-part model your formulation of in-context learning appears
surprisingly restrictive, namely at first limiting the prediction to a convex
combination of context labels and then by the relevant theorem further
limiting to the specific modelling choice of using a softmax distribution
based on query similarity for weighting the labels.

It seems to me that mechanisms for in-context learning in a transformer are
unlikely to be limited in this way, and the ability to use alternative
mechanisms would generally serve to broaden the model class under
consideration and therefore put your performance lower bounds into question.
In other words, these restrictions on the ICL sub-model seem to be the source
of the conclusion that IWL is eventually preferable to ICL, which does not so
much answer the question of why IWL is eventually preferred as raise the
question of why we should believe that ICL is limited in this specific way.

**W5. Gap between theory and experimental study.**
The experimental methodology involves comparing one transformer's performance
with that of two other 'baseline' transformers, one trained on data that is
supposed to incentivise ICL and another trained on data that is supposed to
incentivise IWL. The comparisons between the 'middle' transformer and these
baselines are then used to corroborate the predictions of the theory.
However, the use of pretrained transformers as baselines creates a confound
that the 'IWL transformer' may not resemble the IWL model in the theory and
(even more plausibly) the 'ICL transformer' may not resemble the ICL model in
the theory. This would potentially undermine the connection between the
theory and the experimental results (and, in turn, the relevance of the
theory). This gap appears not to have been acknowledged in the paper.

Given that you have details models of these learners and have posited
specific training algorithms capable of achieving your regret bounds, and in
the synthetic data experiments your set up seems to match the theoretical
assumptions (apart from the tabular input space, though presumably the
synthetic data setting could be modified to satisfy this assumption along
with any others I have overlooked), it seems like it would have been possible
to implement your theoretical models directly and compare the performance of
these models to the baseline transformer (or the middle transformer).


**W6. Unclear relevance and novelty of LLM experiments.**
The LLM experiments are interesting in their own right, however I came away
feeling unimpressed for several reasons:

1. I felt you could have done more to connect these experiments and their
   results to the theoretical framework developed in section 2. At the moment
   I don't see what the LLM experiments say about the theory and I don't see
   what the theory says about the LLM experiments.

2. The total number of examples tested is small (if my understanding is
   correct and all examples are listed in the appendices). No statistical
   analysis has been performed on any specific questions about the results to
   see if the trends reported are statistically significant.

3. Though it is not my area, I have the feeling that there exists at least
   a small number of papers having already studied the topic of the
   competition between knowledge from fine-tuning and knowledge from contexts
   in determining the predictions of LLMs. (I regret I have not been able to
   locate any examples, but one that comes somewhat close is Wei et al.,
   2023, arXiv:2303.03846, which is actually cited in passing in the
   introduction but is not discussed in any detail). I would have liked to
   see you contextualise their experiment and results within this literature.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 1
questions: Questions
1. How could the real vs invented (name, city) pair result be explained? If the model is being fine-tuned, why can’t it memorize invented (name, city) pairs? Is there an intuition or is this simply the case?
2. How applicable is the theory to real LLMs?

Suggestions:
1. The current figures were quite confusing for me, the axis and plot labeling constantly changed through Fig 2,3,7,8. It would be great if the plot colors/style can be improved so that they can be better understood.
2. I think Figure 16 is main-text material, perhaps more so than Figure 8.
soundness: 3
strengths: Overall, the paper’s motivation is clear and the flow is consistent. Here is a list of strengths:

S1: The paper connects an observed phenomenon (ICL’s transience into IWL) with a synthetic model and theory.

S2: The intuitive interpretation that new training samples are effectively test samples under the assumptions made in Sec. 2 is useful.

S3: This work clearly demonstrates how noise level and the fraction of rare tasks can affect the ICL versus IWL.

S4: Unlike concurrent work exploring similar phenomena with similar methodology (https://openreview.net/forum?id=XgH1wfHSX8, https://openreview.net/forum?id=INyi7qUdjZ, https://openreview.net/forum?id=LbceJJc9h2), this work provides experiments with real language models. However see below for some criticism as well. Despite the criticism, the investigation of how well findings on synthetic data generalize to real models is valuable.
summary: This paper tries to understand how trained Transformers choose between in-weights learning (IWL) and in-context learning (ICL). The paper is well grounded in recent literature studying the contrast of phenomenology and inference time behavior of IWL and ICL, especially with Singh et al. 2024’s finding that ICL can be transient. The paper first sets up a mathematical framework where vectors are input and noisy labels are used in a classification setup. The paper then discusses analytic predictors using ICL and IWL, and explains why IWL is asymptotically superior and why a neural network can find the optimal solution by gradient descent. The paper conducts experiments comparing a transformer’s validation loss to the ICL and IWL predictors to identify that noise level and fraction of rare tasks are critical variables deciding ICL/IWL. Finally, the paper shows, by fine tuning Gemini Nano 1, that fine tuning can induce loss of ICL abilities in real LLMs
weaknesses: The paper has some weaknesses, listed below:

W1: Theory seems to rely on assumptions not met in realistic applications of ICL. The theory assumes that the data are drawn i.i.d, while this is usually not met in many realistic applications of ICL. Thus it is quite unclear whether new training samples can safely assumed as test samples in real LLM training

W2: The setup seems to perform a classification task where x,y are given in pairs. However, this *might* be not a good model of ICL in real LLMs as it does not involve a complex sequence-space computational structure. Recent works, e.g. Akurek et al. 2024 (https://arxiv.org/abs/2401.12973), seems to model the sequence space nature of ICL better. It seems like this seems important to understand ICL \textit{in Transformers} as Tong et al. 2024 (https://arxiv.org/abs/2405.15618 ) points out that MLPs can learn in context.

W3: I was not able to draw these conclusions in line 484 from the given Figure 7: “in line with our theory, the IBD error of the “selected” IC learner is significantly lower for low-frequency classes. Furthermore, both the IC and IW predictors achieve similar IBD errors on CH, suggesting that IWL more easily emerges on common classes even with larger input noise (Figure 7, right).” It would be great if this can be explained further

W4: While I agree with most conclusions drawn from the plots, it would be better to compute explicitly a ICL_vs_IWL ratio as a metric since currently it seems like these conclusions are drawn by matching the experiment curves with predictors by eye.

W5: It is intuitively hard to grasp the experiment. This is solely a presentation issue, it would be very helpful if there was a schematic diagram which describes all of $p_{relevant}$, $p_{high}$, OOBD, IBD, IC model, IW model, etc. The figures could also be improved, as mentioned below in questions.

W6: It is unclear how well the real LLM experiments justify the claims in the main text and the claim in Singh et al. 2024 that ICL can be transient *during pretraining*. The current experiments seem to suggest that fine-tuning can override ICL abilities, but it is very probable that the ability itself is still there while it is simply suppressed superficially. (See Jain et al 2023 (https://arxiv.org/abs/2311.12786, https://arxiv.org/abs/2410.1653 ). This would not be corresponding to the claim in the main text and in Singh et al. 2024, which I believe discusses a pre-training stage transience requiring orders of magnitude more steps than fine tuning.

Meta-review: This paper investigates how transformers balance between in-context learning (ICL) and in-weight learning (IWL) capabilities. The key findings are that ICL emerges when IWL/pretraining has insufficient data, particularly for rare classes, and that ICL can become transient as more training data enables better IWL performance, at which point the LLM will ignore its context. The paper develops theoretical analysis through a simplified model and validates the findings through experiments on synthetic data, Omniglot, and a real LLM. The reviewers appreciated the theoretical framework and controlled experimental validation. The weaknesses were limited applicability of the theory and experiments to real-world LLMs due to certain simplifying assumptions (eg, mixture model). Overall, the reviewers were in agreement to accept the paper.

Decision: Accept (Poster)