Title: The Transient Nature of Emergent In-Context Learning in Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: in-context learning

transformers

emergence

transience
Abstract: Transformer neural networks can exhibit a surprising capacity for in-context learning (ICL) despite not being explicitly trained for it.  Prior work has provided a deeper understanding of how ICL emerges in transformers, e.g. through the lens of mechanistic interpretability, Bayesian inference, or by examining the distributional properties of training data. However, in each of these cases, ICL is treated largely as a persistent phenomenon; namely, once ICL emerges, it is assumed to persist asymptotically. Here, we show that the emergence of ICL during transformer training is, in fact, often transient. We train transformers on synthetic data designed so that both ICL and in-weights learning (IWL) strategies can lead to correct predictions. We find that ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases, indicating an asymptotic preference for IWL. The transient nature of ICL is observed in transformers across a range of model sizes and datasets, raising the question of how much to ``overtrain'' transformers when seeking compact, cheaper-to-run models. We find that L2 regularization may offer a path to more persistent ICL that removes the need for early stopping based on ICL-style validation tasks. Finally, we present initial evidence that ICL transience may be caused by competition between ICL and IWL circuits.

The Transient Nature of Emergent In-Context Learning in Transformers

Abstract

Transformer neural networks can exhibit a surprising capacity for in-context learning (ICL) despite not being explicitly trained for it. Prior work has provided a deeper understanding of how ICL emerges in transformers, e.g., through the lens of mechanistic interpretability, Bayesian inference, or by examining the distributional properties of training data. However, in each of these cases, ICL is treated largely as a persistent phenomenon; namely, once ICL emerges, it is assumed to persist asymptotically. Here, we show that the emergence of ICL during transformer training is, in fact, often transient . We train transformers on synthetic data designed so that both ICL and in-weights learning (IWL) strategies can lead to correct predictions. We find that ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases, indicating an asymptotic preference for IWL. The transient nature of ICL is observed in transformers across a range of model sizes and datasets, raising the question of how much to 'overtrain' transformers when seeking compact, cheaper-to-run models. We find that L2 regularization may offer a path to more persistent ICL that removes the need for early stopping based on ICL-style validation tasks. Finally, we present initial evidence that ICL transience may be caused by competition between ICL and IWL circuits.

1 Introduction

In-context learning (ICL) is the ability of a model to use inputs at inference time to adapt its behavior, without weight updates, in order to solve tasks not present during training. The first examples of this behavior in neural networks were observed in architectures specifically designed and trained for few-shot learning , the capacity to learn a desired behavior from only a few examples [1-3]. 2 Training in these cases involved shuffling the labels corresponding to input exemplars on each 'episode' so that, to perform well on the training set, the model had to recall exemplar-label mappings from context to make future predictions. At test time, novel exemplar-label mappings were provided, and the network had to use these to classify query exemplars.

Research on ICL changed with the development of the transformer [5]. Brown et al. [6] first observed that a transformer-based language model, GPT-3, trained auto-regressively at sufficient scale, exhibited ICL without any specific effort of the authors to promote it via the training objective or data. Since then, a large body of literature has studied [7-12] or reported [13-15] examples of ICL in transformers trained at scale.

(a) In-context learning.

1

2

3

4

5

# train steps

1e7

accuracy

0.2

0.4

0.6

0.8

1.0

pe ig

pe se

chance

(b) In-weights learning.

1

2

3

4

5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(c) Train log loss.

0

1

2

3

4

5

# train steps

1e7

8

6

4

2

0

2

train log loss

口

These compelling findings have sparked a surge of research into the phenomenon of emergent capabilities in large neural networks. Nonetheless, recent work has shown that ICL is not a guaranteed outcome of training transformers. Chan et al. [10] found that specific properties that are typical of language data, such as burstiness and its highly skewed distribution, play an important role in emergent ICL in transformers. When trained on data without these properties, Chan et al. [10] found that transformers tend to resort to in-weights learning (IWL). In the IWL regime, the transformer relies on information stored in the weights of the model, rather than new information provided in-context. 3 Importantly, ICL and IWL often appear to be in opposition, with ICL emerging most readily when training data has a large number of tokens or classes and when that data is bursty, with items appearing in clusters rather than being uniformly distributed.

Such controlled studies based on known data-generating distributions are critical for better understanding the ICL phenomenon in transformers. At the same time, a complementary body of work studies emergence in massive models trained directly on organic web-scale data, with the takeaway that impressive capabilities such as ICL are more likely to emerge in large models trained on more data [14, 17-19]. However, this reliance on massive models poses serious practical challenges, such as how to innovate rapidly, how to train energy-efficiently and in low-resource settings, and how to run efficiently at deployment. Thus, a growing line of research [20-22] has focused on achieving comparable performance (including emergent ICL) in smaller transformer models.

Overtraining is currently the favored approach for training small yet performant transformers. These small models are trained on more data, possibly with repetition, beyond what is prescribed by scaling laws and compute budget [13, 15]. At its core, overtraining relies on an assumption that is implicit in

(a) Example sequences and outputs.

327

589

2

589

327

710

327

589

Train

data:

ICL

eval:

IWL

eval:

0

1

1

0

1

0

1

0

2

0

589

710

221

478

846

895

context

query

327

327

0

327

221

327

Strategy used

ICL

IWL

example

outputs

(b) Model schematic.

embed embed

exemplar label

exemplar

label exemplar

embed embed embed

most (if not all) recent explorations of ICL in LLMs: persistence . That is, it is assumed that once a given model has been trained sufficiently for an ICL-dependent capacity to emerge, that capacity will be retained as training progresses, provided the training loss continues to decrease.

Here, we show that the general assumption of persistence is false. We do so by adapting a standard image-based few-shot dataset [23], which allows us to evaluate ICL carefully under controlled conditions. We characterize simple cases where ICL emerges, only to be subsequently lost while the model's loss continues to decrease. In other words, while ICL is well-known to be an emergent phenomenon, we should also think of it as a potentially transient effect (Section 3, Figure 1). We find this transience occurs across a range of model sizes (Section 4.1), dataset sizes (Section 4.2), and dataset types (Section 4.3), though we do identify some properties (Section 4.4) that can postpone transience. In Section 5, we study ways to alleviate this risk, and show that regularization may offer a path to persistent ICL. In Section 6, we present preliminary evidence that ICL transience is caused by competition with IWL circuits.

In general, if networks are trained carelessly for longer and longer, we find that ICL can disappear as readily as it emerges, leaving models without capabilities that we are increasingly expecting to find in modern AI systems.

2 Experimental Setup

2.1 Dataset construction

In order to study the transience of ICL across a range of model and dataset sizes in a controlled fashion, we use a synthetic data-generation process inspired by Chan et al. [10], which allows us to properly assess in-context versus in-weights learning. Our data generators are primarily built from the Omniglot dataset, a standard benchmark for few-shot learning, which consists of images of handwritten characters from different alphabets [23, MIT License]. The Omniglot dataset consists of 1623 character classes, with 20 exemplars each. In experiments with more than 1,600 classes, we construct additional image exemplars by applying rotations and flips of the original Omniglot images.

Inputs to the transformer model are sequences of exemplars and labels. We use 8 exemplar-label pairs (the 'context') followed by one 'query' exemplar, for an input sequence length of 17 (Figure 2a). The model is trained to minimize the loss in predicting the label for the query exemplar (Figure 2b).

To explore the generality of our results, we also extend to exemplars derived from language model token embeddings. We use pre-trained token embeddings from the LLaMa family [15] of large language models, which use a vocabulary size of 32,000 tokens. We extract the input embedding matrices from all four open-source LLaMa 1 models. We then subset and cluster [24] these token embeddings to form 3,200 classes of 5 tokens each. For each LLaMa model size, we construct two such datasets, with different clustering seeds. The full procedure is described in Appendix C.

(a) In-context learning.

(b) In-weights learning.

1.0

1.0

P(bursty)

0.8

0.8

0.0

accuracy

0.6

2

accurac

0.6

0.5

0.9

1.0

0.4

0.4

0.2

0.2

0.0

1

2

3

4

5

0.0

1

2

3

4

5

chance

#train steps

1e5

#train steps

1e5

Training sequences. The training task (see top row in Figure 2a) can be solved via ICL or IWL. The context for each training sequence contains 3 exemplar-label pairs from the query class ('bursty' sequences), which prior work has found to incentivize ICL [10]). Bursty sequences allow the network to look for matching exemplars in-context to output a label for the query example, an example of ICL. To avoid repetition biases, the training context also includes 3 exemplar-label pairs from a distinct, distractor class. We keep exemplar-label mappings fixed across sequences, so the network could learn a mapping from exemplars to labels in-weights (without needing to find matching exemplars in-context). This design allows us to observe whether the model prefers ICL or IWL at different points in time, as a function of model size or in response to the properties of the training data.

Evaluation sequences. We follow Chan et al. [10] and construct two kinds of evaluation sequences that measure the extent to which a trained network relies on ICL versus IWL. To evaluate ICL (see second row of Figure 2a), we use sequences with 4 exemplar images from each of two classes, but we set the class labels to either 0 or 1 at random for each sequence (unlike during training, where the exemplar-label mappings are fixed). Accuracy on this evaluator is measured across 0 and 1 as possible outputs (so chance-level accuracy is 50%). As these labels were not associated with these exemplars during training, the only way to achieve above-chance accuracy on this evaluator is for the network to refer back to the context and copy the label corresponding to matching exemplars. To evaluate IWL (see third row of Figure 2a), we use sequences where none of the context exemplars come from the same class as the query, but all of the exemplar-label mappings are the same as during training. In this case, ICL is not useful (as there are no matching exemplars in-context), so the network must rely on exemplar-label mappings stored in-weights (IWL) for above-chance accuracy. Other evaluation techniques for ICL and IWL are considered in Appendix B.

2.2 Model details

For our transformer model, the 'default' settings consist of 12 layers, with an embedding dimension of 64 and an additive, sinusoidal positional encoding scheme [5]. Each element of the sequence is passed through an embedder, either a standard embedding layer for the label tokens, a ResNet encoder for the exemplar images, or a simple linear layer for exemplars created from LLaMa tokens. (Figure 2b). Both embedders are trained jointly with the network. In our experiments, we vary multiple aspects of this architecture and observe the effect on the transience of ICL. All experiments were run with 2 seeds and used the Adam optimizer [25]. We did not observe qualitative differences in behavior between seeds. We ran most experiments for 5e7 iterations, but truncated some early when signs of transience were clear. More details (e.g., specific hyperparameter choices, and links to code) can be found in Appendix A.

(a) In-context learning.

1

2

3

4

5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

1

2

3

4

5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

# layers

6

12

18

3 In-context learning is transient

Figure 1a shows that ICL is transient even when using data-distributional settings previously found by Chan et al. [10] to strongly incentivize ICL. We annotate various aspects of the ICL curve, such as the peak height, peak onset, and decay slope. Prior work [10] stopped training at 5e5 training steps, before peak onset. Training for much longer in an identical setting leads to the disappearance of ICL. Notably, ICL is slowly replaced by IWL (as seen by the steadily increasing accuracy on the IWL evaluator, Figure 1b), all while the training loss continues to decrease (Figure 1c).

Chan et al. [10] found some transient ICL in other data regimes, such as when some small percentage of sequences are non-bursty, where the context does not contain the query (Figure 3). In these settings, the eventual rise of IWL is to be expected, as ICL cannot minimize the loss on the non-bursty sequences. Notably, our results (Figure 1) show surprisingly that ICL transience extends even to cases where ICL is 'sufficient' to fully solve the training task (such as when the context always contains a relevant exemplar-label mapping that can be used). When comparing to the results from Chan et al. [10] (Figure 3), we see that higher levels of burstiness lead to a higher peak height and gentler decay slope. When training for fewer iterations, it might appear that ICL is persistent , but our results from training longer in Figure 1 demonstrate that ICL is actually transient .

4 In-context learning is transient across settings

4.1 Effect of model size

We investigate whether the ICL transience effect is modulated by model size, given the current prevalence of extremely large transformer models. We find that there is no consistent effect of model depth on ICL transience (Figure 4a-b). In fact, deeper models can lead to lower peak heights (e.g., the 18-layer model has a lower peak height than the 12-layer model). Notably, the decay slope is similar across different model depths, indicating that the ICL is unlikely to persist simply by scaling model depth.

We also experiment with the width of the model in Section 6, which sheds some light on the causes of ICL transience.

4.2 Scaling dataset size

Next, we investigate the effects of dataset size on ICL transience. There are two primary ways of varying the dataset size: increasing the number of exemplars per class (a form of increasing in-class variation), or increasing the number of classes.

Chan et al. [10] found that in-class variation was crucial for ICL to emerge, and we find a similar trend. When the number of exemplars per class is increased (from 10 to 15 to 20) while keeping the number of classes fixed, we see an increase in ICL peak height (Figure 5a). However, despite higher peak heights, we also see steeper decay slopes, indicating that increasing in-class variation is unlikely to make ICL persistent.

(a) In-context learning.

0.0

0.5

1.0

1.5

2.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

0.0

0.5

1.0

1.5

2.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

# exemplars per class

10

15

20

(a) In-context learning.

0.0

0.5

1.0

1.5

2.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

0.0

0.5

1.0

1.5

2.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

12800 classes, 20

exemplars each

1600 classes, 160

exemplars each

1600 classes, 20

exemplars each

When increasing the number of classes, we again find a similar trend to Chan et al. [10] with peak heights increasing (Figure 6a). In this case, we also find that decay slopes become more gradual. To better illustrate the differences between increasing in-class variation and increasing the number of classes, we do another experiment where the total number of exemplar-label pairs is fixed to 12 , 800 × 20 = 1 , 600 × 160 = 256 , 000 , with one run having more classes and less in-class variation, and another run having fewer classes and more in-class variation. We find that increasing the number of classes is a more effective way to promote ICL (higher peak height, shallower decay slope) than increasing the in-class variation, perhaps due to each class appearing less often during training. This result also indicates that, beyond a certain point, in-class variation actually hurts ICL performance (lower peak height, steeper decay slope, as seen in Figure 6a). To conclude, ICL remains transient as we increase the number of classes, but training for longer and longer becomes necessary to make it go away, given the shallower decay slope.

4.3 Extending to language model token embeddings

To explore the generality of our results, we replaced the image exemplars with exemplars derived from language-model token embeddings (procedure described fully in Appendix C). On all of these

(a) In-context learning.

(b) In-weights learning.

1.0

1.0

Embeddingsfrom

0.8

0.8

LLaMalmodelsize:

7B

accuracy

0.6

2

0.6

13B

ccurac

chance

30B

0.4

0.4

70B

0.2

0.2

0.0

8.0

0.5

1.0

1.5

2.0

0.0+

8.0

0.5

1.0

1.5

2.0

lchance

#train steps

1e6

#train steps

1e6

token-embedding derived datasets, we find that whenever ICL emerges, it is subsequently transient (Figure 7). Furthermore, the peak height is lower than in our other experiments. In terms of timing, the peak onset as well as subsequent transience occurs on a quicker timescale (note the shorter x -axis scale in Figure 7). These results also further corroborate our findings on dataset size (Section 4.2), as the constructed token embedding-label dataset is smaller ( 3 , 200 × 5 = 16 , 000 exemplar-label pairs, as opposed to 1 , 600 × 20 = 32 , 000 pairs in our Omniglot experiments), so we would expect lower heights and sharper decay slopes. Overall, these experiments are a promising step towards connecting our findings to natural language, and demonstrate that the transience phenomenon is not limited to image-based exemplar-label datasets.

4.4 Changing the data distribution

Until now, we have trained on data for which the marginal distribution over classes was uniform. However, a common property of naturalistic data is a Zipfian (power-law) skew [26]. Zipfian data contains a few 'common' classes that appear very often, and many 'rare' classes that appear less often than they would if the class distribution were uniform. In Section 4.2, we found that more classes (corresponding to each class appearing less often) led to shallower decay slopes. We explored whether data with a Zipfian distribution could play a similar role in reducing the transience of ICL.

Indeed, in Figure 8, we find this to be the case. We implemented different levels of skew across data classes according to a Zipfian distribution p ( X = x ) ∝ x -α ( x corresponds to the rank of an input class-e.g., α = 0 corresponds to a uniform distribution). At Zipf exponent α = 1 , we find a severely delayed ICL peak onset, thereby delaying the subsequent transience, and a shallow decay slope following this delayed onset (similar to that seen with a large number of classes in Figure 6a). Furthermore, corroborating the findings of Chan et al. [10], we find that stopping training early, when using skewed distributions, can lead to a combination of ICL and IWL in the same network (as evidenced by the increased IWL at Zipf exponent α = 1 as compared to Zipf exponent α = 0 , Figure 8b). However, it is important to note that if the Zipfian exponent is too extreme (e.g., α = 2 ), ICL does not emerge at all, and the network uses IWL to solve the task. We conclude that moderately skewed Zipfian data does not eliminate transience, but mitigates it by necessitating training for an even longer time to observe transience (similar to when there is a large number of classes).

(a) In-context learning.

0

1

2

3

4

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

0

1

2

3

4

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

Zipf exponent

0.0

1.0

2.0

5 Regularization may eliminate ICL transience

If we consider that ICL is a more generally useful solution than IWL, then ICL transience is, in a sense, the opposite of the grokking phenomenon, where a model changes from a less to a more general solution after long periods of training [27]. Regularization (or implicit regularization) may help to drive grokking [28, 29] (and some work even shows an impact of dataset size as well [30], analogous to what we find above). Thus, would we see regularization mitigating the ICL transience phenomenon?

(a) In-context learning.

0

1

2

3

4

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

0

1

2

3

4

# train steps

1e7

0.00

0.02

0.04

0.06

0.08

0.10

accuracy

L2 regularization

0

1e-05

5e-05

1e-04

We see in Figure 9 that this is indeed the case, with L2 regularization seeming to even eliminate ICL transience entirely, at least on the timescales that we tested. Without any regularization, we see transient ICL as before. However, as we add increasing levels of L2 regularization, we see that the decay slope of ICL performance goes to 0. On the other hand, we actually observe some transience of in-weights learning (purple and maroon curves, Figure 9b). These results indicate that the ICL circuit is a lower norm solution than the IWL circuit. Thus, regularization may offer a path to persistent ICL. However, with values of regularization that are too high ( > 0 . 0001 ), the network is over-regularized and ICL transience returns, but, notably, IWL accuracy never rises above zero either (see Appendix D for more train loss curves and more extreme values of regularization).

6 ICL transience may be caused by competition with IWL circuits

We performed initial investigations to understand the origins of ICL transience: why does ICL fade when the IWL solution emerges? Do the ICL and IWL circuits compete with each other for resources, for example, in the transformer's residual stream [16]? If so, we would expect wider transformers with larger embedding size (and therefore greater residual stream capacity) to be less afflicted by ICL transience. In Figure 10a, we see that this is the case: Greater model embedding sizes lead to higher peak heights and gentler decay slopes. However, one confound to consider is that larger embedding may just independently enhance ICL abilities (as they enhance IWL abilities, Figure 10b), rather than reduce competition between circuits.

To better understand how scaling the embedding dimension enhances ICL and IWL, we trained models on data that could be solved via ICL only or IWL only, rather than being solvable by both strategies. For ICL-only training, we used the same type of sequences used in our ICL evaluator, where the network must use ICL to solve the query-prediction task. For IWL-only training, we used the same type of sequences used in our IWL evaluator, where the network must use IWL to solve the query-prediction task. Figure 10d shows that larger embedding dimensions enhance the model's ability to perform IWL, but do not consistently help ICL (Figure 10c). Thus, these results indicate that the increased peak height and shallower decay slope in Figure 10a are likely due to decreased competitive interactions with IWL circuits, rather than an enhancement of ICL capabilities.

We arrived at convergent evidence by building on the regularization experiments from Section 5. We applied weight decay selectively to either the ResNet embedding layers, the MLP layers, or the self-attention layers. When we applied weight decay to only the self-attention layers, ICL was still transient. When we applied weight decay to the MLP or ResNet layers, however, ICL transience was mitigated (Appendix Figure 16). These results can be interpreted in light of prior work indicating that in-weights information is stored in MLP layers [31, 32]. By selectively penalizing this behavior, we enable ICL to persist, thus providing convergent evidence that - when no weight decay is applied - ICL fades due to competition with IWL circuits.

7 Related Work

This work builds on several papers that seek to understand in-context learning by analyzing small networks and/or controlled data sources. Xie et al. [7] consider an implicit definition of ICL (defined in terms of the model's loss) and show that the effect can be framed as a form of Bayesian inference. Chan et al. [10] show via experimentation with controlled synthetic datasets that, e.g., the bursty and

(a) In-context learning.

1

2

3

4

5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

1

2

3

4

5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

Embed dim

32

64

128

0.0

0.5

1.0

1.5

2.0

2.5

3.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

0.0

0.5

1.0

1.5

2.0

2.5

3.0

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

Embed dim

32

64

128

Zipfian nature of text data could play an important role in the emergence of ICL. Many other works consider how ICL can be used to learn certain function classes using synthetic data, and how ICL may be implementing a form of gradient descent to accomplish this [33-35].

As with the aforementioned work, here we do not experiment directly with billion-parameter transformers trained on web-scale corpora. Nonetheless, our work should be of relevance to studies that aim to derive ICL in increasingly small and compact text-based language models. By training language models of many different sizes on different amounts of data, Hoffmann et al. [13] identified a relationship between the amount of training and model size that appeared compute-optimal with respect to the training loss. This correspondence motivated Chinchilla, a 70B-parameter language model whose downstream performance (on tasks that require ICL) was comparable to far larger models. More recently, Touvron et al. [15] took this further, 'overtraining' smaller models to achieve lower training loss according to a given, inference-time, compute budget. Perez et al. [36] point out that in such work, tasks that require ICL are typically included in the models' validation set in order to optimise hyperparameters directly for ICL. Our results indicate that, without appropriate validation and early stopping, training for a long time may eventually lead to ICL gradually disappearing, perhaps even without recognition from model developers. A large number of 'classes' or Zipfian data distributions (both properties of natural language [37]) can mitigate this by delaying the disappearance. Moreover, regularization may completely eliminate this risk. Indeed, Hoffmann et al. [13] find that the use of optimizers with weight decay [38] leads to models that perform better, but this benefit only manifests at later stages of training (perhaps when ICL would otherwise decay).

8 Discussion

In-context learning is often transient. The key insight of this work is the empirical finding that, if in-context learning (ICL) emerges in a transformer network, it may not necessarily persist as the model continues to be trained. That is, while ICL is often emergent, it may be in many cases also be transient (Section 3). In fact, since the loss of ICL is gradual, its creeping disappearance may not even be known to model developers. This emphasizes the importance of evaluating appropriate validation metrics over the course of training [e.g., 39]. Furthermore, these results suggest that it may be desirable to stop training some transformer models before a compute budget is reached-to

preserve ICL-even if training losses are still decreasing. This would need to be ascertained on a case-by-case basis. Other methods for preventing the disappearance of ICL, like regularization, may also be critical for delivering flexible, in-context learners.

What are strategies for mitigating ICL transience? In Section 4, we found that data distributional properties (increasing the number of classes, adding a Zipfian skew) can strengthen ICL and postpone the transience of ICL, and so can increasing embedding size (and least in the ranges we tested). The fact that most transformers are trained on language, which naturally has both of these datadistributional properties [37], may explain why ICL transience hasn't been documented in LLMs yet. However, as the trend of overtraining continues, there's a risk of losing these impressive ICL capabilities. We found that the most robust method to make ICL persistent was L2 regularization (Section 5).

Why does ICL emerge at all, if it is not asymptotically preferred? Since GPT-3 [6], the emergence of ICL has become an expected phenomenon in large-scale transformer models. Our experiments underline how curious it is that ICL emerges in such models given that it does not seem to be asymptotically preferred. Possible explanations may concern optimization quirks (similar to the slingshot mechanism proposed as an explanation for grokking [29]) or perhaps lottery tickets contained in common initialization methods [28, 40] that incentivize ICL (when trained with relevant data-distributional properties). Future work could explore these possible explanations by considering the effect of alternate optimizers or initialization schemes on the transience of ICL.

Given that ICL does emerge, why does it fade in favor of IWL? On our training data, either ICL or IWL can reach good performance. While we do not know for certain why ICL emerges in a network, the question remains of why it fades if it is a viable strategy to solve the training task. Our results in Section 6 suggest that part of the explanation may lie in a competition between IWL and ICL circuits for resources in the transformer residual stream [16]. Under the assumption that IWL is asymptotically preferred, this competition would explain why ICL fades after emerging.

Why is IWL asymptotically preferred over ICL, when both solve the task? Induction heads [9] are a possible circuit that may be responsible for ICL, and comprise of two interacting components: finding a match in-context, then copying some token forward. The match operation is fundamentally limited by the softness of the attention operation [41-43]. On the other hand, IWL relies on learning the exemplar-label mapping in-weights, which we show is feasible in Figure 10d. We postulate-and hope to investigate in future work-the possibility that although ICL and IWL can achieve perfect accuracy on the task, this 'imperfect match' to prior context asymptotically incentivizes solutions (when training with cross-entropy loss, as is standard) that do not rely as much on context and instead learn in-weights.

How do we get ICL and IWL to co-exist asymptotically in the same model? Large language models clearly show a co-existence of ICL and IWL. Chan et al. [10] found that training on Zipfian data allows the network to perform ICL on rare classes, but learn common classes in-weights. Regularization was the best strategy we found to make ICL persistent, but it leads to little-or-no IWL. Given our results, the only effective way currently to get co-existence of ICL and IWL appears to be stopping training at the right time. Future work could investigate other factors that may restore co-existent ICL and IWL, even asymptotically.

Limitations and future directions. Perhaps the most significant limitation is that our experiments were not performed on large language models, one of the most important applications of transformers. Our training resources limited our ability to perform such experiments in the current work, but we believe that this is a necessary future step, to understand the full import of the current results. As a first step toward this, we demonstrated ICL transience on datasets with language model token embeddings (Section 4.3), but our datasets are still restricted to the exemplar-label setup. It would also be interesting to explore the effect of context length: For a given train or inference budget, a system designer could trade off between longer context (possibly enabling better ICL) and a larger model (possibly enabling better IWL). We also note that our definitions of ICL and IWL are relatively specific, and many more important mechanisms are likely at play in transformers-for example, contextual information could be used in conjunction with IWL to narrow down possible outputs via copying [16], which we find some preliminary evidence of in Appendix B. Future work could investigate more mechanistic explanations of ICL transience, the tradeoff between ICL and IWL, and additional recipes for reducing ICL transience.

Acknowledgments and Disclosure of Funding

A.K.S. and T.M. are supported by the Gatsby Charitable Foundation. This work was supported by a Schmidt Science Polymath Award to A.S., and the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/Z/19/Z) and the Gatsby Charitable Foundation (GAT3755). A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.

The authors thank Kira Düsterwald and DJ Strouse for insightful discussions throughout the course of the project. In addition, the authors thank Andrew Lampinen, Pierre Richemond, Roma Patel, and Murray Shanahan for feedback on the draft.

References

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850. PMLR, 2016.

Jane X Wang. Meta-learning in natural and artificial intelligence. Current Opinion in Behavioral Sciences , 38:90-95, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 , 2021.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-inductionheads/index.html.

Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 18878-18891. Curran Associates, Inc.,

URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 77c6ccacfd9962e2307fc64680fc5ace-Paper-Conference.pdf .

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 , 2023.

Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. arXiv preprint arXiv:2303.07895 , 2023.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726 , 2023.

Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models, 2022.

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.

Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 350(6266):1332-1338, 2015. doi: 10.1126/science.aab3050. URL https://www.science.org/doi/abs/10.1126/science. aab3050 .

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980 .

George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology . Ravenio Books, 1949.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. Technical Report arXiv:2201.02177, arXiv, January 2022. URL http://arxiv.org/abs/2201.02177 . arXiv:2201.02177 [cs] type: article.

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023.

Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon, June 2022. URL http://arxiv.org/abs/2206.04817 . arXiv:2206.04817 [cs, math].

Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, 2023.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 30583-30598. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ c529dba08a146ea8d6cf715ae8930cbe-Paper-Conference.pdf .

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022.

Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677 , 2022.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models, 2021.

Steven T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review , 21(5):1112-1130, October 2014. ISSN 1069-9384. doi: 10.3758/s13423-014-0585-6. URL https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC4176592/ .

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR , abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101 .

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR , abs/1803.03635, 2018. URL http://arxiv.org/abs/1803.03635 .

André F. T. Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro M. Q. Aguiar, and Mário A. T. Figueiredo. Sparse and continuous attention mechanisms, 2020.

Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171, dec 2020. doi: 10.1162/tacl_a_ 00306. URL https://doi.org/10.1162%2Ftacl_a_00306 .

David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385 .

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015.

A Model and training procedure details

All experiments used the same model and training procedure unless stated otherwise. The transformer [5] consisted of 12 layers with am embedding dimension of 64 and 8 heads. The images were embedded by a ResNet [44] with two blocks per group and channels per group (16, 32, 32, 64). The ResNet embedder was learnt jointly and not pre-trained. The integer labels were embedded using a standard embedding layer. The input embeddings were augmented with a standard sinusoidal positional encoding [5] with timescale 30 (as our sequences are all length 17). Experiments were run for up to 5e7 training steps with batch size 32, on 16 TPU v2 or v3 cores. They were trained using Adam [25] (with default parameters of β 1 = 0 . 9 , β 2 = 0 . 999 ) and a learning rate schedule with a linear warmup up to a maximum learning rate of 3e-4 at 4,000 steps, followed by an inverse square root decay. L2 regularization was implemented by adding the squared weights of the model (excluding batch norm parameters) to the loss term. All experiments were run with 2 seeds each. In all figures, (shaded) error bars indicate standard deviation around the mean.

Also, we note that, despite the long training times, all our experiments are still in the <1 epoch regime, due to the combinatorial nature of our dataset. We estimate there are over 1 , 600 × 1 , 599 × (1 , 598 × 1 , 597 / 2) × (8! / (3! × 3!)) × (20 9 ) = 1 . 87 × 10 27 possible training sequences, of which our model sees a maximum of 32 × 5 × 10 7 = 6 . 4 × 10 8 .

Source code can be found at github.com/google-deepmind/emergent_in_context_learning and github.com/aadityasingh/icl-transience .

B Alternate evaluation metrics

While the primary evaluations used throughout the paper are inspired by the ICL and IWL evaluators of Chan et al. [10], we did consider a few alternate evaluators. Prior work has shown that transformers are prone to simple copying biases [16] and that the importance of few-shot exemplars might be more in their format rather than correctness [8, 11]. To account for copying biases, we introduce the 'IWL-copy-available' evaluator, wherein we randomly replace one of the labels with the true label. Note that, during training, the correct label always appears in context, which may incentivize the formation of such naive copying circuits. To perform a stricter test of ICL performance, we thus take inspiration from the 'flipped-ICL' evaluation in Wei et al. [11].

For clarity, we illustrate some abbreviated example sequences below:

Train: X 23 Y 24 X 23 X 23 A 0 Y 24 B 1 Y 24 X ?. Note there are 3 instances of X 23, 3 instances of Y 24, and one instance each of A 0, B 1. Exemplar X is always associated to label 23 through training, Y to 24, A to 0, and B to 1.

IWL eval (in main text): A 0 C 2 F 5 B 1 Y 24 D 3 H 7 X ?. Niether the exemplar nor label appears in context.

IWL-copy-available eval: A 0 C 2 F 5 B 23 Y 24 D 3 H 7 X ?. We randomly replace one of the labels with the desired answer label. Note, this task allows the network to benefit from copying, but still provides no information in-context to learn that X should map to 23.

ICL eval (in main text): X 0 X 0 Y 1 X 0 Y 1 Y 1 X 0 Y 1 X ?. Both X and Y's labels are replaces with 0/1. Note, we also tried using other pairs of labels for the ICL evaluator (e.g., 15/20), but found little variance across this choice.

flipped-ICL eval: X 24 X 24 Y 23 X 24 Y 23 Y 23 X 24 Y 23 X ?. Note, if the network outputs 24 here, it demonstrates in-context learning, as it must be using the exemplar-label from context. Alternatively, if the network outputs 23, it demonstrates in-weights learning, as its ignoring the exemplar-label mapping provided in context. Accuracy=1 on this evaluator means the network is doing ICL, Accuracy=0 means the network is doing IWL.

Following the results of Section 4.3, which showed that fixed embeddings exhibit transience on shorter timescales, we conducted some experiments using the above evaluators on a dataset of fixed omniglot embeddings by preprocessing all images through a Imagenet-pretrained-and-frozen Resnet18 encoder [44, 45]. Like the main paper, we considered datasets with either 1,600 or 12,800 classes, with 20 exemplars per class. We did not run these experiments as long as those in the main paper, as the purpose was to test the new evaluators.

(a) In-context learning.

(b) In-weights learning.

1.0

0.8

accuracy

0.6

0.4

0.2

#classes:

1600

0.0+

12800

(c) flipped-ICL evaluator.

(d) IWL-copy-available evaluator.

1.0

0.8

accuracy

0.6

0.4

0.2

0.0

8.00

0.25

0.50

0.75

1.00

1.25

0.00

0.25

0.50

0.75

1.00

1.25

#train steps

1e7

#train steps

1e7

As seen in Figure 11, the alternate evaluators' asymptotic behaviors largely mirror the evaluators used by Chan et al. [10]. That said, there are some interesting differences. For example, under the flipped-ICL evaluator, models trained on 1,600 classes are never able to fully override the mapping of exemplars to labels seen in training (as evidenced by accuracy remaining below 0.5). However, at the scale of 12,800 classes, models are able to (accuracy goes above 0.5 before decaying). Intriguingly, these results may connect to Wei et al. [11], except in the sense of a more diverse dataset rather than a larger parameter count, leading a model to 'do in-context learning differently'.

With respect to transience, we do still observe it in both cases. For 1,600 classes, the increase in performance indicates the network may be overriding its exemplar-label mapping on some sequences, but this fledgling ability is short-lived. For 12,800 classes, while the shallow decay slope makes transience on the standard ICL evaluator not visible due to the short timescale (note, ICL is still transient at this # of classes, as shown in Figure 5a), transience is clearly visible on the flipped-ICL eval.

Furthermore, we see that there is indeed an effect of copying in our networks, as seen by the improved performance of models on the IWL-copy-available evaluator as compared to the IWL evaluator.

For a clearer comparison to prior work on synthetic datasets, we opted to use the same evaluators as Chan et al. [10] in this work, but hope the above analysis shows the importance of good evaluators in diagnosing ICL transience.

C Construction of LLaMa-based exemplar-label datasets

In this section, we provide more detail as to the procedure used to construct the fixed embedding datasets used in Section 4.3.

C.1 Constructing datasets

We save the token embedding matrix for each LLaMa v1 model. These have dimensions 32,000 × d model , where d model = 4096 , 5120 , 6656 , 8192 for the 7B, 13B, 30B, and 70B, LLaMa models, respectively. To go from these raw embeddings to 'classes' that we can use in our setup, we perform the following steps:

Subselect: Many tokens correspond to individual bytes or characters from non-English languages. We observed that the statistics of these token embeddings are often different than

(a) In-context learning.

(b) In-weights learning.

1.0

1.0

Embeddingsfrom

0.8

0.8

LLaMalmodelsize:

7B

accuracy

0.6

2

0.6

13B

chance

accurac

30B

0.4

0.4

70B

0.2

0.2

0.0+

8.0

0.5

1.0

1.5

2.0

0.0+

8.0

0.5

1.0

1.5

2.0

chance

#trainsteps

1e6

#train steps

1e6

the others, so we remove these tokens. Specifically, we use tokens 259 to 29,870, inclusive, for a total of 29,612 tokens.

Cluster: We cluster the token embeddings using K-means clustering and cosine distance using the FAISS library [24]. As FAISS does not offer a valid minimum number of points per cluster, we first cluster into 2400 (or 4800) clusters, then pick all clusters that have more than 10 (or 5) points. This gives us at least 1,600 (or 3,200) clusters. We randomly pick the actual 1,600 (or 3,200) clusters.

Select from cluster: Many of these clusters contain more than 10 (or 5) points. To maximize in-class variation, we pick the furthest 10 (or 5) points from the cluster centroid. Thus, we have our 1,600 classes of 10 examples, or 3,200 classes of 5 examples.

Note that step 2 is highly dependent on the random seed used to initialize cluster centroids. We run the above procedure for all 4 LLaMa models on 2 random seeds.

C.2 Example clusters

If we map the embeddings back to tokens, we can get a sense of what various clusters may represent. Many clusters cover similar tokens, but some clusters can be more semantic or even multilingual! We show some extracted clusters (for 3,200 classes, 5 embeddings per class, on the 70B LLaMa) below:

C.3 The difficulty of emergent ICL

We observed that ICL emergence was more sensitive to model initialization when using LLaMa token embeddings as exemplars. An example seed where ICL did not emerge at all (for 3,200 classes, 5 exemplars per class) is shown in Figure 12. However, in all cases where ICL did emerge (even to a small degree), we found it to be transient (Figures 7, 13). Furthermore, the results presented in Section 4.3 are averaged over two different clustering seeds (see step 2 above), which yield different output datasets. ICL is transient across all 8 datasets tested.

At 1,600 classes, with 10 exemplars per class, we found mild ICL emergence, as shown in Figure 13. We found that 3,200 classes were necessary to get any strong ICL, which is shown in Figure 7.

(a) In-context learning.

(b) In-weights learning.

1.0

1.0

Embeddingsfrom

0.8

0.8

LLaMalmodelsize:

7B

accuracy

0.6

2

0.6

13B

accurac

chance

30B

0.4

0.4

70B

0.2

0.2

0.0+

8.0

0.5

1.0

1.5

2.0

0.0

8.0

0.5

1.0

1.5

2.0

lchance

#train steps

1e6

#train steps

1e6

D Additional regularization results

(a) In-context learning.

0.0

0.5

1.0

1.5

# train steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

accuracy

chance

(b) In-weights learning.

0.0

0.5

1.0

1.5

# train steps

1e7

0.00

0.02

0.04

0.06

0.08

0.10

accuracy

L2 regularization

5e-04

1e-03

0

1

2

3

4

# train steps

1e7

8

6

4

2

0

2

train log loss

L2 regularization

0

1e-5

1e-4

1e-3

# train steps

(a) In-context learning, decay = 1e-5

0.7

0.75

0.8

0.85

accuracy

0

0.02

0.04

0.06

accuracy

0

0.1

0.2

0

0.4

0.8

1.2

1e7

self attention

Weight decay applied to:

MLP layers

ResNet embed

(b) In-context learning, decay = 1e-4

(d) In-weight learning, decay = 1e-4

(c) In-weight learning, decay = 1e-5

0.5

0.7

0.9

1.6

# train steps

0

0.4

0.8

1.2

1e7

1.6

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the in-context learning (ICL) phenomenon in transformer networks and investigates its transience during training. It presents empirical findings that ICL if emerged in transformers, may not persist as the model continues to be trained. \nThe paper highlights the importance of evaluating appropriate validation metrics and suggests stopping training before a computing budget is reached to preserve ICL. \nStrategies to mitigate ICL transience, such as data distributional properties and regularization, are explored. The paper also discusses the competition between ICL and in-weight learning (IWL) circuits and raises questions about the emergence and fading of ICL. \nThe experiments demonstrate that ICL in transformers is often transient. Model size, dataset size, and data distribution can influence the transience of ICL, but persistent ICL is not guaranteed. Wider models with larger embedding sizes show improved ICL, possibly due to reduced competition with IWL circuits. Skewed data distributions mitigate ICL transience but necessitate longer training.\nFuture directions, limitations, and implications for large language models are discussed. Overall, the paper provides valuable insights into the nature of ICL and its implications for transformer networks, contributing to the understanding of in-context learning and its behavior during training.\n\nSTRENGTHS:\nOriginality and Contribution:\n- Empirical findings on the transience of in-context learning (ICL): The paper presents novel empirical evidence that ICL if emerged in transformer networks, may not persist as the model continues to be trained. This finding sheds light on an important characteristic of transformer models and highlights the need for appropriate evaluation and validation metrics.\n- It provides insights into how ICL emerges, disappears, and gives way to in-weight learning (IWL) during training.\n- Discussion of the coexistence of ICL and in-weights learning (IWL): The paper explores the competition between ICL and IWL circuits for resources in the transformer residual stream. It raises intriguing questions about the emergence and fading of ICL and offers potential explanations for the coexistence of ICL and IWL in large language models.\n- Exploration of strategies to mitigate ICL transience: The paper investigates various strategies to mitigate the transience of ICL, including increasing the number of classes, introducing Zipfian data distribution, and using L2 regularization. These strategies provide valuable insights into potential methods for preserving and enhancing ICL capabilities.\n- It is interesting that ICL is transient even under conditions that are deemed conducive to the emergence of ICL.\n- Comprehensive and thought-provoking discussion, posing clear and relevant questions: discussion of future research directions and acknowledgment of the limitations of the conducted experiments. It identifies the need for experiments on large language models, exploration of other datasets and context lengths, and investigation of additional mechanisms in transformers, encouraging further advancements in the field.\n- Consideration of the impact of training duration and stopping criteria: The paper emphasizes the importance of appropriate training duration and stopping criteria to preserving ICL. It suggests that training large transformer models beyond a certain point may result in the gradual disappearance of ICL, even if training losses continue to decrease. This insight has practical implications for model development and training practices.\n\nResearch Problem and Motivation:\n- The research problem is well-defined and clearly stated, focusing on the transience of ICL in transformer networks.\n- The paper provides a convincing motivation for the chosen problem by highlighting the importance of evaluating appropriate validation metrics, preserving ICL, and understanding the factors influencing its persistence.\n- The relevance and importance of the research problem are clearly explained, emphasizing the potential impact on training large-scale language models.\n- The paper raises numerous additional questions, stimulating further scientific inquiry and exploration.\n\nMethodology:\n- The methodology is clearly described and well-founded. The paper outlines the experimental setup, including the models, training procedures, and evaluation metrics used.\n- The algorithms, models, and techniques employed are appropriate for addressing the research problem of studying ICL transience in transformer networks.\n- The paper adequately explains the experimental setups and data collection processes, providing sufficient details for reproducibility.\n\nExperimental Evaluation:\n- The experimental results are presented clearly and concisely, with appropriate figures and tables.\n- The evaluation methodology is rigorous and appropriate, considering various metrics and comparing experimental conditions.\n- While the statistical significance of the experimental results is not explicitly mentioned, the paper conducts a thorough analysis and provides empirical evidence to support the presented findings.\n- The rigorous analyses demonstrate a commitment to scientific rigor and methodology.\n\nRelated Work:\n- The paper comprehensively reviews the relevant literature, discussing previous works on ICL, language models, optimization, and initialization methods.\n- The paper differentiates itself from existing works by focusing on the transience of ICL and investigating strategies to mitigate it.\n- The references and citations are appropriate and up-to-date, incorporating relevant studies in the field.\n\nClarity and Organization:\n- The paper is well-structured with clear sections and subheadings, making it easy to follow the flow of ideas.\n- The ideas and concepts are presented logically and coherently, building upon each other to support the main arguments.\n- The language used in the paper is clear and concise, minimizing excessive technical jargon and enhancing readability.\n\nPresentation and Visualization:\n- The plots are visually appealing, utilizing scalable vector graphics and an attractive gradient color scheme. The figure captions are concise and informative.\n- The writing is of high quality, with minimal errors or inconsistencies.\n- The overall presentation is visually appealing and well-designed, enhancing the readability and comprehension of the paper.\n\nWEAKNESSES:\n- Lack of experiments on large-scale datasets: The experiments mainly focus on controlled synthetic datasets, and the impact of ICL transience on real-world, large-scale datasets is not explored. Including experiments on diverse, natural language datasets would enhance the practical relevance of the findings.\n\n- Limited scope of experiments: The experiments were not performed on large language models, one of the most important applications of transformers. Conducting experiments on larger models would provide a more comprehensive understanding of the phenomenon and its implications.\n\n- Narrow definitions of ICL and IWL: The definitions of in-context learning (ICL) and in-weights learning (IWL) used in the paper are relatively specific and make for a clear-cut distinction that in reality - as acknowledged in a footnote - may not be that clear. Exploring additional mechanisms and interactions between context and learning could contribute to a more comprehensive understanding of transformer models.\n\n- Limited exploration of alternative explanations: The paper acknowledges possible explanations for the emergence and transience of in-context learning but does not thoroughly explore alternative hypotheses. Further investigation into optimization quirks, initialization schemes, or alternate optimizers could provide deeper insights into the phenomenon.\n\nOverall, the above-mentioned points can be seen more as ideas for future work rather than genuine weaknesses of the paper. \n\nMinor points:\n- Lack of statistical significance analysis: Although the experimental results are presented clearly, the paper does not explicitly address the statistical significance of the findings. Including statistical tests or discussing the significance of the observed differences would strengthen the conclusions.\n\n- (minor) Maybe explain in-weight learning in the abstract, contrast it to in-context learning\n- (minor) second to last sentence in the abstract (l. 13), reformulate: how does that question logically follow from the observation?\n- (minor) l. 18 Reformulate definition to be clearer, order of words\n- redo the references, missing information (ls. 373, 394, 408), inconsistent formatting (e.g., ls. 307, 315)\n- Capitalize \"In-Context\" instead of \"In-context\" in the title.\n- Clearer motivation for why researching ICL is interesting and relevant?\n\nQUESTIONS:\n- Were the plots in the paper based on multiple or single runs? If only a single run was used, how was statistical significance ensured?\n- Have you considered the potential interplay between ICL and other important mechanisms in transformers, such as contextual information used in conjunction with IWL for narrowing down possible outputs via copying?\n- Figure 5 shows how many runs were averaged, and what do the error shadings indicate?\n- Could you further explain the statement that the balance between ICL and IWL is a mirror reflection of grokking? Why would ICL, being more generally useful, be transient instead of persisting?\n- Have you explored alternative regularization techniques or approaches to achieve the coexistence of ICL and IWL?\n- When the regularization value exceeds 1e-4, ICL transience returns and IWL also does not work. Does this imply a decrease in accuracy with more training? What is the relationship between regularization, ICL, and IWL?\n- What may be the underlying mechanism by which regularization affects the persistence of ICL? Does it make IWL more difficult or even impossible, leaving only ICL as a viable option?\n- How do you envision your findings' potential real-world applications or implications on ICL and its transience in transformer networks?\n- How might this research impact the development and training of AI models in practical scenarios?\n\nLIMITATIONS:\nWhile the potential negative societal impacts are not explicitly discussed in the provided excerpt, the authors have identified areas for further investigation and improvement in their research, which addresses some limitations of their work. \n\nSince this work focuses primarily on analytical rather than directly enhancing capabilities, there are no immediate negative effects to be considered.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the emergence and persistence of in-context learning (ICL) in Transformer models. Training Transformer models on tasks by Chan et al. [10], the authors find that ICL does not persist, despite an initial emergence. Loss reduction suggests a preference for in-weight learning after some training steps. Through specialized experiments, the authors explore model size, training dataset statistics, and L2 regularization to understand and mitigate this phenomenon. The findings emphasize the importance of L2 regularization in Transformers during extended training. This research provides valuable insights into ICL dynamics, contributing to better understanding of this phenomenon and offering insights into potential improved training strategies.\n\nSTRENGTHS:\n1. The paper builds upon the established dataset by Chan et al. [10], ensuring reproducibility of results. The phenomenon of interest is consistently observed across different scenarios, allowing for comprehensive examination. The inclusion of small-scale experiments facilitates further investigation by researchers. The availability of the code, given that it will be released, will help with reproducibility and transparency of the study.\n2. The authors adopt a systematic approach to address the problem at hand, systematically eliminating various factors that contribute to the lack of persistence in in-context learning (ICL). Through controlled experiments, they explore and analyse multiple aspects that may impact ICL persistence.\n3. The paper has a well-structured and coherent storyline, ensuring clarity of the results. The authors effectively motivate each experiment, providing a clear rationale for their choices. I enjoyed reading the narrative presented by the authors.\n\nWEAKNESSES:\n1. The phenomenon is observed for a single small dataset. Although some experiments are made with respect to aspects of the training data, i.e. number of classes and number of exemplars per class, the small relative size of the training data in conjunction with the large number of training steps tested for here, may potentially lead to conclusions that do not generalize past this setting. Some attempts were made to match data distribution with naturalistic data, but any conclusion that generalize to large language models have to be taken with a grain of salt.\n2. Although a lot of different effects are explored and some correlations explored, no clear answer is given in the end.\n\nQUESTIONS:\n1. Connections to Grokking are not obvious to me. In Grokking [27] memorization at early phase of training is replaced by more general solutions at later stages of training. In your context, the opposite effect seems to be happening, if you assume that ICL are the more general solutions.\n2. What is the effect of different positional encodings to the emergence and persistence of ICL? It would be interesting to study.\n3. In Figure 7 what is the \"chance\" prediction for IWL? Different runs here have different number of classes.\n4. You can match the color scheme between Figures 9,10 and Figure 11.\n5. Line 242-244 \"Furthermore, these results imply that it may be desirable to stop training large transformer models before a compute budget is reached—to preserve ICL—even if training losses are still decreasing.\": the statement is a bit too bold given the experimental setup of the paper.\n6. Line 258-259 \"lottery tickets contained in common initialization methods [27, 37] that incentivize ICL\": I find this a very interesting conjecture that could be explored. \n7. Another potential effect that could be explored may be the \"softness\" of the softmax function. Perhaps the network realizes that to decrease the loss further it needs to resolve to IWL. It would be interesting to track the entropy or \"hardness\" of the attention mechanism. If the softmax is deemed to be the deciding factor, you can think about replacing the softmax with a harder version e.g. [1, 2, 3] and see if the transient effect disappears. \n\n[1] Martins, André, et al. \"Sparse and continuous attention mechanisms.\" Advances in Neural Information Processing Systems 33 (2020): 20989-21001.\n\n[2] Hahn, Michael. \"Theoretical limitations of self-attention in neural sequence models.\" Transactions of the Association for Computational Linguistics 8 (2020): 156-171.\n\n[3] Chiang, David, and Peter Cholak. \"Overcoming a theoretical limitation of self-attention.\" arXiv preprint arXiv:2202.12172 (2022).\n\nLIMITATIONS:\nThe paper discusses a limited scenario where ICL is not persistent. Experiments are performed but always in the context of the provided dataset and for smaller models. More experiments are in general required to verify the existence of this phenomenon elsewhere. The study is nonetheless very interesting and useful for the community.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores In-Context Learning (ICL), a model's ability to adapt its behavior during inference without weight updates, and In-Weights Learning (IWL), where the model relies on information stored in its weights. While early neural networks displayed ICL in controlled scenarios, research changed with the transformer model, particularly with GPT-3, which exhibited ICL without deliberate training efforts. However, it was found that ICL isn't guaranteed when training transformers, with language data properties like burstiness and skewed distribution playing a crucial role. Additionally, ICL and IWL often seem to oppose each other, with ICL emerging more readily when training data has a larger number of tokens or classes and is bursty. Despite the challenges of relying on large models, overtraining is currently a favored approach for training small yet high-performing transformers. The paper challenges the assumption of persistence (that once ICL emerges it remains), showing that ICL can be transient, with the potential to disappear if networks are trained carelessly for longer periods.\n\nSTRENGTHS:\nOn the positive side, the paper is certainly quite interesting and relevant to the major open question these days, which is the emergent capability of large language models. The paper raises a number of interesting open questions (see Discussion in section 7). In fact, I would argue that that is the most interesting section of the paper.\n\nWEAKNESSES:\nOn the negative side, the paper can be viewed as \"half-baked\" at this point. It relies heavily on some empirical results, derived mostly from small models and visual tasks, and it raises a number of questions that are interesting but.. open. I wish the paper could go a bit more deeply to actually answer at least one of those questions. At this point I cannot say with confidence that the paper is ready to be presented at the premier conference of deep learning.\n\nQUESTIONS:\nMy major question is -- is the process of Chan et al. [10], which has been adopted by the authors of this paper, really capable to answer such fundamental questions about ICL and the emergent capabilities of LLMs? The queries that the paper relies on very short -- just 8 exemplar-label pairs and one query image -- i.e., just a sequence of length 17. That also means that the provided \"context\" is very short. this could be a reason that ICL is not persistent. \n\nAnother potential issue is the \"context\" in these experiments is very highly informative regarding the given query. This is why we see that ICL gives a much much larger accuracy than IWL in all plots of the paper. Is this a realistic/pragmatic evaluation regime? I would assume that in practice the context is only partially informative (at least for NLP tasks) and that the network has to rely in anyway much more on trained weights.\n\nLIMITATIONS:\nThe paper is quite open about its limitations, mentioning clearly that it has not utilized very large models and so it is not really certain whether the major claims of this study would be actually still seen in large models such as GPT3 or 4. Additionally the paper does not try to \"oversell\" what it does -- and it identifies many open questions instead of claiming that it already has answers to those questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents empirical evidence illustrating the transient nature of in-context learning (ICL) in transformers, and it shows that the models may shift to in-weights learning (IWL) when overtrained. Furthermore, the transience of ICL is studied across diverse settings such as model size, dataset size, data distributions, and training regularization. Results reveal that adequate L2 regularization can mitigate this transience.\n\nSTRENGTHS:\n1. This paper reveals a characteristic of transformers performing transient ICL.\n2. The proposition that regularization can eliminate the transience is noteworthy.\n3. The provided experiments are clearly organized and easy to follow.\n\nWEAKNESSES:\n1. The paper lacks clarity in its notations, which makes them hard to follow without reference to previous literature. For instance, the term \"in-weights learning\" needs to be related to supervised learning; in the Zipfian distribution equation $p(X = x) \\propto x^{-\\alpha}$, the terms $X$ or $x$ should be explained as representing the rank of input classes.\n2. While the authors provide numerous empirical evidences, they scarcely discuss potential underlying reasons. Some interpretations or conjectures would enrich the discussion around the observed outcomes.\n3. The experiments are constrained to a single dataset and transformer (with different sizes), which is not convincing enough in my opinion. It would be interesting to see if the similar phenomenon is observed with other datasets (e.g., NLP dataset) or other auto-regressive models. The usage of the Omniglot dataset—a stationary supervised dataset—where in-context samples can be regarded as independent entries, makes the existence of IWL unsurprising.\n\nQUESTIONS:\n1. Why are the IWL accuracies consistently low? Would further training iterations increase the IWL accuracy and what is the maximum achievable IWL accuracy?\n2. Have the authors considered that the observed ICL performance drop might be due to the fact that evaluation in-context data distribution is different from training, especially as the label space changes? In Fig 5(c), when evaluation and training use the same data distribution, the performance does not decay. \n3. The connection between Fig5(a)(b) and Fig5(c)(d) seems unclear given the different training datasets. What's more, in Section 4.1, the authors' claim \"Figure 5d shows that larger embedding dimensions enhance the model’s ability to perform IWL, but do not consistently help ICL (Figure 5c).\" seems puzzling. Since ICL accuracies are already close to ones, is there room for improvement? Would reducing the embedding dimension identify a threshold beyond which ICL accuracy is $<1$?\n4. In the Discussion section, the authors note \"However, after training for a long time, we find that even rare classes begin to be learnt in-weights.\" However Section 4.3 does not provide evaluation over rare classes.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}