The Transient Nature of Emergent In-Context Learning in Transformers
Authors: Aaditya K Singh
Stephanie C.Y. Chan
Ted Moskovitz
Erin Grant
Andrew M Saxe
Felix Hill
Venue: NeurIPS 2023 poster (2023)
Keywords: in-context learning
transformers
emergence
transience
TL;DR: Emergent in-context learning in transformers is often a transient effect.

Abstract: Transformer neural networks can exhibit a surprising capacity for in-context learning (ICL) despite not being explicitly trained for it.  Prior work has provided a deeper understanding of how ICL emerges in transformers, e.g. through the lens of mechanistic interpretability, Bayesian inference, or by examining the distributional properties of training data. However, in each of these cases, ICL is treated largely as a persistent phenomenon; namely, once ICL emerges, it is assumed to persist asymptotically. Here, we show that the emergence of ICL during transformer training is, in fact, often transient. We train transformers on synthetic data designed so that both ICL and in-weights learning (IWL) strategies can lead to correct predictions. We find that ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases, indicating an asymptotic preference for IWL. The transient nature of ICL is observed in transformers across a range of model sizes and datasets, raising the question of how much to ``overtrain'' transformers when seeking compact, cheaper-to-run models. We find that L2 regularization may offer a path to more persistent ICL that removes the need for early stopping based on ICL-style validation tasks. Finally, we present initial evidence that ICL transience may be caused by competition between ICL and IWL circuits.

The Transient Nature of Emergent In-Context Learning in Transformers
Aaditya K. Singh" Gatsby Unit, UCL
Stephanie C.Y. Chan" Google DeepMind
Ted Moskovitz Gatsby Unit, UCL
Erin Grant Gatsby Unit & SWC, UCL
Andrew M. Saxe! Gatsby Unit & SWC, UCL
Felix Hill Google DeepMind
Abstract
Transformer neural networks can exhibit surprising capacity for in-context learn- ing (ICL) despite not being explicitly trained for it. Prior work has provided a deeper understanding of how ICL emerges in transformers, e.g., through the lens of mechanistic interpretability. Bayesian inference, or by examining the distributional properties of training data. However, in each of these cases, ICL is treated largely as persistent phenomenon; namely, once ICL emerges, it is assumed to persist asymptotically. Here, show that the emergence of ICL during transformer training is, in fact, often transient. We train transformers on synthetic data designed that both ICL and in-weights learning (IWL) strategies can lead correct pre- dictions. We find that ICL first emerges, then disappears and gives way to IWL, all while the training loss decreases, indicating an asymptotic preference for IWL. The transient nature of ICL is observed in transformers across range of model sizes and datasets, raising the question of how much to "overtrain" transformers when seeking compact, theaper-to-run models. We find that L2 regularization may offer path to more persistent ICL that removes the need for early stopping based on ICL-style validation tasks. Finally, we present initial evidence that ICL transience may be caused by competition between ICL and IWI. circuits.
Introduction
In-context learning (ICL) is the ability of model to use inputs at inference time to adapt its behavior, without weight updates, in order to solve tasks not present during training. The first examples of this behavior in neural networks were observed in architectures specifically designed and trained for few-shot learning, the capacity to learn a desired behavior from only a few examples [1-3].2 Training in these cases involved shuffling the labels corresponding to input exemplars each "episode" so that, to perform well on the training set, the model had to recall exemplar-label mappings from context to make future predictions. At test time, novel exemplar-label mappings were provided, and the network had to use these to classify query exemplars.
Research on ICL changed with the development of the transformer [5]. Brown et al. [6] first observed that a transformer-baset language model, GPT-3, trained auto-regressively at sufficient scale, exhibited ICL without any specific effort of the authors to promote it via the training objective or data. Since then, large body literature has studied [7-12] or reported [13-15] examples ICL in transformers trained at scale.
CCo-first authors; direct correspondence to aaditya. sirra..sinh.... 21@ucl. ac. uk and scychan&google.c com com Co-senior authors
2 Few-shot learning can also implemented via methods that apply gradient updates [c.g., 4], but we focus on implementations of ICL that do not require any weight updates.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) In content learning.
peak height
1.0
0.8
decay slope
accuracy
0.6
chance
0.4
peak onset
0.2
2
3
# train steps
1e7
(b) In ne-weights learning.
(c) Train log loss.
1.0
2
0.8
train log loss
accuracy
0.6
0.4
-6
0.2
0.0
chance
2 # train steps
1
1e7
# train steps
1e7
Figure 1: In-context learning is transient, shown for our "default" settings: 12 layers, embedding dimension of 64, trained on 1,600 classes, with 20 exemplars per class. All training sequences are bursty (see Figure 2a for details). Chan et al. [10] found these settings to strongly incentivize ICL, but did not observe ICL transience (see Figure 3), as they did not train long enough. (a) ICL evaluator accuracy. (b) IWL evaluator accuracy. We note that, while accuracy on train sequences is 100%, accuracy on the IWL evaluator is very slowly increasing, as the test sequences are out-of out-of- di-striibuutonn See Appendix for further investigation. (c) Training log loss. Two colors indicate two seeds used for experiments.
These compelling findings have sparked surge of research into the phenomenon of emergent capabilities in large neural networks. Nonetheless, recent work has shown that ICL, is not guaranteed outcome of training transformers. Chan et [10] found that specific properties that are typical of language data, such burstiness and its highly skewed distribution, play important role in emergent ICL in transformers. When trained on data without these properties, Chan et al. [10] found that transformers tend to resort to in- in-weights learning (IWL). In the IWL regime, the transformer the model, rather than new information provided
relies on information stored in the weights in-context." Importantly, ICL and IWL often appear to be in opposition, with ICL emerging most readily when training data has large number of tokens or classes and when that data bursty, with items appearing in clusters rather than being uniformly distributed.
Such controlled studies based on known data- gentratnnn distributions are critical for better under- standing the ICL phenomenon in transformers. At the same time, a complementary body of work studies emergence in massive models trained directly on organic web-scale data, with the takeaway that impressive capabilities such ICL are more likely to emerge in large models trained on more data [14, 17-19]. However, this reliance on massive models poses serious practical challenges, such as how innovate rapidly, how to train energy-efficiently and in low-resource settings, and how to run efficiently at deployment. Thus, growing line of research [20-22] has focused on achieving comparable performance (including emergent ICL) in smaller transformer models.
Overtraining is currently the favored approach for training small yet performant transformers. These small models are trained on more data, possibly with repetition, beyond what is prescribed by scaling laws and compute budget [13, 15]. At its core, overtraining relies on an assumption that implicit in
We also note that ICL and IWL are not so cleanly distinguishable in many cases: ICL can depend at least partially on information stored in in-weights, and behavior that looks like IWL may still fact using context naively, e.g., via simple copying biases [16].

---

(a) Example sequences and outputs.
(b) Model schematic.
Strategy used
327
ICL
IWL
Train data:
Oses
seee
L
< 327
Ree
327
327
Transformer
eval:
ICL
?
?
C
0
327
embed embed
embed
embed embed
eval:
IWL
648
<i. 2
CL
1ss
222
Saru 478
3ese
221
327
@
327
1
589
<
example outputs
exemplar label
exemolar label exempler
context
query
Figure 2: An overview of our setup. (a) Example sequences during training, ICL evaluation, and IWL evaluation. Example outputs are colored green when correct and red when incorrect. Note that for train sequences, ICL and IWL strategies both result in the correct answer. On ICL eval sequences, the IWL prediction is incorrect. ICL is required to remap exemplars to the randomized 0 or 1 labels. On IWL eval sequences, there are no matching exemplar-labe pairs in context, so IWL is necessary. (b) Model schematic. Training and evaluation focuses on the predicted label for the final exemplar.
most (if not all) recent explorations ICL in LLMs: persistence. That is, it assumed that once given model has been trained sufficiently for an ICL-dependent capacity to emerge, that capacity will be retained as training progresses, provided the training loss continues to decrease.
Here, we show that the general assumption of persistence false. We do so by adapting a standard umsge-based few-shot dataset [23], which allows us to evaluate ICL carefully under controlled conditions. We characterize simple cases where ICL emerges, only to be subsequently lost while the model's loss continues decrease. In other words, while ICL is well-known be an emergent phenomenon, we should also think of it as a potentially fransient effect (Section 3, Figure 1). We find this transience occurs across range of model sizes (Section 4.1), dataset sizes (Section 4.2), and dataset types (Section 4.3), though we do identify some properties (Section 4.4) that can postpone transience. In Section 5, we study ways alleviate this risk, and show that regularization may offer path to persistent ICL. In Section 6, we present preliminary evidence that ICL transience is caused by competition with IWL circuits.
In general, if networks are trained carelessly for longer and longer, we find that ICL can disappear as readily it emerges, leaving models without capabilities that we increasingly expecting to find in modern Al systems.
2 Experimental Setup
2.1 Dataset construction
In order to study the transience of ICL across a range of model and dataset sizes in a controlled fashion, use synthetic gettereennation process inspired by Chan et al. [10], which allows us to properly assess in-context versus in- in-weights learning. Our data generators are primarily built from the Omniglot dataset, standard benchmark for few-shot learning, which consists of images of handwritten characters from different alphabets [23, MIT License]. The Omniglot dataset consists of 1623 character classes, with 20 exemplars each. In experiments with more than 1,600 classes, construct additional image exemplars by applying rotations and flips of the original Omniglot images.
Inputs to the transformer model are sequences of exemplars and labels. We use 8 exemplar- pairs (the "context") followed by one "query" exemplar, for an input sequence length of 17 (Figure 2a). The model is trained to minimize the loss in predicting the label for the query exemplar (Figure 2b).
To explore the generality of our results, we also extend to exemplars derived from language model token embeddings. We use pre-trained token embeddings from the LLaMa family [15] of large language models, which use a vocabulary size of 32,000 tokens. We extract the input embedding matrices from all four open-source L.LMMa models. We then subset and cluster [24] these token embeddings to form 3,200 classes of tokens each. For each I.LaMa model size, we construct two such datasets, with different clustering seeds. The full procedure is described in Appendix C.
3

---

(a) In-context learning.
(b) In- In-weights learning.
1.0
1.0
Pibursty)
0.8
0.8
accuracy
0.5
0.6
accuracy
0.9
0.4
0.2
0.0
2 3
0.0
chance
train steps
le5
train steps
165
Figure 3: (Reproduced with permission from Chan et al. [10].) It was previously shown that ICL can be transient (purple and red curves) when transformers are trained on data that is only weakly conducive to ICL (e.g.. with low levels of burstiness). P(bursty) indicates the fraction of training sequences that are bursty. All curves in this figure train on a dataset with 1,600 classes and 20 exemplars. Note the z-axis scale: At this number of iterations, there no sign of ICL transience for the highest level of burstiness, but our experiments are run for a much larger number of steps. We also note that, when P(bursty) < 1, some of the sequences during training are of the same form as the IWL evaluation sequences. As a result, the IWL evaluator is less out-of- ditribbbbi,,,,nn, which is why the accuracies in (b), when P(bursty) 1, are so high.
Training sequences. The training task (see top row in Figure 2a) can be solved via ICL or IWI. The context for each training sequence contains exemplar-label pairs from the query class ("bursty" sequences), which prior work has found to incentivize ICL [10]). Bursty sequences allow the network to look for matching exemplars in-context output a label for the query example, an example of ICL. To avoid repetition biases, the training context also includes 3 exemplar-label pairs from a distinct, distractor class. We keep exemplar-label mappings fixed across sequences, so the network could learn mapping from exemplars to labels in- ne-eighhts (without needing to find matching exemplars in-context). This design allows us to observe whether the model prefers ICL or IWL at different points in time, as a function of model size or in response to the properties of the training data.
Evaluation sequences. We follow Chan al. [10] and construct two kinds of evaluation sequences that measure the extent to which a trained network relies on ICL versus IWL. To evaluate ICL. (see second row of Figure 2a), we use sequences with 4 exemplar images from each of two classes, but we set the class labels to either 0 or I at random for each sequence (unlike during training, where the exemplar-label mappings are fixed). Accuracy on this evaluator is measured across 0 and as possible outputs (so chance-leve accuracy is 50%). As these labels were not associated with these exemplars during training, the only way to achieve above-chance accuracy on this evaluator is for the network to refer back to the context and copy the label corresponding to matching exemplars. To evaluate IWL (see third row of Figure 2a), we use sequences where none of the context exemplars come from the same class as the query, but all of the exemplar-label mappings are the same as during training. In this case, ICL is not useful (as there are no matching exemplars in-context), so the network must rely on exemplar-labe mappings stored in-weights (IWL) for above-chance accuracy. Other evaluation techniques for ICL and IWL are considered in Appendix B.
2.2 Model details
For our transformer model, the "default" settings consist of 12 layers, with an embedding dimension of 64 and an additive, sinusoidal positional encoding scheme [5]. Each element of the sequence is passed through an embedder, either standard embedding layer for the label tokens, a ResNet encoder for the exemplar images, or a simple linear layer for exemplars created from LLaMa tokens. (Figure 2b). Both embedders are trained jointly with the network. In our experiments, we vary multiple aspects of this architecture and observe the effect on the transience of ICL. All experiments were run with 2 seeds and used the Adam optimizer [25]. We did not observe qualitative differences in behavior between seeds. We ran most experiments for 5e7 iterations, but truncated some early when signs of transience were clear. More details (e.g., specific hyperparameter choices, and links to code) can be found in Appendix A.

---

(a) In-context learning.
(b) In- In-weights learning.
1.0
1.0
# layers
0.8
0.8
6
accuracy
accuracy
12
0.6
0.6
18
chance
0.4
0.4
0.2
0.2
0.0
0.0
chance
2
2
3
# train steps
1e7
# train steps
1e7
Figure 4: (a-b) ICL is transient regardless of model depth, with no clear trend of peak height or peak onset. Decay slopes are roughly similar across model sizes.
3 In-context learning is transient
Figure la shows that ICL is transient even when using data-distributional settings previously found by Chan et al. [10] to strongly incentivize ICL. We annotate various aspects of the ICL curve, such as the peak height, peak onset, and decay slope. Prior work [10] stopped training at 5e5 training steps, before peak onset. Training for much longer in identical setting leads to the disappearance of ICL. Notably, ICL is slowly replaced by IWL (as seen by the steadily increasing accuracy on the IWL evaluator, Figure 1b), all while the training loss continues decrease (Figure Ic).
Chan et al. [10] found some transient ICL in other data regimes, such as when some small percentage of sequences are non-bursty, where the context does not contain the query (Figure 3). In these settings, the eventual rise of IWL is to be expected, ICL cannot minimize the loss on the non-bursty sequences. Notably, results (Figure 1) show surprisingly that ICL transience extends even to cases where ICL is "sufficient" to fully solve the training task (such as when the context always contains a relevant exemplar-label mapping that can be used). When comparing to the results from Chan et al. [10] (Figure 3), we see that higher levels of burstiness lead to a higher peak height and gentler decay slope. When training for fewer iterations, it might appear that ICL is persistent, but our results from training longer Figure demonstrate that ICL is actually transient.
4 In-context learning is transient across settings
4.1 Effect of model size
We investigate whether the ICL transience effect is modulated by model size, given the current prevalence of extremely large transformer models. We find that there is no consistent effect of model depth on ICL transience (Figure 4a-b). In fact, deeper models can lead to lower peak heights (e.g.. the 18-layer model has lower peak height than the 12-layer model). Notably, the decay slope is similar across different model depths, indicating that the ICL is unlikely to persist simply by scaling model depth.
We also experiment with the width the model in Section 6, which sheds some light on the causes of ICL transience.
4.2 Scaling dataset size
Next, we investigate the effects of dataset size on ICL transience. There are two primary ways of varying the dataset size: increasing the number of exemplars per class (a form of increasing in-class variation), or increasing the number of classes.
Chan et al. [10] found that in-class variation was crucial for ICL to emerge, and we find similar trend. When the number of exemplars per class is increased (from 10 to 15 20) while keeping the number of classes fixed, we see an increase in ICL peak height (Figure 5a). However, despite higher peak heights, we also see steeper decay slopes, indicating that increasing in-class variation unlikely to make ICL persistent.
5

---

(a) In-context learning.
(b) In-weights learning.
1.0
1.0
exemplars class
0.8
10
15
chance
20
0.4
0.2
0.2
8.0
1.0
0,0
0.5
chance
train stops
# train steps
1c7
Figure 5: In-class variation improves ICI., previously known and seen here by the higher peak heights, but ICL. nonetheless transient across settings.
(a) In-context learning.
(b) In-weights learning.
1.0
1.0
12800 classes, 20 exeemllass each
0.8
0.8
accure
acquracy
1600 classes, 160 exemplars each
0.6
chance
1600 classes, exemplars each
0.4
0.4
0.2
0.2
8..0
0.5
1.5
0,,0
chance
+ train steps
1c7
train steps
1e7
Figure 6: Effect of dataset size on ICL transience. Increasing the number of classes leads to higher ICL peak height and a more gradual decay slope. Despite the number of exemplar- label pairs being the same in the purple and red curves, note that the ICL and IWL behavior very different when we increase the number of classes (purple) versus the in-class variation (red). Also, note that chance level performance for IWL differs slightly: 1/12,800 for the other curve (purple), and 1/1,600 for the 1,600 classes lines (red and orange).
When increasing the number of classes, we again find a similar trend Chan al. [10] with peak heights increasing (Figure 6a). In this case, we also find that decay slopes become more gradual. To better illustrate the differences between increasing in-class variation and increasing the number of classes, we do another experiment where the total number of exemplar-label pairs is fixed to 12, 800 x 20 1, 600 x 160 256, 000, with one run having more classes and less in-class variation, and another run having fewer classes and more in-class variation. We find that increasing the number of classes is more effective way to promote ICL (higher peak height, shallower decay slope) than increasing the in-class variation, perhaps due to each class appearing less often during training. This result also indicates that, beyond a certain point, in-class variation actually hurts ICL performance (lower peak height, steeper decay slope, as seen in Figure 6a). To conclude, ICL remains transient as we increase the number of classes, but training for longer and longer becomes necessary to make go away, given the shallower decay slope.
4.3 Extending to language model token embeddings
To explore the generality of our results, replaced the image exemplars with exemplars derived from sanguage-mode token embeddings (procedure described fully in Appendix C). On all of these
In-context learning.
(b) In-weights learning.
Embeddings from LLaMa1 model size:
7B
accurai
0.6
accuracy
0.6
138
chance
308
0.4
0.4
708
0.2
0.0
0.0
0.5
1.0
1.5
2.0
train steps
train steps
1e6
Figure 7: ICL is transient across range of input datasets, derived from clusterings of LLaMa I token embeddings from the four different sizes. Error bars indicate standard error across two possible datasets derived from each model's embedding matrix. Further results be found in Appendix C.

---

token-emme eeeen-enbedding derived datasets, we find that whenever ICL emerges, it is subsequently transient (Figure 7). Furthermore, the peak height lower than in our other experiments. In terms of timing. the peak onset as well as subsequent transience occurs on quicker timescale (note the shorter a-axis scale in Figure 7). These results also further corroborate our findings on dataset size (Section 4.2), as the constructed token embedding-labes dataset is smaller (3. 200 x 5 = 16, 000 exemplar-lahed pairs, as opposed to 1, 600 x 20 = 32, 000 pairs in our Omniglot experiments), so we would expect lower heights and sharper decay slopes. Overall, these experiments are a promising step towards connecting our findings to natural language, and demonstrate that the transience phenomenon not limited to image-basee based exemplar-labe datasets.
4.4 Changing the data distribution
Until now, we have trained on data for which the marginal distribution over classes was uniform. However, common property of naturalistic data Zipfian (power-law) skew [26]. Zipfian data contains a few "common" classes that appear very often, and many "rare" classes that appear less often than they would the class distribution were uniform. In Section 4.2, we found that more classes (corresponding to each class appearing less often) led to shallower decay slopes. We explored whether data with a Zipfian distribution could play a similar role in reducing the transience of ICL.
Indeed, in Figure 8, we find this to be the case. We implemented different levels of skew across data classes according to Zipfian distribution p(X x) x x =a (x corresponds to the rank of an input class e.g., a = O corresponds to a uniform distribution). At Zipf exponent a = 1, we find a severely delayed ICL peak onset, thereby delaying the subsequent transience, and shallow decay slope following this delayed onset (similar to that seen with a large number of classes in Figure 6a). Furthermore, corroborating the findings of Chan et al. [10], we find that stopping training early, when using skewed distributions, evidenced by the increased IWL at Zipf exponent a = as compared to Zipf exponent a = 0, Figure 8b). However, it is important to note that if the Zipfian exponent is too extreme (c.g., does not emerge at all, and the network uses IWL to solve the task. We conclude that moderately skewed Zipfian data does not eliminate transience, but mitigates it by necessitating training for an even longer time to observe transience (similar to when there is large number of classes).
lead to combination of ICL and IWL in the same network (as
2). ICL
(a) In-context learning.
(b) In-weights learning.
1.0
Zipf exponent
0.8
0.0
accuracy
accuracy
1.0
0.6
chance
0.6
2.0
0.4
0.2
0.0
2 # train steps
0.0
chance
1e7
train steps
1e7
Figure 8: ICL transience as a function of dataset skew, implemented across classes in accordance with Zipfian distribution. (a) A moderate Zipf exponent of leads significant delay of ICL peak onset followed by a shallow decay slope. implying models would need to be trained for even longer to get full transience. However, an extreme level of skew (Zipf exponent 2) leads the complete loss of ICL altogether. (b) IWL, evaluated on the same skewed distribution over query classes as in the training data (thus high accuracy on Zipf exponent 2 is not unexpected, as the IWL test set is dominated by small number of classes).
5 Regularization may eliminate ICL transience
If we consider that ICL is a more generally useful solution than IWL, then ICL transience is, in sense, the opposite of the grokking phenomenon, where model changes from less to a more general solution after long periods of training [27]. Regularization (or implicit regularization) may help to drive grokking [28, 29] (and some work even shows an impact of dataset size as well [30], analogous to what we find above). Thus, would we see regularization mitigating the ICL. transience phenomenon?
7

---

(a) In-context learning.
(b) In-weights learning.
1.0
0.10
L2 regularization
0.08
acouracy
0.6
accuracy
0.06
Se-05
chance
0.04
1c 04
0.2
0.02
0.00
0.00
1
2
# train steps
1e7
# steps
1e7
Figure 9: L2 regularization potentially eliminates ICL transience. Increasing regularization decreases ICL transience, least up to point. Regularization also seems to lead instead IWL transience. (Note that the y-axis in Figure 9b is zoomed in so that IWL. transience is visible.)
We see in Figure 9 that this is indeed the case, with L2 regularization seeming to even eliminate ICL transience entirely, at least on the timescales that we tested. Without any regularization, we see transient ICL as before. However, as we add increasing levels of L2 regularization, we see that the decay slope of ICL performance goes to O. On the other hand, we actually observe some transience of in-weights learning (purple and maroon curves, Figure 9b). These results indicate that the ICL circuit is a lower norm solution than the IWL circuit. Thus, regularization may offer a path to persistent ICL. However, with values of regularization that are too high (>> 0.0001), the network is over-regularized and ICL transience returns, but, notably, IWL accuracy never rises above zero either (see Appendix D for more train loss curves and more extreme values of regularization).
6 ICL transience may be caused by competition with IWL circuits
We performed initial investigations to understand the origins of ICL transience: why does ICL fade when the IWL solution emerges? Do the ICL and IWL circuits compete with each other for resources, for example, in the transformer's residual stream [16]? If so, we would expect wider transformers with larger embedding size (and therefore greater residual stream capacity) to be less afflicted by ICL transience. In Figure 10a, we see that this is the case: Greater model embedding sizes lead to higher peak heights and gentler decay slopes. However, one confound consider is that larger embedding may just independently enhance ICL abilities (as they enhance IWL abilities, Figure 10b), rather than reduce competition between circuits.
To better understand how scaling the embedding dimension enhances ICL and IWI. we trained models data that could be solved via ICL only IWL only, rather than being solvable by both strategies. For ICL-only training, we used the same type of sequences used in our ICL evaluator, where the network must use ICL to solve the query-prediction task. For IWL-only training, we used the same type of sequences used in our IWL evaluator, where the network must use /WL to solve the query-prediction task. Figure 10d shows that larger embedding dimensions enhance the model's ability to perform IWL. but do not consistently help ICL (Figure 10c). Thus, these results indicate that the increased peak height and shallower decay slope Figure 10a are likely due decreased competitive interactions with IWL circuits, rather than an enhancement of ICL capabilities.
We arrived at convergent evidence by building the regularization experiments from Section 5. We applied weight decay selectively to either the ResNet embedding layers, the MIP layers, or the self-attention layers. When applied weight decay to only the self-attention layers, ICL was still transient. When we applied weight decay to the MLP or ResNet layers, however, ICL transience was mitigated (Appendix Figure 16). These results can be interpreted in light of prior work indicating that in-weights information is stored in MLP layers [31, 32). By selectively penalizing this behavior, we enable ICL to persist, thus providing convergent evidence that when no weight decay is applied - ICL fades due to competition with IWL circuits.
7 Related Work
This work builds on several papers that seek to understand in-context learning by analyzing small networks and/or controlled data sources. Xie et al. [7] consider an implicit definition of ICL (defined terms of the model's loss) and show that the effect can framed as a form of Bayesian inference. Chan et al. [10] show via experimentation with controlled synthetic datasets that, e.g., the bursty and
8

---

(a) cn-context learning.
(b) In weights learning.
1.0
Embeed dim
0.8
0.8
32
accuracy
accuracy
64
0.6
chance
0.6
128
0.2
0.2
0.0
0.0
chance
2
# train steps
1e7
train steps
1e7
(c) in-context learning, when training on data that (d) In-weights learning, when training on data that only
only supports in in-context learning.
supports in-weights learning.
1.0
1.0
Embed dim
0.8
0.8
32
accuracy
accuracy
64
chance
0.6
128
0.4
0.4
0.2
0..0
0.5
1,0
1.5
2.0
2.5
3.0
0.0
2.0
2.5
3.0
< chance
train steps
train steps
1e7
Figure 10: (a-b) We find trend that wider models (larger embedding sizes) have higher ICL peak heights and more gradual decay slopes. (c-d) However, when we trained on data that allowed us to specifically elicit each type of learning, found that embedding size only consistently helpful for enhancing IWL, indicating that mitigation ICL transience is likely due to decreased interaction between the two circuits, opposed to enhancement of ICL capabilities.
Zipfian nature of text data could play an important role in the emergence of ICL. Many other works consider how ICL can be used to learn certain function classes using synthetic data, and how ICL may be implementing a form of gradient descent to accomplish this [33-35].
As with the aforementioned work, here we do not experiment directly with billion- parommeer trans- formers trained on web-scale corpora. Nonetheless, our work should be of relevance to studies that aim to derive ICI. in increasingly small and compact text-based language models. By training language models of many different sizes on different amounts of data, Hoffmann al. [13] identified relationship between the amount of training and model size that appeared compute-optimal with respect to the training loss. This correspondence motivated Chinchilla, 70B-parameter language model whose downstream performance (on tasks that require ICL) was comparable to far larger models. More recently, Touvron et al. [15] took this further, "overtraining" smaller models to achieve lower training loss according
given, inference-timee compute budget. Perez et al. [36] point out that in such work, tasks that require ICL are typically included in the models' validation set in order optimise hyperparameters directly for ICL. Our results indicate that, without appropriate validation and early stopping, training for long time may eventually lead to ICL gradually disap- pearing, perhaps even without recognition from model developers. A large number of "classes" or Zipfian data distributions (both properties of natural language [37]) can mitigate this by delaying the disappearance. Moreover, regularization may completely eliminate this risk. Indeed, Hoffmann et al. [13] find that the use of optimizers with weight decay [38] leads to models that perform better, but this benefit only manifests later stages of training (perhaps when ICL would otherwise decay).
8 Discussion
In-context learning is often transient. The key insight of this work is the empirical finding that, in-context learning (ICL) emerges in transformer network, it may not necessarily persist as the model continues to be trained. That is, while ICL is often emergent, it may be in many cases also be transient (Section 3). In fact, since loss of ICL gradual, creeping disappearance may not even be known to model developers. This emphasizes the importance of evaluating appropriate validation metries over the course of training [e.g., 39]. Furthermore, these results suggest that may be desirable to stop training some transformer models before compute budget is reached

---

preserve ICL- even if training losses are still decreasing. This would need to be ascertained on case-by-case basis. Other methods for preventing the disappearance of ICL, like regularization, may also be critical for delivering flexible, in-context learners.
What are strategies for mitigating ICL transience? In Section 4, we found that data distributional properties (increasing the number of classes, adding Zipfian skew) can strengthen ICL and postpone the transience of ICL, and so can increasing embedding size (and least in the ranges we tested). The fact that most transformers are trained on language, which naturally has both of these data- distributional properties [37], may explain why ICL transience hasn't 't been documented in LLMs yet. However, as the trend of overtraining continues, there's 's a risk of losing these impressive ICL capabilities. We found that the most robust method to make ICL persistent was L2 regularization (Section 5).
Why does ICL emerge at all, if it not asymptotically preferred? Since GPT-3 [6], the emergence of ICL has become an expected phenomenon in large-scale transformer models. Our experiments underline how curious it that ICL emerges in such models given that it does not seem to be asymptotically preferred. Possible explanations may concern optimization quirks (similar to the slingshot mechanism proposed as an explanation for grokking [29]) or perhaps lottery tickets con- tained in common initialization methods [28, 40J that incentivize ICL (when trained with relevant dintaaiitributionaal properties). Future work could explore these possible explanations by considering the effect of alternate optimizers or initialization schemes on the transience of ICL.
Given that ICL does emerge, why does it fade in favor of IWL.? On our training data, either ICL or IWL. can reach good performance. While we do not know for certain why ICI. emerges in a network, the question remains of why it fades if it is a viable strategy solve the training task. Our results in Section 6 suggest that part of the explanation may lie in a competition between IWL and ICL circuits for resources in the transformer residual stream [16]. Under the assumption that IWL is asymptotically preferred, this competition would explain why ICL fades after emerging.
Why is IWL asymptotically preferred over ICL, when both solve the task? Induction heads [9] are a possible circuit that may be responsible for ICL, and comprise of two interacting components: finding a match in-context, then copying some token forward. The match operation is fundamentally limited by the softness of the attention operation [41-43]. On the other hand, IWL. relies on learning the exemplar-label mapping in wniieights, which we show is feasible in Figure 10d. We postulate and hope to investigate in future work- the possibility that although ICL and IWL can achieve perfect accuracy the task, this "imperfect match" prior context asymptotically incentivizes solutions (when training with cross-entropy loss, as is standard) that do not rely as much on context and instead learn in-whights.
How do we get ICL and IWL to co-exist asymptotically in the same model? Large language models clearly show co-existence of ICL and IWL. Chan et al. [10] found that training on Zipfian data allows the network to perform ICL on rare classes, but learn common classes in-weights. Regularization was the best strategy found to make ICL persistent, but it leads to littlc-or-no IWL. Given our results, the only effective way currently to get co-existence of ICL and IWL appears to be stopping training at the right time. Future work could investigate other factors that may restore co-existent ICL and IWL, even asymptotically.
Limitations and future directions. Perhaps the most significant limitation is that our experiments were not performed on large language models, one of the most important applications of transformers. Our training resources limited our ability to perform such experiments in the current work, but we believe that this is a necessary future step, to understand the full import of the current results. As first step toward this, we demonstrated ICL transience on datasets with language model token embeddings (Section 4.3). but our datasets are still restricted to the exemplar-labe setup. would also be interesting explore the effect of context length: For given train or inference budget, system designer could trade off between longer context (possibly enabling better ICL) and a larger model (possibly enabling better IWL). We also note that our definitions of ICL and IWL are relatively specific, and many more important mechanisms are likely at play in transformers-for example, contextual information could be used in conjunction with IWL to narrow down possible outputs via copying [16), which we find some preliminary evidence of in Appendix B. Future work could investigate more mechanistic explanations of ICL transience, the tradeoff between ICL and IWL, and additional recipes for reducing ICL transience.
10

---

Acknowledgments and Disclosure of Funding
A.K.S. and T.M. are supported by the Gatsby Charitable Foundation. This work was supported by a Schmidt Science Polymath Award to A.S., and the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/Z/19/Z) and the Gatsby Charitable Foundation (GAT3755). A.S. CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.
The authors thank Kira Düsterwald and DJ Strouse for insightful discussions throughout the course of the project. In addition, the authors thank Andrew Lampinen, Pierre Richemond, Roma Patel, and Murray Shanahan for feedback the draft.
References
[1] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[2] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Metalleaming with aegmete-ugmented neural networks. In International conference on machine learning. pages 1842-1833 PMLR, 2016.
[3] Jane X Wang. Meta-learning in natural and artificial intelligence. Current Opinion in Behavioral Sciences, 38:90-95, 2021.
[4] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages 1126-113 1135. PMLR, 2017.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL http::::p///c/..... httpps:/pproeeeddngggee. neurips. cc. cccpppprr_p. f files/ wtt.s//apppee//2///// pptpp//pp////i11//1 htt.::/55...bbb477774416111..a..
Paper. pdf.
[7] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.0000 2021.
[8] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
[9] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https:/trannffommr-ccriiii wttp://///oooooooo///aaaanaaaaao heads/index.html.
[10] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang. Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional proper- ties drive emergent in-context learning in transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, Cho, and Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 18878-18891. Curran Associates, Inc.,

---

2022. URL https://prrocoedings. newrips. T7ocaactts62223377766000000oo
pdf.
[11] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303 033446 2023.
[12] Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. arXiv preprint arXiv:2303.07889,, 2023.
[13] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023) 20200.2..111..... 2022.
[15] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[16] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Htaffied-Doddss Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http://taansforme-cccciiii putpeerrrriiiiiisispu2222222 wtwpr/aiiii..p/////////aeennnnia /fram/work//dde...neee htmml.
[17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint TTt::2001.08366 2020.
[18] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. 2021.
[19] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682. 2022.
[20] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot, Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726, 2023.
[21] Yukun Huang. Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models, 2022.
[22] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
[23] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133(6 -31133388, 2015. doi: 10.1126/science.abbb000 aab3050.. URL https: htttps::/www... science. wwt.ooo/////////ioooooo 1126/science aab3050.
[24] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL. http:/ //t::///xxii /aaxiiiiiiaa/aaa//aaaaaaaaaavvaa org/abs/ 111///111..6 6980.
12

---

[26] George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books, 1949.
[27] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitiing on Small Algorithmic Datasets. Technical Report arXiv:2201.0P2 arXiv, January 2022. URL http://arxiv, www...rg/abba. htt//bb//2000 211.2221122211111 arXiv:2201.02177 [cs] type: article.
[28] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023.
[29] Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon, June 2022. URL
org/ abs//2bb..22... 04817. arXiv:2206.04817
[cs, math].
[30] Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, 2023.
[31] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021.
[32] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
[33] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can trans- formers learn in-context?
case study of simple function classes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 30583-30598. Curran Associates, neurips. cc/ppper_fillleppppr//222222 /ptppr//022/filll/
Inc., 2022. URL httpittts///pooceeeiiiiii Papeee-app-----eeeee Paper-Confert Conference .df. 15220000e1661866677755538000
[34] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.1566 2022.
[35] Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mord- vintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677 2022.
[36] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models, 2021.
[37] Steven T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 2125):1 October 2014. ISSN 1069-9384. doi: Mmc//rrtcc1os//MCA1/655
URL https://www.r ncbi nlm. wwh.ni..... gov/
[38] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CORR, abs/1711.05101, 2017. URL http://arxiv.org: http:/abs/11.../.. 05101.
[39] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[40] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR, abs/ 803.03635, 2018. URL http://arxi:: http::///xxxvvvoo/baaaaasee 1t///100300000 http//8883.00355... 03635.
[41] André F. T. Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro M. Q. Aguiar, and Mário A. T. Figueiredo. Sparse and continuous attention mechanisms, 2020.
[42] Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8: 8:156-171, dec 2020. doi: 10. 1662tll/cccaaa a 00306. URL https:/ //opi. org/oo///// /0/ 1tt.11112/taacccaao02o
13

---

[43] David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention, 2022.
[44] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/ abb//551.005 102.1332,,3385, 2015. URL
03385.
[45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015.
14

---

A Model and training procedure details
All experiments used the same model and training procedure unless stated otherwise. The transformer
[5] consisted of 12 layers with am embedding dimension of 64 and 8 heads. The images were embedded by a ResNet [44] with two blocks per group and channels per group (16, 32, 32, 64). The ResNet embedder was learnt jointly and not pre-trained. The integer labels were embedded using a standard embedding layer. The imput embeddings were augmented with standard sinusoidal positional encoding [5] with timescale 30 (as our sequences are all length 17). Experiments were run for up to Se7 training steps with batch size 32, on 16 TPU v2 or v3 cores. They were trained using Adam [25] (with default parameters of [3 = 0.9, .3 0.999) and a learning rate schedule with a linear warmup up to a maximum learning rate of 3c-4 at 4,000 steps, followed by an inverse square root decay. L2 regularization was implemented by adding the squared weights of the model (excluding batch norm parameters) to the loss term. All experiments were run with seeds each. In all figures, (shaded) error bars indicate standard deviation around the mean.
Also, we note that, despite the long training times, all our experiments are still in the <1 epoch regime, due to the combinatorial nature of our dataset. We estimate there are over 1, 6,000 x 1, 599 x (1, 598 x 1, 597/2) x (81/(3) x 3!)) x (209) = 1.87 x 102T possible training sequences, of which our model sees maximum of 32 x 5 x 10? = 6.4 x 10
Source code can be found at witthubccom/goog..... deepeeeddeeeedddeeeeeeeee headd//eeereeeeiine cttpt/iiinoottttteeeeeeeeee and gi thub. com/ com/aadi
B Alternate evaluation metrics
While the primary evaluations used throughout the paper are inspired by the ICL and IWL evaluators of Chan et al. [10], we did consider a few alternate evaluators. Prior work has shown that transformers are prone to simple copying biases [16] and that the importance of few-shot exemplars might be more in their format rather than correctness [8, 11]. To account for copying biases, we introduce the evaluator, wherein we randomly replace one of the labels with the true label. Note that, during training, the correct label always appears in context, which may incentivize the formation of such naive copying circuits. To perform a stricter test of ICL performance, we thus take inspiration from the "flipped-ICL" evaluation in Wei et al. [11].
"IWL-coyy-vvallaai"
For clarity, we illustrate some abbreviated example sequences below:
* Train: X 23 Y 24 X 23 X 23 A 0 Y 24 B 1 Y 24 X ?. Note there are 3 instances of X 23, 3 instances of 24, and one instance each of O, B 1. Exemplar X is always associated to label 23 through training, Y to 24, A to O, and B to 1.
IWI. eval (in main text): A 2 5 B Y 24 D 3 H Niether the exemplar nor label appears in context.
IWL-copy-aviilabll eval: AOCCF522 23 24 We randomly replace one of the labels with the desired answer label. Note, this task allows the network to benefit from copying, but still provides no information in-context to learn that X should map to 23.
ICL eval (in main text): X O X O Y 1 O 1 Y X O Y 1 X ? Both X and Y's labels are replaces with 0/1. Note, we also tried using other pairs of labels for the ICL evaluator (e.g.. 15/20), but found little variance across this choice.
fCipped-ICL eval: X 24 X 24 Y 23 X 24 Y 23 Y 23 X Y X ?. Note, if the network outputs 24 here, demonstrates in-context learning, as it must be using the exemplar-label from context. Alternatively, if the network outputs 23, it demonstrates in-weights learning, as its ignoring the exemplar-label mapping provided in context. Accuracy=1 on this evaluator means the network is doing ICL, Accuracy=0 means the network is doing IWL.
Following the results of Section 4.3, which showed that fixed embeddings exhibit transience on shorter timescales, we conducted some experiments using the above evaluators on dataset of fixed omniglot embeddings by preprocessing all images through mmagenet-pretrained frozenn Resnet18 encoder [44, 45]. Like the main paper, we considered datasets with either 1 1,600 or 12,800 classes, with 20 exemplars per class. We did not run these experiments as long as those in the main paper, as the purpose was to test the new evaluators.
15

---

(a) In-context leamiigg
(b) In-weights learning.
1.0
0.8
accuracy
0.6
0.2
classes:
1600
0.0
12800
(c) flipped-ICL evaluator.
(d) awalcopy evaluator.
1.0
accuracy
0.6
0.4
88.00
0.25
1.25
train steps
train steps
1e7
Figure 11: Results of alternate evaluators on fixed embedding Omniglot data. Note the shorter a-axis scale. (a) ICL evaluator, same main text. (b) IWL evaluator, same as main text. (c) flippedIICL evaluator. Note how signs of transience are visible sooner on this stricter measure. (d) IWL-copy-available evaluator. Similarly, IWL-copy-available starts growing sooner than the main IWL evaluator, offering earlier indications of transience
As seen in Figure 11, the alternate evaluators' asymptotic behaviors largely mirror the evaluators used by Chan al. [10]. That said, there are some interesting differences. For example, under the flpppe-ICL evaluator, models trained on 1,600 classes are never able to fully override the mapping of exemplars labels seen in training (as evidenced by accuracy remaining below 0.5). However, the scale of 12,800 classes, models are able to (accuracy goes above 0.5 before decaying). Intriguingly, these results may connect to Wei al. [11], except the sense more diverse dataset rather than larger parameter count, leading model to "do in-context learning differently".
With respect to transience, do still observe in both cases. For 1,600 classes, the increase in performance indicates the network may be overriding its exemplar-label mapping some sequences, but this fledgling ability is short-lived. For 12,800 classes, while the shallow decay slope makes transience on the standard ICL evaluator not visible due to the short timescale (note, ICL is still transient this # of classes, as shown Figure 5a), transience clearly visible on the flipped-ICL eval.
that there indeed effect of copying in networks, performance of models on the colaaola-avallable evaluator as compared to the IWL. evaluator.
Furthermore,
by the improved
For clearer comparison to prior work synthetic datasets, opted to use the same evaluators as Chan et al. [10] in this work, but hope the above analysis shows the importance of good evaluators in diagnosing ICL transience.
C Construction of LLaMa-based exemplar-label datasets
In this section, we provide more detail to the procedure used to construct the fixed embedding datasets used Section 4.3.
C.1 Constructing datasets
We save the token embedding matrix for each LLaMa v1 model. These have dimensions 32,000 x ddmodet, where dmoode = 4096, 5120, 6656, 8192 for the 7B, 13B, 30B, and 70B, LLaMa models, respectively. To go from these raw embeddings to "classes" that we can use in our setup, we perform the following steps:
Subselect: Many tokens correspond to individual bytes or characters from non-English languages. We observed that the statistics of these token embeddings often different than
16

---

(a) In co-tontext learning.
(b) In wn-weights learning.
1.0
Embeddingg from
0.8
LLaMal model size:
accuracy
0.6
138
chance
30B
0.4
708
0.2
0.2
0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
chance
train steps
1e6
train steps
1e6
Figure 12: A model initialization seed for which no ICL emerges on LLaMa-derived datasets
the others, we remove these tokens. Specifically, we use tokens 259 to 29,870, inclusive, for total of 29,612 tokens.
2. Cluster: We cluster the token embeddings using K-means clustering and cosine distance using the FAISS library [24]. As FAISS does offer a valid minimum number of points per cluster, we first cluster into 2400 (or 4800) clusters, then pick all clusters that have more than 10 (or 5) points. This gives us least 1,600 (or 3,200) clusters. randomly pick actual 1,600 (or 3,200) clusters.
3. Select from cluster: Many of these clusters contain more than 10 (or 5) points. maximize in-class variation, we pick the furthest 10 (or 5) points from the cluster centroid. Thus, we have our 1,600 classes of 10 examples, or 3,200 classes examples.
Note that step 2 highly dependent on the random seed used to initialize cluster centroids. We run the above procedure for all LLaMa models on 2 random seeds.
C.2 Example clusters
we map the embeddings back to tokens, we can get a sense of what various clusters may represent. Many clusters cover similar tokens, but some clusters can be more semantic or even multilingual! We show some extracted clusters (for 3,200 classes, embeddings per class, on the 70B LLaMa) below:
_erm _room rm _Rom _rm _entirely |
.utter _extremely -partially |
Comple
hover _mysqli gz RGB rgb
async _predicate Observer _deleg _delegate
ive ous ccable rable | ibbl
FLAG I _flag _Flag | flag Flag
_specified _provided I _supplies _uppll
_supplied
ring _dent dent _rent RENT
Derby | Dit I _pedig I _ort _Orr
Britain England _Madrid _Spanish
Espagne
C.3 The difficulty of emergent ICL
We observed that ICL emergence was more sensitive to model initialization when using LLaMa token embeddings exemplars. An example seed where ICL did not emerge at all (for 3,200 classes, exemplars per class) is shown in Figure 12. However, in all cases where ICL did emerge (even to small degree), found it to be transient (Figures 7, 13). Furthermore, the results presented in Section 4.3 are averaged over two different clustering seeds (see step 2 above), which yield different output datasets. ICL transient across all 8 datasets tested.
At 1,600 classes, with 10 exemplars per class, we found mild ICL emergence, as shown in Figure 13. We found that 3,200 classes were necessary to get any strong ICL, which is shown in Figure 7.
17

---

(a) In co-cone learning.
(b) In we-weights learning.
1.0
Embeddings from
0.8
LLaMa1 model size:
accuraty
0.6
accuracy
0.6
138
chance
30B
0.4
70B
0.2
0.2
0. 0.00
0.5
1.0
1.5
2.0
0.0
0.5
1.5
chance
train steps
1e6
train steps
1e6
Figure 13: Mild emergence and subsequent transience of ICL on model trained on the LLaMa derived datasets with 1,600 classes and 10 exemplars per class.
D Additional regularization results
(a) In-context learning.
(b) In-weights learning.
L2 regularization
0.08
Se 04
accurat)
0.6
chance
accuracy
0.06
1e-03
0.4
0.04
0.2
0.02
0..0
0.5
1.0
train steps
1e7
0.0
1.0
1.5
train steps
1e7
Figure 14: For extreme values L2 regularization (> le-04), model seems to be over-regularized: We observe return to ICL transience, as well as poor performance IWL.
L2 regularization
train log loss
1e:5
-4
-8
train steps
107
Figure 15: Total train loss (including L2 penalty) for sampling of experiments. Regularization seems to make the training loss smoother, as well as causing train loss to plateau at higher value (rather than continuing to drop).
18

---

(a) In-context learning, decay = 1e-5
(b) In-context learning, decay 1e-4
0.85
0.9
accuracy
0.8
0.75
0.7
0.7
0.5
(c) In-weight learning, decay 1e-5
(d) In-weight learning, decay 1e-4
0.06
accuracy
0.2
0.04
0.1
0.02
0
0.4
0.8
1.2
1.6
0
0.4
0.8
1.2
1.6
# train steps
1e7
# train steps
1e7
Weight decay applied to:
ResNet embed
MLP layers
self attention
Figure 16: Selectively applying weight decay to different sets network weights reveals that IWL relies MLP layers. When apply weight decay only self-attention layers (blue), ICL is still transient. When we only apply weight decay to MLP layers, ICL transience is mitigated (red). These results can be interpreted in light of prior work indicating that in in-weights information is stored in MLP layers 32]. By selectively penalizing this behavior, we enable ICL to persist, thus providing convergent evidence that when no weight decay is applied ICL fades due to competition with IWL circuits.
19


Reviews:
[{"review_id": "Y64ZQ0voA7", "reviewer": "Reviewer_oS5Q", "summary": "This paper presents empirical evidence illustrating the transient nature of in-context learning (ICL) in transformers, and it shows that the models may shift to in-weights learning (IWL) when overtrained. Furthermore, the transience of ICL is studied across diverse settings such as model size, dataset size, data distributions, and training regularization. Results reveal that adequate L2 regularization can mitigate this transience.", "questions": "1. Why are the IWL accuracies consistently low? Would further training iterations increase the IWL accuracy and what is the maximum achievable IWL accuracy?\n2. Have the authors considered that the observed ICL performance drop might be due to the fact that evaluation in-context data distribution is different from training, especially as the label space changes? In Fig 5(c), when evaluation and training use the same data distribution, the performance does not decay. \n3. The connection between Fig5(a)(b) and Fig5(c)(d) seems unclear given the different training datasets. What's more, in Section 4.1, the authors' claim \"Figure 5d shows that larger embedding dimensions enhance the model’s ability to perform IWL, but do not consistently help ICL (Figure 5c).\" seems puzzling. Since ICL accuracies are already close to ones, is there room for improvement? Would reducing the embedding dimension identify a threshold beyond which ICL accuracy is $<1$?\n4. In the Discussion section, the authors note \"However, after training for a long time, we find that even rare classes begin to be learnt in-weights.\" However Section 4.3 does not provide evaluation over rare classes.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper reveals a characteristic of transformers performing transient ICL.\n2. The proposition that regularization can eliminate the transience is noteworthy.\n3. The provided experiments are clearly organized and easy to follow.", "weaknesses": "1. The paper lacks clarity in its notations, which makes them hard to follow without reference to previous literature. For instance, the term \"in-weights learning\" needs to be related to supervised learning; in the Zipfian distribution equation $p(X = x) \\propto x^{-\\alpha}$, the terms $X$ or $x$ should be explained as representing the rank of input classes.\n2. While the authors provide numerous empirical evidences, they scarcely discuss potential underlying reasons. Some interpretations or conjectures would enrich the discussion around the observed outcomes.\n3. The experiments are constrained to a single dataset and transformer (with different sizes), which is not convincing enough in my opinion. It would be interesting to see if the similar phenomenon is observed with other datasets (e.g., NLP dataset) or other auto-regressive models. The usage of the Omniglot dataset—a stationary supervised dataset—where in-context samples can be regarded as independent entries, makes the existence of IWL unsurprising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and questions! Please find responses to your comments below:\n\nW1: We believe in maximizing the clarity of our work, and appreciate your suggestions for doing so. We will clarify the definition of Zipfian distributions. We will also add additional explanation regarding the relationship between \"in-weights learning\" and supervised learning. \n\nW2: We agree that these novel results provoke interesting new questions about the mechanisms of in-context learning in transformers. The experiments on embedding size provide a clue as to why we observe ICL transience (lines 134-151) – we carefully designed those experiments to help us see that IWL and ICL circuits are in competition for capacity in the transformer. We have also run additional experiments elucidating how and why weight decay mitigates transience (see general rebuttal and Figure 3, attached page) – these results provide additional evidence for ICL-IWL competition as a mechanism driving ICL transience. We will update the paper with these results, and emphasize this takeaway more clearly in the paper. We also give some further conjectures on why ICL is transient in the Discussion (lines 253-274), though we refrain from over-speculating. \n\nW3: The scope of our paper is focused on transformers, given that ICL has very different mechanisms and behavior in transformers vs RNNs (e.g., as found in Chan et al. (2022)). We do not make any claims that ICL transience extends beyond transformers, and we will emphasize this in the paper. With regards to datasets, we extended to fixed embeddings (Figure 1, attached page) and have started runs with one-hot data and Gaussian in-class variation. We maintain a paired exemplar-label setup as it allows for careful variation and controls (see general rebuttal and attached page). Such synthetic approaches are common in the study of ICL, as noted in our related work (lines 208-214). Extending to uncontrolled, web-scraped natural language is an exciting future direction, but is beyond the scope of our current work.\n\nBy design, our training task can indeed be solved quite naturally by IWL, but also by ICL. The critical insight of our work is that the IWL and ICL solutions trade off in surprising ways over the course of training, even on this “stationary” dataset – specifically, ICL emerges and then fades away. This behavior has not been previously reported.\n\nQ1: We believe further iterations would continue to increase IWL accuracy, as the slope is positive and shows no sign of plateauing. IWL accuracy is low as the IWL eval sequences are out-of-distribution (purposefully) for the model. For example, models are known to exhibit naive copying biases (as noted by reviewer 8T18, and Elhage et al. (2021)), which are no longer useful on the IWL sequences, which could be contributing to the lower accuracy – we note this in Footnote 2 (page 2).\n\nQ2: The paper focuses on in-context learning that is *emergent*, i.e. which appears even when the training data does not require in-context learning as a solution (and therefore where it is out of distribution with respect to the training data). This emergent phenomenon is observed widely in transformers (e.g., Brown et al. (2020), Chan et al. (2022)). The results in Figure 5(c) are only meant to demonstrate that the in-context learning task is \"solvable\". Our main claim is that transformers, when trained on data that *does not* require ICL, first learn to do ICL (corroborating others' findings of emergent ICL), but then subsequently lose ICL abilities, which may have implications for the current trend of training transformer models for longer and longer (c.f. Touvron et al. 2023).\n\nWith regards to the specific 0-1 label space, we'd conducted early experiments across different random label spaces and found only a small amount of variance. Specifically, across 11 additional random relabelings (in addition to the 0-1 relabeling), we calculated the 95% confidence interval to be +/- 2% around the peak, and +/- 1% on the decay slope. Given these, we opted for the simpler, more reproducible setup of using just the 0-1 relabeling. We will update the paper to explain this choice.\n\nQ3: In Figures 5(a)(b), we find that larger embedding dimensions mitigate ICL transience. Figure 5(c)(d) are follow-up experiments that are carefully designed to disambiguate between two possible explanations for this finding. Specifically, we argue that ICL is not inherently worse at smaller embedding dimensions (as seen in 5c), so the mitigated transience (seen in 5a) is likely due to less competition to IWL circuits. There is likely an embedding size threshold below which ICL does not emerge, but our main purpose in providing these figures was to suggest a possible explanation – competition in the residual stream – for why ICL may be transient. We will clarify the paragraph where we make these points (lines 143-151).\n\nQ4: The overall increase in IWL accuracy led us to believe that accuracy was increasing across classes, but we agree that this isn't necessarily the case and thank you for pointing this out. We will look into the specific IWL accuracies for classes according to rarity, updating Section 4.3 and this part of the discussion accordingly.\n\nThank you once again for your detailed comments and feedback. We hope that we’ve managed to address your concerns. If not, we’d be more than happy to continue the discussion!"}, {"review_id": "KJ8q96AkOe", "reviewer": "Reviewer_iyV5", "summary": "The paper explores In-Context Learning (ICL), a model's ability to adapt its behavior during inference without weight updates, and In-Weights Learning (IWL), where the model relies on information stored in its weights. While early neural networks displayed ICL in controlled scenarios, research changed with the transformer model, particularly with GPT-3, which exhibited ICL without deliberate training efforts. However, it was found that ICL isn't guaranteed when training transformers, with language data properties like burstiness and skewed distribution playing a crucial role. Additionally, ICL and IWL often seem to oppose each other, with ICL emerging more readily when training data has a larger number of tokens or classes and is bursty. Despite the challenges of relying on large models, overtraining is currently a favored approach for training small yet high-performing transformers. The paper challenges the assumption of persistence (that once ICL emerges it remains), showing that ICL can be transient, with the potential to disappear if networks are trained carelessly for longer periods.", "questions": "My major question is -- is the process of Chan et al. [10], which has been adopted by the authors of this paper, really capable to answer such fundamental questions about ICL and the emergent capabilities of LLMs? The queries that the paper relies on very short -- just 8 exemplar-label pairs and one query image -- i.e., just a sequence of length 17. That also means that the provided \"context\" is very short. this could be a reason that ICL is not persistent. \n\nAnother potential issue is the \"context\" in these experiments is very highly informative regarding the given query. This is why we see that ICL gives a much much larger accuracy than IWL in all plots of the paper. Is this a realistic/pragmatic evaluation regime? I would assume that in practice the context is only partially informative (at least for NLP tasks) and that the network has to rely in anyway much more on trained weights. \n\n", "limitations": "The paper is quite open about its limitations, mentioning clearly that it has not utilized very large models and so it is not really certain whether the major claims of this study would be actually still seen in large models such as GPT3 or 4. Additionally the paper does not try to \"oversell\" what it does -- and it identifies many open questions instead of claiming that it already has answers to those questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "On the positive side, the paper is certainly quite interesting and relevant to the major open question these days, which is the emergent capability of large language models. The paper raises a number of interesting open questions (see Discussion in section 7). In fact, I would argue that that is the most interesting section of the paper.", "weaknesses": "On the negative side, the paper can be viewed as \"half-baked\" at this point. It relies heavily on some empirical results, derived mostly from small models and visual tasks, and it raises a number of questions that are interesting but.. open. I wish the paper could go a bit more deeply to actually answer at least one of those questions. At this point I cannot say with confidence that the paper is ready to be presented at the premier conference of deep learning. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful feedback, and are glad you were excited by the questions our work raises. We believe that answering each of these questions is a difficult endeavor in itself, and that answering them is a challenge for the whole community that will happen incrementally. Our work takes the crucial first steps by documenting the phenomenon of ICL transience, providing some explanation (lines 143-151, competition in the residual stream), and providing hypothesis-driven mitigations (Section 5, weight decay). We have also performed new experiments that provide additional convergent evidence that ICL transience is driven by competition with IWL circuits (Fig 3, attached), which we will add to the paper. Finally, we end with an honest discussion of open questions, which we believe will motivate future work.\n\nIn response to your question about context length, we conducted some short runs with twice the context length (Figure 2, attached page) – 16 image-exemplar pairs (for a sequence length of 33), 6 of which come from the query class, 6 from a distractor class, and 4 random. We had difficulty getting ICL to emerge consistently in this setting, across 4 seeds, only 2 showed above chance ICL at all (both with a lower peak height). Consistent with our experiments though, in both cases where ICL appeared, it was transient. We believe this longer context might actually be less informative, perhaps due to a more \"imperfect match\" (one of the theories we suggest for ICL transience, lines 266-273) given the larger number of distractor exemplar-label pairs, or recency biases known to exist in transformers due to the fixed sinusoidal positional embedding scheme.\n\nWith regards to the \"highly informative\" context of our original experiments, we believe this makes the transience of ICL even more surprising! If ICL fades even with such useful context, we believe it likely that overtraining with less informative contexts, all else the same, would also lead to transience."}, {"review_id": "p4jUAQbd56", "reviewer": "Reviewer_wdqh", "summary": "The paper investigates the emergence and persistence of in-context learning (ICL) in Transformer models. Training Transformer models on tasks by Chan et al. [10], the authors find that ICL does not persist, despite an initial emergence. Loss reduction suggests a preference for in-weight learning after some training steps. Through specialized experiments, the authors explore model size, training dataset statistics, and L2 regularization to understand and mitigate this phenomenon. The findings emphasize the importance of L2 regularization in Transformers during extended training. This research provides valuable insights into ICL dynamics, contributing to better understanding of this phenomenon and offering insights into potential improved training strategies.", "questions": "1. Connections to Grokking are not obvious to me. In Grokking [27] memorization at early phase of training is replaced by more general solutions at later stages of training. In your context, the opposite effect seems to be happening, if you assume that ICL are the more general solutions.\n2. What is the effect of different positional encodings to the emergence and persistence of ICL? It would be interesting to study.\n3. In Figure 7 what is the \"chance\" prediction for IWL? Different runs here have different number of classes.\n4. You can match the color scheme between Figures 9,10 and Figure 11.\n5. Line 242-244 \"Furthermore, these results imply that it may be desirable to stop training large transformer models before a compute budget is reached—to preserve ICL—even if training losses are still decreasing.\": the statement is a bit too bold given the experimental setup of the paper.\n6. Line 258-259 \"lottery tickets contained in common initialization methods [27, 37] that incentivize ICL\": I find this a very interesting conjecture that could be explored. \n7. Another potential effect that could be explored may be the \"softness\" of the softmax function. Perhaps the network realizes that to decrease the loss further it needs to resolve to IWL. It would be interesting to track the entropy or \"hardness\" of the attention mechanism. If the softmax is deemed to be the deciding factor, you can think about replacing the softmax with a harder version e.g. [1, 2, 3] and see if the transient effect disappears. \n\n[1] Martins, André, et al. \"Sparse and continuous attention mechanisms.\" Advances in Neural Information Processing Systems 33 (2020): 20989-21001.\n\n[2] Hahn, Michael. \"Theoretical limitations of self-attention in neural sequence models.\" Transactions of the Association for Computational Linguistics 8 (2020): 156-171.\n\n[3] Chiang, David, and Peter Cholak. \"Overcoming a theoretical limitation of self-attention.\" arXiv preprint arXiv:2202.12172 (2022).", "limitations": "The paper discusses a limited scenario where ICL is not persistent. Experiments are performed but always in the context of the provided dataset and for smaller models. More experiments are in general required to verify the existence of this phenomenon elsewhere. The study is nonetheless very interesting and useful for the community.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper builds upon the established dataset by Chan et al. [10], ensuring reproducibility of results. The phenomenon of interest is consistently observed across different scenarios, allowing for comprehensive examination. The inclusion of small-scale experiments facilitates further investigation by researchers. The availability of the code, given that it will be released, will help with reproducibility and transparency of the study.\n2. The authors adopt a systematic approach to address the problem at hand, systematically eliminating various factors that contribute to the lack of persistence in in-context learning (ICL). Through controlled experiments, they explore and analyse multiple aspects that may impact ICL persistence.\n3. The paper has a well-structured and coherent storyline, ensuring clarity of the results. The authors effectively motivate each experiment, providing a clear rationale for their choices. I enjoyed reading the narrative presented by the authors.", "weaknesses": "1. The phenomenon is observed for a single small dataset. Although some experiments are made with respect to aspects of the training data, i.e. number of classes and number of exemplars per class, the small relative size of the training data in conjunction with the large number of training steps tested for here, may potentially lead to conclusions that do not generalize past this setting. Some attempts were made to match data distribution with naturalistic data, but any conclusion that generalize to large language models have to be taken with a grain of salt.\n2. Although a lot of different effects are explored and some correlations explored, no clear answer is given in the end. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your insightful comments. You have clearly thought carefully about the paper, and we very much appreciate it.\n\nW1: As noted in the general rebuttal, we are running experiments with modifications of the dataset, including running on more symbolic data (Figure 1, attached page). We would also like to note that the data is not actually that \"small\" in size due to the combinatorial construction of our sequence prediction task (17 input tokens -> one output token). Specifically, there are over 1600 * 1599 * (1598 choose 2) * (8! / (3! * 3!)) * (20^9) = 1.87 * 10^27 possible training sequences!\n\nW2: We will update the paper to emphasize evidence we do provide for mechanisms behind ICL transience – specifically, competition in the residual stream (lines 143-151) – as well as mitigations and what they might indicate about underlying mechanisms (e.g., Figure 3, attached page). We also believe that raising questions and initial evidence is just as important to the process of science as taking the final steps to reach the final answers.\n\nQ1: Yes indeed! This was our intended meaning with  \"ICL transience is a mirror reflection of the grokking phenomenon\" – we will make this clearer with different wording, that it seems to be the opposite of grokking. This was the observation that motivated our use of weight decay (to try and retain the more \"generalizing\" ICL solution).\n\nQ2: This is definitely an exciting direction for future work. We ran some preliminary experiments in the rebuttal period on a smaller, 2-layer model and found that positional embeddings can lead to differences in peak height and decay slope. ICL remains transient across settings where it emerges (Figure 4, attached page).\n\nQ3: Good question – the chance predictions here would be 1/# classes (so 1/12800 for the purple line, and 1/1600 for the other two). Due to the scaling of the plot (which we wanted to keep as 0-1 for consistency), we just included the 1/1600 chance line. We will add extra clarification to the text and image captions.  \n\nQ4: Thank you for pointing this out. We will make this change.\n\nQ5: We will soften that statement – thank you for the recommendation! We will modify it to the following (please let us know if you recommend further changes): \"Furthermore, these results suggest that it may be desirable to stop training some transformer models before a compute budget is reached—to preserve ICL—even if training losses are still decreasing. This would need to be ascertained on a case by case basis. At the least, evaluations should be performed to ensure that ICL performance remains within desired levels throughout training.\" \n\nQ6: We find this very interesting as well, and hope to explore this idea in future work!\n\nQ7: Agreed! We thank you for the additional references, which we will add to our discussion of the \"imperfect match\" theory (lines 266-273). We are also excited to try replacing the attention mechanism in future work, as you suggest."}, {"review_id": "osHN1g95aZ", "reviewer": "Reviewer_8T18", "summary": "The paper explores the in-context learning (ICL) phenomenon in transformer networks and investigates its transience during training. It presents empirical findings that ICL if emerged in transformers, may not persist as the model continues to be trained. \nThe paper highlights the importance of evaluating appropriate validation metrics and suggests stopping training before a computing budget is reached to preserve ICL. \nStrategies to mitigate ICL transience, such as data distributional properties and regularization, are explored. The paper also discusses the competition between ICL and in-weight learning (IWL) circuits and raises questions about the emergence and fading of ICL. \nThe experiments demonstrate that ICL in transformers is often transient. Model size, dataset size, and data distribution can influence the transience of ICL, but persistent ICL is not guaranteed. Wider models with larger embedding sizes show improved ICL, possibly due to reduced competition with IWL circuits. Skewed data distributions mitigate ICL transience but necessitate longer training.\nFuture directions, limitations, and implications for large language models are discussed. Overall, the paper provides valuable insights into the nature of ICL and its implications for transformer networks, contributing to the understanding of in-context learning and its behavior during training.", "questions": "- Were the plots in the paper based on multiple or single runs? If only a single run was used, how was statistical significance ensured?\n- Have you considered the potential interplay between ICL and other important mechanisms in transformers, such as contextual information used in conjunction with IWL for narrowing down possible outputs via copying?\n- Figure 5 shows how many runs were averaged, and what do the error shadings indicate?\n- Could you further explain the statement that the balance between ICL and IWL is a mirror reflection of grokking? Why would ICL, being more generally useful, be transient instead of persisting?\n- Have you explored alternative regularization techniques or approaches to achieve the coexistence of ICL and IWL?\n- When the regularization value exceeds 1e-4, ICL transience returns and IWL also does not work. Does this imply a decrease in accuracy with more training? What is the relationship between regularization, ICL, and IWL?\n- What may be the underlying mechanism by which regularization affects the persistence of ICL? Does it make IWL more difficult or even impossible, leaving only ICL as a viable option?\n- How do you envision your findings' potential real-world applications or implications on ICL and its transience in transformer networks?\n- How might this research impact the development and training of AI models in practical scenarios?", "limitations": "While the potential negative societal impacts are not explicitly discussed in the provided excerpt, the authors have identified areas for further investigation and improvement in their research, which addresses some limitations of their work. \n\nSince this work focuses primarily on analytical rather than directly enhancing capabilities, there are no immediate negative effects to be considered.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and Contribution:\n- Empirical findings on the transience of in-context learning (ICL): The paper presents novel empirical evidence that ICL if emerged in transformer networks, may not persist as the model continues to be trained. This finding sheds light on an important characteristic of transformer models and highlights the need for appropriate evaluation and validation metrics.\n- It provides insights into how ICL emerges, disappears, and gives way to in-weight learning (IWL) during training.\n- Discussion of the coexistence of ICL and in-weights learning (IWL): The paper explores the competition between ICL and IWL circuits for resources in the transformer residual stream. It raises intriguing questions about the emergence and fading of ICL and offers potential explanations for the coexistence of ICL and IWL in large language models.\n- Exploration of strategies to mitigate ICL transience: The paper investigates various strategies to mitigate the transience of ICL, including increasing the number of classes, introducing Zipfian data distribution, and using L2 regularization. These strategies provide valuable insights into potential methods for preserving and enhancing ICL capabilities.\n- It is interesting that ICL is transient even under conditions that are deemed conducive to the emergence of ICL.\n- Comprehensive and thought-provoking discussion, posing clear and relevant questions: discussion of future research directions and acknowledgment of the limitations of the conducted experiments. It identifies the need for experiments on large language models, exploration of other datasets and context lengths, and investigation of additional mechanisms in transformers, encouraging further advancements in the field.\n- Consideration of the impact of training duration and stopping criteria: The paper emphasizes the importance of appropriate training duration and stopping criteria to preserving ICL. It suggests that training large transformer models beyond a certain point may result in the gradual disappearance of ICL, even if training losses continue to decrease. This insight has practical implications for model development and training practices.\n\nResearch Problem and Motivation:\n- The research problem is well-defined and clearly stated, focusing on the transience of ICL in transformer networks.\n- The paper provides a convincing motivation for the chosen problem by highlighting the importance of evaluating appropriate validation metrics, preserving ICL, and understanding the factors influencing its persistence.\n- The relevance and importance of the research problem are clearly explained, emphasizing the potential impact on training large-scale language models.\n- The paper raises numerous additional questions, stimulating further scientific inquiry and exploration.\n\nMethodology:\n- The methodology is clearly described and well-founded. The paper outlines the experimental setup, including the models, training procedures, and evaluation metrics used.\n- The algorithms, models, and techniques employed are appropriate for addressing the research problem of studying ICL transience in transformer networks.\n- The paper adequately explains the experimental setups and data collection processes, providing sufficient details for reproducibility.\n\nExperimental Evaluation:\n- The experimental results are presented clearly and concisely, with appropriate figures and tables.\n- The evaluation methodology is rigorous and appropriate, considering various metrics and comparing experimental conditions.\n- While the statistical significance of the experimental results is not explicitly mentioned, the paper conducts a thorough analysis and provides empirical evidence to support the presented findings.\n- The rigorous analyses demonstrate a commitment to scientific rigor and methodology.\n\nRelated Work:\n- The paper comprehensively reviews the relevant literature, discussing previous works on ICL, language models, optimization, and initialization methods.\n- The paper differentiates itself from existing works by focusing on the transience of ICL and investigating strategies to mitigate it.\n- The references and citations are appropriate and up-to-date, incorporating relevant studies in the field.\n\nClarity and Organization:\n- The paper is well-structured with clear sections and subheadings, making it easy to follow the flow of ideas.\n- The ideas and concepts are presented logically and coherently, building upon each other to support the main arguments.\n- The language used in the paper is clear and concise, minimizing excessive technical jargon and enhancing readability.\n\nPresentation and Visualization:\n- The plots are visually appealing, utilizing scalable vector graphics and an attractive gradient color scheme. The figure captions are concise and informative.\n- The writing is of high quality, with minimal errors or inconsistencies.\n- The overall presentation is visually appealing and well-designed, enhancing the readability and comprehension of the paper.", "weaknesses": "- Lack of experiments on large-scale datasets: The experiments mainly focus on controlled synthetic datasets, and the impact of ICL transience on real-world, large-scale datasets is not explored. Including experiments on diverse, natural language datasets would enhance the practical relevance of the findings.\n\n- Limited scope of experiments: The experiments were not performed on large language models, one of the most important applications of transformers. Conducting experiments on larger models would provide a more comprehensive understanding of the phenomenon and its implications.\n\n- Narrow definitions of ICL and IWL: The definitions of in-context learning (ICL) and in-weights learning (IWL) used in the paper are relatively specific and make for a clear-cut distinction that in reality - as acknowledged in a footnote - may not be that clear. Exploring additional mechanisms and interactions between context and learning could contribute to a more comprehensive understanding of transformer models.\n\n- Limited exploration of alternative explanations: The paper acknowledges possible explanations for the emergence and transience of in-context learning but does not thoroughly explore alternative hypotheses. Further investigation into optimization quirks, initialization schemes, or alternate optimizers could provide deeper insights into the phenomenon.\n\nOverall, the above-mentioned points can be seen more as ideas for future work rather than genuine weaknesses of the paper. \n\nMinor points:\n- Lack of statistical significance analysis: Although the experimental results are presented clearly, the paper does not explicitly address the statistical significance of the findings. Including statistical tests or discussing the significance of the observed differences would strengthen the conclusions.\n\n- (minor) Maybe explain in-weight learning in the abstract, contrast it to in-context learning\n- (minor) second to last sentence in the abstract (l. 13), reformulate: how does that question logically follow from the observation?\n- (minor) l. 18 Reformulate definition to be clearer, order of words\n- redo the references, missing information (ls. 373, 394, 408), inconsistent formatting (e.g., ls. 307, 315)\n- Capitalize \"In-Context\" instead of \"In-context\" in the title.\n- Clearer motivation for why researching ICL is interesting and relevant?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful and impressively thorough review! We have factored in the suggested minor edits, and we address the questions below:\n\nQ1: Plots in the paper were based on two runs. We observed little variance between these two runs (as shown by the lighter colored error bars around curves – these are often hard to see because of the low variance!), and thus decided to not do more random seeds, given the significant cost of each run. Nonetheless, we observed ICL transience across many runs when compiled across conditions, and this replicability of ICL transience across conditions indicates robustness of the overall phenomenon.\n\nQ2: Yes! We believe this to be a possible mechanism at work in our models, and plan to investigate it further. However, we'd like to emphasize the focus of the present work is on the transience of ICL.\n\nQ3: Error shadings indicate standard error of the mean.\n\nQ4: By mirror reflection here we mean that ICL transience is the *opposite* of grokking. In grokking, networks transition from less general to more general circuits, while ICL transience appears to be the opposite (assuming ICL is viewed as \"more general\") – networks transition from ICL solutions to IWL solutions. We will reword this part of the text so that it is clearer.\n\nQ5: Not yet, but this is an exciting avenue for future work.\n\nQ6: Above that scale (our run in the appendix with 1e-3), the regularization is too high for the network to retain either circuit – if we train for long enough, the network will lose ICL and IWL, as indicated by the train loss plateauing at a higher value (Figure 11). \n\nQ7: That is what we believe to be happening, though a mechanistic analysis (with ablations of subcircuits) would likely be necessary to verify. To further investigate these results, we have performed an additional analysis, applying regularization to different parts of the network (Figure 3, attached page). We find that regularization mitigates ICL transience only when it is applied to the dense layers, and not to the attention layers. \n\nQ8-9: We hope our findings will encourage model developers to be more conscious of evaluations of in-context learning throughout training, as compared to just watching the train loss. Given the trend of overtraining models for longer and longer (Touvron et al. 2023), we believe this especially relevant."}]

Author rebuttal:
We thank the reviewers for their detailed feedback and questions. Their comments and suggestions have enabled us to make significant improvements to the work.

We are glad the reviewers found the work  "interesting and relevant to the major open question these days" and that they enjoyed the "clarity of the results" and "comprehensive and thought-provoking discussion, posing clear and relevant questions". We believe that our work contributes significantly to the scientific body of work around understanding transformers by (a) identifying an interesting new phenomenon (ICL transience), (b) providing initial explanations as to why it arises (Section 4.1), and (c) giving a hypothesis-driven solution for how to mitigate ICL transience (Section 5). 

We share the reviewers' enthusiasm to see these results in LLMs. Given our resource restrictions, we opted for the data generating process of Chan et al. (2022), prioritizing more controls and experimental conditions, as compared to limited large-scale runs on uncontrolled internet-scale language data. While our work does not directly consider LLMs, we hope it will convince large-scale practitioners to consider the possible transience of ICL, which may motivate more careful evaluations of ICL throughout training. 

We also share the reviewers' desire to understand the mechanism behind this interesting phenomenon. Our existing results suggest that as IWL increases, ICL gets crowded out due to competition in the residual stream (lines 134-151). We have conducted additional experiments with weight decay applied selectively to MLP layers of the network (which Geva et al. (2021) indicate may be primarily responsible for in-weights information). We found that in these cases, IWL does not arise, and ICL persists (red line, Figure 3, attached page). We believe these results provide compelling observational evidence that ICL fades due to competition with IWL circuits, and will emphasize this in the paper. Our work documents a hitherto unknown phenomenon and also takes the crucial first steps towards elucidating underlying mechanisms, which we hope will spur further inquiry in the academic community on the theories governing in-context learning.

To address reviewers' questions, we present additional experimental results in our one page report, which we will also incorporate into our paper. To this end, we have extended to a dataset with fixed, rather than learned, embeddings (Figure 1). We also conducted experiments with longer context lengths (Figure 2, reviewer iyV5) and experiments on the effect of position embeddings (Figure 4, reviewer wdqh). 

Meta-review: The paper presents new empirical understandings of in-context learning (ICL) in transformers. The main finding is that the transformer shifts its mechanism from ICL to in-weights learning (IWL), when the pretraining data can be solved by both mechanisms, and the transformer is trained for long enough. The paper also provides ablations and proposes mitigation strategies.

The opinions about this paper are mixed. The reviewers have concerns about the setting being a bit too specific, but agree that the message is interesting and worthy to share with the community. After further discussions, the reviewers are supportive of this paper and are optimistic about the discussions it would raise, despite the concerns. Therefore, I recommend acceptance.

I encourage the authors to incorporate the additional results (in rebuttal) and the reviewers' suggestions into the revision, to further improve the paper for the publication.
Decision: Accept (poster)