Strategy Coopetition Explains the Emergence and Transience of In-Context Learning
Authors: Aaditya K Singh
Ted Moskovitz
Sara Dragutinović
Felix Hill
Stephanie C.Y. Chan
Andrew M Saxe
Venue: ICML 2025 oral (2025)
Keywords: Mechanistic interpretability
transformers
in-context learning
transience
dynamics
Machine Learning
strategy
cooperation
competition
coopetition
Primary area: deep_learning->attention_mechanisms
TL;DR: We find and model cooperative and competitive dynamics (termed "coopetition") that explain the emergence and subsequent transience of in-context learning.

Abstract: In-context learning (ICL) is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent ICL as a transient phenomenon that can sometimes disappear after long training times. In this work, we sought a mechanistic understanding of these transient dynamics. Firstly, we find that—after the disappearance of ICL—the asymptotic strategy is a remarkable hybrid between in-weights and in-context learning, which we term “context-constrained in-weights learning” (CIWL). CIWL is in competition with ICL, and eventually replaces it as the dominant strategy of the model (thus leading to ICL transience). However, we also find that the two competing strategies actually share sub-circuits, which gives rise to cooperative dynamics as well. For example, in our setup, ICL is unable to emerge quickly on its own, and can only be enabled through the simultaneous slow development of asymptotic CIWL. CIWL thus both cooperates and competes with ICL, a phenomenon we term “strategy coopetition”. We
propose a minimal mathematical model that reproduces these key dynamics and interactions. Informed by this model, we were able to identify a setup where ICL is truly emergent and persistent.

Strategy Coopetition Explains the Emergence and Transience of In-Context Learning
Aaditya K. Singh" Ted Moskovitz 2 Sara Dragutinović Felix Hill Stephanie C. Y. Chan *
Andrew M. Saxe 1
Abstract
and solve tasks not seen in training. ICL contrasts with in- weights learning (IWL), which is standard learning through weight updates. ICL is striking not only for its power, but also because it emerges without being explicitly trained for (as Brown et al. (2020) first observed, when training transformer models on internet-scale language corpora.). Newer work has shown that ICL can sometimes in fact disappear after emerging, i.c. it can be transient over the
In-context learning (ICL. is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent ICL. as a transient phenomenon that can sometimes dis- appear after long training times. In this work, sought mechanistic understanding of these transient dynamics. Firstly, we find that-after the disappearance of ICL-the asymptotic strat- egy is remarkable hybrid between in-weights and in-context learning, which we term "context- constrained in wniweis learning" (CIWL). CIWL is in competition with ICL, and eventually re- places it the dominant strategy of the model (thus leading to ICL transience). However, we also find that the two competing strategies actu- ally share sub-circuits, which gives rise to coop- erative dynamics as well. For example, in our setup, ICL is unable to emerge quickly on its own, and can only be enabled through the simultaneous slow development of asymptotic CIWL. CIWL thus both cooperates and competes with ICL, phenomenon we term "strategy coopetition". We propose a minimal mathematical model that re- produces these key dynamics and interactions. In- formed by this model, we were able to identify a setup where ICL is truly emergent and persistent.
course of training (Singh et al. 2023 Anand al. 2024 He et al. 2024).
This evolving picture of ICL necessitates a deeper under- standing of the dynamics of ICL emergence (and transience). ICL is often viewed to be in competition with other strate- gies (Nguyen & Reddy 2024 Park et al., (024), such as IWL, with the tradeoff thought to be modulated by data properties (Chan al. 2022
2024), model size (Wei et al.) 2023). and/or training time (Singh et al. 2023). While com- petition may explain why ICL gives way to other strategies through the course of training, the question remains: why does emerge in the first place (if only to fade away)?
In this work, we aim extend the mechanistic understand- ing of ICL, which currently focuses on induction heads OOsssn et al. 2022) and their emergence dynamics (Singh) et al. 2024), to a richer dynamical setting involving multiple strategies cycling in and out over the course of learning. To do so, we reproduce and investigate the key transience result in a simplified synthetic data setting with a 2-layer attention- only transformer. Using behavioral evaluators, we find the asymptotic strategy after the disappearance of ICL is not pure in- in-weights learning. Rather, it is surprising hybrid strategy that we term context-constraine in-weights learn- ing (CIWL, Section 4 The implementation of CIWI. takes the form of skip-trigrams (Elhage et al.) 2021) distributed across multiple heads in a form of superposition (Elhage et al.) 2022). Perhaps even more remarkably, we find that even though CIWL dominates over ICL asymptotically, both strategies share critical sub-circuits (Section 5.I), indicating cooperative dynamics between these scemingly competitive mechanisms- phenomenon we term "strategy coopeti- tion." " We borrow the term "coopetition" from game theory, where describes situations where competitors simultane-
1. Introduction
Transformer-based large language models (LLMs) show an impressive propensity for in-context learning (ICL) the ability to use inputs at inference time to adapt behavior
'Equal contribution Gattsb Computational Neuroscience Unit, University College London 'Anthropic AI. work com- pleted while the Gatsby Unit, UCL *University of Oxford "Google DeepMind. Correspondence to: Aaditya K. Singh <aa- ditya.singh.2 21 @ucllac.uk>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Strategy coopetition explains the emergence and transience of in-context learning
Strategy used
24
1.0
ICL CIWL
Layer 2 "Induction Head"
Train:
@
24
37
24
24
0.8
(ICL + CIWL)
In-context accuracy
37 or 37 or
IWL:
37
S
41
гер 24
rep 37
ICL:
0
1
C
0
0
0
Train
0.4
ICL eval
CIWL
0
37
S
24
37
CIWL eval
Layer (ICL vs
24
0.2
Flip eval chance
Flip:
0
24
37
37
24
X
CIWL)
a)
context
query
example outputs
b)
sequeeces seen
2.0
C)
24
37
t
Figure 1. (a) Example sequences seen during training and evaluation. Training data is "bursty", enabling both in-context and in (n-weights strategies (the context always contains an exemplar from the same class as the query, but alsoo exemplar-label mappings are fixed throughout training). Evaluation sequences (below dotted line) are designed to measure the presence of different strategies. ICL relies the exemplar-tabel mapping in context. IWL. depends solely on in-weights information. ZlWriiqqire the correct label in context, but not the query exemplar. The Flip evaluator measures the balance between ICL and CTWL. (1.0 means pure ICI, 0.0 means pure CTWI.). Bolding indicates OOD exemplar- label pairings. Grayed outputs indicate random selection between the two in-context labels. (b) Accuracy on sequences from (a), over the course of training. "In-context accuracy" is computed by restricting the network's outputs the two labels present in context-this ensures the same chance level (0.5) for all plotted evaluators. ICL transience is clearly visible in blue. IWL is not shown, found little-to-no IWL in the networks (Appendix canonical "induction head"" 2. ICL strategy dominates network output, as evidenced by peak in the Flip evaluator (red); 3. CIWL strategy matches strength of ICL, as indicated by 50% performance on Flip evaluator; 4. CTWL strategy dominates network output, leading Flip evaluator (red) to be 0 and CIWL evaluator (green) to be 1. (c) Illustration of competitive (Layer 1) and cooperative (Layer 2) interactions we find between ICL and CIWL strategies. Both strategies are present varying amounts through training. as represented by the varying line weights in the Layer| circuits: when Layer 1 acts as previous token heads, the network exhibits ICL, but when Layer I heads attend to self, the network exhibits CTWL. Crucially, the computation Layer remains largely unchanged after its initial formation, despite the strategy switch from ICI. to CTWL.
We annotate four points: 1. the formation of Layer circuits, the
ously cooperate and compete with each other Cooperation enables the emergence of ICL (despite not being asymp- totically preferred), while competition leads to its eventual transience (as previously indicated by (Singh et al. (2023)) Notably, ICL emergence can only occur when CIWL is not fully formed (Section 5.4). which may explain the tradeoff seen by Chan et al. (2022): under certain data properties, CIWL forms too "quickly," preventing ICL emergence.
emerge (and possibly fade) when training transformers.
2. Experimental setup
2.1. Training details
We train 2-layer attention-only transformers (Vaswani et al. 2017 Elhage et al., 2021) on a synthetic few-shot learn- ing task. We use dmodel = 64, with 8 heads per layer and learned absolute positional embeddings. As is common in mechanistic work (Olsson et al.) 2022; Singh et al., 2024). we chose this minimal setting as sufficed to reproduce key phenomena & Ba 2015) with 31 = 0.9, B2 = 0.999, a learning rate of 10 and batch size of 32 sequences. All models were trained in JAX (Bradbury et al. 2018). All code is open- sourced at
We formalize intuitions from this case study into min- imal mathematical model capturing coopetition dynamics and explaining the transience behavior. Our model moti- vates further experiments to modulate the tradeoff between strategies through learning, like reducing the asymptotic bias towards one strategy via data properties. For the case of ICLw vs CIWL, we find that matching context and query exemplars removes the asymptotic bias towards CIWL and leads to the persistence of the "faster" ICL strategy.
? We used the Adam optimizer (Kingma
aadi tyat ingh
icl icl-dynamics
Our work represents a step forward in understanding how different strategies trade off during learning, through mech- anistic investigations on the transience of ICL. We hope our work inspires further work on such coopetitive, dynamical phenomena and enhances intuitions around how capabilities
2.2. Dataset
Our few-shot learning task consists of sequences of exemplar-label pairs, where image exemplars are drawn from the Omniglot dataset of handwritten characters (Lake) ct al. 2015). Each character class contains 20 image exem-
A classic historical examples is Hollywood studios the carly 20th century, who competed aggressively for talent and audiences, but also collaborated to establish industry-wide standards and jointly negotiate with labor unions.
We still consider various alternative architectures, such as those with MLPs or RoPE (Su et al. 202T), in Appendix B
2

---

Strategy coopetition explains the emergence and transience of in-context learning
plars, and is assigned to different one-hot label. Images were embedded using Resnet18 encoder that was pre- trained on ImageNet (He et al. 2015 Russakovsky et al. 2015), before being imput to the transformer (Singh et al.
model's preference between ICL and CIWL strategies. The two context exemplars have their labels flipped relative to training (e.g. if exemplars X and Y were trained with label mappings X:24 Y:25, the in-context mappings would instead be X:25 Y:24 for this evaluator). The query comes from one of those two classes. If the network prefers the ICL strategy, accuracy on this evaluator would be the network prefers CIWL, accuracy would be O, as it would instead output the label that was paired with the query exemplar during training. This evaluator is especially useful in measuring which strategy is dominant at each point in training (even when both strategies are present in some form).
2024). While the original Omniglot dataset has 1623 classes,
we follow prior work (Chan et al, 2022) and augment it to 12984 classes by applying flips and rotations. Of these, we use a random 12800 for training. In Appendix [B.3 we also considered using different #'s of classes exemplars, observing similar modulations to (Singh et al. (2023) for the duration, timing, and magnitude of the transience effect.
Each training sequence consists of two exemplar-label pairs (the "context"), ) followed by a "query" exemplar. The model tasked with outputting the correct label for the query (cross-entropy loss). We train on the "bursty" sequences from Chan et al. 2022), where an exemplar-label pair from the same class as the query exemplar is always present in context. This kind of burstiness reflects real-world data dis- tributions like language, and permits both in-weights strate- gies (since exemplar-labe mappings are fixed throughout training) and in-context strategies (since the query exem- plar always comes from the same class as one of the two "context" exemplars) (Figure Ia top row).
3. Reproducing the transience of emergent ICL and the convergence to CIWL
Figure Dp shows a reproduction of the key transience phe- nomena in our simplified setting, with an extended figure in the appendix (Figure TT.. ICL emerges and then disappears, as evidenced by above-chance performance on the ICL eval- uator (blue). The disappearance of ICL corresponds to rise in accuracy on the CIWL evaluator (green), indicating that the network is somehow using just the label information from context, in combination with some form of in-weights information, to get the right answer.
2.3. Evaluators
Notably, there is a significant period of time where ICL. and CIWL co-exist (from ~3e6 to ~2e7 sequences seen). During this time, the balance between the two strategies shifts from ICL to CIWL, evidenced by the decrease in the Flip evaluator (red). Early in training, ICL dominates (annot. 2). Then, ICL and CIWL are roughly balanced (annot. 3), before ICL fades and CIWL dominates (annot. 4). This corresponds to a switch in the network's ' behavior, which first bases its output on the exemplar- label mappings from context, and then eventually to the exemplar-label mappings from training (in combination with the label provided in context).
To measure in-context and in-weights strategies, we consider four
evaluation sets (Figure
In the "ICL." evaluator, we replace labels from training with O and 1. This invalidates any exemplar-label mappings previously stored in weights. To perform above chance, the model is instead forced to use the mappings provided in context, i.e. an in-context learning (ICL) strategy.
In the "IWL" evaluator, the query exemplar (and its label) does not appear in the two context pairs, thus preventing the model from using context to perform the task. This forces the model to use knoweldge stored in weights, i.e. a pure in-weights learning (IWL.) strategy. Performance on this evaluator barely rises above chance (Appendix C.I].
4. The asymptotic "context-constrrinet" in-weights learning" (CIWL) mechanism
The "CIWL" evaluator is like the IWL evaluator, except that while matching exemplar the query is not in context, the correct label is in context. Following Singh al. (2023). the correct label is randomly paired with one of the context exemplars. Like the IWL evaluator, this evaluator can be solved by pure IWL. However, this evaluator also permits a mixed strategy where in-context label information can be combined with in in-weights information. We refer to the strategy that achieves above chance on this evaluator (but chance the IWL evaluator) as context-constrainet in- weights learning (C/WL). It requires the correct label token in context, but not the full exemplar-label pairing.
Prior work (e.g.. Chan et al., 2022, 2024, Nguyen & Reddy 2024 Anand et al. (0224) has contrasted ICL to "pure" IWL strategies that are completely independent of context. How- ever, we find that the dominant asymptotic mechanism in our networks is dependent on in-weights information but also context-con comtext-comstrained, requiring the presence of the correct label in context. In this section, we thus focus on the mecha- nisms underlying the CIWL strategy, and relegate discussion of auxiliary pure in weighights strategies to Appendix C.I
Finally, the "Flip" evaluator can be seen as testing for the
3

---

Strategy coopetition explains the emergence and transience of in-context learning
Attention strength
1
Single head in-context accuracy
1.0
related with output predictions (Fig (Figg)), causally indicating that each 1.2 head is serving to copy input labels to the out- put. The combination of attending to the correct label and copying it forward give rise to the CIWL mechanism.
0.8
0.6
Layer 2 HHod
3
Layer 2 skip-trigrams may be represented in superposi- tion. While the averaged results in Fig 2 may indicate that heads are performing similar functions each other (in con- trast to to/Singh al. (2024), who found differing strengths for different induction heads), we find more heterogeneity when inspecting responses on individual data points (Fig 15a). It's also surprising that heads do not perfect their attention pat- tern to the correct label (again differing from the induction heads in Singh et al. (2024)). We suspect that the large num- her of classes present ( (12800), compared to the dimension of the model (64) and number of heads (8) lead to the in- dividual skip-trigrams (for each exemplar-label pair) being stored in superposition (similar to sparse features stored in Transformer MLPs (Bricken et al. 2023) ; preliminary evidence in Appendix C.2).
Layer Head #
5
5
0.2
2
6
3
7
8
0.0
1 3
a)
Index in CIWL eval sequence
b)
-1.0
Attention delta correct vs incorrect label
-0.5
0.0
0.5
1.0
Figure 2. CIWL strategy is implemented via Akip-trigram-1 like mechanisms in Layer 2 (I.2), with substantial K- and V- com- position to Layer 1 (I.I). (a) Average attention patterns for 1.2 heads at the end of training. Attention is measured from the query token (index 5) to each token in context. is computed over CIWL sequences where the correct label at index 2 (results for label at index 4 in Fig [2p. We see that, at the end of training, L2 heads attend to the correct label, regardless of what exemplar it is paired with in context. (b) Task performance as function of clamped attention delta (difference between post-softmax attention weight) to correct vs. incorrect label token, calculated over 5000 CIWL sequences, when only the given head is active. CIWL accuracy increases as attention to the correct label increases.
4.2. Layer 1 engages in K- and V- composition, despite not being necessary
While LI heads do not directly influence network output, ablating them completely (without preserving patterns and values in 1.2) leads to big drop in performance. This indicates substantial composition (Elhage et al. (02TT) be- tween the two layers (i.e. L2 attention heads are dependent inputs from L1), despite the fact that skip-trigrams can be implemented with just single layer: Appendix Fig 13 shows that a I-layer model can learn CIWL using skip- trigram- like circuits, in line with prior work that describes l-layer mechanisms for skip-trigrams (Elhage et al. 2021
4.1. Layer 2 heads are skip-trigram-copiers
While label copying might appear to be a separate additive mechanism, we actually find that CIWL is implemented through the use of attention heads in Layer 2 (L2), which act as skip-trigram-copiee (Elhage et al. 2021). This mech- anism attends to the correct label, and then copies it forward to the output: "... [label] ... [query] > [label]."
Fig p shows attention patterns on the CIWI. evaluator. At the end of training, the model correctly attends from the query exemplar the in-context label that was paired with the query during training (regardless of which exemplar that label currently appears with in context). We also find such trends on in-distribution training sequences and Flip evaluators. Skip-trigram heads only appear in L2, with Layer (LI) heads not contributing directly to the output: ablating L1 -> output connections, via the pattern preserving method of Singh al. (2024), results in negligible drop in accuracy (98.7% 7% to 98.5%) on the CIWL evaluator.
To dig into the specific composition between the two layers, we consider various ablations of L.1 head outputs on the CIWL evaluator data, while preserving some combination of values, keys, or queries in L2. Recall that accuracy at the end of training on this evaluator is 98.7%, with chance level being 50%. When we preserve everything but values, performance drops to 59.3%, indicating LI outputs influ- ence L2 values in a crucial way (known as V-composition) [ When we preserve everything but keys, performance drops to 57.5%, indicating K-composition. When we preserve everything but queries, performance drops to 95.4%, in- dicating a very small amount of Q-composition (if at all), Thus, the primary reliance of L.2 on LI is in the calculation of L2 keys and values, begging the question: Given that a one-layer network could implement skip-trigram-copiee why does this observed K- and V- composition between layers emerge?
The second part of this mechanism is the copying forward of the label from context to output. We demonstrate this through the use of causal ablations: we clamp the attention pattern for a given head to give a certain weight to the "cor- rect" label token and the remaining weight to the "incorrect" label token. When considering each L.2 head in isolation F1 we find that modulating this weight is nearly perfectly cor-
"As defined hy Elhage al. 2021 1, the degree of k-, q- and composition is the degree that previous layer contributes to keys, queries, and values (respectively) of a following layer.
Thiis is performed by taking the output of a single 1.2 head after our clamp, and feeding it through the rest the network.
4

---

Strategy coopetition explains the emergence and transience of in-context learning
1.0
1.0
0.5
Layer 2
Head #
Induction strength
In-context accuracy
0.0
0.5
0.5
Train
Label eval
01 eval
Flip eval
-0.5
0.0
0.0
a)
sequences seen 1e7 b)
1e7 C)
# sequences seen
sequences seen
1e7
Figure 3. The transience of emergent ICL is driven by changes in the function of Layer 1. (a) Average induction strength (attention delta to correct ICL token vs incorrect ICL. token) on Flip eval data of each Layer 2 head, through the course training. We see induction circuits emerge then flip, matching the end-of-training attention patterss shown in Fig 2 2 weights to be those from the end of training, and plot performance on each of our evaluators. Using the Layer 2 weights from the end of training (darker curves) reproduces the original behavior (lighter curves; matches Fig D) at all points in training after the initial flat portion. This indicates that Layer 2 is not meaningfully changing during the transition from ICL to CIWL. (c) For each available checkpoint, we fix the Layer weights those from specific checkpoint (marked by the dotted orange vertical lines). After the Layer I weights are fixed in this way (darker curves), network hehavior doesn't 't change, as evidenced by the flat lines on all the data we considered.
(b) For cach available checkpoint, we fix Layer
5. ICL emerges, despite not being asymptotic, due to cooperative interactions with CIWL.
2023 We take each checkpoint from training, and then fix subsets of the network to the weights from a different checkpoint from the same run. In Fig 3. we show that fixing the first half of the network (embedding LI) to that of a checkpoint at given point (orange dotted lines) leads to very little change in network behavior after that point (flattening of the dark lines as compared to the lighter lines). Conjointly, if we fix the second half of the network (L2 + unembedding) to the checkpoint from the end of training (FggB)) we see little difference in behavior after the initial phase change, which indicates that Layer 2 changes are not meaningfully affecting the network after that point.
Induction circuits (which typically underpin ICL strategies, c.f., Olsson al. (2022) also show characteristic K- and V- composition, and seem be present during the transient emergence of ICL in our experiments (p). Could the K- and V- composition for the two strategies be related? Do these seemingly competing strategies actually share circuit elements? Could that be why ICL emerges, despite not being asymptotically preferred? In this section, we find the answer is, remarkably, yes.
Notably, this means the canonical "induction heads" (in L2) remain, but they become part of the computational strategy CIWL rather than ICL,, due to the change in LI. These results connect to a broader emerging notion that few-shot ICL (Brown et al. 2020) may lie on a spectrum of ICL abilities (Lampinen et al. 2024 Yin & Steinhardt 2025). which now show may be connected on the mechanistic level via shared subcircuits.
5.1. The ICL to CIWL transition is explained by L2 reuse and L1 dynamies
Induction circuits are typically comprised of a "previous token" head in the 1st layer that copies information from the previous token into the next token, and an "induction head" in the 2nd layer that uses the st layer to find tokens preceded by the present token. This enables a common form of ICL::"[A]]U] "[A*][B=] ... [A] + [B]" OOlsson et al. 2022).
5.2. Learning the ICL strategy is enhanced by the availability of asymptotic CIWL
Surprisingly, we find here that the L2 induction heads in the ICL induction circuits are re-used by CIWL with little change. The transition from ICL to CIWL is implemented by transition in the role of LI heads, which switch from previous-token attention to attending-to-se (Fig 10) 5
Why does ICL emerge at all, if it is not asymptotically preferred? We find that ICL's emergence may actually be enabled by ICL's shared L2 subcircuit with CIWL. Thus, shared subcircuits may lead to cooperative interactions, on top of known competitive interactions (Singh et al. 2023 Park et al. 2024).
This is supported by attention patterns in L1 (Fig 14] and L2 2)) and also by two sets of additional experiments, inspired by the notion of progress measures (Nanda et al.
To show this, train networks on "ICL-only" data, where we randomize the exemplar-label mappings across contexts, but keep the mappings consistent within each context. This data can only be solved by ICL, and not by CIWL (since
"Aloorithmically we believe these heads are essentially learn ing to map label tokens to prototypical embeddings. It's hard to directly verify such hypothesis, given the heads' rotation invari- ance and observed superposition (Appendix C.2). but our evidence for the semantic stationarity of L2 points to such an algorithm.
5

---

Strategy coopetition explains the emergence and transience of in-context learning
Step from CIWL-onl run
8M
16M
32M
64M
No Intervention
ICL Evaluator Loss
Second half clamped
1.0
In-context accuracy
0.8
Flip Evaluator
ICL
ICL Evaluator
2-
0.6
0.4
CIWL
0.2
0
0.0
a)
# ICL-only sequences seen 1e7 b)
ICL-only sequences seen 1e7 C)
2
0.0
0.5
total sequences seen
1.5
2.0
1e7
2.5
Figure 4. ICL emergence is enabled due to cooperative interactions with CIWL. (a) In this plot, we train networks on "ICL-only" data, i.e. where ICL is viable strategy but CIWL is not. Without any interventions (black), we hit a loss plateau that greatly slows learning (S.f.[[Singh ct al. (204)). However, if replace the Layer and unembedding weights with those from end-of-training on our standard training data (green), the network learns quickly. We thus see that these weights, which were part a CTWL. strategy, are reusable for learning ICL. (b) We further consider using Layer unembedding weights from different checkpoints of "CIWL-only" run, and then training on "ICL-ouly" data. Early and late checkpoints lead to no learning of ICL, but middling checkpoints (from 9.8M to 31.5M, identified via binary search) do enable ICL learning.
(c) We continued training different checkpoints from the "CIWL-only" run our standard training data. Once CIWL has formed (later checkpoints), ICL does not re-emerge even when switching to the bursty data (which otherwise permits ICL).
there are no fixed mappings to learn in weights). When training from scratch, we see the network struggle learn 4h, black). Yet when we previously trained on bursty data (where CIWL was viable strategy), we saw ICL emerge quite quickly (Fig [p). Combined, these form striking reS actually emerges more easily when it is not the only viable strategy! Presence of an asymptotic CIWL mechanism may be enabling ICL. emergence.
then take checkpoints from different points in training of this CIWL-only run, and use the 1.2 + unembedding weights to repeat the experiments of Section 5.2 namely, clamped training on ICL-only data. Essentially, we're asking if the L2 weights when learning CIWL strategy, without influ- ence from ICL., can still enable ICL strategies. Surprisingly, we find that the answer is, for brief period, yes.
Fig 4p shows training curves of our clamped networks on ICL-only data. Each curve is colored based on the iteration from the CIWL-only run that the clamped weights come from. We see that there is region from about 10M 32M sequences seen where ICL can emerge. After this point, L2 weights likely specialize too strongly on CIWL strategy (as they't 're being trained on data that only permits such strategies), and thus can no longer be re-used for ICL.
To further test this hypothesis, we again trained on ICL- only data, but clamped the weights of the second half of the network (I.2 unembedding), using those from the end of training from our "standard run" on bursty data, after the network has converged to a CIWL strategy. Through this causal intervention on dynamics (using the framework from Singh et al. (2024), ), we show that ICL. can emerge quite rapidly (Fig 4a, green) with the help of the CIWL weights for Layer 2. These findings connect to the notion that loss plateaus may arise from multiple sub-circuits needing to go right (Singh et al. 2024). When the Layer 2 subcircuits are provided, ICL emerges quickly.
5.4. If CIWL is fully formed, ICL will not emerge
To understand why and when ICL emerges, a final critical factor is that CIWL. is "slow" to emerge on certain data distributions, relative to ICL. If this were not the case, the competitive interactions might dominate and CIWL might prevent ICL. from ever emerging. Indeed, Chan al. (022 found many data distributions where in-weights strategies appear quickly and dominate, and in-context strategies do not emerge at all (reproduced in our setup in Appendix (.3))
5.3. ICL is "close to" the path towards CIWL
One confound in the previous experiment is that CIWL weights at the end-of-training were likely influenced by the earlier presence of ICL (a possible path dependence). Here, we consider a stricter version of the experiment.
To show more directly that ICL cannot emerge once CIWL has fully formed, we take various checkpoints from the "CIWL-only" run, and continue training but with bursty data (Fig When initializing from early "CIWL--only" check- points, we see transient emergence of ICL before CIWL again dominates. At later checkpoints, CIWL. persists, with no emergence of ICL. This indicates that ICL could only
We start by training networks on "CIWL-only" data: se- quences drawn from the same distribution as the CIWL evaluator, which can be solved by CIWL but not ICL. We
'hii resembles the setup used by Singh et al. 2024), except with 12800 classes and 20 exemplars per class.
6

---

Strategy coopetition explains the emergence and transience of in-context learning
emerge at the earlier training times because CIWL was not yet fully formed.
gla, b, c, d) Mechanism
Train
ICL eval
0.4
Mechanism
CIWL eval
5.5. Strategy coopetition
Loss
Less
5
0.2
Taken together, these results reveal surprisingly rich inter- action between ICL and CIWL: ICL is enabled by CIWL, yet also competing with CIWL for resources in the network. Prior work has pointed to competition between ICL and CIWL, which we now mechanistieally observe in Layer 1. In contrast, Layer 2 exhibits cooperative interactions that lead ICL to emerge, despite not being asymptotic. This emergence dynamic has a few requirements:
0.0
4K Steps
8k
0.0 0.5 sequences seen
a)
b)
1e7
Figure 5. Minimal mathematical model captures key phenomena of strategy racing and coopetition. (a) Toy model dynamics. (b) Loss of real transformer, corresponding to Fig [p. Notably. both curves exhibit transience behavior in Mechanism (ICL). Intriguingly. the toy model also captures nonmonotonicity in the emergence of Mechanism 2 (CTWI.), highlighted in orange.
1. Useful: ICL's ability to reduce loss "bursty data."
2. On the way: ICL is "close" to the path of 2L network learning an asymptotic CIWL strategy.
loss is also computed on tensor products of multiple vectors. The tensor product encodes the intuition that multiple sub- circuits need to simultaneously "go right" in order for the entire circuit to be Iearned. The model of (Sinnn al. (2024) is equivalent to the loss for single mechanism in model. Our full model has multiple circuits (ICL and CIWL) in competition, racing to minimize loss (Section 5.4. but also cooperating (Sections 5.15 5.3) via shared sub-circuits (b, c).
3. Fast: The CIWL strategy emerging "slowly" enough to allow the "faster" ICL to make a transient appearance.
The first factor follows from the dataset design. The second requirement is supported by Sections 5.1
The third requirement is supported by Section 5.4 and prior work
5.3
(Chan et al. 2022] Singh et al. 2023 Nguyen & Reddy 2024).
We model these race dynamics using two mechanisms that interact multiplicatively (lines T and (2). If the offset PL O. then minimizing either the loss of Mechanism or 2 will lead to 0 loss. By setting M1 > O, we enforce an asymptotic preference for one mechanism over the other. Competition is added via the second term (line (3), which drives a or d to 0 in order to achieve O loss (if a > 0). This corresponds to the network pressure we observe for Layer to be part of an ICL or CIWL strategy. To model cooperative interactions in Layer 2, we re-use vectors b, c in both mechanisms. Finally, to model the relative speeds of the two mechanisms, we use different dimensions for and d (larger vectors are slower to learn).
6. A toy model of strategy racing + coopetition
To crystallize these intuitions, we propose a minimal mathe- matical model that captures the competitive and cooperative interactions at play. Specifically, we consider learning four vectors a, b, c, d via gradient descent on the following loss function, where at, b,, c,, d are the true values:
C(a, c.)
|a @ b 8 b cE + #1
(1)
Putting it all together, we simulate the toy model dynam- ics numerically for the setting of dim(a) = dim(b) = dim(c) 20, dim(d) = 160, /11 = 0.1, 0.1. We (2) show results in Fig 5p, with additional seeds and details in Appendix D Our simulation shows transience of Mech- anism 1, where its loss first quickly drops with the initial phase change, but then goes back up, indicating that a is nearly learned but then eventually goes back to O. This return to 0 is caused by the asymptotically preferred emerging on a slower timescale and the competition term (a > 0). These emergence and subsequent transience dy- namics mimic those of ICL, with sub-cireuits b, c represent- ing the shared Layer 2 "induction heads."
Mechanism 1 (ICL) Loss
x
||d 2 b* & &
cIfE
Mcchanism (CTWL) Loss
|a Competition
(3)
where 20 0 is a parameter modulating the strength of competition and and μ1 2 0 modulates the relative asymptotic preference of mechanism 2 over mechanism 1. If a. μ1 > 0, loss is minimized when a 0, b- = b" = - d addd" d".
This loss function builds on the minimal model of phase change dynamics proposed by Singh et al. (2024), where
Curiously, the toy model dynamics showed a small divot in the formation of Mechanism 2, corresponding to CIWL (highlighted in orange, Fig 3p.. Specifically, when Mech- anism is nearly fully learned (~ 5k iterations), there's a
An alternate interpretation is that is fixed version of I Lagrange multiplier for satisfying the constraint that a is O.
7

---

Strategy coopetition explains the emergence and transience of in-context learning
Normal
Match exemplars
chance
persists after its initial emergence! Furthermore, this finding extends to the larger-scale setup considered by |Singh et al. (0233) well 10 indicating that the intuitions from the toy model may extend to larger networks.
innontest accuracy
1.0
Flip evaliator
0.5
CIWL
In combination with Chan et al. (2022), this result would point toward data properties that not only lead to the emer- gence of ICL CCL-bussty data, with high diversity (# of classes) and variation (# of exemplars) across sequences- but its persistence as well: strong correlations in-context- exact matching exemplars [T] Overall, we found this interven- tion, motivated by our minimal mathematical model, a com- pelling illustration of how a better dynamical understanding may lead to data interventions that can control the cycling in and out of strategies throughout training.
0.0
0.5
1.0
1.5
2.0
0
2 3
# sequences seen 1e8 Figure 6. Matching context and query exemplars lcads persistent ICL in small (left) and large (right) models, as evidenced by the darker red line converging to accuracy one on the flip evaluator. Larger models are trained with the setup of et al. (2023). The timescale of transience much longer in larger models (lighter line, right plot), so we don't It see full fading of ICI. Notably, the matched exemplars run shows no indicating of tuming downwards.
# sequences seen
1e7
mild worsening of Mechanism 2, before the eventual learn ing that causes transience of Mechanism 1 and asymptotic dominance of Mechanism 2. When we reexamined the loss curves of our actual transformer, (whose corresponding in- context accuracies are plotted in Fig D)) we found a similar divot in the loss on CIWL evaluator (Fig [ While we do not fully understand what leads to this brief divot we believe this mirroring between model and empirics lends credence to this simple mathematical model.
8. Discussion
When there are multiple ways to solve problem, when and why does model "choose" between each strategy?
In this work, we have explored this question by building on previous work demonstrating the transience of in-context learning (ICL) in transformers
ICL emerges but
then disappears after long training times (Singh et al. 2023 Anand et al. 2024; He et al. 2024 see extended Related Work in Appendix A..
7. ICL can be made asymptotically preferred
Equipped with a better understanding of transience, we re- turn to revised version of the question from (Chan al.] b Are there data properties that incentivize the asymp- totic emergence of ICL, but don't 't require it? We tackle this question using intuition from our toy model.
We uncovered a few surprising findings. On a task that was designed to be solvable by both in-weights learning and in-context learning, we find that the model asymptotically prefers an unexpected strategy that is combination of the two, which we call "comtext-constrained in-weights learning" (CIWL), After first showing this behaviorally through OOD evaluations, we elucidate the mechanistic implementation of this strategy (Section 4. which may have connections to notions of task recognition (Lin & Lee| 2024) and superpo- sition in larger models (Templeton et al. 2024
Specifically, if μu = 0, the faster Mechanism (corre- sponding to ICL) would win asymptotically (Fig 17). Thus, when two strategies are possible, one can be made preferred asymptotically if it's 1) faster and 2) as good asymptotically. We've ve already established that ICL is faster to emerge than CIWL (Section 5.4) when training on our bursty data, so all that remains is to see if we can find data properties that make ICL "not worse" at the task than CIWL.
We find that cooperative interactions between strategies may lead to the transient emergence of one strategy, even though it is asymptotically dominated by the other strategy. In par- ticular, ICL and CIWL have shared sub-circuitry that enable such cooperation. For the case of transient ICL, we show
The key difference in the mechanisms we find for ICL and CIWL is that CIWL matches query exemplars to static la- bels, while ICL matches to a variable context exemplar. This variation may make the "matching" harder for ICL. We ablate this by considering data where the relevant context exemplar is the exact same as the query exemplar 9 In Fig 6 we find that training on such data makes ICL asymptotic it
10 SSeeiifically. 12-layer transformers, with MLPs, trained on bursty sequences of 8 exemplar-label pairs, with the 3 bursty con- text exemplars matching the query in our intervention.
Il A possible mechanistic explanation could be that the matching performed by softmax attention cannot be as precise, which leads to an asymptotic preference for CIWI. when the bursty exemplars don't exactly match the query.
*We speculate that as Mechanism I (ICL) forms, the competi- tion forces brief regression to Mechanism 2 (CIWL), which is slowly learning in parallel.
12 "Specifically, the compression of 12800 class-specific skip- trigrams into 8 attention heads with superposition could motivate research on an analog of sparse autoencoders [Bricken et al. (2023) for attention heads (e.g.. going from 8 heads to 12800 sparse heads that may individually represent each skip-trigram feature).
"Note: there are still 20 exemplars per class, making the exemplar-label mapping many-to-one (leaving CIWL the same), it's just that the bursty context exemplar exactly matches the query exemplar (making ICI. more effective).
8

---

Strategy coopetition explains the emergence and transience of in-context learning
that its emergence is due to three key properties useful (ICL helps reduce loss the task, by construction), on the way (ICL is close to the path of a CIWL solution, Sec- tion 5.3), and fast (ICL only emerges if CIWL is sufficiently unformed, Section 5.4). Eventually, CIWI. does dominate and ICL fades due to competitive interactions, as suggested by prior work (Singh et al. 2023). We borrow the term "coopetition" describe these simultaneously cooperative and competitive strategies.
specifically highlighted here.
Acknowledgements
We'd like to thank Andrew Lampinen, DJ Strouse, Kira Düsterwald, Jirko Rubruck, Dan Roberts, Basile Con- favreaux, Jin Lee, Yedi Zhang, and Murray Shanahan for useful discussions and feedback on early drafts.
A.K.S. and T.M. are funded by the Gatsby Charitable foun- dation. This work was supported by Schmidt Science Polymath Award to A.M.S., and the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/2/19/Z) and the Gatsby Charitable Foundation (GAT3850). A.M.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.
We crystallize these intuitions from the case study to a more general mathematical model of strategy racing that cap- tures cooperative and competitive interactions. This toy model motivates experiments that identify data properties that do lead to persistent ICI., despite not requiring it, an- swering some of the original questions of Chan et al. (2022).
The shared sub-cireuitry we find may also point to different forms of ICL. (e.g., task recognition and task learning) not being as different as previously perceived. While earlier works have argued a tradeoff via competitive interactions (Lin & Lee 2024] Nguyen & [aadyy]20244 Park et al. 2224). our work suggests a more nuanced take and provides mech- anistic evidence for more recent notions that ICL abilities
Dedication
In loving memory of Felix Hill, whose mentorship, vision, and enthusiasm made this entire research program on in- context learning possible (Chan et al. (0022); Singh et al. (2023) (024), and now this paper). Felix's s ideas continue to influence our field and will resonate for many years to come. More importantly, Felix made immeasurable and lasting impact on the lives of the researchers he mentored and inspired.
lie on spectrum (Lampinen al. 2024
Overall, we hope our work serves to deepen the understand- ing of ICL and how it may interact with other strategies through the course of training. Beyond ICL, we view our work as a case study demonstrating a few themes that are important to keep in mind, for those of us who wish to un- derstand model behaviors: (1) Models often learn surprising and counterintuitiiv strategies, even for simple tasks. (2) Models are highly dynamical and we cannot assume that their strategies remain constant over training, even if they appear stable at a given time. (3) These training dynamics are affected by a kind of backwards hysteresis, where later strategies can affect the development of earlier strategies (4) Alternate strategies are not always strictly in competition and can exhibit coopetition by boosting each other.
References
Anand, S., Lepori, M. A.. Merullo, J., and Pavlick, E. Dual Process Learning: Controlling Use of In- Context vs. In- In-Weights Strategies with Weight Forget- ting, May 2024. URL httt:://aaxiv.or////eaaa
2406.00053 arXiv:2406.00053 [cs] version: 1.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman- S.,, and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL
jax
Impact Statement
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield- Dodds. Z., Tamkin, A., Nguyen, K., McL.an, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transformer- wewptttiiieftttuue//eeeeee htmm..
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be
In addition to our "positive" evidence for the found explana- tion, we reject various alternate hypotheses in Appendix F "Saxee et al. 2022 also considered the notion of different path- ways in a network competing to minimize loss, using the formalism of gated deep linear networks, and found similar intuitions about the "fastest" pathway winning.
'ee note that this backwards hysteresis complements notions of path dependence, such as the CIWL strategy being influenced by ICL earlier, for which correlates have been found in larger-scale language models (Yin & Steinhardt 2025).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,
9

---

Strategy coopetition explains the emergence and transience of in-context learning
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020. URL http://paoceeiinsseeuiiee.. paper htt.fffeeeeaaapppee////0///11/11
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. CoRR, abs/1512.003385, 2015. URL http:://aaaxiiorrrraaa///S123333
He, T., Doshi, D., Das, A., and Gromov, A. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks, June 2024. URL /ttpb/tp://
a /ttp////.abb.22255500 [cond-mat, physics:hep-th, stat].
arXiv:2406.02550
Hendrycks, D. and Gimpel. K. Gaussian error linear
Paper.pdf
units (gelus), 2023. URL |nttps://arxiv.org]
abs 008/1606.08413
Chan, B., Chen, X., György, A., and Schuurmans, D. Toward Understanding In-context vs. In-weight Learn- ing, October 2024. URL httpp://axxivoorg abs/ 2410.23042 arXiv:2410.23042 [es].
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: 7 /arxxi.orr abs abs/1412.6980;
Chan, S., Santoro, A., Lampinen, A., Wang. J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distributional properties drive emergent in-context learning in transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho. K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 18878-18891, Curran Associates, Inc., 2022. URL
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through prob- abilistic program induction. Science, 350(6266); 1332-1338, 2015. doi: 10.1 10.11166/scicmcc..a33050
URL 10.126/scienc..ab3555
paper_/iiees/pppee//00/////11
Lampinen, A. K., Chan, S. C. Y., Singh, A. K., and
77c6caccfa9622222111448888 Ssper-Confereece.ppp
Shanahan, M. The broader spectrum of in-context
learning, 2024. URL htttps://rriiv.org//bbt
2412.03782
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hermandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K. Amodei, D., Brown, T., Clark, J., Kaplan, J.. McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https:/transformer- httii/puuuu/22//aaeeooooooo
Lin, Z. and Lee, K. Dual operating modes of in-context learning, 2024. URL http:://azx.v..og///b/
2402.18819
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt, 2023.
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability, 2023.
html.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition. Transformer Cir- cuits Thread, 2022. URL.
Nguyen, A. and Reddy, G. Differential learn- ing kinetics govern the transition from memoriza- tion to generalization during in-context learning, November 2024. URL http://arxiivorr/ 2412.00104| arXiv:2412.00104 [cs].
Olsson, C., Elhage, N. Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., HatfieldDoddss, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, I.,, Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learn- ing and induction heads. Transformer Circuits Thread, 2022.
Frankle, J. and Carbin, M. The lottery ticket hy- pothesis: Training pruned neural networks. CoRR, 0D/11803.06335 2018 URL http://aaxxiv.or// abs Abs//803.03335
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories, 2021.
html.
10

---

Strategy coopetition explains the emergence and transience of in-context learning
Park, C. F., Lubana, E. S., Pres, I,, and Tanaka, H. Com- petition Dynamies Shape Algorithmic Phases of In- Context Learning, December 2024. URL http:// arXiv:2412.01003
Yin, K. and Steinhardt, J. Which attention heads mat- ter for in-context learning?, 2025. URL https://
axxivorg/abs//5022.10
aux.v..rrg/ab//2112.0000 [es].
Zhang. Y., Singh, K., Latham, P. E., and Saxe, A. Training dynamics of in-context learning in linear at- tention, 2025. URL https: https: http:s///axxiv.rr//aaa/
Reddy, G. The mechanistic basis data dependence and abrupt learning in an in-context classification task, 2023.
2501.16263
Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.. Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge, 2015.
Saxe, A. M., Sodhani, S., and Lewallen, S. The neural race reduction: Dynamics of abstraction in gated networks, 2022.
Singh, A. K., Chan, S. C. Y., Moskovitz, T., Grant, E., Saxe, A. M., and Hill, F. The Transient Na- ture of Emergent in-Context Learning in Transformers, December 2023. URL http::////// //a a/rxx//riiii// v.org/abs.. 2311.08360 arXiv:2311.08355 [cs].
Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y., and Saxe, A. M. What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation, 2024.
Su. J., Lu, Y., Pan, S., Wen, B. and Liu, Y. Ro- former: Enhanced transformer with rotary position em- bedding. CoRR, abs/2104.0998.. 2021. URL httpp: //arxiv.orr/abs/2200..2
Templeton, A., Conerly, T., Marcus, J., Lindsey, J. Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., Mc- Dougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, Scaling monosemantie- ity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL
http:://ttansfemmerrrooooooooppp
scal hagallag..nooooeaaaattttttnnnnno
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At-
tention all you need. Advances in neural information processing systems, 30, 2017.
Wang, X., Tang, X., Zhao, W. X., and Wen, J.RR. Inves- tigating the pre-training dynamics of in-context learn- ing: Task recognition task learning, 2024. URL
Wei. J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.c 03846. 2023.

---

Strategy coopetition explains the emergence and transience of in-context learning
A. Extended related work
Different forms of ICL Since its first documented emergence at scale (Brown et al. 2020), few-shot ICL in transformers became widely researched. ICL is often contrasted in- in-weights learning, where networks learn tasks over the course of training in-weights, rather than adapting context. However, some (Lin & Lee 2024 have argued that few-shot learning may be more akin task recognition, where few-shot prompts serve more identify task that the model already knows, rather than learn new task. This tradeoff may even be modulated by scale (Wei et al.) (023) or training time (Wang ct al. 2024). While induction heads (Olsson ct al. (022) may serve as a mechanistic explanation for few-shot task learning, mechanisms for naskrreogniitonn-iie phenomena have been lacking. In our work, we provide evidence for a context-constrained in-weights mechanism which may help elucidate rask-recognition like phenomena by indexing into the label space in context. Furthermore, we hope the shared subcireuitry between different levels of context-usage (CIWI. and ICL) may enhance mechanistic understanding of the full spectrum of in-context learning abilities (Lampinen et al. 2024).
A dynamical understanding of ICL Since the finding that the emergence of ICL may be a transient phenomena (Singh et al. (2023), reproduced by He et al. ( (Anand et al.] (2024) in other settings), many have focused on an enhanced dynamical understanding of ICL. Some have studied emergence dynamics (Reddy 2023 [Singh et al. 2024) of ICL, though only on data that requires ICL to minimize loss. Newer work has considered the tradeoff between strategies throughout the course of training in small (Nguyen& & Reddy] 2024; Park et al., 2024) and larger models (Wang et al., (024), or centered on theoretical modeling (Chan et al. 2024). These works largely focus on comperitive dynamics between strategies. Our work goes beyond prior work by providing an explanation for the emergence of ICL when not asymptotically preferred or necessary due to cooperative interactions between different strategies.
B. Extended behaviors seen in small settings
In this section, we provide additional results on alternative setups we considered speciffiadee.ccilllalll...... using other types of positional embeddings, interaction with data properties considered by Chan et al. (2022). and potentially using MLP layers 2L network (with absolute positional embeddings). For further exploration of the transience phenomena, we refer readers the original work of Singh et al. (2023) our work here focuses on finding a smaller scale setting that allows for study of the key phenomena (with insights that we show may carry to larger settings, Section
All our plots in this section feature accuracy on the IWL evaluator as well. Note that in-context accuracy on this evaluator is meaningless, as the correct label does not appear in context. As a result, we just plot the direct accuracy (for which chance level would be #classes ). Most of the times, this accuracy stays at chance level, though we note settings where it seems to rise. Further exploration of mechanisms that may be responsible IWI. are provided in C.T
Runs in this section are also often run for longer than our main experiments-we we truncated the plots in the main paper to focus the relevant phenomena. Here, we present the full timecourses we ran in case it's useful for future researchers. All code can be found at httpt:tt///ggtt tww..cooobccoooooo httt:p:://g/ccabb.com///ee aad tyas ngh/ ic cclclloo -dynami cs
B.1. Different random seeds
To confirm our reproduction, we experimented over few seeds (for both model initialization and data generation/ordering) see Figure 7 We found data seed to make little-to-no difference. Model intialization can change the exact timing of transience, but the general profiles of all the curves are the same.
B.2. Other positional embeddings
We considered absolute sinusoidal positional embeddings (introduced by Vaswani et al. (2017) and used in the original works of Chan et al. (002) and Singh et al. (023)) as well as rotary positional embeddings (RoPE, introduced in (Su et al. 202T) and used by (Singh et al,, 2024)). However, we found that, for two-layer attention only networks, learned absolute positional embeddings were needed to reproduce the emergence and transience of ICL. While larger networks don't 't require this added flexibility, we found necessary to elicit emergent ICL, which we speculate may be due to the difficulty of learning previous token heads in Layer 1 otherwise (Olsson et al. 2022).
12

---

Strategy coopetition explains the emergence and transience of in-context learning
seed: 5, Data seed:
Init seed:
Data seed:
Init seed: 5, Data seed: 4
1.0
In-context accuracy
Train
0.8
ICL eval
CIWL eval
Flip eval
0.4
IWL eval"
chance
0.2
0.0
sequences seen
# sequences seen
sequences seen
Init seed: 6, Data seed: 2
Init seed: 6, Data seed:
Init seed: 7, Data seed: 2
1.0
In-context accuracy
0.8
0.6
0.4
0.0
6
0
2
sequences
1c7
sequences
1e7
sequences
1e7
Figure 7. Reproduction of our main setup (Section 2
over random seeds. Top left figure is same as Figure 1b.
Learned (absolute)
Sinusoidal (absolute)
Rotary/RoPE (relative)
1.0
Train
ancurnertacccurayyy
In-onneett accuracy
0.8
eval
CIWL eval
Flip eval
0.6
IWL eval"
chance
0.4
6
sequences seen 1e7
# sequences seen
1e7
sequences seen
1e7
Figure Changes in behavior when using different positional embedding schemes. Left figure is same Figure Tp. Note that learned absolute positional embeddings needed to get ICL emergence in 2L attention-only transformer on our task.
13

---

Strategy coopetition explains the emergence and transience of in-context learning
12800 classes
1600 classes
1600 classes
exemplars/class
20 exemplars/class
5 exemplars/class
Train
In-context accuracy
0.8
ICL evall
CIWL eval
Flip eval
IWL eval"
0.4
chance
0.0
0.0
0.5
1.0
0.0
0.5 sequences seen
1.0
0.0
0.5 sequences seen 1e8
1.0
sequences seen 1e8
1e8
Figure 9. Changes in hehavior under different data properties, largely reproducing what was seen by Singh et al. (003
12800 classes
12800 classes
1600 classes
1600 classes
20 exemplars/class
5 exemplars/class
20 exemplars/class
exemplars/class
1.0
Train
In-context accuracy
0.8
ICL eval
CIWL eval
0.6
Flip eval
eval"
0.4
chance
0.0
2
sequences seen 1e8
sequences seen
1e8
# sequences seen
1e8 # sequences seen
1e8
Figure 10. Changes in behavior when training 2- 2-layer transformers with MLPs. In our main setting (left), we see similar trends, but when training on 12800 classes with 5 exemplars each, we get transient and resurgent ICL (unlike the corresponding run without MLPs, Figure left) In the rightmost plot, we see significantly increased pure in-weights learning low numbers of classes and exemplars (as
compared Figure(9). which may be due to the hypothesized role of MLPs such pathways (Geva el al. 2021 Meng et al.) [2023] Singh
al. 2023
B.3. ICL does not emerge with lower classes or fewer exemplars
We consider different #'s of classes and different #'s of exemplars. Chan et al. (2022) found that higher numbers of both incentivized ICL emergence, and Singh et al. (2023) found that more classes slowed down transience. In Figure we find that larger number of classes are necessary for ICL to dominate over CIWL for a time (Flip evaluator accuracy > 0.5). We still see smaller transience effects with 1600 classes, as long as exemplars are used. When reducing both (1600 classes, exemplars), we see no ICL emergence, and even some asymptotic strengthening of pure in in-weights mechanisms.
These findings match what mechanism emerges "too quickly" for ICL to show up (see Section 5.4
by (Singh al. (2023). and we hypothesize that when # of classes is lower, the CIWL
B.4. Behavior with MIPs
We also considered the use of 2-layer transformers with MLPs well, typically used in practice (Vaswani et al.) 2017). We use standard 4x expansion factor and GeLU activations (Hendrycks & Gimpel 2023). Results in Figure T0
While our main setting (12800 classes, 20 exemplars) seems to have similar trends, we observed transience and resurgence in other settings (12800 classes, exemplars), which make it difficulty to claim asymptotic behaviors in these small models with MLP. We thus restricted our main analysis to the attention-one only models, as is common for mechanistic work on transformers (Elhage al. 2021 Olsson et al. 2022. Singh al. 2024).
14

---

Strategy coopetition explains the emergence and transience of in-context learning
Train
eval
0.8-
Flip eval
IWL eval
Accuracy
0.6
chance
0.2
0.0
0.0
0.5
1.0
1.5 sequences seen
2,0
2.5
3.0
1e8
Figure Same run Figure [p. but rum long. We plot accuracy on y-axis (across 12800 labels) as opposed in-context accuracy to more compellingly demonstrate saturation of CTWI. (green) and minimal learning of TWI. (gray). This plot lets us include that networks are fact learning pure mechanism, but rather context-constrained one which requires the correct label in context.
Attention strength
1
Layer 2 Head #
3
4
5
7
8
3
4
Index in CIWL eval sequence
Figure 12. Analogous plot to Figure 2 except the label inserted at index as opposed to index 2.
15

---

Strategy coopetition explains the emergence and transience of in-context learning
Attention strength
Actention strength
Single head In-context accuracy
1.0-
accuracy
0.8
Train
*
ICL eval
0.6
0.6
CIWL eval
0.4
Flip eval
Layer
6 5
Layer 1 Head #
IWL eval
5
2
6
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
5
-1.0
-0.5
0.0
0.5
1.0
a)
sequences
1e7
b)
eval sequence
Index CIWL
C)
Index in CIWL
sequence
d)
Attention delta
Figure 13. A I-layer transformer trained on our bursty training data can learn skip-trigram piggrarmccopers. just as well as the 2-layer model. (a)
Metries indicate that the model learns CIWL strategy. Notably, ICL does emerge, would expect for I-layer transformer Elhagee et al. 2021). (b-c) Analogous plots to Figure and Figure [2] showing that all heads learn attention from query token (index 5)
the correct token in-context (index 2 4 for b, respectively). (d) Analogous plot to Figure showing that all heads copiers.
Layer attention patterns on data from: CIWL eval
Avg. attenbion to self Avg. attention to prev. token for the two label tokens
Train
ICL eval
Flip eval
for the two label tokens
0.8
0.4
0.0
0.8
Layer 1 Head #
0.4
0.0.0
1.5
0.0
0.5 1.0
2.0 0.0
0.5 sequences seen 1e7
2.0 0.0
1.0
1.5
2.0
sequences seen 1e7
# sequences seen 1e7
# sequences seen 1e7
Figure Per-Layer- lhhad average attention patterns over different data subsets through course training. Top row depicts average attention from label token (at index 2,44) to the previous token (index 1, 3, respectively). Bottom row depicts average attention from label token (at index 2, 4) to itself. Most of attention on these tokens made up of these two pieces (attention previous and attention to self), though tokens attend to all previous tokens (standard causal transformer setup). Attention patterns look similar similar subsets, indicating that Layer heads operate similarly on our out of distribution evaluators. Layer Head appears to have transient previous token bchavior, suggesting may be partly responsible for the emergence of ICI..
16

---

Strategy coopetition explains the emergence and transience of in-context learning
1.0
picorrect from context) on train data
with only one Layer 2head eciive
plcorret from context) on train dala
Layer
Head
0.9835
0.80-
with all heads active
Head Attention correct incorrect
0.9830
0.75
0.9825
0.9820
0.0
0.70
0.9815
0.65
0.9810
-0.5
0.5
0.0
0.5
1.0
10a1
10" Head temperature
10-1
10 temperature
a)
Layer Attention delta
correct over incorrect taken
b)
C)
Figure 15. Preliminary evidence of superposition. (a) While average attention patterns across heads look identical (Figure 21). patterns on individual points
quite different (e.g.. shown for Head and 2, where each point in the plot represents attention deltas of these two heads for single evaluation sequence). (b) Decreasing the temperature of any Layer 2 head's 's attention operation, if is the only Layer 2 head active, increases performance. (c) The same temperature decrease does consistently affect performance if all heads are active, indicating non-additive interactions.
C. Additional results
C.1. Networks barely exhibit pure in-weights learning (IWL)
Figure TT depicts accuracy on a pure IWI. evaluator (Figure gray). which differs from the CIWL, evaluator in that the correct label token does not appear in context. This evaluator is the same as that used by Chan et al. (2022) Singh et al. (2023). Like (Singh et al. 2023), we find that networks perform relatively poorly on thie evaluator, instead employing a CIWL strategy (as evidenced by the delta between green and gray curves in Figure
That said, the performance on the IWL. evaluator at the end of Figure is 0.08%, which is above chance level (which would be 1/12800 0.008%, indicating that the network may have picked up tiny amount of pure IWL (or could be noise). This prompted to investigate further, and we did uncover very minor in n--eiights mechanism.
Specifically, if we ablate all attention heads (so that the network is just the embedding -> unembedding connection), we find that in-context accuracy on CIWL is 62.46% (compared to 98. 74% with no ablation). Note, with this no-heads ablation, the network can't actually attend to any tokens context, so the "in-context accuracy" can be interpreted as "how likely is the model to output the correct class over a given random class." " This value (62.46%) being above chance (50%) indicates the model able do this comparison somewhat accurately using just the embedding - unembedding pathway.
To make this pure IWI. mechanism isn't playing key role in our results, consider the opposite ablation: zero-out the embedding - unembedding pathway, and leave the rest of the network unchanged. We do this by using pattern-andvaale preserving ablation, using two passes through the network. In the first pass, no intervention is used, and patterns and values are cached. In the second pass, we out the embeddings, but use the cached activations for the attention heads, having the intended effect. This ablation leads to 2% drop in accuracy (98.74% to 96.72%), indicating that the CIWL mechanism, implemented via skip-trigram-copying attention heads, activate, the embedding - unembedding pathway quite auxiliary.
To conclude, networks do small amount of pure IWL, when trained on "bursty data," but the contribution of this strategy is negligible compared to the more dominant ICL and CIWL trading off through training.
C.2. Layer heads may be acting in superposition
In Section 4 identify that the asymptotic strategy of the network relies Layer heads acting skip-trigram-copiers and thus implementing CIWL strategy. However, we observe some curious features:
1. All heads appear be doing the same thing average (Figure 2
2. All heads are imperfect on average average-attentioon attention to the correct token never converges (instead plateauing around 0.8).
17

---

Strategy coopetition explains the emergence and transience of in-context learning
Loss
Loss
Loss
Steps Competition
Steps
Steps
Steps
C(a. b, d)
Mechanism
Mechanism
Figure 16. Simulations of the toy model over 12 sceds, using the same settings as those for Figure 3 dim(a) = dim(b) = dim(c)
20, dim(d) = 160, μr 0.1, 0.1. We additionally show the competition term in gray.
In this section, we lay out preliminary evidence for how these heads may be acting in superposition. Our hypothesis was inspired by Elhhage et al. (2022), ), who find that transformer MLP hidden layers may represent sparse features in superposition. The skip trigrams our network learning are sparse (each sequence only requires comparing 2), and numerous (12800 classes), while the network weights that can be used represent these are limited (eight 8- 8-dimensional heads).
First, we find that while heads have the same attention patterns average, their patterns given sequence can be quite different (Figure 15a). This would point to some form of distributed computation (which is hidden if only considering average patterns).
The most compelling evidence we find is by considering the interactions between different heads, and their imperfect average attention. If we consider a head in isolation (with all the other heads ablated), we can ask if the imperfection is "optimal." Specifically, we consider artificially decreasing the temperature of the softmax attention on this head To see its performance better worse. We see in Figure 15p that uniformly decreasing the temperature improves performance. which begs the question: Why doesn't 't the network learn to do this?
The answer comes from considering how heads interact. If we leave all heads active and then decrease the temperature of a head, we find that overall performance often goes down. Each head on its own gets better at solving the task, but the inter-head interactions can often make the network overall worse! These non-additive interactions hint at a tight coupling between heads, and the notion that heads are meaningfully exploiting their dynamic range attention (instead of the simplified, binary notion of "heads learn to attend to the correct token"). These results are stark contrast the additivity of heads found by (Singh et al. (2024), and suggest this is due form of superposition.
We conclude by noting that this suggestion is somewhat speculative, backed by preliminary evidence. We hope this heads-in-sung -nr-ii-ppupppttttonnn hypothesis, and preliminary evidence supporting it, can motivate further work on understanding individual attention head function.
Note when conducting these experiments, we artificially also only allow head attend label tokens. We know this is what they in practice, and we wanted to avoid high temperature performance from being artificially hurt by increased attention to non-label tokens.
"B performance here, mean performance the task if only this head active. Alternatively, this value he viewed readout the contribution of this head to the output..
18

---

Strategy coopetition explains the emergence and transience of in-context learning
E E
Loss
Loss
Steps
Steps
Steps
Steps
Competition
C(a. b, d)
Mechanism
Mechanism
Figure 17. Simulations of the toy model over seeds, using settings: dim(a) = dim(b) dim(cc :) 20, dim(d) = 160, #1
a 0.1. Without the asymptotic bias yu the faster Mechanism gets learned first and dominates. d - 0 due the competition term, which leads to increasing loss of Mechanism (green).
D. Additional details and seeds on toy model
All vectors in our toy model (Section 6) are initialized randomly with each element being drawn from (O, 1). Vectors evolve through gradient descent, with learning rate of 1. In practice, where normalize for the size of the tensors in each expression, to avoid needing to tune the learning rate. Mathematically, this rescaling is isomorphic to corresponding setting of μı, a and learning rate.
scaled Frobenius norm in the loss function,
In Figure T6 we show simulations on 12 random seeds (to supplement the single seed shown in Figure 5). While the exact timing in magnitudes of the curves can shift around, the overall dynamical profile (transience of Mechanism 1, the divot in Mechanism 2 formation) is largely preserved. For some seeds, we found that the vectors did exit the loss plateau (likely caused by saddle point in the loss landscape, c.f. (Singh al. 2024)) for the duration of our simulation.
In Figure 17 show simulations 12 random seeds with the same settings Figure 16 except that μı instead of 0.1. Eliminating the asymptotic bias for Mechanism 2 gets rid of transient behavior, with the faster mechanism (Mechanism 1) emerging and dominating. These intuitions motivated our experiments in Section
All code is open-sourced at htttps://iithub.com htt.://ww..c..aaaa.e.///en///aaa see if this toy model can model other types of dynamics (e.g., recent work from Zhang et al. (0055 mentions similar interactions their discussion).
We're especially excited to
E. Rejected hypotheses for the transience of emergent ICL
In the scientific process of figuring out what may cause ICL emergence, given that it's not asymptotically preferred, we considered (and subsequently rejected) many alternative hypotheses. In the spirit of sharing the path used to arrive at a conclusion, we share some of these hypotheses here.
F. Hypothesis 1: Earlier ICL is necessary for CIWL to emerge
Given the experiments the main paper, this hypothesis already seems a bit suspicious Figure 1 and Section 5.4 we show that CIWL can be learned without earlier ICL emergence. But what somchow bursty data requires ICL to emerge first order reach a CTWL solution?
19

---

Strategy coopetition explains the emergence and transience of in-context learning
1.0
chance
chance
Baseline No PT attn
Baseline
In-cantext accuracy
Ablate L1H1,4,7
Flip evaluator
0.6
0.4
0.00
0.25 0.50 0.75 1.00 1.25 1.50 1.75
2,00 0.00
0.25 0.50 0.75 1.00 1.25 sequences
1.50
1.75
2.00
sequences seen
1c7
1e7
Figure 18. Various experiments to reject altemnative hypotheses for why ICI. emerges, despite not being asymptotic. Recall that accuracy of the Flip evaluator means ICT. is dominant, and accuracy of means CIWT. is dominant. (a) ICI. is not necessary for CTWT. emergence. Darker line indicates evolution from training run where Layer heads were prevented from attending to the previous token, effectively disabling ICL. Lighter line is from standard training (same as Figure This clamp has relatively little effect on the dynamics of CIWL emergence. (b) ICL is likely not initialization artifact. Darker line indicates evolution from training run where Layer I heads 1, 4, and (which play a crucial role ICL in standard training) are ablated. While ICL emergence slightly delayed, still occurs, and in fact transience is bit slower as well.
We test (and reject) this hypothesis by considering experiment where we prevent Layer I heads from attending to the previous token, and then training on bursty data. This experiment is a causal intervention on dynamics (Singh et al. 2024). as it prevents ICL emergence but lets other strategies develop naturally. We show results in Figure T8h. Notably, we get similar curves to standard training, with the exception that ICL just doesn't 't emerge the hypothesis that transient ICL necessary to arrive asymptotic CIWL.
emerge (since it was blocked). Thus, we reject
E.2. Hypothesis 2: ICL emerges due to a lottery ticket
Another idea may be that ICL simply emerges since typical transformer initializations just happen to be "close" to an ICL solution-so rather than the dynamics leading the model through an ICL solution transiently, just starts near one and the transience we observe more of an "initialization effect." Though not quite the same, the notion of lottery tickets (Frankle & Carbin 2018 is what inspired this hypothesis.
Given our earlier experiments, this hypothesis does feel a bit suspicious as well-for example, we find that ICL transience is quite robust over initialization seeds (Appendix B.I. To offer more evidence against this hypothesis, we again consider a more causal intervention on dynamics (Singh et al. 2024):
ICL is transient due to initialization effects, this would imply that the circuits being used in ICL are already "largely present" the start of training. Through ablations on checkpoints from standard training, we identify that (for the main run we analyze throughout the text and presented in Figure [[) the Layer heads contributing previous-token behavior (Figure 14p to ICL are heads 1, 4. and 7. We establish this by first inspecting attention patterns, and then ablating these heads in checkpoints from around the peak performance of ICL (~3e6 to ~ 1 1.5e7). Such an ablation on these checkpoints climinates ICL behavior (as verified by performance on the ICL evaluator dropping to chance level).
We then re-train the model from scratch, with the same initialization, but with these Layer heads clamped off throughout training. Presumably, this would eliminate any galready-eissting -ew-sting----innttllzzaatteteete at-initialization" ICL circuits. Doing so minorly delays ICL emergence (Figure 18p), but does not prevent it. This indicates that their is strong dynamical pressure for ICL to emerge. regardless of initialization, providing evidence against this hypothesis.
20


Reviews:
[{"review_id": "fSahy6dyDl", "reviewer": "Reviewer_Bv3n", "summary": "This paper investigates why in-context learning (ICL), a capability that emerges in transformer models without explicit training, sometimes disappears after extended training periods. The authors study this phenomenon using a simplified experimental setup with 2-layer attention-only transformers trained on a synthetic few-shot learning task based on Omniglot handwritten characters.\nThe research demonstrates that after ICL disappears, the model does not simply revert to traditional in-weights learning (IWL). Instead, it adopts what the authors term \"context-constrained in-weights learning\" (CIWL) - a hybrid strategy that requires the correct label to be present in the context but does not need the full exemplar-label pairing that ICL uses. This CIWL strategy is implemented through skip-trigram mechanisms in Layer 2 of the network.\n\nMost notably, the paper uncovers that ICL and CIWL simultaneously compete and cooperate within the model architecture, a dynamic the authors call \"strategy coopetition.\" While the two strategies compete in Layer 1 (where heads switch from attending to previous tokens for ICL to self-attention patterns for CIWL), they share critical subcircuits in Layer 2. This sharing explains why ICL emerges at all, despite not being asymptotically preferred by the model. The authors further develop a minimal mathematical model that reproduces these key dynamics. Their model captures how competition drives ICL's eventual replacement by CIWL, while cooperation enables ICL's initial emergence. Using insights from this model, they identify data conditions where ICL becomes persistent rather than transient - specifically, when context exemplars exactly match query exemplars.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We really appreciate your acknowledgement of the strong and thorough evidential support for our main claims.\n\n## We respond here to the main criticisms:\n> The claim that ICL can be made persistent by matching context and query exemplars is supported by Figure 6, though the mechanistic explanation for why this works could be more developed.\n\nWe understand this desire, especially in the context of so many mechanistic explanations for other observed behaviors! However, given the large number of experiments and results, we believe it would be fair to leave this to future work.\n\n> The authors briefly discuss the possibility that Layer 2 heads act in \"superposition\"... manipulating head temperature... would solidify this intriguing hypothesis.\n\nThese results are in Appendix C.2, Figure 15b and c. The overall claim about superposition is meant to be preliminary (and is stated clearly as such). It is not critically related to the main narrative, hence its appearance in the appendix. We chose to include these mentions in the paper to hopefully spur future work in this intriguing direction.\n\n> (Section 7)... Explicit evidence from larger-scale empirical studies or tasks closer to practical applications would substantially reinforce this claim.\n\nFigure 6 shows that our findings scale to other commonly used setups in literature (the 12L transformers and data of Singh et al., 2023, which used the same data as the widely cited Chan et al., 2022). However, we were careful never to claim that such dynamics operate in LLMs – this would need to be left to future work.\n\nAs a wider point, there is by now a long tradition of this kind of work on smaller models and simplified setups, which has been well appreciated within the field, and which has indeed shown transfer to larger models and diverse scenarios. We believe it is important to value and undertake this type of work, because it allows us as a field to discover insights which can then be applied and tested in other setups in future work.\n\n## There are also a few points in the remaining review that indicate that the reviewer may have overlooked some of our analyses:\n\n> Some aspects that could affect the validity of the analyses include reliance on averaged attention patterns that might mask individual variations in head behavior,\n\nWe plot individual attention patterns in Figure 15a, in the preliminary evidence for superposition\n\n> The authors should verify Figure 4c, as the directional arrows for \"ICL\" and \"CIWL\" appear to be reversed compared to what's described in the text.\n\nAs explained on Lines 160-163 when introducing the Flip evaluator, these lines are correct (and simply meant to remind the reader of this indication).\n\n## Some additional responses:\n\n> In Figure 14, there are references to undefined appendices that should be clarified or removed.\n\nWe thank the reviewer for pointing this out (it was simply old text we forgot to edit) and have made the corresponding change.\n\n> The preliminary results in Figure 6 suggest some generalization, but what additional evidence or theoretical arguments support the claim that similar dynamics operate in state-of-the-art models?\n\nWe were careful never to claim that such dynamics operate in LLMs, though we do show extension to 12L transformers and common setups from literature (Singh et al., 2023). Our work is meant to build intuitions using rigorous analysis in smaller settings that may inspire work on larger models. For example, https://arxiv.org/abs/2502.14010 points to similar dynamics in larger models (with less mechanistic rigor on dynamics, given the difficulty of such experiments when using LLMs).\n\n> Could you clarify whether you see CIWL as fundamentally the same phenomenon as task recognition, or whether there are important distinctions?\n\nWe see them as closely related intuitively, with further mechanistic work in LLMs needed to establish equivalence. Generally, we are wary of overclaiming.\n\n> Have you conducted ablation studies or circuit analyses to understand why this modification equalizes the asymptotic preference between ICL and CIWL?\n\nWe believe the \"what\" has to precede the \"why\" – our paper mostly focused on the \"why\" of ICL transience, building intuitions that led us to a setting where ICL is persistent (a new \"what\"). We believe rigorous investigation of mechanistic explanations here (beyond the intuitions provided in the paper) is beyond the scope of our work, but would be excited for future work to tackle it.\n\n> how sensitive is this reproduction to parameter settings?\n\nWe found the model quite robust to parameter settings, with the caveats we mention in the paper (and repeat here for clarity): When mu1=0, the faster strategy would be persistent. When alpha=0, there would be no transience since there's no competition.", "claims_and_evidence": "The submission presents several key claims generally well-supported by evidence, but with some major limitations.\n\nThe transience of in-context learning (ICL) is convincingly demonstrated in Figure 1b, reproducing previous findings from Singh et al. (2023) and others. The characterization of the asymptotic \"context-constrained in-weights learning\" (CIWL) strategy is well-supported by:\n\n- Behavioral evidence through specialized evaluators (Figure 1b)\n- Mechanistic evidence of skip-trigram-copiers in Layer 2 (Figure 2)\n- Ablation studies showing minimal pure in-weights learning (Section C.1)\n\nThe core \"strategy coopetition\" claim is substantiated through experimental interventions such as:\n\n- Figure 3b shows that fixing Layer 2 weights to end-of-training values preserves the overall behavioral trajectory, indicating Layer 2 circuits are shared between strategies\n- Figure 3c demonstrates that fixing Layer 1 weights locks in behavior, suggesting competition happens primarily in Layer 1\n- Figure 4a-c provides compelling evidence that CIWL enables ICL emergence despite eventually replacing it\n\nThe mathematical model in Section 6 reproduces key behavioral patterns observed in the transformers, including the unexpected \"dip\" in CIWL formation. This strengthens the theoretical understanding of the observed dynamics.\n\n## Problematic claims:\n1. The claim that ICL can be made persistent by matching context and query exemplars is supported by Figure 6, though the mechanistic explanation for why this works could be more developed.\n\n2.  The authors briefly discuss the possibility that Layer 2 heads act in \"superposition\"—that multiple sparse features might share attention heads simultaneously (Appendix C.2). This claim is intriguing and potentially important, yet the current evidence is labeled by the authors as \"preliminary.\" While their exploratory analyses do hint at complex non-additive interactions among heads, this aspect lacks conclusive experimental support. Further investigations—for example, systematically manipulating head temperature or conducting head-specific targeted ablations—would solidify this intriguing hypothesis.\n\nThis is concerning  because superposition could significantly impact the authors' interpretation of the mechanisms underlying CIWL, further robust experiments would substantially strengthen this claim. Without more conclusive evidence, this point remains somewhat speculative.\n\n3. The paper claims that its results and explanations have potential implications for larger transformer models and realistic training scenarios (Section 7). However, the presented evidence is mostly limited to synthetic tasks using small-scale, simplified transformer setups. While the authors briefly show preliminary evidence from larger transformer models trained on similar synthetic tasks, they stop short of providing evidence on realistically scaled language modeling tasks or non-synthetic datasets.\n\nWhile the mechanisms discovered might generalize conceptually, the direct relevance to state-of-the-art transformers trained on natural datasets remains unclear. Explicit evidence from larger-scale empirical studies or tasks closer to practical applications would substantially reinforce this claim.\n\n4. The paper proposes that ICL emerges transiently because it is \"close to the path\" toward the asymptotic CIWL solution. Although the experiments showing reuse of Layer 2 heads strongly support shared mechanisms between ICL and CIWL, the notion of “closeness” to the path is presented somewhat informally. The authors clearly show that certain intermediate CIWL-only checkpoints allow rapid ICL emergence, but a more explicit or quantitative measure of \"closeness\" in model space or loss landscape would strengthen this point.\n\nWithout a clearly defined notion of \"closeness\" or detailed visualization of training trajectories (for example, using linear interpolation or functional similarity metrics), the explanation is somewhat abstract. Explicit analysis or visualization of model parameters or activations as training progresses would improve clarity and reinforce this important conceptual claim.", "methods_and_evaluation_criteria": "The methods and evaluation criteria employed in this paper are well-suited to investigate the transience of in-context learning in transformers. The authors use a simplified architecture (2-layer attention-only transformers) which is methodologically sound for mechanistic studies. This choice follows established practices in transformer interpretability research and allows for clearer attribution of roles to specific components. While this simplification limits generalizability, the authors address this by demonstrating that their key findings extend to larger models (Figure 6, right panel), striking an appropriate balance between interpretability and relevance.\nThe synthetic few-shot learning task based on Omniglot characters provides a controlled environment where multiple learning strategies are viable, making it an excellent testbed for studying strategy competition. The bursty data design intentionally permits both in-context and in-weights learning, which is crucial for their research questions.\n\nThe specialized evaluators (ICL, IWL, CIWL, and Flip) are well-designed for this study. They enable precise measurement of distinct strategies through behavioral signatures:\n\n- The ICL evaluator isolates pure in-context learning by invalidating weight-based exemplar-label mappings\n- The CIWL evaluator tests for a specific hybrid strategy requiring context constraints but not full exemplar-label pairing\n- The Flip evaluator quantifies the relative dominance between strategies\n\nThese behavioral measures are complemented by mechanistic analyses that strengthen the evidence:\n\n- Attention pattern analyses that reveal the underlying computational mechanisms\n- Causal ablation studies that establish the functional roles of specific components\n- Layer-fixing experiments that isolate the contributions of different model parts\n\nThe mathematical model serves as both a theoretical framework and an additional evaluation criterion. By reproducing key behavioral patterns observed in the transformer experiments, it validates the proposed explanation and generates testable predictions.", "theoretical_claims": "This paper does not present formal mathematical proofs that require verification.", "experimental_designs_or_analyses": "I've examined the experimental designs and analyses in this paper and find them generally sound and well-executed. The authors' specialized evaluators (ICL, IWL, CIWL, and Flip) create controlled conditions that effectively isolate specific strategies, allowing clear attribution of model behavior to different learning mechanisms. The Flip evaluator is particularly innovative as it quantifies the relative dominance between strategies rather than just measuring their presence.\n\nThe mechanistic analyses provide convincing evidence for the paper's claims. The attention pattern analyses effectively demonstrate the skip-trigram copying mechanisms underlying CIWL, and the authors establish causal relationships through interventions rather than relying solely on correlational evidence. For instance, the attention clamping experiments establish that Layer 2 heads functionally copy label tokens, while the layer-fixing experiments convincingly demonstrate that Layer 2 remains largely static after initial formation while Layer 1 drives strategy changes.\n\nThe strategy-specific training experiments offer strong evidence for the coopetition hypothesis. Training on ICL-only data shows difficulty learning, but using CIWL-trained Layer 2 weights enables ICL learning—a key finding that supports the authors' central claim about strategy cooperation. Similarly, the CIWL-only training followed by bursty data effectively demonstrates that ICL emergence depends on CIWL not being fully formed. The paper demonstrates robustness through replication across multiple random seeds, different architectural variants, and various data configurations. \n\nSome aspects that could affect the validity of the analyses include reliance on averaged attention patterns that might mask individual variations in head behavior, the simplified nature of the toy mathematical model compared to actual transformer dynamics, and questions about generalizability from a 2-layer attention-only transformer to more complex architectures. Despite these minor limitations, the experimental designs and analyses provide good support for the paper's claims about strategy coopetition in transformer learning dynamics.", "relation_to_broader_scientific_literature": "- The identification of \"context-constrained in-weights learning\" (CIWL) as the asymptotic strategy that replaces ICL builds directly upon Singh et al.'s (2023) discovery of ICL transience, providing a mechanistic explanation for what happens after ICL disappears. CIWL also relates conceptually to Lin and Lee's (2024) work distinguishing between \"task recognition\" and \"task learning\" modes of in-context learning. What the authors identify as CIWL shares similarities with the task recognition paradigm, where context serves more to identify what knowledge to retrieve from weights rather than providing new pattern-completion information.\n\n- The paper's core contribution—the \"strategy coopetition\" framework—extends several research threads. It complements Nguyen and Reddy's (2024) and Park et al.'s (2024) work on competition between strategies, but adds the crucial insight about cooperative interactions between seemingly competitive mechanisms. This reframes prior findings from Chan et al. (2022) regarding how data properties modulate the emergence of different strategies, suggesting that bursty data promotes cooperation between strategies before asymptotic competition takes over.\n\n- The mechanistic analysis showing shared subcircuits between strategies represents a significant advance over previous work. While Olsson et al. (2022) characterized induction heads and their role in ICL, and Elhage et al. (2021) described skip-trigram mechanisms, this paper shows how these circuit motifs can be repurposed between different computational strategies. This relates to Elhage et al.'s (2022) work on superposition in transformers, suggesting that limited model capacity leads to shared computational resources between different capabilities.\n\n- The toy mathematical model extends Singh et al.'s (2024) minimal model of phase changes in transformer learning, incorporating both competitive and cooperative dynamics. It also connects with Saxe et al.'s (2022) theoretical work on the \"neural race reduction\" and dynamics of abstraction in neural networks, providing further evidence that different learning dynamics can coexist and interact in complex ways during training.\n\n- Finally, the identification of data conditions leading to persistent ICL builds upon Chan et al.'s (2022) investigations of how data properties affect strategy adoption. It also connects to Lampinen et al.'s (2024) recent work on the \"broader spectrum of in-context learning,\" suggesting that different manifestations of context-sensitivity may exist along a continuum rather than as discrete capabilities.", "essential_references_not_discussed": "Based on my review, the paper generally provides a thorough discussion of relevant literature.", "other_strengths_and_weaknesses": "**Originality**\n\nThe paper demonstrates novelty in introducing the concept of \"strategy coopetition\" to explain a previously observed but poorly understood phenomenon of transient in-context learning. By identifying context-constrained in-weights learning (CIWL) as a distinct hybrid strategy and characterizing its mechanistic implementation, the authors provide novel insights beyond what was previously known about ICL transience. The discovery that competing strategies share subcircuits in Layer 2 while competing in Layer 1 represents a conceptual breakthrough in understanding transformer learning dynamics. However, the paper's originality is somewhat constrained by its foundation on existing work on ICL transience and the relatively straightforward extension of previous mathematical models. \n\n**Significance**\n\nThis work makes a significant contribution by providing a mechanistic understanding of how transformers transition between learning strategies during training. By explaining why ICL emerges despite not being asymptotically preferred, the authors address a fundamental question about capability emergence in modern AI systems. The finding that ICL can be made persistent through specific data modifications has potential implications for training methodologies, especially if these dynamics extend to larger models as preliminary results suggest. The concept of strategy coopetition could influence how researchers think about capability emergence and circuit formation in neural networks more broadly. However, the significance is somewhat limited by the focus on simplified models and synthetic tasks, raising questions about generalizability to real-world language modeling scenarios. While the authors demonstrate some extension to larger models, more comprehensive validation across diverse architectures would strengthen the work's broader impact.\n\n**Clarity**\n\nThe paper presents complex findings with great clarity through well-structured progression from phenomena reproduction to mechanistic understanding to mathematical modeling. The authors effectively use specialized evaluators to isolate and measure different strategies, providing clear operational definitions that facilitate understanding. The visualizations of attention patterns and strategy dynamics effectively communicate key insights, while causal intervention experiments clearly demonstrate functional roles of model components. However, the paper contains dense technical content that assumes substantial familiarity with transformer architecture and mechanistic interpretability techniques. The multiple interrelated experiments and detailed analyses could be challenging for readers to track without careful study. Some important methodological details are relegated to appendices, and the rejection of alternative hypotheses section, while valuable, could be better integrated into the main narrative to strengthen the central claims.", "other_comments_or_suggestions": "- The authors should verify Figure 4c, as the directional arrows for \"ICL\" and \"CIWL\" appear to be reversed compared to what's described in the text. \n- In Figure 14, there are references to undefined appendices that should be clarified or removed.\n- The mathematical notation in Section 6 could be more consistently aligned with the mechanism descriptions in earlier sections to help readers make connections between the empirical findings and theoretical model.\n- Finally, some figures (particularly Figures 14 and 15) contain dense information that could be simplified or restructured for clarity.", "questions_for_authors": "1. Your demonstration of strategy coopetition is compelling in 2-layer attention-only transformers, but how confident are you that this mechanism explains ICL transience in larger, more complex models? The preliminary results in Figure 6 suggest some generalization, but what additional evidence or theoretical arguments support the claim that similar dynamics operate in state-of-the-art models?\n\n2. The CIWL strategy you identify bears conceptual similarities to what some researchers call \"task recognition\" (as opposed to \"task learning\"). Could you clarify whether you see CIWL as fundamentally the same phenomenon as task recognition, or whether there are important distinctions?\n\n3. Your explanation for ICL persistence when context exemplars match query exemplars is intriguing but somewhat underexplored mechanistically. Have you conducted ablation studies or circuit analyses to understand why this modification equalizes the asymptotic preference between ICL and CIWL?\n\n4. Your toy mathematical model reproduces key dynamics observed in transformers, but how sensitive is this reproduction to parameter settings? Is there a range of parameters for which the model fails to exhibit transience, and what would this tell us about conditions where ICL might naturally persist?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5vcYnbIEY", "reviewer": "Reviewer_7egA", "summary": "Architecture: 2-layer attention-only transformer (appendix has other models)\n\nDataset: Omniglot, augmented to 12k+ classes. The majority of the classes are used for training, but the remaining 184 classes are used for testing.\n\nTraining sequences are set up with a few shot learning favor and constructed using the exemplars from the training classes. Specifically, each training sequence consists of a context (which contains two exemplar-label pairs) and a query. Training sequences follow the “bursty” structure, requiring that at least one of the exemplars in the context always belongs to the same class as the query.\n\nTesting sequences are constructed using the exemplars from the test classes and they are crafted to evaluate various strategies including ICL, in weight learning (IWL), a new strategy CIWL, and a balance between ICL and CIWL named FLIP.\n\nThe main finding is that in the beginning of training, ICL and CIWL cooperate and ICL can emerge. At some point ICL disappears and CIWL comes to the forefront.\n\n## update after rebuttal\nI maintain my score and positive impression of the paper.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are happy they think our work is \"an excellent example of rigorous deep learning science.\" We've also noted and updated the paper based on the suggestions, with responses to specific questions below:\n\n> How do the authors expect the findings on ICL persistence to generalize beyond Omniglot? Would similar results hold for more complex datasets, such as natural language or vision tasks?\n\nGiven that recent works have found generalization of insights from the Omniglot setting to more naturalistic settings (e.g., language model token embeddings of Singh et al., 2023, and RL setups of Raparthy et al., 2023), we are cautiously optimistic w.r.t generality. For example, recent works on LLMs (https://arxiv.org/abs/2502.14010), point to related phenomena.\n\n> In the experiments with deeper networks and MLP layers (Appendix B), did the authors observe any systematic trends in how model depth or capacity influences the timescale of ICL transience? Specifically, does increasing depth delay the transition from ICL to CIWL, or does it accelerate it? If such trends exist, could they be described in terms of scaling laws similar to those seen in other deep learning phenomena?\n\nWe didn't study the timescales as a function of architectural choices, as these were already discussed at scale by Singh et al., 2023 (from reading their paper, the analyses do seem motivated by scaling laws work, though not quantified as exactly, as the reviewer suggests). We did have some earlier experiments related to Layer 1 capacity (e.g., if training with fewer heads active) that showed fewer active heads lead to delayed transience (up to a point – fewer than 3 heads blocked ICL since the network effectively becomes a 1L model). We believe bridging the gap between the rigorous mechanistic understanding of our work, and scaling laws on larger models is an exciting direction for future research.", "claims_and_evidence": "Yes, the setup is clear and well-justified. The training data design ensures that both ICL and CIWL have an opportunity to emerge, while the test evaluations (e.g., the FLIP test) directly measure whether ICL remains dominant or fades. We see strong empirical evidence for the claim that ICL transience is linked to “coopetition” with CIWL.", "methods_and_evaluation_criteria": "The main method employed here is really setting up a minimal environment where we can observe “coopetition”. There’s also a portion of the paper dedicated to performing mechanistic interpretability. Here it is discovered that ICL and CIWL share sub-circuits.", "theoretical_claims": "The paper does not make traditional theoretical claims in the form of formal theorems.", "experimental_designs_or_analyses": "Yes, the experimental design is well crafted. The evaluation framework is particularly strong, as it systematically isolates different strategies through controlled test sequences. For example, the FLIP evaluator directly measures whether the model relies on ICL or shifts toward CIWL, providing a clear signal of strategy transition.", "relation_to_broader_scientific_literature": "Recent work has shown ICL to be a transient phenomenon in that it can disappear after long training time. This paper shows that cooperation with CIWL enables the emergence of ICL in the first place, while competition leads to its eventual disappearance and replacement by the CIWL strategy. To the best of my knowledge, the identification of the CIWL strategy is novel.", "essential_references_not_discussed": "Not that I’m aware of.", "other_strengths_and_weaknesses": "This is an excellent example of rigorous deep learning science. The work is compelling and should have broad appeal. A key strength is that it reveals a previously unknown strategy—context-constrained in-weights learning (CIWL)—and its relationship with ICL. The paper exemplifies how well-designed experiments and mechanistic interpretability can lead to novel insights in deep learning.", "other_comments_or_suggestions": "* The term asymptotic is used very frequently throughout the paper. Asymptotic in what is never really spelled out.   \n* The caption to Figure 1 is very dense. I found it a bit daunting to read in its position. I returned to it after reading far beyond it.  \n* The x-axis in Figure 1b represents the number of training sequences seen, which implicitly corresponds to training time. Since emergence and disappearance are often framed in terms of training time, making this connection clearer in the text might help readers.  \n* \\\\citep should be used in Line 28 and 29 for Olsson and Singh citations  \n* I’m not sure the footnote on Hollywood studios is a good use of real estate. It’s enough to know that coopetition is a term from game theory and not a gratuitously silly word made up for this paper.", "questions_for_authors": "1. How do the authors expect the findings on ICL persistence to generalize beyond Omniglot? Would similar results hold for more complex datasets, such as natural language or vision tasks?  \n2. In the experiments with deeper networks and MLP layers (Appendix B), did the authors observe any systematic trends in how model depth or capacity influences the timescale of ICL transience? Specifically, does increasing depth delay the transition from ICL to CIWL, or does it accelerate it? If such trends exist, could they be described in terms of scaling laws similar to those seen in other deep learning phenomena?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VgBtpEbsS3", "reviewer": "Reviewer_u2HU", "summary": "This paper systematicallly studies transient dynamics of in-context learning (ICL) in transformers. In particular, the authors identify that after ICL disappears, a hybrid strategy between in-weights and in-context learning called \"context-constrained in-weights learning\" (CIWL) emerges, which competes with and eventually replaces ICL. Despite this competition, the two strategies share sub-circuits, leading to cooperative dynamics. The paper also proposes a minimal mathematical model to explain these interactions and highlights a setup where ICL remains persistent after long training times.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and are glad they found our work \"particularly compelling\". We've factored in their suggestions and respond to their question below:\n\nThe mathematical model presented was largely just our first attempt at extending the \"single mechanism\" model from Singh et al. (2024) – that's where the tensor product of 3 vectors came from. Our intuition was that, since ICL and CIWL can both \"solve the task\", we could use the product of the losses as an \"OR\" operation (a cool symmetry to how the tensor product within a mechanism's loss represents an \"AND\", as per Singh et al. (2024)). The offset mu1 was added as our results (and the original findings of Singh et al. (2023)) indicated CIWL was always asymptotic. The competition term is between $\\mathbf{a}$ and $\\mathbf{d}$ since Singh et al. (2024) use $\\mathbf{a}$ to correspond to Layer 1.\n\nFurther investigation of the toy model would be an interesting avenue for future work – we mostly introduced it to crystallize intuitions and since we were also (pleasantly) surprised by how it captured more nuanced dynamical features (the \"divot\"). Specifically, we think connecting it to the Neural Race Reduction ideas of Saxe et al. (2022) could be particularly noteworthy.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem.", "theoretical_claims": "I didn't check the proofs closely.", "experimental_designs_or_analyses": "The experimental designs and analyses seem sound/valid to me.", "relation_to_broader_scientific_literature": "The paper built upon prior work such as Singh et al.'23 and Reddy'24, which shows a few new findings related to the transcience of ICL, such as the surprising CIWL and its asymptotic dominance, and its cooperation with ICL.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "**Strength**\n\n1. The exploration of the transience of ICL is fascinating, and the nuanced findings regarding the existence of CIWL, its interaction with ICL, and its asymptotic dominance provide novel insights into the internal mechanisms of attention-based models and their emergent behaviors. These findings are particularly compelling to me.\n\n2. In addition to the rich results, I appreciate that the authors have developed a simple mathematical model that (1) replicates real-world phenomena and (2) demonstrates some predictive power.\n\n**Weakness** \n\n1. While I appreciate the attempt to develop a mathematical model, I find it somewhat difficult to grasp, particularly in terms of practical interpretation. For instance, the meaning of $\\mu_1=0$ is not entirely clear to me—specifically, how it can relate to a characteristic of the dataset (lines 386-392).", "other_comments_or_suggestions": "1. Some in-text citations should be done with \\citep instead of \\cite: line 28-29.\n\n2. The term \"attention delta\" is used several times in the paper. I can get its meaning from context, I think it would be better to explicitly define the term to avoid confusion.", "questions_for_authors": "1. I’m a bit confused about how the mathematical model in Section 6 was developed, as it appears to come up rather abruptly. I understand that there is a pattern match between the model and the ICL experiment described in lines 379-351, but I’m curious about how the specific form of the objective was chosen initially. For instance, why was the tensor product chosen as a component? Why tensor product with three vectors?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates a phenomenon that has been observed with in transformer models in which in-context learning (the phenomenon by which transformers learn from context without needing to update the weights of the model) can disappear after long periods of training. The authors study this problem  by studying 2-layer attention-only transformers.

Reviewers universally had a positive opinion of this paper and engaged with the authors to clarify certain points that could improve the final version. Nevertheless this paper is a clear accept.
Decision: Accept (oral)