An evolutionary perspective on modes of learning in Transformers
Authors: Alexander Ku
Thomas L. Griffiths
Stephanie C.Y. Chan
Venue: ICLR 2026 Poster (2026)
Keywords: transformers
in-context learning
in-weights learning
learning dynamics
evolutionary biology
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Evolutionary biology provides insight into the factors that govern whether Transformers learn in-context or in-weights.

Abstract: The success of Transformers lies in their ability to improve inference through two complementary strategies: the permanent refinement of model parameters via _in-weight learning_ (IWL), and the ephemeral modulation of inferences via _in-context learning_ (ICL), which leverages contextual information maintained in the model's activations.
Evolutionary biology tells us that the predictability of the environment across timescales predicts the extent to which analogous strategies should be preferred. Genetic _evolution_ adapts to stable environmental features by gradually modifying the genotype over generations. Conversely, environmental volatility favors _plasticity_, which enables a single genotype to express different traits within a lifetime, provided there are reliable cues to guide the adaptation.
We operationalize these dimensions (environmental stability and cue reliability) in controlled task settings (sinusoid regression and Omniglot classification) to characterize their influence on learning in Transformers.
We find that stable environments favor IWL, often exhibiting a sharp transition when conditions are static. Conversely, reliable cues favor ICL, particularly when the environment is volatile.
Furthermore, an analysis of learning dynamics reveals task-dependent transitions between strategies (ICL $\to$ IWL and vice versa). We demonstrate that these transitions are governed by (1) the asymptotic optimality of the strategy with respect to the environment, and (2) the optimization cost of acquiring that strategy, which depends on the task structure and the learner's inductive bias.

Published as a conference paper at ICLR 2026
AN EVOLUTIONARY PERSPECTIVE ON MODES OF LEARNING IN TRANSFORMERS
Alexander Y. Ku Thomas L. Griffiths
Griffithss,Stephnnt Stephanie C.Y. Chan'
GGoogle DeepMind
2 Department of Psychology, Princeton University
(alexku, ecychan)&google com, tongéprinceton.edd
ABSTRACT
The success of Transformers lies in their ability to improve inference through two complementary strategies: the permanent refinement of model parameters via in-weight learning (IWL), and the ephemeral modulation of inferences via in- context learning (ICL), which leverages contextual information maintained in the model's activations. Evolutionary biology tells us that the predictability of the environment across timescales predicts the extent to which analogous strategies should be preferred. Genetic evolution adapts to stable environmental features by gradually modifying the genotype over generations. Conversely, environmental volatility favors plasticity, which enables a single genotype to express different traits within lifetime, provided there are reliable cues to guide the adaptation. We operationalize these dimensions (environmental stability and cue reliability) in controlled task settings (sinusoid regression and Omniglot classification) to char- acterize their influence on learning in Transformers. We find that stable environ- ments favor IWL, often exhibiting sharp transition when conditions are static. Conversely, reliable cues favor ICL. particularly when the environment is volatile. Furthermore, analysis of learning dynamics reveals task-dependent transitions between strategies (ICL- -> IWL and vice versa). We demonstrate that these tran- sitions are governed by (I) the asymptotic optimality of the strategy with respect to the environment, and (2) the optimization cost of acquiring that strategy, which depends on the task structure and the learner's inductive bias.
INTRODUCTION
Transformers Vaswani et al. 2017) underpin the success of modern large language models (LLMs), demonstrating impressive performance across a diverse set of tasks (Brown et al., 2020). key emergent capability of these models is in-context learning (ICL), which allows them to perform novel tasks specified by examples in the input prompt, without updating model parameters (Brown al. 2020 Dong al. 2022 Liu et al. 2023). This stands in sharp contrast standard in-weights learning (IWL). where knowledge is gradually encoded into the model parameters during training. Research into the nature ICL extensive, ranging from mechanistic accounts including induction heads (Olsson et al., 2022) and function vectors (Todd et al. 2023), to theoretical frameworks that interpret ICL as a form of implicit optimization or Bayesian inference /Von Oswald et al.) 2023 Dai
et al. 2022 Xie et al. 2021). Most relevant to this work, however, is the finding that the emergence of ICL relies on specific statistical properties the training data (Chan et al. 2022).
While the emergence of ICL can be productively understood in terms of rational adaptation to en- vironmental pressures (Wurgaft et al., 2205).. this perspective has limitations. By focusing on what the asymptotically optimal strategy is (i.c., ICL or IWL), it struggles to account for the learning dynamics, particularly the factors that precipitate transitions between strategies. This gap is signif- icant because empirical studies show that ICL can be transient, emerging early in training only to diminish or be superseded by IWL as the model converges (Singh et al. 2023).
In this paper, argue that evolutionary theory offers powerful explanatory lens for understanding these learning dynamics. Specifically, we examine the environmental factors that drive transitions between analogous biological strategies: plasticity and evolution. Plasticity is the capacity of

---

Published as a conference paper at ICLR 2026
single genotype to produce different phenotypes (e.g., morphological behavioral changes) in response to environmental cues. It allows for adaptation within an individual's lifetime, minroring how ICL modulates inference based on a prompt. Conversely, evolution involves the slow modifica- tion of the genotype across generations via selection to produce specific adaptive traits, analogous to updating model parameters via IWI., Furthermore, just evolution gives rise plasticity, the capacity for ICL is acquired through the process of IWL.
Evolutionary theory suggests that the reliability of information across timescales determines which adaptive strategy is favored (Stearns) 1989). Plasticity is generally selected for in fluctuating en- vironments, provided reliable cues are available Stephens, 1989). However, plasticity tends to be selected against when the environment is stable enough to obviate the need for flexibility, when reliable cues are unavailable, or when the inherent costs of plasticity are prohibitive (Stearns 1989, Stephens 1989 Wett et al. 1998). Furthermore, genetic assimilation describes the evolutionary process by which plastic traits become fixed, even- tually rendering their expression insensitive to the cues that originally produced them (Waddington 1953 Pigliucci et al. 2006). This phenomenon parallels the empirically observed transience of ICL, where the model's 's reliance the prompt diminishes the task is consolidated into weights (Singh
guide the adaptive response (Stearns) 1989
ct al. 2023).
We hypothesize that analogous factors govern the competition between ICL and IWL in Transform- ers, determining which strategy dominates at different stages of training. To test this, we operational- ize two key dimensions: prompt to specify the target task, and environmental stability, defined as the invariance of the target task across different prompts encountered during training. We manipulate these dimensions in two controlled settings: parametric sinusoid regression and few-shot Omniglot classification. Our results confirm that the evolutionary principle regarding environmental predictability across timescales ac- curately predicts the learning dynamics of Transformers. This suggests that an ecological view of training environments can offer principled guidance for developing new training methodologies.
reliability, defined as the sufficiency of information within single
2 ENVIRONMENTAL PREDICTABILITY
Our general approach relies on drawing correspondence between aspects of biological adaptation and the learning dynamics observed in Transformers. Fundamentally, we view both systems as optimizing for environmental predictability: the strategy selected depends on the timescale at which information is most reliable. make these parallels concrete, we consider two examples that illustrate the specific phenomena investigate.
Plasticity. The classic example of plasticity is the defense adaptation of the water flea, Daphnia Tollrian & Harvell 1999). When Daphnia detect chemical cues (kairomones) released by predators, they develop a protective helmet and tail spine. In the absence of these cues, individuals with the exact same genotype develop standard, non-defensive morphology conserve energy. Just as the Daphnia phenotype is determined by the interaction of a fixed genotype with an environmental cue, Transformer's output is determined by the interaction of fixed weights with the context provided in the prompt. In both cases, the system adapts its behavior to the current environment without modifying its underlying genotype or weights.
Genetic assimilation. The phenomenon where ICL emerges early in training but is later superseded by IWL parallels genetic assimilation, a concept famously demonstrated by Waddington's 's selection experiments on Drosophila [Waddington] 1953). In these experiments, Waddington exposed fruit fly pupae to heat shock (an environmental cue). which caused fraction of them to develop a cross- veinless wing structure (a plastic response). By selectively breeding only those individuals that exhibited the trait, he eventually produced a lineage where the veisseveineess trait appeared without the heat shock. Similarly, Transformer initially reliant on prompts (ICL) will, given stable training conditions, consolidate the capability into its weights (IWL), rendering the inference insensitive to the contextual cues that originally guided it (Singh al. 2023)
At its core, these two phenomena hinge on distinct aspects of environmental predictability: the reliability of cues guide adaptation within lifetime, and the stability of the environment to fa- cilitate evolution across generations. Both dimensions ultimately derive from the predictive power of information at different timescales specifically, how accurately experience (whether gathered
2

---

Published as a conference paper at ICLR 2026
within single lifetime accumulated across generations) anticipates optimal behavior. If this cor- respondence between biological adaptation and learning in Transformers holds, then systematically manipulating predictability different timescales (e.g., via noise) should allow us to determine the preferred learning strategy in Transformers. The following section details how we operationalize this experimentally in controlled training environments.
3 EXPERIMENTAL SETUP
ICL is widely viewed as emergent form of meta-learning. Accordingly, we use two standard benchmarks from the meta-learning literature: parametric sinusoid regression (Finn et al. 2017) and few-shot Omniglot classification (Lake et al. (015). These environments allow to systematically manipulate our proposed dimensions of environmental predictability (cue reliability and environ- mental stability) across both continuous and discrete domains.
3.1 SINUSOID REGRESSION
In the sinusoid regression task (Finn al.) 2017). the objective is to predict the output of target function f((x) = A, sin(x + 0i) (see Figure Each training episode consists of a prompt of N 10 examples, {(r4 yd J==1 followed by query input 2i for which the model must predict the ground truth = fe(T)). The inputs v are sampled uniformly from n, m].
stability reliability
stabilty 1 reliability
stability reliability
stabiiity reliability
Figure Sinusoid regression under varying predictability conditions. Each panel shows prompt examples (black dots) for given target function timestep i (solid line), and the function at the preceding timestep 1 (dashed line). Top row depicts volatile environments, meaning the target function changes significantly at each step. Bottom row depicts stable environments, where the tar- get function more stable across time. Left column depicts unreliable cues, where prompt examples noisy. Right column depicts reliable cues, where prompt examples are less noisy and closer to the target function.
Cue reliability. We manipulate this by adjusting the variance, a², of the Gaussian observation noise *N o²) added the prompt targets y;- A lower x² corresponds to higher cue reliability, as the prompt examples more faithfully reflect the true underlying function ft.
Environmental stability. We manipulate this by controlling the parameter a in an AR(1) process that governs the evolution of the task parameters 6, = [A,, oz across training steps t. The parame- ter a [0, 1] dictates the correlation between successive tasks, where a indicates high stability. The update rule is given by:
0: a0,-1 1 (1 a)0.
3

---

Published as a conference paper at ICLR 2026
where de represents random innovations drawn from the base task priors: amplitude A ~ 4/0.5. 1.3] and phase ~ 20 2 This formulation ensures mean-reversion towards the prior distribution, allowing us vary stability without drifting outside the solvable task distribution.
3.2 OMNIGLOT CLASSIFICATION
We adapt the few-shot Omniglot benchmark (Lake et al. 2015) to construct a dynamic binary clas- sification task (see Figure 2 At each training step t, a global mapping M, hidden binary label to every character class in the Omniglot vocabulary C' ((C) = 1623). We define the target function (x( = (cx),, where cz is the character class of image x.
{0, assigns
Each episode consists of prompt with N = 2 image-label pairs v)); and query image Zg.. The query image belongs to character class Cq. and the model must predict its current true label Me(c,). Crucially, we construct the prompt such that one example matches the query's class (Cs- = Cy). This setup explicitly creates competition between strategies: the model can solve task via ICL (copying the label y from the matching prompt example) or via IWL (relying the mapping Mcg),) stored in its weights).
(rolia ble cue)
(unreliable cue) yo
yi -
-2
Y:4 -
y-
Y/ 0
2
2
Omniglot classification episodes. The prompt the left contains reliable cue (prompt label y1 for character c1 Cq matches Mc(cg)). The prompt on the right contains an unreliable cue (prompt label 1/2 for Co Cq is flipped relative to M,(c,)).
Figure 2:
Cue reliability. We operationalize reliability by controlling the probability, p. that prompt labels are accurate. For each example i in the prompt, the provided label yi corresponds to the true mapping Mc(c;) with probability p. and is flipped otherwise:
Mc(c)) yb = 1 - Me(c:)
with probability P with probability 1 1-
(2)
Environmental stability. We operationalize stability by controlling the persistence of the global mapping M, over time. The initial mapping Ma(e) is sampled from Bernoulli(O.5) distribution. For subsequent steps t > 0, each character's label persists with probability 7:
M.(c) M (c) Mc-((e)
with probability ?
(3)
with probability -
higher y indicates greater stability. This process induces a first-order temporal autocorrelation of = 2- 1 for each character's 's label trajectory.
3.3 MODEL AND TRAINING
All experiments use a decoder-only Transformer with 4 layers, 4 attention heads per layer, an em- bedding dimension of 128, and learned positional encodings. Input processing varies by task. For the sinusoid regression task, scalar inputs and outputs are linearly projected to the embedding dimen- sion. For Omniglot classification, character images processed by shallow ResNet with three stages, each with residual blocks, and outputting 16, 32, and 64 channels per stage, respectively. This ResNet is trained jointly with the Transformer.
Models are optimized to minimize either the mean squared error (MSE) for sinusoid regression or the binary cross-entropy (BCE) for Omniglot classification, calculated on their predictions for the query item (gg). We use the AdamW optimizer (Loshchilov & Hutter (017) with a peak learning rate x 10 subject to cosine decay schedule following 1,000 warmup steps. All models are trained for total of 50,000 training steps, using a batch size of 128.
4

---

Published as a conference paper at ICLR 2026
3.4 EVALUATION
To quantify the model's preference for ICL versus IWL, we use an evaluation protocol inspired by Chan al. (2022). This involves constructing evaluation prompts where the underlying target task explicitly conflicts with the current training environment.
Evaluation prompts. We generate these prompts by sampling an evaluation task, fe. from the prior distribution such that it is independent of the current training task, ft. We then construct prompt {T y4 )i) using labels derived from this evaluation task (yi = fe(())).
Target definitions. For given query Tig. this setup defines two conflicting prediction targets:
The ICL target, ViCl = fe(g)), corresponds to the label implied by the prompt: model predicting this value is relying on context (ICI.).
The IWL target, VIWL = f( corresponds to the label implied by the current training environment; model predicting this value disregards the prompt in favor of its weight- based encoding of the environment (IWL).
Quantifying strategy preference. We quantify the model's strategy preference by computing the prediction error relative to both targets (EICL and EIw1) using the task-appropriate metric (MSE for Sinusoid, BCE for Omniglot). We define the ICL preference score as:
EiWL
SICL = EICL + ElwL
(4)
An SiCL ss 1 indicates pure reliance on context, while SiCL ss indicates pure reliance on weights.
4 RESULTS
Our experiments reveal that the learning strategies adopted by Transformers systematically shaped by the predictability of their training environment, in ways that closely parallel adaptive mechanisms in evolutionary biology. We first examine how environmental stability and reliabil- ity governs the asymptotic preference for ICI. versus IWI. We then explore the learning dynamics between these strategies, revealing task-dependent patterns of transience (ICL + IWL and vice versa). Finally, we show that the cost of acquiring ICL versus IWL governs these dynamics. To- gether, these results reconcile diverse set of learning phenomena in Transformers with principles of biological adaptation.
4.1 DETERMINANTS OF ASYMPTOTIC STRATEGY
Figure shows how dimensions of environmental predictability influence the model's asymptotic strategy (l. = 50, 000), supporting our core hypothesis that information reliability across timescales governs the trade-off between ICI and IWI..
Stable environments favor ICL. In sinusoid regression (Figure 3 top). increasing environmental stability (t a) leads a precipitous decline in ICL preference. As - 1, the model shifts almost entirely to IWL. This recapitulates the biological principle of genetic assimilation: when environ- mental features are stable across generations, selection favors the permanent encoding of traits over incurring cost of plasticity (Stearns) 1989; Wett et al. 1998). Similarly, in the Omniglot task (Figure 3] bottom), ICL preference plummets as + 1. This reflects a transition where the global mapping becomes invariant, allowing the model to consolidate the task into its weights and render the prompt redundant.
Volatile environments with reliable cues favor ICL. Conversely, when environmental stability is low, the model relies heavily on ICL, provided that reliable cues are available. sinusoid regres- sion, reliable cues (L o², lighter lines) fosters strong ICL preference. This aligns with the evo- lutionary view that plasticity is favored specifically when environmental fluctuations render fixed traits maladaptive. but only if accurate cues exist to guide the plastic response (Stephens, 1989. Scheiner 1993). In Omniglot, this effect even more pronounced: across a broad range of instabil. ity (a: < 0.95), reliability is the primary determinant of strategy.
5

---

Published as a conference paper at ICLR 2026
Relabilly Ge*)
0.00
0.4
0.75
1.00
122
0.00
0.10
0.30
0.50 Stabilty 62)
0.70
0.80
0.90
0.95
1.00
Omniglct (t= 50, 000)
0.8
Hatabilky
0.5
0.5
1.0
6.0
Stabilly (2)
The asymptotic strategy converged models (f 50, 000). The ICL preference score (SIcl) is plotted as function of environmental stability (x-axis) and cue reliability (line brightness). The dashed line at SiG. = 0.5 marks the crossover point. sinusoid regression (top), stability parameterized by the AR(I) coefficient Cue reliability increases with decreasing noise variance o² (lighter lines). In Omniglot classification (bottom), stability is parameterized by the mapping persistence (logit scale). Cue reliability increases with label correctness probability p (lighter lines).
Figure 3:
When strategies are equally effective. Interestingly, observe deviation from biology in the Omniglot task. With perfectly reliable cues (p = 1), the model maintains a strong preference for ICL even when the environment is perfectly stable (Figure bottom right). Evolutionary theory suggests that plasticity selected against in stable environments due to maintenance costs WDeWitt) et al. 1998). The persistence of ICL here implies that, for Transformers, the maintenance cost of ICI. may be negligible or, crucially, that the acquisition cost of the alternative strategy (IWL.) is prohibitively high. This suggests that when both strategies are asymptotically effective, the model's preference is driven by the difficulty of learning each solution. We formalize this intuition in Sec- tion
4.2 TASK-DEPENDENT PATTERNS OF TRANSIENCE
Beyond asymptotic strategies, the learning trajectories reveal shifts strategy preference during training a phenomenon known as transience (Singh et al. [023]] Panwar et al. 2023] [Singh
et al., 2025). As shown Figure direction of change between ICL and IWL is highly task- dependent.
ICI. transience (ICL IWI.). In the Omniglot task under high environmental stability (Figure top), observe the phenomenon of ICL transience (Singh et al. 2023). The model exhibits strong preference for ICL early in training. However, as training progresses, the model gradually shifts towards IWL solution. This progression parallels genetic assimilation, where trait initially induced by environmental cues (plasticity) becomes genetically encoded under prolonged, stable selection, eventually rendering the redundant (Waddington 1953 Pigliucei al.) 2006).
IWL transience (IWL + ICL). contrast, the sinusoid regression task (Figure 4 bottom) exhibits the opposite trajectory: IWI. transience (or delayed ICI.). Under medium stability (a 0.8), the

---

Published as a conference paper at ICLR 2026
Omniglot
Rellailty
0.9
0.4
0.2
10000
20000
30000
40000
50000
Step (1)
Sinusoid le 0.8)
1.0
Ralability (o7)
0.8 0.00
0.75
0.6
1.00
1.25
1.50
0.0
Figure 4: The shift in ICL preference (Sict.) over training steps (t) reveals opposing directions of transience. Omniglot classification (top) demonstrates ICL transience initially relies on ICL but gradually consolidates the task into weights. Sinusoid regression (bottom) demonstrates IWL transience (a 0.8). The model initially relies on IWL but gradually improves inferences by using the prompt.
1). The model
model initially relies on IWL (approximating the function via weights) and exhibits a low preference for context. A strong preference for ICL emerges only later in training, particularly when cues are highly reliable (L o²). This suggests that for sinusoid regression, an IWL approximation of the target function is easier to learn early on, whereas the circuitry required for ICL develops more slowly. This pattern relates to phenomena like grokking, where generalization capability (here, ICL) emerges abruptly after a long period of fitting the training distribution (Power et al. 2022).
4.3 COST-BASED ARBITRATION BETWEEN STRATEGIES
The contrasting trajectorics observed in Section 4.2 raise the question of why Transformers initially prefer ICL in Omniglot (ICI. IWL.) but IWL in Sinusoid (IWI. ICI.) These results suggest that learning dynamics are governed not only by the asymptotic optimality determined by the environment), but also by the cost of acquiring that strategy. We posit that the strategy that is cheaper to learn (i.e., whose inductive bias closely fits the task structure) will emerge earlier in training, regardless of whether it is the long-term optimal solution. This aligns with the broader literature on simplicity bias (Arpit et al., 2017 Goldblum et al. 2023 Deora et al. 2025) while highlighting that simplicity is contingent on the correspondence between the model and the training data.
strategy (which is
Cost of learning for each strategy. Intuitively, the relative difficulty of IWL and ICL differs across our two tasks. In Omniglot, IWL requires memorizing global mapping for large vocabulary (CC) = 1623). This high-capacity memorization problem with high sample complexity. In contrast, ICL requires only a local pattern-matching circuit (comparing the query the prompt) OOlsson et al. 2022 Singh et al., 2024), which is simple for the attention mechanism to discover. Thus, for Omniglot, ICL cheaper than IWL. In sinusoid regression, the dynamic is reversed. An IWL solution requires learning a single global sine wave approximation, which neural networks parameterize rapidly. Conversely, ICL requires implementing regression algorithm within the

---

Published as a conference paper at ICLR 2026
forward pass (e.g.. gradient descent or ridge regression via attention (Von Oswald et al. (023)). This a complex circuit that is difficult to learn. Thus, for sinusoid, IWL is cheaper than ICL.
Quantifying learning cost. To formalize this intuition, adopt the minimum description length (MDL) perspective for quantifying the cost of learning particular dataset, using the prequen- tial codelength (Blier & Ollivier 2018). Prequential codelength the cumulative negative log- likelihood (NLL) of each token, computed sequentially by model strategy (ICL or IWL) trained on all preceding tokens. See Appendix AA.3 for formal description. Intuitively, a strategy that is easier to learn for the particular dataset (more aligned inductive bias) will achieve lower log-likelihoods early in training, resulting in a shorter prequential codelength
Because our goal is to evaluate the inductive bias of different learning strategies the transformer
can adopt (ICL or IWL), rather than the Transformer as a whole, we consider training environments
where either ICL or IWL are strongly favored, For ICL, this means volatile environment (a: = 0.0)
with reliable cues (p 1.0, x² 0.0). And for IWL, this means a stable environment (a 1.0)
with unreliable cues (p = 0.5, o² 1.5). We then computed the prequential codelength of each strategy (ICL and IWL) for each task (simusoid and Omniglot) by keeping a running sum of the NLL (i.e., the loss) during the entire course of training As shown in Figure 5 the results confirm our intuition. For Omniglot, the cost for ICL is significantly lower than for IWL. Conversely, for sinusoid, the cost for IWL is significantly lower than for ICL. This metric predicts the starting point of the learning trajectories in Figure the model adopts the cheaper strategy first.
Sinusoid
Omniglot
MML
ICL
250
500
750
1000
1250
1500
1750
2000
1000
2000
3000
4000
5000
Cumullatvv MSE
Camulitive ICE
Figure 5: Cost of learning IWL ICI. measured the prequential codelength (cumulative MSE/BCE loss accrued over 50,000 training steps). IWL less expensive than ICL for Sinusoid, and the reverse is true for Omniglot, showing that the relative cost of IWL ICL corresponds to transience dynamics (IWL-first or ICL-first) observed experimentally for the two tasks.
Manipulating learning cost. To causally validate that learning cost drives these dynamics, manipulated the difficulty of the IWL strategy in the Omniglot task. The sample complexity of IWL all
(memorizing labels for all characters) is governed by the coupon collector problem N = -CC characters at least K times requires O(K log N) episodes. By reducing the vocabulary size [C), reduce the cost of the IWI. strategy.
As Figure 6 shows, reducing |C] from 1623 to 100 drastically changes the learning dynamics. In the standard setting (dark line), the model starts with high ICL preference. However, in the reduced setting (lightest line), the dynamics reverse: the model starts with a preference for IWL (Sict. < 0.2) before eventually switching ICL. This induced IWL transience (reproducing the dynamic seen in the Sinusoid task) shows that the initial preference of strategy is determined by its acquisition cost.
Our findings align with recent work linking ICL transience task structure (Singh al. 2023)
and relative acquisition speeds (Nguyen & Reddy] 2024] Singh et al. 2025) Wurgaft et al. 2025
as well as mechanistic studies on circuit efficiency and competition dynamics (Thilak et al. 2022
Nanda et al.) 2023) Varma et al. 2023 Park et al.) 2024). Together, they imply that Transformers arbitrate between strategies based on a distinct optimization cost. The framework of resource ratio- nal analysis (Lieder & Griffiths] 2020), introduced in cognitive science but inspired by earlier work
Prequential codelength also exhibits a "catch-up up phenomenon" that may be interesting to compare with transience observations here complex models have higher cost initially and later require more samples "catch up", even after they become more optimal (Erven al. 2012).
'It is worth noting that for the Omniglot task, the binary cross entropy loss equivalent to the NLL; however for sinusoid regression, the mean squared emror loss is monotonically related to the NLL under a Gaussian likelihond.
8

---

Published as a conference paper at ICLR 2026
Omniglot 0.999, 0.95)
1.0
0.4
200
400
800
0.2
800
1000
10000
20000
30000
40000
50000
Step (1)
Figure 6: We manipulate the learning cost of the IWL strategy by varying the Omniglot vocabulary size (IC)). We plot ICL preference (SicL. across training steps (t) in stable environment (a 0.999) with reliable cues (p 0.95). In the standard setting (IC) 1623, dark line), the model exhibits sustained preference for ICL. Drastically reducing the vocabulary size (e.g., to CI 100, lightest line) lowers the cost of IWL, causing the model initially favors IWL before eventually switching to ICL, thereby reversing the pattern of transience previously observed in this task.
in Al (Horvitz. 1987 Russell & Subramanian 1994). may provide a of tools for formalizing this tradeoff and offering further insight into the behavior of Al models.
DISCUSSION
In this paper, we have demonstrated that the competition between ICL and IWL in Transformers is governed by the same principles of environmental predictability that shape biological adaptation. While stability and cue reliability dictate the asymptotic strategy (favoring IWL or ICL, respec- tively), our analysis of transience reveals that the trajectory of learning is driven by the cost of acquiring each strategy. We find that Transformers consistently adopt the casier strategy first (as determined by their inductive bias for a particular task) while the more costly but asymptotically optimal strategy develops over a longer timescale.
5.1 LIMITATIONS AND FUTURE DIRECTIONS
While this work offers foundational insights into the learning dynamics of Transformers, we ac- knowledge specific limitations that define its scope, as well promising avenues for future research.
Task simplification. Our sinusoid and Omniglot environments are, by design, simplifications of the complex linguistic environments that LLMS are trained in. However, such controlled settings are essential for isolating variables (c.g., environmental predictability) and their causal influence on learning dynamics. Future work should aim to verify these principles in more complex domains.
Levels of analysis. Our framework constitutes a computationall-evel analysis (Marr 1982 Ku et al. 2025). We characterize the adaptive problem a system faces (optimizing prediction under varying uncertainty) and the functional logic of its solution (ICL vs. IWL). The analogy to evolutionary biology operates this functional level of adaptation to statistical structure in the environment; we do not claim direct mechanistic equivalence between IWI. and ICL in Transformers and the specific genetic or developmental pathways of biological adaptation.
The Baldwin effect. Beyond merely managing uncertainty, key evolutionary function of plasticity to accelerate adaptation on rugged fitness landscapes (the Baldwin Effect). Plasticity smooths the fitness landscape by enabling individuals survive in new environments, guiding the population toward regions where genetic assimilation can subsequently fix the trait (Hinton et al. 1987 Fer- nando et al., 2018). An exciting direction for future research is to determine if analogous dynamic exists in Transformers - does the early emergence of ICL serve as a scaffold for accelerating the acquisition of IWL solutions for complex tasks?

---

Published as a conference paper at ICLR 2026
5.2 CONCLUSION
By viewing Transformers through the lens of evolutionary biology, we show that their learning dy- namics are governed by the statistical structure of their environments. We find that the preference for ICL versus IWL is strongly determined by dimensions of environmental predictability (specifically, environmental stability and cue reliability). Furthermore, our analysis of learning dynamics reveals that while the environment dictates the asymptotically optimal strategy, the learner's inductive bias leads to transient phases where the model exploits a lower-cost solution first. This framework estab- lishes more ecological perspective model behavior, paving the way for training methodologies that cultivate systems capable of robustly and flexibly navigating the diverse and dynamic environ- ments they increasingly encounter.
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics statement. This work constitutes basic research into the fundamental learning dynamics of Transformers. Our experiments rely exclusively synthetic data and the publicly available Omniglot benchmark; consequently, the work involves no personally identifiable information and we do not anticipate any direct negative societal impacts.
Reproducibility statement. To ensure the reproducibility of our findings, we have provided detailed description of the model architecture, training protocol, and task generation processes in Section Comprehensive hyperparameter configurations and details regarding compute resources listed in Appendix A Furthermore, the complete codebase for data generation, training, and analysis will be made publicly available upon publication.
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International conference machine learning, pp. 233-242. PMLR, 2017.
Léonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamnrr. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in in-context learn- ing in transformers. Advances in Neural Information Processing Systems, 35: (8878-1889 2022.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint ar arXiv:22/2.1000 10559, 2022.
Puneesh Deora, Bhavya Vasudeva, Tina Behnia, and Christos Thrampoulidis. In-context occam's razor: How transformers prefer simpler hypotheses the fly. arXiv preprint arXiv:2506. 250.:2250..99311, 2025.
Thomas J Wetttt, Andrew Sih. and David Sloan Wilson. Costs and limits of phenotypic plasticity. Trends in Ecology & Evolution, 13(2):77-81, 1998.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, al. A survey on in-context learning. arXiv preprint arXiv:230J.002333 2022.
Tim van Erven, Peter Grünwald, and Steven de Rooij. Catching up faster by switching sooner: predictive approach adaptive estimation with an application to the AIC-BIC dilemma.
10

---

Published as a conference paper at ICLR 2026
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(3)3661-
URL https:// 417. 2012. ISSN 1467-9868. doi: 01111//.1467-988822111.000 2011 2011.01025..
molinelibbarr wwp.ey.wwille www.y.lllo...oo.....o....eo..... eprint: http:////w...iilaaaaa....///oooa 140000009666.2111000000
abs ottp:////...o.//////...//////11.
Chrisantha Fernando, Jakub Sygnowski, Simon Osindero, Jane Wang. Tom Schaul, Denis Teplyashin, Pablo Sprechmann, Alexander Pritzel, and Andrei Rusu. Meta-learning by the bald- win effect. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1313-1320, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp- 1126-1135. PMLR, 2017.
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning, arXiv preprint arXiv:2304.05366, 2023.
Geoffrey E Hinton, Steven J Nowlan, et al. How learning can guide evolution. Complex systems, (3):495-502, 1987.
Eric Horvitz. Reasoning about beliefs and actions under computational resource constraints. Pro- ceedings of the third conference on uncertainly in artificial intelligence, pp. 429-44, 1987.
Alexander Ku, Declan Campbell, Xuechunzi Bai, Jiayi Geng, Ryan Liu, Raja Marjieh, R Thomas McCoy, Andrew Nam, Ilia Sucholutsky. Veniamin Veselovsky, et al. Using the tools of cogni- tive science to understand large language models at different levels of analysis. arXiv preprint arXiv:2503. 13401, 2025.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266)::332213388 2015.
Falk Lieder and Thomas L Griffiths. Resource-rations analysis: Understanding human cognition as the optimal use limited computational resources. Behavioral and Brain Sciences, 43:e1, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: systematic survey of prompting methods in natural language pro- cessing. ACM computing surveys, 55(9):1-35, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Henry Holt and Co., Inc., New York, NY, USA, 1982.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05211, 2023.
Alex Nguyen and Gautam Reddy. Differential learning kinetics govern the transition from memo- rization generalization during in-context learning, arXiv preprint arXiv:2412.00104, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. /895, 2022.
Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. arXiv preprint arXiv:2306. 044494,, 2023.
Core Francisco Park, Ekdeep Singh Lubana, Itamar Pres, and Hidenori Tanaka, Competition dy- namics shape algorithmic phases of in-context learning. arXiv preprint arXiv:2412.01003, 2024.
Massimo Pigliucei, Courtney Murren, and Carl D Schlichting. Phenotypic plasticity and evolution by genetic assimilation. Journal of Experimental Biology, 209(12):236222) 2367, 2006.

---

Published as a conference paper at ICLR 2026
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Stuart Russell and Devika Subramanian. Provably bounded-optimal agents. Journal of Artificial Intelligence Research, 2:575-609, 1994.
Samuel M Scheiner. Genetics and evolution of phenotypic plasticity. Annual review of Ecology and Systematics, 24(1):35-68, 1993.
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36:27801-27819, 2023.
Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C. Y. Chan, and Andrew M. Saxe. What needs to go right for induction head? A mechanistic study of in-context learning circuits and their formation, April 2024. URL http:://arx: wttp://wwww...gp//////..... arXiv:2404.07129 [cs].
Aaditya K Singh, Ted Moskovitz, Sara Dragutinovic, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. Strategy coopetition explains the emergence and transience of in-context learning. arXiv preprint arXiv:2503300661,, 2025.
Stephen C Stearns. The evolutionary significance of phenotypic plasticity. Bioscience, 39(7):436- 445, 1989.
DW Stephens. Variance and the value of information. The American Naturalist, 134(1):128-144 1989.
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint arXiv:2206.0485 2022.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15113. 2023.
Ralph Tollrian and C Drew Harvell. The ecology and evolution of inducible defenses. Princeton University Press, 1999.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Informa- fion Processing Systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174 2023.
Conrad H Waddington. Genetic assimilation of an acquired character. Evolution, 7(2):118-122 1953.
Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, and Noah D Goodman. In-context learning strategies emerge rationally. arXiv preprint arXiv:2506.1 17859, 2025.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit Bayesian inference. arXiv preprint arXiv:2111.02080, 2021.
12

---

Published as a conference paper at ICLR 2026
A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
A. PARAMETER SWEEPS AND REPLICATION
To investigate the effects of environmental predictability, we performed dense parameter sweeps for the Sinusoid regression and Omniglot classification tasks. Unless otherwise specified, all results reported in figures are the mean across runs using different random seeds, and plotted error bars represent +1 standard error of the mean (SEM).
Sweep Sinusoid regression
The following grid of parameter values was used:
a E {0.0, 0.1, 0.3, 0.5, 0.7, 0.8, 0.95, 0.99, 0.999, 1.0)
x² (0.0, 0.005, 0.01, 0.02, 0.05, 0.1, 0.1,0,0 0.3,
0.75, 1.0, 1.5)
Sweep 2: Omniglot classification
The following grid was used, where a is determined from via 27 1, as described in the main text.
0.4, 0.9, 0.95, 0.99, 0.999, 1.) . E (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 0.99, 1.}
. E (0,40.0,,.22004,
Sweep 3: Omniglot vocabulary size
To generate Figure environmental stability and cue reliability were held at high values (a 0.999, = 0.95), while IWL cost was varied via the vocabulary size:
CC (100, 200, 400, 600, 800, 1000, 1200, 1400, 1623)
These sweeps comprise (11 12) + (11 12) + 9 273 unique experimental configurations. Including the 3 random seeds per configuration, this resulted a total of 819 model training runs.
A.2 COMPUTE RESOURCES
All models were trained on single TPUv3 cores. Each of the 819 training runs took approximately 30 minutes to complete the 50,000 training steps (detailed in Section 3). The total compute usage was approximately 410 TPU-hours.
A.3 PREQUENTIAL CODELENGTH
The standard and formal definition of prequential codelength is (Blier & Ollivier 2018):
12
L(D)
Σ
log P(yelori
where D ((x1, (i(1,99),--. V1),.., (Znny,)) V..)) sequence of observed data points and 0,-1 represents the model parameters trained on the data observed up to time t 1: ((11,8)).. 372-1,81-11
B USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, large language model was used to assist with editing the prose for clarity and aid in literature discovery. This latter function was particularly valuable for exploring the evolutionary biology literature that provides the conceptual framework for this study. All scientific claims and the final content of the paper were written and verified by the authors.
13


Reviews:
[{"review_id": "42Y1AJ5ORh", "reviewer": "Reviewer_yWAK", "summary": "The paper investigates when decoder-only Transformers rely on in-context learning (ICL) versus in-weights learning (IWL). Using an evolutionary analogy (phenotypic plasticity vs genetic encoding), the authors operationalize cue reliability and environmental stability and perform dense parameter sweeps on two controlled tasks (sinusoid regression and Omniglot binary classification). They introduce a preference score (SICL) comparing errors against ICL- and IWL-targets, present asymptotic preference maps and training-time transience dynamics, and posit a qualitative 'relative-cost' hypothesis to explain strategy emergence.", "questions": "Please correct and clarify the SICL definition and interpretation. If it was a typesetting mistake, state the intended formula and re-run affected figures and analyses. As a sanity check, include results for a synthetic pure-ICL and pure-IWL predictor showing the corrected SICL behaves as intended.\n\nProvide explicit pseudocode for the evaluation protocol: how EICL and EIWL are constructed, how evaluator prompts are sampled, number of evaluation examples per measurement, and how conflict trials are generated and scored.\n\nIncrease robustness: re-run key configurations (those showing sharp transitions or notable transience) with >=5-10 seeds and report SEMs/confidence intervals and statistical tests for main claims.\n\nPerform the following ablations/controls: (a) vary prompt length N; (b) freeze and/or pretrain the ResNet encoder for Omniglot to localize effects; (c) sweep model capacity (smaller/larger Transformers); (d) include explicit conflict trials during evaluation and report how often models follow prompt vs internal mapping.\n\nQuantify the relative-cost hypothesis: measure steps-to-target-error for ICL-only and IWL-only baselines, parameter-efficiency, or representational complexity, and test whether these predict observed transience directions.\n\nAdd basic mechanistic analyses: track layerwise weight changes over training, analyze attention-head patterns, or use linear probes/lesioning to show distinct circuitry for ICL vs IWL and to support any assimilation claims.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper comes with a clear, motivating framing that yields testable experimental axes (stability and cue reliability).\n\nThe scientific claims are substantiated with systematic empirical evaluation with dense sweeps and temporal analyses across many configurations.\n\nCoherent qualitative findings across two tasks: stability favors IWL; reliable cues favor ICL; transience depends on relative difficulty.\n\nEfficient experimental design allowed broad exploration and reproducibility in principle; hyperparameter grids and compute are reported.", "weaknesses": "I might be mistaken but is there a critical and central inconsistency?: SICL is defined as SICL = EIWL / (EICL + EIWL + eps) but interpreted (and plotted) as higher SICL meaning more ICL.\n\nInsufficient robustness: only 3 seeds per configuration; many key effects (thresholds, transience) need more seeds and statistical tests.\n\nLimited external validity: only two simplified tasks and a single small Transformer; applicability to larger models and potentially LLMs or naturalistic domains is untested.\n\nMissing important ablations/controls: prompt-length (N) sweep, encoder-freeze/pretrain ablations for Omniglot, model-capacity sweep, and explicit conflict trials to directly distinguish ICL vs IWL.\n\nMechanistic evidence is lacking: no attention/head diagnostics, weight-change tracking, probes, or lesioning to support claims of circuit-level implementation or assimilation into weights.\n\nThe relative-cost hypothesis is qualitative and unquantified; no direct cost or sample-complexity metrics are provided to predict transience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80eXf3g3Vy", "reviewer": "Reviewer_XQNw", "summary": "This paper investigates how environmental predictability influences the balance between in-context learning (ICL) and in-weights learning (IWL) in Transformers. The paper draws many analogies to evolutionary biology's phenotypic plasticity and genetic encoding and the circuit learned in transformers. The authors set up two model systems: sinusoidal regression and Omniglot and set up variables representing environmental stability (task consistency across training) and cue reliability (how informative are in-context examples). Key findings include: (1) high stability favors IWL while high cue reliability enhances ICL, (2) learning dynamics show task-dependent transience patterns (both ICL→IWL and IWL→ICL transitions, this latter seems quite novel), and (3) a \"relative-cost hypothesis\" suggesting that the computational ease of acquiring each strategy determines preference and transition dynamics.", "questions": "It would be good to cite some more papers which also explore IWL vs ICL:\n\nhttps://arxiv.org/abs/2306.04891 <- this paper seems to co-pioneer the findings on transience, although they didn't focus on presenting it that way.\nhttps://arxiv.org/abs/2412.01003 <- seems directly related to the cost of memorizing more processes and also discuss that circuit complexity slows ICL.\nhttps://arxiv.org/abs/2506.17859 <- also seems related to the relative cost hypothesis.\nhttps://arxiv.org/abs/2506.19351 <- discusses an Occam's razor on complexity.\n\nQ1: I'm not so sure if this is possible, but is it possible to decompose the model prediction in the style of https://arxiv.org/abs/2412.01003, i.e. decomposing the probability itself by the ICL vs IWL probabilities?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: The experiments are quite novel and interesting, even though some past studies study curriculum learning where the loss landscape keeps changing, studying this together with the question of ICL vs IWL is genuinely interesting.\n\nS2: The **quality** of the paper's presentation is great, even though the framing seems questionable (see W1). The paper is well-written and figures are clear. The experimental design is clean and easy to understand.\n\nS3: The identification of IWL->ICL transience (in addition to previously documented ICL transience) seems novel and interesting.", "weaknesses": "W1: This is my only major concern of the paper. The connection to evolutionary biology seems to be just at the very high level. The framing makes sense, but does not actually offer any falsifiable statements or predictions. While this could have been a nice discussion point, I don't see why this should be a main theme of the paper instead of simply framing it as a study of circuit competition on non stationary losses. Furthermore the training method lacks any kind of evolutionary mechanism such as selection, mutation, reproduction. The authors themselves do acknowledge that there is no direct equivalence. It seems like the paper could be much better linked to other concepts like learning theory, meta-learning frameworks, complexity theory, etc. I really like the experiments and the experiments were genuinely interesting, but I am left confused why such a superficial connection to evolution was made, without discussing core concepts of evolution: G-P mapping, mutations, genetic drift, etc.\n\nHowever, I am happy to discuss this further, perhaps there is a connection which is genuinely helpful that I'm missing.\n\nW2: Beyond the weaknesses discussed in W1, it seems like the relative-cost hypothesis is a good intuition to have, but at the same time doesn't seem to be too novel compared to classic discussions in simplicity bias, circuit complexity, memorization budget in deep learning.\n\nW3: The choice of learning rate scheduling is questionable when the data distribution is non-stationary. Perhaps this makes the results harder to interpret since it artificially slows down the speed of learning. However, I don't think this will qualitatively change the results too much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odZRKybbx3", "reviewer": "Reviewer_ER3k", "summary": "The authors investigate in-context learning versus in-weights learning in Transformers (though I don't see why the experiments couldn't be done with other memory-based architectures, like RNNs).\n\nAdopting meta-learning settings,  both Omniglot and sinusoid regression, they independently manipulate the stability of the (outer-loop, across tasks) task distribution, and the reliability of the (inner-loop, within-task) contextual cues.\n\nThen, testing with a new task allows then to estimate how much the network relies on in-weights or in-context learning, based on whether its answer to the final query input is closer to that predicted from current in-context cues, or from the ongoing training task, respectively. \n\nThey show that higher contextual reliability favors in-context learning, while higher task stability favors in-weight learning. Different tasks have different dynamics. Increasing the number of possible classes decreases preference for in-weights learning in the Omniglot task.", "questions": "If the authors can make their hypotheses a bit more precise and/or actionable it would probably increase the reach of the paper. Other than that I have no pressing questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The question is interesting. The experiments are informative. The review of the literature (on both evolution and ML) is nice.", "weaknesses": "There seems to be no fatal flaw in the paper that I can see.\n\nIt may be argued that some of the results are not really earth-shattering (\"more stability = more overfitting?\"), though the additional experiments in Figures 4 and 5 provide more details on the dynamics.\n\nIt's not clear how much the \"relative cost\" hypothesis helps, because there seems to be no precise definition of \"cost\", except for a-posteriori hardness on IWL? (I note that the parameter used to tune Omniglot task is the total number of classes; however, the sinusoid task has an infinite number of classes, yet it doesn't seem to clearly favor ICL or IWL more than Omniglot, but rather it seems to incline differently for various regimes of stability/reliability).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)