Prior Forgetting and In-Context Overfitting
Authors: Sungyoon Lee
Venue: NeurIPS 2025 poster ()
Keywords: in-context learning
transformer
task recognition
task learning
pretraining
prior forgetting
in-context overfitting
Primary area: deep_learning

Abstract: In-context learning (ICL) is one of the key capabilities contributing to the great success of LLMs. At test time, ICL is known to operate in the two modes: task recognition and task learning. In this paper, we investigate the emergence and dynamics of the two modes of ICL during pretraining. To provide an analytical understanding of the learning dynamics of the ICL abilities, we investigate the in-context random linear regression problem with a simple linear-attention-based transformer, and define and disentangle the strengths of the task recognition and task learning abilities stored in the transformer model’s parameters. We show that, during the pretraining phase, the model first learns the task learning and the task recognition abilities together in the beginning, but it (a) gradually forgets the task recognition ability to recall the priorly learned tasks and (b) relies more on the given context in the later phase, which we call (a) \textit{prior forgetting} and (b) \textit{in-context overfitting}, respectively.

Prior Forgetting and In-Context Overfitting
Sungyoon Lee Department of Computer Science
Hanyang University sungyoonlee@hanyang ac kr
Abstract
In-context learning (ICL) is one of the key capabilities contributing to the great success of LLMs. At test time, ICL is known to operate in the two modes: task recognition and task learning. In this paper, we investigate the emergence and dynamics of the two modes of ICI. during pretraining. To provide an analytical understanding of the learning dynamics of the ICI. abilities, we investigate the in-context random linear regression problem with simple linear-attention based transformer, and define and disentangle the strengths of the task recognition and task learning abilities stored in the transformer model's 's parameters. We show that, during the pretraining phase, the model first learns the task learning and the task recognition abilities together in the beginning, but it (a) gradually forgets the task recognition ability recall the priorly learned tasks and (b) relies more the given context in the later phase, which we call (a) prior forgetring and (b) in-context overfitting. respectively.
Introduction
Large language models (LLMs) show an emergent behavior (Wei et al. 2022]. known as in-context learning (ICL) [Brown al. 2020 Kaplan al., 2020). that they can learn (without updating any model parameters) a new unseen task from a few demonstrations of input-output pairs given at test time. While models trained with traditional supervised learning for a given task/mapping learn task-specific features that might be spurious or irrelevant for other tasks, the ICL ability enables the models learn stronger representations with task-agnostic architecture and task-agnostic data and to efficiently learn diverse tasks via text interaction in flexible way. Thus, it is often considered as one of the key capabilities contributing to the great success of LLMs.
At test time, in-context learning is known to operate the following two modes: given an in-context task, the model (i) recalls similar functions and concepts learned (priorly) in the pretraining phase and (ii) smoothly adapts to and implicitly learns the (observed) in-context task (Xie et al.) 2022 Raventós al. 2024 Pan et al., 2023 Lin and Lee 2024]. These dual Bayesian modes of TCL.
often called (i) task recognition and (ii) task learning Pan et al. 2023). In other words, the pretrained model (i) knows the prior task distribution and (ii) has the ability select and perform proper task
corresponding to the given demonstrations.
The power of each mode of ICL is not the same and often dominates the other depending many factors. To separately investigate the effects of the task recognition and task learning abilities, it has been proposed irrelevant labels). Min et al. 2002]. Lyu et al. [2023) show that randomly replacing output labels in the demonstrations does not significantly affect ICL performance which indicates that the task recognition dominates the task learning. On the other hands, Yoo et al. 20022), WWii et al.) (2023) SShi
the experimental setups with noisy output labels (random semantically
al. (2023 show that larger models are easily distracted by wrong or irrelevant demonstrations and Pan et al. [2023) show that larger models exhibits a better task learning ability than smaller ones and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

- increasing In Context Strength
In Context Overfitling
risks
Co-Ceeexx Overfitting
Prior Forgetting
Gener alicalion Risk Generalization Gap
Decreasing Strength
training steps
Figure Conceptual sketch of in-context overfitting and prior forgetting. (Left) Following the gradient flow (the c-shaped trajectory (() from magenta to lime) in the parameter space, we observe monotonic increase in-context strength (L) during training, which we call in-context overfitting, and a decrease in prior strength (-) in the later phase of the training, which we call prior forgetting. The exact meaning of the axes will become clear in Figure 2 (Bottom). (Right) We observe monotonic increase in the generalization gap (blue dotted line) and u-shaped curve of the generalization risk (black solid line). We also call these two phenomena in-context overfitting (-+) since the generalization gap is directly linked with the in-context strength.
improves with more demonstrations. other words, the task learning ability emerges as we scale up the model and it plays more important role than the task recognition for larger models.
As shown above, most studies attempt to analyze the behaviors of the pretrained transformers at test time, but to deeper understand the two modes of ICL, we need to investigate their emergence and dynamics in the pretraining phase. Thus, we raise the following questions:
How do in-context learning abilities emerge (and disappear) during pretraining?
To answer this question with analytical understanding of the learning dynamics of the ICL abilities. we investigate the in in-context random linear regression problem [Garg et al.) [202]] Akyürek et al.)
2023 Von Oswald al. 2023a Li et 2023b] with simple
based transformer
[Schlag et al. 2021 Von Oswald et al. 2023a] Moreover, to mathematically model setup similar
to the experimental settings considered Min al. [2022], Lyu al. [2023], Yoo et al. (2022), Wei et al. [2023) . Shi et al. 2203). Pan et al. 2023 and to disentangle the strengths of the task recognition and task learning abilities, we introduce new settings we call demonstration-quer task irrelevance and noncentral task model. With these small modifications from previous works, we can derive a simple yet interesting dynamics regarding the ICL abilities.
To be specific, we show that, during the pretraining phase, the model first learns the task learning and task recognition abilities together in the beginning, but (a) gradually forgets the task recognition ability to recall the priorly learned tasks and (b) relies more on the given context in the later phase, which we call (a) prior forgetting and (b) in-context overfitting respectively. Figure [ illustraates prior forgetting and in-context overfitting from the perspectives of model parameters (Left) and generalization loss (Right).
Related Work
After it was demonstrated that large transformers Vaswani et al. 2017 such GPT-3 can perform in-context learning (Brown et al. 2020 Kaplan et al. 2020J, there has been a growing interest in understanding the underlying mechanisms of the ICL ability.
Garg et al. [2022 empirically show that, for in-context linear regression, the trained transformers match the performance of the optimal least squares estimator, i.e., the solution obtained by gradient flow on in-context examples. This work is followed by Von Oswald al. 20033a], [Akyürek
et al. (2033 in which the authors provide a construction of transformer that can implement gradient descent for in-context linear regression. Especially, the construction in Von Oswald et al. [2023a]
only requires single linear self-attention layer (Schlag et al. 2021 to implement single iteration

---

of gradient descent. After that, many works consider linear transformer as a simple proxy for the softmax-based transformers to theoretically investigate their complex behaviors (Mahankali et al.)
2024
Ahn et al.
2024a b
Zhang et al. 2024
Ahn al. 2024a show that, for certain training objective over random instances of linear regression, the global minimizers can implement preconditioned gradient descent. Here, the parameter configura- tion of the minimizer is the same (up a constant factor) the one constructed in Von Oswald et al.
2023a Moreover, ZZang et al. 2024] show that such global minimizers can be achieved by running gradient flow under a certain initialization inducing the balancedness condition [Arora al. 2019a 2018 2019b
Du et al. 2018 throughout the training.
Beyond the linear regression, Garg al. [022] also explore more complex function classes such as two-layer neural networks and decision trees, and empirically show that the trained transformer in-context learn these function classes. Some theoretical work extend the in-context linear
regression problem to exponential regression [Gao et al. 2023], softmax regression (Li et al.) 2023a
or autoregressive learning [Von Oswald et al. 2023b Sander et al.) 2024 Zheng et al. 2024]
Moreover, Dai et al. [2023 empirically explore language models on real NLP tasks such as sentiment
classification, topic classification, and natural language inference.
Likewise, many studies attempt to analyze the behaviors of the pretrained transformers at test time, e.g., the two Bayesian modes of ICL [Xie et al.
[0222] Wies et al.
[023 Jiang 2023 Wang et al.)
2024b Raventós et al., [2024 Pan et al. [2023] Lin and Lee 2024 and their relative performance
(especially when we scale the model) [Min al. 2022 Lyu al. [023 Yoo et al.. 2022 Wei et al.,
2023 SSh et al., 2023 Pan et al.
2023 Shi et al. [024]. On the other hand, we focus more on the
emergence and -eeellnd dynamics of the test-time performance of the two modes during pretraining.
To model the task distribution, Raventós et al. 2024 Lin and Lee 20244 introduce probabilistic mixture model of multiple task groups with task desendennddnt input distributions, while we analyze a simple unimodal task distribution.
There are some more interesting work on the power balance between the two Bayesian modes of ICL. |Wang et al. (2023) show similar results with Min et al. [2022) for thain-of-thougee prompting that invalid reasoning steps do not hurt performance on multi-step reasoning tasks. [Reynolds and McDonel| [202T] show that zero-shot prompts can match and even outperform few-shot prompts,
which implies that the task recognition ability plays a more important role than from the task learning
ability for some tasks.
Wang al. 2024a| design metric called competition intensity to explore the emergence of ICL and empirically show that the two modes of ICL are competitive during pretraining, while we measure the strengths of the two modes from the transformer's 's parameters and theoretically investigate how they emerge and disappear.
3 Settings
In this section, we first investigate the in-context random linear regression problem with a simple innear-attention-basseee transformer, following GGagg et al. 2022 Akyürek al. [023]] Von Oswald ct al. 2023a Li et al. 2023b Schlag et al. 202 in Section 3.1 Then, we introduce some new
settings and definitions explore robustness shift that we are given demonstrations irrelevant to
the query task as similar to the empirical settings considered in Min et al. [0022] Lyu et al.) [2023).
Yoo et al. [2022) Wei et al. [2023). Shi al. [2023] Pan al. 2023 in Sections 3.2 3.3
See
Appendix A for a quick reference for the notations.
3.1 In-Context Linear Regression with Linear Transformer
We train transformer with the training set, which consists of the input context matrices and the corresponding target responses. The input context matrix
2
x Y
X
x (n+1)
(1)
x(2)
(n)
a(a+1))
E r (dd+1)x(++1)
Y
0
(1)
2(2)
in
0
3

---

is generated by drawing + 1 d-dimensional covariates x(1) and an in-context task vector w representing a linear function fw I + w I and computing the target responses y( as follows: Lid. ~
x
(1)
Dx. w ~ Dw, = w T xx) (i=1,-- 1,.., , n 1),
where 200) Rd ( R. = XX æ(ntl) E Rdx(n+11), , [x(1) x [n) E Rd
Y [Y 0J Rix(nt)) = (y))) yna) E Rixn Here, the (1))ss for i S n and (n++1)
called the in-context covariates and the query input, respectively.
We consider single-layer linear transformer TP.Q with linear self-attention (LSA) [Schlag et al. Von Oswald et al. 2023a parametrized by two matrices P and Q and residual connection He
2021
al.
2016]:
Tr.(22) = Z 1 LLSAP.O(2))
(I)
dd11m+1
(S)).Q(2) = PZMZ QZ,
where
O Oaxd P=
Oa Q
Q 9
0.] [g(+4)x((+)) M 0
[n 0,
0,
E R (((++)x(a+t))
P
0
Here, the first d rows Pi:d; of P and the last column Q:a-1 of Q do not affect the output Tr (Z) of the transformer (see Appendix [B i.e., during the gradient-based training they remain the same the initial values which are usually very small, and thus we simply put Pi:d,: (dx(d-1) and Qia++ 02+1-
Note that the usual self-attention (SA) can be expressed as
SAw, Wi Wo (Z) WvZMo WF
where is usually the column-wise softmax [Bahdanau al. 2015 Vaswani et al. 2017]. Here, the linear self-attention is a special case of the self-attention when P Wv, Q = W Wo. (Z) = Z.
We say the transformer performs a task v if (2, = (x w T (1+1) For the parametrization in I we can express the task
i (Q
+ qw GG, kw),
where G2 = - EE
x 10 (i) See Appendix for details.
To learn to predict the target response (1 w (1+1) for the query input, the training loss (also called the training risk) is given as
(w (((11)) - { Ew.x [(w- 2)) TT)TTx(()) x(+11)) }]]
Lucauu (P, Q) = EE.
3.2 Demonstration-Querr Task Irrelevance and Noncentral Task Model
Zhang et al. [2024] consider the following three distribution shifts:
task shifts: the tasks provided the pretraining phase and the tasks at test time follow different distributions.
query shifts: the in-context covariates Fest and the query input æest distributions.
(i)
(a+1) follow different
covariate shifts: the in-context inputs X in the pretraining phase and the test phase follow different distributions.
We depart from these distribution shifts and the vanilla setup (without any distribution shifts) consid- ered in previous work, and introduce another class of scenario, which we call demonstration- task irrelevance, where the demonstration task u and the query task wy (both test time) are different (independent) but sampled from the same distribution:

---

Assumption 3.1 (Demonstration -uuery Task Irrelevance).
iid. Dw-
w, Wq
In other words, we are given input context matrix
Zest
X
lest
Pest
(1)
Fiest
Fies
(a+1)
Yiest
w
Drest
w
Wiest
0
with the demonstration task wc but the target response determined by another independent query task wg. not by the demonstration task w:
(x+1)_ Diest w0
(n+1)
Piest
Note that this assumption is designed not model a real-world scenario but to disentangle the power of the two ICL abilities because it similar to the empirical settings considered in Min al. [022].
Lyu al. [2023], Yoo al. [2022]. Wei et al. [2023) Shi et al. [2023] Pan et al. 2023 that
given demonstrations irrelevant to the query task.
To investigate the robustness and generalization to the demonstration-queyy irrelevacce we consider the following generalization risk (also called test risk):
T (P, Q) = Eur Xisx (wo7
T
(n+1) (f (
and the generalization gap between the training risk and the generalization risk defined as follows:
AL(P,Q) Lee(P.Q) Lurai P)Q).
(3)
If the task learning plays more important role than the task recognition, then the model relies more on the demonstration task that irrelevant to the query task and thus it shows a higher generalization risk and gap. Therefore, by measuring the generalization risk or gap, we can separately analyze the task learning ability from the task recognition ability.
Similarly, we also want to separately investigate the power of the task recognition ability with which, given a demonstration task, the model recall similar concepts from prior knowledge infer the query task. However, the two tasks are independent. Therefore, we need to consider shared concept in the prior knowledge between the two tasks.
To this end, we simply examine a non-centeral task distribution Dy with nonzero mean μa + Og- Here, the task center ya explains the prior task distribution in the sense that / and Wy = f + 8g share the (non- zero) prior knowledge μ and they have their own knowledge s and sg- By using this prior knowledge represented by nonzero vector, knows and utilizes this prior, i.c., the power of the task recognition ability (see
measure how much the model
Assumption 3.2 (Isotropic Covariate and Noncentral Task). We assume that
(i) the covariate distribution is Dx = (Os,,, Ia) and
(ii) the task distribution Dw (H, Ew) is noncentral with || and isotropic with Σw = o² Ia.
Here, we name (r(Ew) ad the task dispersion.
Under the above isotropic covariate assumption (i), we have
Lygan (PQ)) E.w.x [(w ((w=- -6) a (n() 27 = E x |||l - =0|
(4)
((10g -(10)-1)) TT To (m(tt))) )²] = Emg x [1] - =6/l3)
x Lresa P, Q) = Eurg w,
(5)
Note that we assume the isotropic covariate assumption (i) for simplicity which can be casily relaxed to obtain similar equations with and up to a constant multiple, since E ( (n++))))] ir(Ex) [[|a|[]] if the covariate has zero-mean and covariance of Σx instead of Ta- Moreover, with the noncentral task assumption (ii), we define the following generalization risks:
5

---

Definition 3.3 (Generalization Risk at Zero). If the transformer performs the zero-task v = Oa. i... (x)1)) = 0, then the corresponding generalization risk is called the generalization risk at zero defined
Lo = Eng w [1] |4| (r(Ew) = b.b
When the parameters near the small initialization, the generalization risk is about Lo- Definition 3.4 (Generalization Risk at Random). If the transformer performs a random task w Dw, i.e., TP (a.(((())))) w an(+1), then the expected generalization risk over w' Dw is called the generalization risk at random defined as
= E. [||wr w'11²] = 2tr(Ew)= 2b.
Definition 3.5 (Generalization Risk at Optimum). When the transformer with the parameter (P", Q*) that minimizes the training loss Luains the train/test risks are called the train/test risks at optimum defined as
[ is optimally trained
Lisin = Lurain Q"),,, Liox = Leea(" Q²),
and the gap between the two denoted by AT," Iijest Ligain generalization risk.
Note that Lisst is not optimal
3.3 Two-Parameter Transformer and Prior/In-Context Strength
Given j Rd, we consider the following simple transformers parametrization with two scalars and k:
P
Oaxd oxp
0g
Q
Ta
04
R((+1)x((+11
0a
which, if we put Od, reduces to the parameter construction in Von Oswald et al. [203a to implement a single step gradient descent. From now on, we will use the wwo-parameter notation To with 0 = [a, instead of T(P.Q)-
The two parammreeterr transformer To performs the following task wū i..., (o(x(a+1)) ~ x (n++1);
i (apu
prior strength, -x: in-context strength.
Here a and K are the weights of the task center μe (independent of the demonstrations) and the in-context task w, respectively. Thus, we refer to -a and -x (or just a and k) as the (semantic/bask) prior strength and the in-context strength, respectively. For example, magnitude, then t ignores the demonstrations from w, relying heavily on the task prior Dw (task recognition) represented by μ. On the other hand, if K is much larger than magnitude, then tt only relies on the demonstration task w (task learning) and the model cannot recover irrelevant query task.
is much larger than k in
This two-parameter model enables us to disentangle the two modes of ICL, the task recognition and task learning abilities, with the distinct parameters, the prior strength and in-context strength K. respectively. Thus, in the next section, we will explore the evolution of these two parameters.
Main Results
In this section, first we show that our objective function is quadratic (Theorem 4.1) and then, for this quadratic loss, the gradient flow is linear ODE which shows a simple dynamics (Theorem 4.2). Using this dynamics, we can analyze how the prior/in-context strengths and generalization gap/risk evolve during pretraining. Morcover, we also discuss how the value task dispersion affects the learning dynamics (Theorem 4.3)
4.1 Training Dynamics
The training loss function for the wo-parameterr transformer, from 4 and (7), can be expressed as follows:
Liain (0) E. x ||| a [| E w| + G Ga kw))]
From this equation, it can be easily shown that the training loss function is quadratic with respect to the two variables and k. The following theorem states the details of this quadratic loss function:

---

dff
kiil
500
1400
1500
100
1600
1500
500
1600
1500
-03
48
375
-1.0
84 -10
150
10
12
-3.
-10
-05
0.0
0.5
1.0
Figure 2: Evolution of the two parameters, and K, for different b's. We train the two-parameter transformer using SGD with learning rate of 0.01 and batch size of 4,000. We also use = 10, d = 5. and σ 0.2, 0.4, 0.8, i.e., the task dispersion b o²d 0.2, 0.8, 3.2 (from Left to Right). Top: Empirical results with SGD (solid lines) and theoretical results of (11) and (12) with gradient flow (dashed lines). Bottom: Training trajectory (from initialization
the optimum A) drawn with
quadratic loss landscape (elliptic level sets) on the ax-pllaee The color of trajectory indicates the training loss value. See Figure T (Left) together.
Theorem 4.1 (Quadratic Loss). For the two-parameter transfoorr the training loss is quadratic with respect to the parameter 0 |a, K]
R2:
Liroin
0
0 2
C20 + CTO 0 Co. where
(8)
n ++ 1 C2
R
C1 = 1 [+b]] ER Co²,,o=1
1+b
n
and b b
ddc. The training loss Linan (0) is minimized at 0 0", where
aa" K
d + + + Lo-
0"
and Liminn
(9)
12
++++1
Note that, when = O2, the global minimum 6* reduces to the one in Ahn et al. [202aa] as special case. The proof deferred to Appendix [
Figure 2 (Bottom) (Bottoo) shows the quadratic loss landscape with elliptic level sets and a unique minimum (since the Hessian matrix C2 positive-definit)) 0" on the k-axis (shown with A). The geometry of the loss landscape highly affects the learning dynamics as will be detailed in the following theorem and the later sections.
Theorem 4.2 (Training Dynamics), Under the same setting of Theorem by solving the following linear differential equation (gradient flow) starting from (0) = 0:
θ = - Imiin (0) = C20 C1.
we can obtain the following solution:
0(t) CE -+t -++
D4
CE d- uu u +
(10)
n
c =
(n d 1)
where
A 2 - / 6z
b /4 + 62
and 0+
1,
2
2

---

Kaam (thwary (fhiees)
itheary
100
200
703
600
100
200
305
+50
136
200
700
1a-
302
203
100
200
362
600
100 200
302
330 200
made
Figure 3: ICL loss curves for different parameterizations (each row) and different b's (each column). We train (Top) the two-parameter transformer (6). (Middle) the full-parameter transformer in ( and (Bottom) practical multi-head (h 8) multi-layer (E 12) transformer with softmax attention and residual connection. We use n = 10, d = and o² = 0.04, 0.12, 0.16, i.c. 6 = 0.2, 0.6, 0.8 (from Left to Right). It shows empirical results with gradient descent (solid lines) and thcoretical results with gradient flow (colored dotted lines). The horizontal dotted lines indicate
the four equations, Lo = 1 b, Liminn = d+1 training loss (red) and generalization loss (black) start from Lo and the training loss monotonically decreases Lizin On the other hand, the generalization gap (blue) monotonically increases A As result, the generalization loss shows u-shaped curve and converges to Lust We use learning rate of 0.01 and batch size of 4,000. We SGD for the two-parameter transformer, but use AdamW for the full-parane palamerreeter transformer and practical models. See Figurel I (Right) together.
n+d++] Lo. Lies (d+1)LotnL, n+d+1 AL* = -2bx.. The
are the eigenvalues and the corresponding eigenvectors of C22 respectively. To be specific, we have
a(t) -2ce -8t sinh S 0,
(11)
x(t) = 2ce Gst cosh(?6t- -Tb) + K 2 R",
(12)
where B,
2+h 2 2 76
14+12 and arctanh
b
/44
Note that the context length n can only scale the dynamics through the constant c and does not affect the shape of the dynamics. The proof is deferred to Appendix D
Figure shows at),, xct) (Top) and the trajectory (t) on the avs-plane (Bottom). Our theory II]
and 12 with gradient flow (Top, dashed) explains the empirical results with SGD (Top, solid) well.
4.2 Prior Forgetting and In-Context Overfitting
Theorem 4.2
illustrates the analytical dynamics of the two parameters att) and x(t). First, from 11 we can show that (x)) decreases till < to for some to > and then increases back to 0 as t goes to oo since the derivative
a' (t) -2ce - 9.4 ( (-B, sinh(?ot) + cosh(>nat) = 2ce 3nt sinh(yyt - 7b)

---

changes its sign (from negative to positive) at t = to Fo/70- i.e. a'(u)) = O, where fu
arctanh (7h/ (b) > O. Here the critical point to is a decreasing function of b (see Figure in
Appendix and as expected, Figure 2 (Top) shows an earlier critical point as we increase b.
Second, from [12.
show that k(t) monotonically decreases H² -n/(n
since
k² (1) 2ce Pat (-3, cosh(nal - m.) 7L sinh(7al - (b)) -2e B&t cosh 76 7m) < 0.
Figure 2 (Top) demonstrates the above two phenomena: (i) axl) decreases in the beginning and then increases back to 0 and (ii) k(t) monotonically decreases K², In other words, (i) the prior strength -a gets stronger in the beginning, but gets weaker, reaching 0 in the later phase, which we call prior forgetting. and (ii) the in-context strength -x increases and it leads the increase in the generalization gap (we will show this in the next section) which we call in-context overfitring.
4.3 Generalization Gap
Moreover, for the generalization gap of the model during the pretraining, from have
5
we
xl(wg- - -w) -w) Gz + kw)) -2bb AL = Em <11lag - ib| w| x0| = 2 ng 20,
(13)
since E/w,] w| the in-context strength -K.
E ww w] || (11 +bb b. Thus, the generalization gap increases with
Figure (blue curves) shows the increase in the generalization gap AL during pretraining, which we also call in-context overfitting. It empirically shows that the full parammetrrr transformer (T) (bottom) behaves similar analytical model (dotted lines) for the two- wwo-parameter transformer, also demonstrating in-context overfitting. Note that, without the two-parameter restriction, the training and generalization losses become a little smaller.
4.4 Different Dynamics Depending on Task Dispersion b
For the trajectory ((t) in 10 we have the derivative
6' cc A+ 24 + ch_ d cx
with the two orthogonal components 04 and v_ At time i, the coefficients are @(Aexp(-(A)) with respect each and have
{> 0, i <0,, ift>
(exp(-tM) = (1-t)))))) 0x
Thus, as shown in Figure 2 (Bottom), in the beginning < 1), the flow is mostly aligned with the sharper direction u. of the quadratic loss landscape because X4 > A__. On the other hand, in the later phase (L 1), the opposite holds and the flow gets more aligned with the flatter v_..
The following theorem provides intuitive pictures for the loss landscape and the gradient flow, and their changes due to variations in the value of task dispersion b.
Theorem 4.3 (Evolution of Task Recoonnition/Learnnn
The eigenvalues X and the corresponding
eigenvectors v+ of the Hessian O2 of the training loss Luain are: (i) for small b = o² d < 1,
h_ = 2 + (b) 2,
v = [1,1+ ~ [1,1],
h_ = (b)) & Xt,
0 = (1,-1+ [1,
and, (ii) for a large b >
X_ (b)),
v [1, 0(b)] 7 [0.1]
X_ 1 0 & Xt,
v. = [1, 0(1bb) Jا [1, 0],
respectively, where W means that and approximately parallel.

---

The proof is deferred to Appendix D
Figure 2 (Bottom) shows that when b is small, e.g., 0.2 (Leftt, the main directions of the elliptic level sets are 04 R [1, 1]T and v x [1,-1] and the gradient flow is first aligned with the sharper direction B4 then later it follows the flatter direction v.. And as b gets larger, e.g., 0.8
(Right), the two directions becomes more like 04 | [1,0]' and w_ | O].
Figure 2 (Top) shows that, as b gets larger (from Left to Right), a(t) gets to have relatively smaller dip as the sharper direction is nearly aligned with [0, 1] orthogonal to the direction increasing
4.5 Generalization Risk
1.8
0.0
a nation risk
1.6
x
1.4
005
init. (t= 0)
opt. (t a))
1.2
test opt.
-1.0
1.0
-1.0
-0.5
0.0
0.5
Figure 4: Generalization risk (red ellipses) and training risk (blue ellipses). The minimizers for the generalization risk and training risk are [x=", trajectory indicates the generalization risk value which decreases in the beginning, but increases back (in-context overfitting). See Figure together.
and [0, x*] (4). respectively. The color of
From 13), the generalization risk (0) ()= = Lerain (0) 2bk can also be expressed as quadratic form sumilar (8)
T ( C + C. C 0
Co
(14)
with Cj C1 + [0, 2b] 2[1, 1] and arg ming Lusa -
[K" T
addition, List = Liriin
20*** (d+1)Lotnle is in between Lg (at zero) and Lp (at random) from x+d+1
9.. If we are given many demonstrations with large n, then the trained model nearly performs the demonstration task which is a random task for the irrelevant query task.
Figure 4 visually demonstrates how the generalization risk behaves on the trajectory 0(t). From the generalization risk has the minimizer (0) on the a-axis, while the training risk (A) on the k-axis. With the two elliptic level sets for the generalization risk (red) and training risk (blue), can expect that the generalization risk decreases as Ott) flows along the sharper direction -V+, and then it increases as (t) get close 0* following v.
5 Conclusion
In this paper, we investigate how the two modes of ICL emerge and disappear during pretraining. By introducing new simple settings, demonstration-query task irrelevance and noncentral task distribution, we can separately analyze the two modes of ICL and show two interesting phenomena: prior forgetting and in-context overfitting. Due to the simplicity of the analysis, we hope that our insights will motivate the future work toward understanding ICL.
10

---

Acknowledgments
We thank the anonymous reviewers for insightful reviews. This work was partially supported by Institute of Information & communications Technology Planning & Evaluation (IIPP) grants (RS-2020-11201373, Artificial Intelligence Graduate School Program (Hanyang University); ) RS- 2023-002206284, Artificial intelligence for prediction of structure-based protein interaction re- flecting physicochemical principles), the National Research Foundation of Korea (NRF) grants (RS-2023-00244896, Implicit bias of optimization algorithms for robust generalization of deep learning; the BK21 FOUR (Fostering Outstanding Universities for Research) project; NRF- 0024SAA53AA020
Socio- Technological Solutions for Bridging the Al Divide: A Blockchain and Federated Learning-Baan Based AI Training Data Platform) and Korea Institute for Advanced Study (KIAS) grant funded by the Korean government (MSIT).
References
K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024a.
K. Ahn, X. Cheng, M. Song, C. Yun, A. Jadbabaie, and S. Sra. Linear attention is (maybe) all you need (to understand transformer optimization). In The Twelfth International Conference on Learning Representations, 2024b. URL htttps://///perrrrr// htt.:///ss....trrrrooooooo.O11//
E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL |ttLpssp///ppeeeeeeeiee htwp://wwwwe.eeeeoorrrrdddddaeee not//forrun77dd00000044111111111
S. Arora, N. Cohen, and E. Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. International conference on machine learning, pages 244-253. PMLR, 2018.
S. Arora, N. Cohen, N. Golowich, and W. Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2019a. URL https://openrevier. www.nnnttoooum.dd-/ntiua...
S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization, Advances in Neural Information Processing Systems, 32, 2019b.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 118887799991 2020.
D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma. Z. Sui, and F. Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers.
A. Rogers, Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4005-4019, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v12223..nmmaa findings-acl..27..
www.gy..rrg/2200 2023/ .nne3...ine findings- acl httpthttt:///aaaaataaaa
247
S. S. Du, W. Hu, and J. D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
Y. Gao, S. Mahadevan, and Z. Song. An over-parameterized exponential regression. arXiv preprint arXiv:2303. 2023.
S. Garg, D. Tsipras, P. S. Liang, and G. Valiant. What can transformers learn in-context? a case study of simple function classes. Advances Neural Information Processing Systems, 35:30583-30598, 2022.

---

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
H. Jiang. latent space theory for emergent abilities large language models. arXiv preprint arXiv:2304.09900 2023.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 .00361, 2020.
Li, Z. Song, Xia, T. Yu, and T. Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304. 13276, 2023a.
Y. Li, M. E. Ildiz, D. Papailiopoolos, and S. Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565-19594, PMLR, 2023b.
Z. Lin and K. Lee. Dual operating modes of in-context learning. In Forty-first International Confer- ence on Machine Learning, 2024. URL
X. Lyu, S. Min, Beltagy, I.. Zettlemoyer, and Hajishirzi. 7-icl: Zero-shot in-context learning with demoosttdmmmnstttiionnnss In The 6l Annual Meeting Of The Association For Computational Linguistics, 2023.
A. V. Mahankali, T. Hashimoto, and Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL https::/
S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyrr. Re- thinking the role of demonstrations: What makes in-context learning work? In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064 Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10. 22533/112222..emppmmaii 7599 URL https://aclant
org/ 2022. emnlp-main 759
J. Pan, T. Gao. H. Chen, and D. Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 2998-831,, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 11813/1112222..... 26663////222...fnnnnngee122222 URL https: http:/acllla /at///acaanntooooy.oooooo org 202g/2222...2 findings-accll. 527
A. Raventós, M. Paul, F. Chen, and S. Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems. pages 1-7, 2021.
M. E. Sander, R. Giryes, T. Suzuki, M. Blondel, and G. Peyré. How do transformers perform in-context autoregressive learning In Forty-first International Conference on Machine Learning. 2024. /ttLptttss//opeereiiie
I. Schlag, K. Irie, and Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355-9366, PMLR, 2021.
F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227. PMLR, 2023.
Z. Shi, J. Wei, Z. Xu, and Y. Liang. Why larger language models do in-context learning differently? In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44991-45013. PMLR, 21-27 Jul 2024. URI. https::/prooee
mlr. wtrr.ppss////e vatse/v23///// shi24f html
12

---

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need, Advances neural information processing systems, 30, 2017.
Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMLR, 2023a.
J. Von Oswald, M. Schlegel, A. Meulemans, S. Kobayashi, E. Niklasson, N. Zucchet, N. Scherrer, N. Miller, M. Sandler, M. Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint urXiv:2309.05858 2023b.
B. Wang, S. Min. X. Deng, J. Shen, Y. Wu, L. Zettlemoyer, and H. Sun. Towards understanding chain-of-thougee prompting: An empirical study of what matters. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
X. Wang, X. Tang, W. X. Zhao, and J.-R. Wen. Investigating the pre-training dynamics of in-context learning: Task recognition vs. task learning. arXiv preprint arXiv:2406. /4022, 2024a.
X. Wang, W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36, 2024b.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL htt.t:t://ppereeiieeeeee http:///nrrrviiee.eeeeoooorrraaa fett/ Survey Certification.
Wei, J. Wei, Y. Tay, Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang. D. Zhou, et al. Larger language models do in-context learning differently. arXiv preprini arXiv:2303.03844 2023.
N. Wies, Y. Levine, and A. Shashua. The learnability of in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview net/ forum?id=f3JNQdd7
S. M. Xie, A. Raghumathan, Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. International Conference on Learning Representations, 2022. URL
https://openrevier. mett/Eorumi.dddddJCCCC
K. M. Yoo. Kim, H. Kim, H. Cho, H. Jo. S.-W. Lee, S.-g. Lee, and T. Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 2422 2437, 2022.
R. Zhang, S. Frei, and I.. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55 2024.
C. Zheng, W. Huang, R. Wang. G. Wu, J. Zhu, and C. Li. On mesa-optimization in autoregressively trained transformers: Emergence and capability. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:///pperreveew.ettttt net.iiewwwweetteeee forrommmi E2BYPreuU8
13

---

A Some Notations
dEN n N
(covariate/task dimension) (number of demonstrations)
Z, Zest E (((+1)((++1)
(input context matrix)
X, Xus Rdx(n+t)) X EdR
Y, Yest E BIx(nt1), 8 E R1x((+1)
x (1)
Prest E Rdd
(in-context covariate; = ... 1.,,
(n+ 1)
2 Best
(n+1)
E Rd
(query input)
x NOa. La)
(covariate distribution)
G2 1 XX
Σ
x (0 x (
E IR
dxd
((uncentered) sample covariance matrix)
1=1
E R
(in-context task)
w E Rd
(query task)
1 E gd
(task performed by the transformer)
Dw NH, Ew)
(task distribution)
14 E Rd
(task center)
Ew I²1a
(task covariance)
tr(Ew) = 0² dd E
(task dispersion)
())) w T x (1) E RR
(target response; = 1.2, 1)
Pust = w Fisst
(0)
(6) E ER
(i = 1.2,...,n)
(n+1) Wy Festt
(a+1)
Viest
E 1R
TrQQ ((d+1)x(++1) - R
(single-layer (full-parameter) linear Transformer)
To (x)+)( + R
(two-parameter Transformer)
LSAp PZMZ QZ
(linear self-attention)
P
Oaxd
0g
E (d+1))x(4+1)
(P for the full-parameter transformer)
P
Oaxd
0g
R (d++))((++))
(P for the two-parameter transformer)
ap
Q
Q
0g
R [dd+1x(d+1)
(Q for the full-parameter transformer)
q
0
Q
Laxd
0g
((++))((++))
(Q for the two-parameter transformer)
0
0
M
In
0
(((+1))((+D)
(masking matrix)
0
0
0 Ja, k]
RR2
(two-parameter parameter)
E R
(prior strength)
-K E R
(in-context strength)
Luain E R
(training loss)
Liest E R
(generalization/test loss)
AL Lusst Lirain R
(generalization gap)
14

---

B Transformer Parameterization and Task s
As shown below, the first d rows of and the last column of Q? do not affect the output Tr: so we put them O. We also show that the transformer performs vi -QQ + qwi G + kw (
Z + I LLSA = Z+ 7
PZMZ TIZZQQQ QZ
71
X
In
On
[XT
Y
Q
X
Y
T
P
K
Y
X
1
T
On
/
x
[x-
Y
Y
p
T
X
+ xY
p
0
9
y
X
0g
X
Y
Q
...]
x
Y
+
p
x
x((+1)T
Y
1
(z) = + ISA
n
d+1,411
XT
PT
0
X
Yent1
pp [
0]
x(a+1)T
Y
En+1
71
XT YT
Q
1)
x 0l
(n+1)T
n
9
T Y q
(n+1)
[D + KKP 0]
n
c(++)TTQ
0
In+
100 X +X8)( D+
-x (n+1)T
[Q XX + + XYT qr X p + kgy Y y
(n+1)T
Q
E ( x
T
p + ky T (1) + 7 qu ( + kqy
2m1
(x+1T
Q
(i) m
T
p + kw x a (0) + w x gz (1) + ugqw" T (1) w
(n+IT
Q x (8) quo T x (20 (1) m)
t=1
(x+)T
+ qw T )0 (0T (p + kw)
(++1T
[- + qw Gr (p kw)]
which proves the equation 0 p ax,, we have
Moreover, for the two-parameter transformer with Q = Ia.
i -QQ +ww ]G.(P + kw) = Gatu + kw)
which proves the equation
15

---

C Higher Moments of Multivariate Gaussian
Lemma C.I. For of and a matrix A [dddd we have the first four moments as follows:
E E(w] = p..
E ww |1 = ppl + L =: Ma.
R || [=l?) ||a412 + +o
E w Aw] μ Ap + + (T(),,
www w] = (2o² + do² + |||l3) Mr
E www Aw] = (A + A 1 + (o² Tr(A) + u An)
E www ww 2M 2m + (do² ||al1))Mo.
E/w www T w) (2 + do²) + (do² [||a113)
E(ww Awn
= M,(A A A.M - m(A+ A Am + Tr(M,
where m pu
If 2 1. then we have
E E(w]] μ,
E www
Ia =: M,
E w
[2]
|||||2 + d,
w Aw] - / "Aμ + Tr(A),
E ww w] = (2 + d ||m1)²
E Now Aw) (A + A AT) ")=== + (Tr(A) + 1." T. Ap)p,
E. wow
waw 2M² 2m² + (d+ ||
E.
wn 222/| 2 + d) + (d+ 11m11111
E Tow
Aww
= ((1 + A
M - m(A + A J.mm + Tr(MA)M.
we further assume μ Oa then we have
E [w]
0
IE wow
ld:
E [ d,
E ww Aw] = Tr(A),
E www w] = Od:
E www Awj Od:
wuw ww T (2 + d)Ia,
Elw ww w = 2d + d?
E www Aww
= A+ A + Tr(A)
16

---

D Proofs
Proof of Theorem 4.1 From
have
Limin P. Q)
E X
[||w =0]1²]
w
|||
|
1|200 11
2w 200
[]] 2F
2F
w,
120
2.:
10
[13]
1st
2nd
From 7 we can get the first term as follows:
④ Ga(a + kww)
||0]]| = (ap - kw) x
n² G² w + 2kah T G.w + a² TCH. G.H,
x [||00||]] x²w Ex G2 + 2xa4 Ex Ex|C2 a² TT x
E 20
|||00||² n² + or( 2naG, + a²G,
(Lemma
=m+ + x² ir)))
+ x( d+1
"
n+d+1 la x] i 1 1+b
1 [
K
0 T2O,
where
1 E (oa: axx (n 1) 1)arx" xx
G= [x
n1
1
(id + 2) + (n 1)Ta]
(Lemma C1
n + d + Id.
n + d+1 d
tr(G)
11 + +1 G, = / T Gp
n
The second term is
w i0 = w T Gz + kw)
w G.p kw G.
ww Ga + kw G.w,
Ex (w" 1d aw E[ μ + kw T E
aww μ + kw" T w.
E. Ew, & (w [0] a + xi1 + b)
= [1 1 + b)
1
2 CTO. o.
The last term is Ex ||| 1 1+0 b Co.
17

---

Next, we want to calculate the minimizer 0" and the corresponding minimum value. First, as C2 > 0, the training loss has only one critical point which is the minimizer:
V Limin (0") = C20 + Ci = 0,
0¹ = C2 C1
n
1
n + d + 1
1 +
n
[ + b
-1
+ d + 1 6
71
[0]
+ d +
[b]
7
[0]
+ d d+1 [1
We can calculate the corresponding minimum value at the minimizer by plugging in 0" = -C-20 and 0 00, -n/(n/ d+ 1)] into Lirin (0*) as follows:
0 C2 + C. 0
Lurain
Sain(00 (0')
Co
1 c C +o
1 C +1+b
=-(1+ b)
+1+b
n n+ d (d+b)(++1)
1
n+d+1
d+1
n+d+1 Lo.
Proof of Theorem
4.2
0 V LuLini(()) -C20 C1,
((t) ai Ail + cgedal 0"
cge
140%
#++++
where \1, A2 are eigenvalues of Co2, i.c. d = -x+, - and C1 C2 are determined by the initial condition
0
((0) = C1
bi /
+ 02
+
=
0
++++1
Therefore, solving the following system of linear equations
C1 + C2
0
C
C2
2
n+d+1
0
we have
n
C = C2
(n + d + 1) 4+
18

---

Therefore, we have
n ((1) n n d+1 1
V4+
6
Art
b444+
v b
Ast
4
+
[
0
2
(tt)
n
1
Vaysz
V4+32
+ +
4 D
7
V4+02
sinh
t
d
+ V4
8
x(t)
n
1
2+h
/4182
e
V4482
V4+ 6a
E
e
- + { ))
+ K
n n d
V 82
2
2
7
2
e
24b
sinh
V4+
/
+
V 62
cosh
V4+ 62
+ K²
n ++1
/4 + 82
2
2
2
71
2
sTnh(TT) sinh
V4+
t
+
cosh(1)) cosh
V
t
1) + n
n d +
/ 62
2
2
V4+
12
2+h
cosh
t
Tb
+ +x²
n n+d+1
/4 8
2
Proof of Theorem 4.3
If is small, then we can ignore higher order terms and obtain
(4 (2+12)1/2 2(1 (b/2) + O(b²)
and 4
+1 + (b)) For large have
(4 + 133) b(4b 1)1/2= 0
and thus, b+ v 14+1 17 (b)) and v 12 = @(b-1). To summarize, we have the eigenvectors
[11 1+ 1+0(
i
4+
b+ /4+
[11, ((6)]
ifb 1'
[[1, -1+ ((6]
if <<
8_
64
[1. ((166]
ibbi 1
2
Similarly, for the cigenvalues, we have d 2+b+y14
=1
1+ and thus 2
O2+)(()
if
A
(0(b)
ifb>1'
(O(b)
ifb
__
(1+0(1
ifb>1
E Extra Figures
19

---

3.0
2.5
2.0
1.5
0.00
0.25
0.75
1.00 b
1.25
1.75
2.00
Figure 5: The critical point 1
arc6anh(70/86) V arctanh
VI+62 2+
that a'()) changes
7o
its sign from negative to positive becomes earlier as we increase the value of 6.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our theoretical analysis limited to the 2- 2-parameter model, but empirical results demonstrate that the full-parameter model also exhibits similar behavior. See Fig 3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
See Assumptions 3.1 and 3.2 and Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the captions of each Figure (e.g., Fig
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We modify the code from www.gttttt.ooeeoeeeeoooooottttet www.co////wcnnnnnceeeeeeee
LinearTransformes
See the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /niiss.cc/pulli//guu guwdes/ddee/eee Coc//ooooeeesiseeeeeeo hwwpbbmmsssioooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the captions of each Figure (e.g.. Fig
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: There is no experiment that requires such error bars.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our random linear regression experiments do not require a lot resources. We used single A40 GPU, but much smaller one would suffice.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips http:///waappcccooooooooccccaaaa
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is theoretical paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This is theoretical paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See the supplementary material for the license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This is theoretical paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This is theoretical paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLM is used.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur/i: /ttpaarrpp..cc//ooofeeeeeee////s for what should or should not be described.
27


Reviews:
[{"review_id": "iu0jgfIzsB", "reviewer": "Reviewer_EWen", "summary": "This paper focuses on the in-context learning (ICL) ability of large language models (LLMs), particularly during the pretraining phase. The authors investigate the emergence and dynamics of two learning modes, task recognition and task learning. It also reveals two key phenomena: Prior Forgetting and In-Context Overfitting.", "questions": "1. Based on Figure 1 (left) and the definition of in-context overfitting in lines 250–251, there seems to be a one-to-one correspondence between increasing in-context strength and overfitting. However, in the early stage of training, models are typically underfitting. How should this apparent contradiction be understood?\n2. Is there a connection between the minimum point of the $L_{test}$ curve and the minimum of $\\alpha(t)$? If so, could this be explained or analyzed more clearly?\n3. Why do Figures 2 and 3 not use the same task dispersion setting? Using a consistent setup would help make comparisons more interpretable.\n4. Can the findings in this paper offer any practical guidance for improving pretraining strategies or designing better ICL prompts?", "limitations": "No, although limitations are mentioned in the checklist, they are not clearly explained in the main text.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n \n- **Originality:** The study explores ICL dynamics from a pretraining perspective, which provides a fresh and insightful perspective. The identification of Prior Forgetting and In-Context Overfitting as emergent phenomena is novel and thought-provoking.\n\n- **Significance:** Understanding how ICL abilities emerge during pretraining can have a substantial impact on the future design of language models, especially regarding pretraining efficiency and generalization capabilities. \n\n- **Quality:** The paper includes thorough and detailed analysis and experiments, which help substantiate the proposed claims and theoretical framing.\n\n- **Clarity:** This paper is well-written and easy to follow. However, there is room for improvements as I mention below.\n\n\n**Weaknesses:**\n\n1. The proposed framework is conceptually novel, especially in its focus on the emergence and dynamics of task recognition and task learning during pretraining. However, much of the empirical analysis relies on simplified models (e.g., two-parameter or single-layer linear transformers), which may limit the extent to which the framework can be fully validated. This also raises concerns about the generalizability of the findings to large-scale, real-world LLMs.\n\n2. **Clarity**\n- 2.1 Figure 1 (left) lacks clarity: the meaning of the axes, curves, and color coding is not well-explained. \n- 2.2 Notational consistency can be improved. For example, distinguishing vectors (e.g., $\\textbf{x}^{(i)}$ ) from scalars (e.g., $y^{(i)}$) using bold font would enhance readability.", "was_revised": true, "final_justification": "My concerns have been addressed. I maintain my rating.", "rebuttal": ">The proposed framework is conceptually novel, especially in its focus on the emergence and dynamics of task recognition and task learning during pretraining. However, much of the empirical analysis relies on simplified models (e.g., two-parameter or single-layer linear transformers), which may limit the extent to which the framework can be fully validated. This also raises concerns about the generalizability of the findings to large-scale, real-world LLMs.\n\n[A1]\nThank you for the feedback. Beyond the simplified model, we trained a more complex and practical **multi-head ($h=8$) multi-layer ($\\ell\\geq12$) Transformer with softmax activation and residual connection with larger $n\\geq 256$ and about a billion training tokens ($\\geq$ 1B)** (we used AdamW to accelerate the optimization) to validate our claim, and **observed in-context overfitting phenomenon** similar to what is shown in Figure 3 ($L_{\\text{test}}$ decreases and then increases). We will include an additional discussion in the revised version.\n\n---\n>2.1 Figure 1 (left) lacks clarity: the meaning of the axes, curves, and color coding is not well-explained.\n\n[A2.1] We will clarify the exact meaning of the axes, curves and color coding in detail.\n\nFigure 1 (Left) and Figure 2 (Bottom, Left) are the same ($\\alpha$-$\\kappa$ graph), but they are slightly rescaled (we used the same setting; SGD, $n=10,d=5, \\sigma=0.2$, etc.).\n\nWe intentionally chose not to name the axes (e.g., $\\alpha,\\kappa$) in Figure 1 saying \"decreasing prior strength\" and \"increasing in-context strength\", because Fig 1 is in the introduction section where we didn't yet introduce the definition of $\\alpha,\\kappa$.\nWe already explained the color coding in Figure 1 (the trajectory \"from magenta to lime\").\n\n\n>2.2 Notational consistency can be improved. For example, distinguishing vectors (e.g., $\\mathbf{x}^{(i)}$) from scalars (e.g., $y^{(i)}$) using bold font would enhance readability.\n\n[A2.2] Thank you for the suggestions. We will enhance readability in the revised version.\n\n---\n>Based on Figure 1 (left) and the definition of in-context overfitting in lines 250–251, there seems to be a one-to-one correspondence between increasing in-context strength and overfitting. However, in the early stage of training, models are typically underfitting. How should this apparent contradiction be understood?\n\n[A3] In the early stage of training, model are not overfitting yet. In-context overfitting **emerges in the later phase**.\n\n- In the early stage of training, the model learns (i) the task prior distribution [task recognition (TR); increasing prior strength $|\\alpha|$] and (ii) how to **fit** to the demonstration data [task learning (TL); increasing in-context strength $|\\kappa|$]. \n- In the later phase, as the prior strength $|\\alpha|$ decreases (prior forgetting), the two ICL abilities (TR and TL) **become imbalanced**, and the model becomes **overly** reliant on the demonstration, which we call in-context **over**fitting.\n\n| phase | prior strength $\\vert\\alpha\\vert$ | in-context strength $\\vert\\kappa\\vert$ | TR | TL |\n|---|---|---|---|---|\n| early phase | $\\vert\\alpha\\vert{\\uparrow}$ (prior learning) | $\\vert\\kappa\\vert{\\uparrow}$ (in-context fitting)| TR$\\uparrow$ | TL$\\uparrow$ |\n| later phase | $\\vert\\alpha\\vert{\\color{red}\\downarrow}$ (prior ${\\color{red}\\text{forgetting}}$) | $\\vert\\kappa\\vert{\\uparrow}$ (in-context ${\\color{red}\\text{over}}$fitting) |TR${\\color{red}\\downarrow}$ |TL$\\uparrow$ |\n\nWe will clarify this point by moving the starting point of the downward \"In-Context Overfitting\" arrow in Figure 1 (left) slightly lower to indicate that in-context overftitting happens in the later phase.\n\n---\n> Is there a connection between the minimum point of the $L_{\\text{test}}$ curve and the minimum of $\\alpha(t)$? If so, could this be explained or analyzed more clearly?\n\n[A4] Figure 4 in Appendix E shows the behaviors of $L_{\\text{test}}$ and $\\alpha(t)$ together.\nAs shown in Figure 4, the point when $\\alpha(t)$ reaches its minimum does **NOT match** the minimum of $L_{\\text{test}}$ on the level set. \n\nInstead, we can provide the following connection between the behaviors of $L_{\\text{test}}$ and $\\alpha(t)$.\nWe have \n$$L_{\\text{train}}(\\theta)=\\frac{1}{2}\\theta^\\top C_2\\theta+C_1^\\top\\theta+C_0,$$\n$$L_{\\text{test}}(\\theta)=\\frac{1}{2}\\theta^\\top C_2\\theta+{\\color{red}{C_1'}}^\\top\\theta+C_0$$\nwhere ${\\color{red}{C_1'}}=C_1+[2,2]^\\top$ and\n$$\\frac{dL_{\\text{test}} }{d\\theta} = C_2\\theta+{\\color{red}C_1'}=\\frac{dL_{\\text{train}} }{d\\theta}+[2,2]^\\top=-\\dot\\theta+[2,2]^\\top$$\nwhich yields the following:\n$$\\frac{dL_{\\text{test}} }{dt}=\\frac{dL_{\\text{test}} }{d\\theta}\\frac{d\\theta}{dt}= (C_2\\theta+{\\color{red}C_1'})^\\top (C_2\\theta+C_1)=(\\dot\\theta-[2,2]^\\top)^\\top\\dot\\theta$$\n$$ = \\\\|\\dot\\theta^\\top-[1,1]^\\top\\\\|^2-2= (\\alpha'(t)-1)^2+(\\kappa'(t)-1)^2-2$$\nTherefore, if **the condition $(\\kappa'(t)-1)^2=1$ (e.g., $\\kappa'(t)=0$) holds**, then $\\alpha'(t)=0$ implies $\\frac{dL_{\\text{test}}}{dt}=0$, i.e, $L_{\\text{test}}$ is minimized at the minimum point of $\\alpha(t)$.\n\n---\n> Why do Figures 2 and 3 not use the same task dispersion setting? Using a consistent setup would help make comparisons more interpretable.\n\n[A5] Thank you for the suggestion. We will include the plots for $b=0.2,0.6,0.8,3.2$ which cover the settings in the both Figure 2 ($b=0.2,0.8,3.2$) and Figure 3 ($b=0.2,0.6,0.8$).\nWe intentionally used different task dispersion settings for each Figure for better visualization. Figure 2 (trajectory dynamics) is less sensitive to $b$ compared to Figure 3 (loss curves). Thus we use a wider range of $b=0.2,0.8,3.2$ for Figure 2, rather than $b=0.2,0.6,0.8$ for Figure 3.\n\n---\n> Can the findings in this paper offer any practical guidance for improving pretraining strategies or designing better ICL prompts?\n\n[A6] \nWe can imagine many possible examples due to the simplicity of the analysis. For example, based on our understanding, ...\n- we may build a new objective or regularizer (e.g., early stopping, penalty on a small $|\\alpha|$) that mitigates the prior forgetting.\n- we may understand a failure mode of LLMs, especially when irrelevant context is given, e.g., Shi et al. (2023).\n- we may design a method to predict the generalization performance of LLMs by monitoring the parameters.\n\nAs our main goal is to provide a simple analysis, we leave their validation in a practical setting as a future work."}, {"review_id": "EHDkmCwCvp", "reviewer": "Reviewer_NtKW", "summary": "This paper investigates the training dynamics of ICL with transformer, with simplified linear transformer and synthetic linear regression tasks. To decouple and evaluate the task recognition and task learning ability, this paper propose to investigate with demonstration-query task irrelevance. Results mainly show that the model first learns the task learning and the task recognition abilities together in the beginning, but it gradually forgets the task recognition ability to recall the priorly learned tasks and relies more on the given context in the later phase.", "questions": "Q1. With some specific settings in this paper, will it be possible to directly identify the global minimum (like [3]), thus illustrate the main results as the process of parameter convergence?\n\nQ2. Why not train in auto-regressive manner? And will the context length matter the main results?\n\n[3] Ahn K, Cheng X, Daneshmand H, et al. Transformers learn to implement preconditioned gradient descent for in-context learning[J]. Advances in Neural Information Processing Systems, 2023, 36: 45614-45650.", "limitations": "NA", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The topic is interesting and important. Understanding ICL, from early works which focus on how a learning algorithm can be expressed, to recent works (including this paper) which focus on how the ability emerges from pre-training, helps us to understand, explain and improve LLMs.\n\nThe first noticeable (but not that important) weakness is that latest related works about understanding ICL with transformer are not discussed nor mentioned, e.g., [1][2].\n\nThe main weakness of this paper is critical: the setting has departed too far from real world LLMs to be meaningful. Three levels of departures can be recognized: \n\nFirst, the linear transformer and synthetic linear regression tasks. Though different from real LLMs, the reviewer can accept this one in most cases, as it facilitates research.\n\nSecond, pre-training with explicitly arranged ICL input ([xy,xy,x?]). The reviewer think this is not  to investigate how the ICL ability emerges from pre-training. Because this is not reduction problem any more, as modern LLMs do not pre-train in such cases.\n\nThird, some unique settings in this paper. The reviewer could not understand what the value of loss between an random label that is independent with input means (Demonstration-Query Task Irrelevance). And the mean value of linear weights is specific to linear regression task (quadratic loss), while LLMs do pre-training neither with such tasks nor with such loss.\n\nBased on above judgement, the reviewer has no longer check the result and proof details. \n\n[1] Wu S, Wang Y, Yao Q. Why In-Context Learning Models are Good Few-Shot Learners?[C]//The Thirteenth International Conference on Learning Representations.\n[2] Yang L, Lin Z, Lee K, et al. Task Vectors in In-Context Learning: Emergence, Formation, and Benefit[J]. arXiv preprint arXiv:2501.09240, 2025.", "was_revised": true, "final_justification": "This is a technically solid paper. But some settings and assumptions, and how closely they are related to real-world LLMs, to be valuable, worth further discussion.", "rebuttal": "> The first noticeable (but not that important) weakness is that latest related works about understanding ICL with transformer are not discussed nor mentioned, e.g., [1][2].\n\n[A1] Thank you for the suggestions. We will include **more discussion** on latest related works (e.g., [1], [2]).\n\n---\n> First, the linear transformer and synthetic linear regression tasks. Though different from real LLMs, the reviewer can accept this one in most cases, as it facilitates research.\n\n[A2] Yes, there has been extensive research exploring the **linear transformer** [Ahn et al., 2024a,b, Dai et al., 2023a, Mahankali et al., 2024, Sander et al., 2024, Schlag et al., 2021, Von Oswald et al., 2023a, Von Oswald et al., 2023b, Zhang et al., 2021, Zheng et al., 2024] and the **in-context linear regression task** [Ahn et al., 2024a,b, Akyürek et al., 2023, Garg et al., 2022, Li et al., 2023a, Li et al., 2023b, Lin and Lee, 2024, Mahankali et al., 2024, Raventós et al., 2024, Von Oswald et al., 2023a, Zhang et al., 2024].\n\nRelaxing the linear attention setting, we also trained a more realistic **multi-head ($h=8$) multi-layer ($\\ell\\geq12$) Transformer with softmax activation and residual connection with larger $n\\geq 256$ and about a billion training tokens ($\\geq$ 1B)** (we used AdamW to accelerate the optimization), and **observed in-context overfitting phenomenon** similar to what is shown in Figure 3 ($L_{\\text{test}}$ decreases and then increases). We will provide more discussion for the practical settings (including the above) in the revised version.\n\n---\n> Second, pre-training with explicitly arranged ICL input ([xy,xy,x?]). The reviewer think this is not to investigate how the ICL ability emerges from pre-training. Because this is not reduction problem any more, as modern LLMs do not pre-train in such cases.\n\n[A3] The objective we (and many others) considered is designed to model the **next-token prediction objective** typically used to pretrain modern LLMs.\n\n- We (and many others [Ahn et al., 2024a,b, Mahankali et al., 2024, Von Oswald et al., 2023a, Zhang et al., 2024]) use the prompt $$Z=\\begin{bmatrix}x^{(1)} &\\cdots &x^{(n)} &x^{(n+1)}\\\\\\\\ f(x^{(1)}) &\\cdots &f(x^{(n)}) &0 \\end{bmatrix}\\in\\mathbb{R}^{(d+1)\\times (n+1)}$$\nto treat $(x,y)$ as a combined context token to simplify the analysis.\n\n- Garg et al. (2022) use the prompt \n$$Z=\\begin{bmatrix} x^{(1)}, &\\begin{bmatrix} f(x^{(1)}) \\\\\\\\ 0_{d-1}\\end{bmatrix}, \\cdots, x_n,\\begin{bmatrix} f(x^{(n)}) \\\\\\\\ 0_{d-1}\\end{bmatrix}, x^{(n+1)}\\end{bmatrix}\\in\\mathbb{R}^{d\\times (2n+1)}$$\n\n- Akyurek et al. (2023) use the prompt\n$$Z=\\begin{bmatrix} x^{(1)} & 0_d & \\cdots &x^{(n)} &0_d &x^{(n+1)}\\\\\\\\ 0& f(x^{(1)}) &\\cdots& 0 & f(x^{(n)}) &0 \\end{bmatrix} \\in \\mathbb{R}^{(d+1)\\times (2n+1)}$$\n\n---\n> Third, some unique settings in this paper. The reviewer could not understand what (i) the value of loss between (ii) an random label that is independent with (iii) input means (Demonstration-Query Task Irrelevance). And (iv) the mean value of linear weights is specific to linear regression task (quadratic loss), while LLMs do pre-training neither with such tasks nor with such loss.\n\n[A4] (i) $L\\_{\\text{test}} = \\mathbb{E}[||w\\_q-\\hat w||^2]$, (ii) $w_q$, (iii) $w$, (iv) $\\mu$\n\nOur goal is to **disentangle** the strengths of the task recognition (TR) and task learning (TL) abilities and quantitatively analyze them **separately**.\n\n- The usual setting is to use the same query task as the demonstration task ($w_q=w$) just like our $L_{\\text{train}}$. Then we cannot isolate and quantify each effect independently (**TL+TR**).\n- However, if we use the query task **different** from the demonstration ($w_q\\neq w$) just like our $L_{\\text{test}}$, then we can measure how much the model relies on the prior (**TR**) as it cannot rely on the demonstration.\n\nIn other words, ...\n- [phase 1] $L_{\\text{train}}$ and $L_{\\text{test}}$ are both good (small). This indicates that the model relies not only on the demonstration, but also on the prior (described by the mean vector $\\mu$) that both $w_q=\\mu+s_q$ and $w=\\mu+s$ share  (**TL+TR**).\n- [phase 2] $L_{\\text{train}}$ is good but $L_{\\text{test}}$ is not. This indicates that the model relies more on the demonstration (**TL**).\n\nOur main results show that \n- In the early stage of training [phase 1], the model learns (i) the task prior distribution [TR; increasing prior strength $|\\alpha|$] and (ii) how to **fit** to the demonstration data [TL; increasing in-context strength $|\\kappa|$] **(TR+TL $\\uparrow$)**. \n- In the later phase [phase 2], as $|\\alpha|$ decreases (prior forgetting), the two ICL abilities (TR and TL) **becomes imbalanced**, and the model becomes **overly** reliant on the demonstration, which we call in-context **over**fitting **(TR $\\downarrow$, TL $\\uparrow$)**.\n\n| phase | prior strength $\\vert\\alpha\\vert$ | in-context strength $\\vert\\kappa\\vert$ | TR | TL |\n|---|---|---|---|---|\n| early phase | $\\vert\\alpha\\vert{\\uparrow}$ (prior learning) | $\\vert\\kappa\\vert{\\uparrow}$ (in-context fitting)| TR$\\uparrow$ | TL$\\uparrow$ |\n| later phase | $\\vert\\alpha\\vert{\\color{red}\\downarrow}$ (prior ${\\color{red}\\text{forgetting}}$) | $\\vert\\kappa\\vert{\\uparrow}$ (in-context ${\\color{red}\\text{over}}$fitting) |TR${\\color{red}\\downarrow}$ |TL$\\uparrow$ |\n\n\n---\n> Q1. With some specific settings in this paper, will it be possible to directly identify the global minimum (like [3]), thus illustrate the main results as the process of parameter convergence?\n\n[A5] Yes, with our specific settings, we are able to directly identify the global minimum in [3] **as a special case when $\\mu=0$**.\n\nWith $\\mu=0$, our parameterization $P=\\begin{bmatrix} 0_{d\\times d}& 0_d\\\\\\\\ 0_d^\\top& \\kappa \\end{bmatrix}$ and $Q=\\begin{bmatrix} I_{d}& 0_d\\\\\\\\ 0_d^\\top& 0\\end{bmatrix}$ (see eq (6)) yields the convergence point (see eq (9)):\n$$P^\\ast=\\begin{bmatrix} 0_{d\\times d}& 0_d\\\\\\\\ 0_d^\\top& \\kappa^\\ast \\end{bmatrix}, Q^\\ast=\\begin{bmatrix} I_{d}& 0_d\\\\\\\\ 0_d^\\top& 0\\end{bmatrix}\\\\ \\text{with}\\\\ \\kappa^\\ast=-\\frac{n}{n+d+1}$$\nThis is equivalent to the following, since $P,Q$ represent the same model up to rescaling $P\\leftarrow \\gamma P$ and $Q\\leftarrow \\gamma^{-1}Q$ for a scalar $\\gamma$:\n\n- [A] **Our global minimum** ($\\mu=0$)\n$$P^\\ast=\\begin{bmatrix} 0_{d\\times d}& 0_d\\\\\\\\ 0_d^\\top& 1 \\end{bmatrix}, Q^\\ast=\\begin{bmatrix} {\\color{red}{\\kappa^\\ast}} I_{d}& 0_d\\\\\\\\ 0_d^\\top& 0\\end{bmatrix} \\text{with}\\ \\color{red}{\\kappa^*=-\\frac{n}{n+d+1}}$$\nThis reduces to Theorem 1 in Ahn et al. (2024a) and their equation (7) when $\\Sigma=I$ is shown below:\n\n- [B] **Global minimum in Ahn et al. (2024a) ($\\Sigma=I$)**\n$$P^\\ast=\\begin{bmatrix} 0_{d\\times d}& 0_d\\\\\\\\ 0_d^\\top& 1 \\end{bmatrix}, Q^\\ast=-\\frac{1}{\\frac{n-1}{n}+(d+2)\\frac{1}{n}}\\begin{bmatrix} I_{d}& 0_d\\\\\\\\ 0_d^\\top& 0\\end{bmatrix}=\\begin{bmatrix} {\\color{red}{-\\frac{n}{n+d+1}}} I_{d}& 0_d\\\\\\\\ 0_d^\\top& 0\\end{bmatrix}$$\n\n- Therefore, we have **[A]=[B]**. One more thing we want to emphasize is that, using a (relatively) general parameterization,  Ahn et al. (2024a) can only derive the global optimum. On the other hand, using a simple parameterization, we can derive the exact dynamics of pretraining and obtain the convergence point (optimum).\n\n---\n> Q2-1. Why not train in auto-regressive manner?\n\n[A6] \nAs answered in [A3], Garg et al. (2022) and Akyurek et al. (2023) train the model in auto-regressive manner. Similarly, the objective we (and many others) used also aims to model the next-token prediction objective typically used to pretrain modern LLMs, but we treat $(x,y)$ as a **combined context token** to simplify the analysis.\n\n---\n> Q2-2. And will the context length matter the main results?\n\n[A7] Theorem 4.1-4.2 and Figure 2-3 show that the task dispersion $b=\\mathrm{tr}(\\Sigma_{\\mathcal{W}})=\\sigma^2 d$ plays an important role in determining the objective $L_{\\text{train}}(\\theta)$ and thus the dynamics $\\theta(t)$. The context length $n$ also matters the main results, but does not affect the **trajectory shape** of the dynamics.\n- We can rewrite eq (10) as follows:\n$$\\theta(t)=c*\\underbracket{ (e^{-\\lambda\\_+t}v\\_+-e^{-\\lambda\\_-t}v\\_-+[0,\\sqrt{4+b^2}]^\\top) }\\_{\\text{only depends on $b$, not on $n$}}=c*f(t;b),\\ \\text{where the scaling factor}\\ c=\\frac{n}{n+d+1}\\frac{1}{\\sqrt{4+b^2}}$$\nsince $\\lambda_{\\pm}$ and $v\\_\\pm$ only depend on $b$, not $n$.\nThe context length $n$ can only **scale** the dynamics and does not affect the **shape** of the dynamics. Even in the case of $n\\gg d$, the scaling factor $c\\approx \\frac{1}{\\sqrt{4+b^2}}$ is almost constant with respect to $n$, and $n$ has little to no effect to control the scale."}, {"review_id": "teOkm11ad7", "reviewer": "Reviewer_25wX", "summary": "This theoretical paper considers in-context learning (ICL) and argues, using a simple theoretical model, that two distinct modes of ICL dominate different stages of pretraining. In the early stage, the model relies more on task identification than on in-context examples. In the later stage, in-context examples become more important. These two ICL modes have been previously observed and described. The main novelty is in developing a simple model that captures these behaviors and how they emerge and dominate different pretraining stages. The theory is validated with some experiments.", "questions": "- In Fig 3, the generalization gaps for two and full-parameter attentions are very similar. Is this a coincidence, or are there reasons to expect this result?\n- The demonstration-query task irrelevance appears similar to _concept shift_, which can drive a specialization-generalization transition; see, eg, arXiv:2409.15582. What is the connection between _prior forgetting & in-context overfitting_ and this concept-shift-induced transition?\n- The paper title, _\"Prior Forgetting and In-Context Overfitting\"_, suggests that this ICL behavior is something we want to avoid, but I understand that it is just another mode of ICL which is sometimes desirable (even though it increases generalization gaps in the settings considered in this paper). Do I miss something important that the title is supposed to convey?\n- What are the lessons from this paper that might translate to practical settings? I understand that this paper is purely theoretical, but a discussion of the potential impact (beyond revealing and characterizing interesting phenomena, which I appreciate) would significantly strengthen the paper.\n- The training plateau is quite common in ICL training but seems absent from the current analysis. Can the authors comment on this absence?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- A simple model allows for a controlled study of ICL behavior, generating insights that have the potential to generalize to more complex settings\n- A simple theory captures task recognition vs task learning competition during pretraining\n- The paper is well-structured and mostly clear\n- Derivations are easy to follow\n\n**Weaknesses**\n- Lack of comparison with benchmark algorithms for linear regression like ridgeless regression and/or Bayes optimal for the noncentred prior consider in the paper; it would be instructive to understand when ICL outperforms naive ridgeless regression by specializing to prior and whether prior forgetting makes ICL solution closer to ridgeless regression (which is optimal for centred isotropic prior)\n- The experiments are limited to linear attention; it would be good to see the results validated, even qualitatively, in more realistic settings \n- If accepted, the authors should consider using the extra page to explain and discuss what their insights mean for more practical settings; do we see a transition from task recognition to task learning during pretraining in these settings?", "was_revised": true, "final_justification": "The authors have sufficiently addressed my questions and clarified my points of confusion. I maintain my rating.", "rebuttal": "> Lack of comparison with benchmark algorithms for linear regression like ridgeless regression and/or Bayes optimal for the noncentred prior consider in the paper; it would be instructive to understand when ICL outperforms naive ridgeless regression by specializing to prior and whether prior forgetting makes ICL solution closer to ridgeless regression (which is optimal for centred isotropic prior)\n\n[A1] We are not sure whether we understand your point and would appreciate clarification if we have misinterpreted it.\n\n- Ridgeless regression refers to solving least squares in the overparameterized regime ($d>n$) by selecting the minimum-norm solution. However, if $d>n$, then $n$ demonstrations $\\\\{(x^{(i)},w^\\top x^{(i)})\\\\}_{i=1}^n$ are not enough to determine the true solution $w$. In the paper, we assume $n\\ge d$ and we will clarify this point in the revised version.\n\n- Note that our prior is not designed to pick a Bayes optimal solution among many possible solution, but to model the task distribution. There is only one unique solution when given $n\\ge d$ demonstrations.\n\n- If we use least squares with $n>d$, then we can always find the **unique** optimal solution with zero training loss.\nHowever, our single-layer Transformers are not expressive enough to solve the least squares and cannot achieve zero training loss ($L^*\\_{\\text{train}}\\neq 0$).\n\n---\n> The experiments are limited to linear attention; it would be good to see the results validated, even qualitatively, in more realistic settings. If accepted, the authors should consider using the extra page to explain and discuss what their insights mean for more practical settings; do we see a transition from task recognition to task learning during pretraining in these settings?\n\n[A2] Thank you for the suggestions. Relaxing the linear attention setting, we trained a more realistic **multi-head ($h=8$) multi-layer ($\\ell\\ge12$) Transformer with softmax activation and residual connection with larger $n\\ge256$ and about a billion training tokens ($\\ge$ 1B)** (we used AdamW to accelerate the optimization), and  **observed in-context overfitting phenomenon** similar to what is shown in Fig 3 ($L_{\\text{test}}$ decreases and then increases).\nFor the general Transformer, it is hard to analyze the dynamics of $\\theta(t)$ as we’ve done in the paper with $[\\alpha(t),\\kappa(t)]$.\n\nWe will use the extra page to further elaborate on practical settings in the revised version.\n\n---\n> In Fig 3, the generalization gaps for two and full-parameter attentions are very similar. Is this a coincidence, or are there reasons to expect this result?\n\n[A3] Yes, the generalization gaps for two- and full-parameter attentions are very similar.\nThis similarity arises because \nthe full-parameter representation\n$$P=\\begin{bmatrix}0_{d\\times d}&0_d\\\\\\\\ p^\\top&\\kappa\\end{bmatrix},Q=\\begin{bmatrix} \\bar Q&0_d\\\\\\\\ q^\\top&0\\end{bmatrix}$$\ntends to **converge to the parameters similar to the two-parameter representation**\n$$P=\\begin{bmatrix}0_{d\\times d}&0_d\\\\\\\\ \\alpha\\mu^\\top&\\kappa\\end{bmatrix},Q=\\begin{bmatrix}I_d&0_d\\\\\\\\ 0_d^\\top&0\\end{bmatrix}$$\nup to scaling. A precise analytic explanation remains an open question. We will include the visualization of learned weight for the full-parameter training.\n\n---\n> The demonstration-query task irrelevance appears similar to concept shift, which can drive a specialization-generalization transition; see, eg, arXiv:2409.15582. What is the connection between prior forgetting & in-context overfitting and this concept-shift-induced transition?\n\n[A4] Thank you for the reference. Its settings seem similar to ours ($w(=\\beta),w_q(=\\tilde\\beta)\\sim\\mathcal{D}\\_\\mathcal{W}$).\n\n- We can also validate the main claim \"More data hurts performance when concept shift is strong\" in [c] from our $L_{\\text{test}}^\\*$: \n$$L_{\\text{test}}^\\*=\\frac{d+1}{n+d+1}(1+b)+\\frac{n}{n+d+1}2b$$\n$\\rightarrow$ more data (larger $n$; more weight on $2b$) hurts ($2b>1+b$) if shift is strong ($b$ is large; $b>1$)\n\n||D-Q task irrelevance|concept shift [c]|\n|-|-|-|\n|task distribution|noncentral|central|\n|focus |training dynamics|performance of pretrained model|\n|$d,n$|non-asymp.|asymptotic ($d,n\\rightarrow\\infty,d/n\\rightarrow\\gamma$)|\n|shift measure|task dispersion $b$ of task distribution|$\\kappa,\\theta$ (pairwise; $\\beta\\leftrightarrow\\tilde\\beta$)|\n\n- We also checked other papers [a,b] on specialization-generalization transition. \nWe focus on **the pretraining dynamics**, i.e., we draw graphs of parameter ($\\theta$) or performance ($L$) vs training steps ($t$), while [a,b] focus on the test performance affected by **the diversity of the pretraining task distribution measured by $\\phi$**, i.e., they draw graphs of performance (test error) for a range of $\\phi$.\n\n||prior forgetting| specialization-generalization transition [a,b]|\n|-|-|-|\n|setting (train) |$w=w_q\\sim\\mathcal{D}\\_\\mathcal{W}$ |$w=w_q\\sim\\mathcal{D}\\_\\mathcal{W}$|\n|setting (test) |$w, w_q\\sim_{i.i.d.}\\mathcal{D}\\_\\mathcal{W}$ (independent vectors from the same $\\mathcal{D}\\_\\mathcal{W}$) |$w=w_q\\sim\\mathcal{D}\\_\\mathcal{W}{\\color{red}'}$ (same vector, but the test task distribution is shifted: $\\mathcal{D}\\_\\mathcal{W}\\rightarrow \\mathcal{D}\\_\\mathcal{W}{\\color{red}'}$)|\n|focus |training dynamics|performance of pretrained model|\n|phenomena|TR$\\downarrow$ and $\\vert\\kappa\\vert\\downarrow$ as $t\\uparrow$ in the later phase | Test error $\\downarrow$ as $\\phi\\uparrow$ (especailly, $\\phi\\geq 90^\\circ$)|\n|indep. variable|training time ($t$)|task diversity ($\\phi$)|\n|dep. variable|performance ($L_{\\text{test}},L_{\\text{train}},\\Delta L$) and parameter ($\\theta$)|performance (test error)|\n\n[a] Goddard et al., Specialization-generalization transition in exemplar-based in-context learning (NeurIPSW 2024)\n\n[b] Goddard et al., When can in-context learning generalize out of task distribution? (ICML 2025)\n\n[c] arXiv:2409.15582\n\n---\n> The paper title, \"Prior Forgetting and In-Context Overfitting\", suggests that this ICL behavior is something we want to avoid, but I understand that it is just another mode of ICL which is sometimes desirable (even though it increases generalization gaps in the settings considered in this paper). Do I miss something important that the title is supposed to convey?\n\n[A5] Task recognition (TR) and task learning (TL) abilities are both desirable. \nThe thing is that, in the later phase of training, the TR ability gets **weaker** (prior forgetting). \nYou're right that weak TR ability and a large generalization gap in our setting does **not negatively affect the training risk** (actually the training risk keeps decreasing).\nHowever, we still want to avoid this behavior, especially when given query task requires prior knowledge or the demonstration is not very informative for the task.\nThe statement \"in-context overfitting = increasing TL ability\" is overly simplified (it is always better with the higher TL ability). We will clarify this with \"in-context overfitting = the two ICL abilities (TR and TL) becomes imbalanced and the model becomes overly reliant on the demonstration as TR gets weaker\" (Yes, this still does not negatively affect the training risk). \n\n|phase|prior strength $\\vert\\alpha\\vert$|in-context strength $\\vert\\kappa\\vert$|TR|TL|\n|-|-|-|-|-|\n|early|$\\vert\\alpha\\vert{\\uparrow}$ (prior learning)|$\\vert\\kappa\\vert{\\uparrow}$ (in-context fitting)|TR$\\uparrow$|TL$\\uparrow$|\n|later|$\\vert\\alpha\\vert{\\color{red}\\downarrow}$ (prior ${\\color{red}\\text{forgetting}}$)|$\\vert\\kappa\\vert{\\uparrow}$ (in-context ${\\color{red}\\text{over}}$fitting)|TR${\\color{red}\\downarrow}$|TL$\\uparrow$|\n\n---\n> What are the lessons from this paper that might translate to practical settings? I understand that this paper is purely theoretical, but a discussion of the potential impact (beyond revealing and characterizing interesting phenomena, which I appreciate) would significantly strengthen the paper.\n\n[A6]\nWe can imagine many possible examples due to the simplicity of the analysis.\nFor example, based on the understanding, ...\n- we may build a new objective or regularizer (e.g., early stopping, penalty on a small $|\\alpha|$) that mitigates the prior forgetting.\n- we may understand a failure mode of LLMs, especially when irrelevant context is given, e.g., Shi et al. (2023).\n- we may design a method to predict the generalization performance of LLMs by monitoring the parameters.\n\n---\n> The training plateau is quite common in ICL training but seems absent from the current analysis. Can the authors comment on this absence?\n\n[A7] We also observed a training plateau (before abrupt loss drop) when we used **SGD** (instead of Adam or AdamW) to train the **full-parameter** Transformer. \n\nThis plateau is predicted by the theory of layerwise linear neural network [d,e].\nEspecially, [e] shows that single-layer linear transformer is equivalent to the two-layer linear neural network (LNN) with cubic feature input.\nLet's consider the simplest setup: we want to minimize $L(a,b)=(1-bax)^2$ where $a,b$ represent the parameters of each layer in the two-layer LNN, $x$ the input, and 1 the target ($x\\mapsto ax\\mapsto bax$). Then, $\\dot a =-\\frac{\\partial L}{\\partial a}=2(1-abx)bx$ and $\\dot b=-\\frac{\\partial L}{\\partial b}=2(1-abx)ax$. Thus, $c=ab$ \nfollows the dynamics $\\dot c = a\\dot b+ b\\dot a = 4(1-cx)cx$ which has a sigmoidal solution of $c(t)=\\frac{1}{x(C_1\\exp(-4xt)+1)}$. Thus, the loss starts with a (long) plateau, exhibits a sudden drop, and saturates. This intuition also holds for general LNNs.\n\nNote that we didn't use SGD to train the full-parameter transformer in Fig 3.\nWe will add the corresponding plots that showed a plateau (by using SGD to train the full-parameter Transformer), together with the above discussion.\n\n[d] Saxe et al., Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (ICLR 2014)\n\n[e] Zhang et al., Training Dynamics of In-Context Learning in Linear Attention (ICML 2025)"}, {"review_id": "qa6oAPhXwC", "reviewer": "Reviewer_LaxR", "summary": "This paper investigates two modes of in-context learning (ICL) exhibited by pre-trained Transformer models during testing, with a particular focus on their emergence and dynamic evolution during the pre-training phase. \nThe study first analyzes a simple random linear regression problem using a linear attention mechanism, and then extends to more complex function classes such as two-layer neural networks, decision trees, exponential regression, softmax regression, and autoregressive learning. \nIn addition, the authors introduce a new setting called \"demonstration-query task irrelevance\" to examine the model's robustness and generalization under various distribution shifts.\nThey emphasize the balance of power between the two ICL modes and propose a measurement method to evaluate their strength and how they emerge or fade over time. \nThrough theoretical analysis and experimental validation, the authors demonstrate the phenomenon known as \"in-context overfitting\", where the model's generalization gap increases with greater context strength.", "questions": "See Weaknesses\n\n\nIf the authors could **provide experiments on larger-scale LLMs and more complex datasets**, I would be willing to raise my score.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n+ The paper provides a deep theoretical understanding of the two in-context learning (ICL) modes in pre-trained Transformers, supported by rigorous empirical validation across a variety of task distributions and function classes.\n\n+ The authors introduce a new evaluation setting called \"demonstration-query task irrelevance\", which enables the study of model robustness and generalization under distributional shifts.\n\n**Weaknesses**\n\n+ While the paper provides a detailed theoretical analysis of the two-parameter Transformer model and demonstrates similar behavior to the full-parameter model through experiments, **this simplification may overlook some dynamic changes and characteristics present in more complex models**. Consequently, the findings might not fully represent the behaviors of the more intricate Transformer architectures used in practical applications. Additionally, it's worth noting that LLMs' pre-training typically involves training on datasets at the billion-token scale. Although this study claims to investigate phenomena during the pre-training phase, **its setup does not entirely align with conventional LLM pre-training practices**, which could be considered one of its primary limitations.\n\n+ It would be beneficial for the authors to employ **more complex datasets** to analyze the dynamics of in-context learning and evaluate Task Recognition and Task Learning using setups similar to those in other relevant studies [1, 2].\n\n+ Although the paper mentions several hyperparameters, such as prior strength and in-context strength, there could be more extensive experimentation to explore how these parameters influence model performance.\n\n[1] What In-Context Learning \"Learns\" In-Context: Disentangling Task Recognition and Task Learning, ACL 2023.\n\n[2] Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning, ICLR 2025.", "was_revised": true, "final_justification": "My concerns are fully addressed.", "rebuttal": "> While the paper provides a detailed theoretical analysis of the two-parameter Transformer model and demonstrates similar behavior to the full-parameter model through experiments, this simplification may overlook some dynamic changes and characteristics present in more complex models. Consequently, the findings might not fully represent the behaviors of the more intricate Transformer architectures used in practical applications. Additionally, it's worth noting that LLMs' pre-training typically involves training on datasets at the billion-token scale. Although this study claims to investigate phenomena during the pre-training phase, its setup does not entirely align with conventional LLM pre-training practices, which could be considered one of its primary limitations.\n\n> It would be beneficial for the authors to employ more complex datasets to analyze the dynamics of in-context learning and evaluate Task Recognition and Task Learning using setups similar to those in other relevant studies [1, 2].\n\n[A1+A2] Thank you for the feedback. Relaxing the simple setting, we trained a more complex and practical **multi-head ($h=8$) multi-layer ($\\ell\\geq12$) Transformer with softmax activation and residual connection with larger $n\\geq 256$ and about a billion training tokens ($\\geq$ 1B)** (we used AdamW to accelerate the optimization), and **observed in-context overfitting phenomenon** similar to what is shown in Figure 3 ($L_{\\text{test}}$ decreases and then increases). We will include a discussion for more complex models (including the above) in the revised version.\n\nDue to limited time and resources, we were unable to analyze more complex datasets, but it is also expected to observe transient nature of in-context learning for more complex tasks other than the simple regression task as similar phenomena have already been demonstrated empirically in previous studies (e.g, [a] for a synthetic Omniglot classification tasks and [b] for a Markov mixture tasks).\n\n[a] Singh et al., The transient nature of emergent in-context learning in transformers (NeurIPS 2023)\n\n[b] Park et al., Competition dynamics shape algorithmic phases of in-context learning (ICLR 2025)\n\n\n---\n> Although the paper mentions several hyperparameters, such as prior strength and in-context strength, there could be more extensive experimentation to explore how these parameters influence model performance.\n\n[A3] First of all, prior strength $|\\alpha|$ and in-context strength $|\\kappa|$ are **not hyperparameters** that we set before training. They are **parameters** that the model learns during training.\n\n- (How $\\kappa$ influence model performance) For example, the generalization gap $\\Delta L = -2b \\kappa=2b|\\kappa|$ is proportional to $|\\kappa|$ so the generalization gap increases with the in-context strength $|\\kappa|$ (in-context overfitting).\n\n- (Hyperparameters 1)\nTheoretically, we have the following four hyperparameters: $\\sigma^2,d,n$, and $b=\\sigma^2 d$.\nTheorem 4.1-4.2 and Figure 2-3 show that the task dispersion $b=\\sigma^2 d$ (controlled jointly by task variance $\\sigma^2$ and task dimension $d$) plays an important role in determining the objective $L_{\\text{train}}(\\theta)$ and in shaping the trajectory $\\theta(t)$.\nEq (10) can be reformulated as follows:\n$$\\theta(t)=c*\\underbracket{ (e^{-\\lambda\\_+t}v\\_+-e^{-\\lambda\\_-t}v\\_-+[0,\\sqrt{4+b^2}]^\\top) }\\_{\\text{only depends on $b$, not on $n$}}=c*f(t;b),\\ \\text{where the scaling factor}\\ c=\\frac{n}{n+d+1}\\frac{1}{\\sqrt{4+b^2}}$$\nsince $\\lambda_{\\pm}$ and $v\\_\\pm$ only depend on $b$, not $n$.\nTherefore, the task dispersion $b=\\sigma^2 d$ ($\\sigma^2$ and $d$ together) **control the shape of the trajectory $\\theta(t)$**, but the context length $n$ can **only scale the dynamics** and does not affect the shape of the dynamics. \nEven in the case of $n\\gg d$, the scaling factor $c\\approx \\frac{1}{\\sqrt{4+b^2}}$ shows minimal sensitivity to $n$, and $n$ has little to no effect to control the scale.\n| Hyperparam. | $b= \\mathrm{tr}(\\Sigma_{\\mathcal{W}})=\\sigma^2d$ | $n$ |\n| --- | --- | --- |\n| Name | task dispersion = (task variance)*(task dimension) | number of demonstration (context length) |\n| Control | shape of the trajectory | scale of the trajectory |\n\n- (Hyperparameters 2) Empirically, we can name the following additional hyperparameters and **summarize their influence on the dynamics**:\nbatch size, learning rate, optimizers (and their internal hyperparameters), initialization, number of layers, non-linear activation (e.g., softmax).\n    - [batch size and learning rate] As our theory is based on the gradient flow, when we used smaller batch size or larger learning rate, we observed more noisy curves for Figure 3, but they, if smoothed, still follow the theoretical curve and the fundamental understanding remains the same.\n    - [optimizer] As our theory is based on the gradient flow, when we used adaptive methods (e.g., Adam or AdamW), we observed loss curves somewhat different from the one in Fig 3, but still observed in-context overfitting with similar values of $L_{\\text{train}}^\\ast,L_{\\text{test}}^\\ast,\\Delta L^\\ast$.\n    - [initialization] not very sensitive and it just needs to be small\n    - [number of layers] Using multi-layer Transformer, we observed a loss plateau (before a sudden loss drop). This plateau is predicted by the theory of layerwise linear neural network [c,d]. Especially, [c] shows that single-layer linear transformer is equivalent to the two-layer linear neural network (LNN) with cubic feature input. Let's consider the simplest setup: we want to minimize $L(a,b)=(1-bax)^2$ where $a,b$ represent the parameters of each layer in the two-layer LNN, $x$ the input, and 1 the target ($x\\mapsto ax\\mapsto bax$). Then, $\\dot a =-\\frac{\\partial L}{\\partial a}=2(1-abx)bx$ and $\\dot b=-\\frac{\\partial L}{\\partial b}=2(1-abx)ax$. Thus, $c=ab$ follows the dynamics $\\dot c = a\\dot b+ b\\dot a = 4(1-cx)cx$ which has a sigmoidal solution of $c(t)=\\frac{1}{x(C_1\\exp(-4xt)+1)}$. Thus, the loss starts with a (long) plateau, exhibits a sudden drop, and saturates. This intuition also holds for general LNNs. Note that we didn't use SGD to train the full-parameter transformer in Figure 3. We will add the corresponding plots that showed a plateau (by using SGD to train the full-parameter Transformer), together with the above discussion.\n    - [non-linear activation (softmax-based attention)] Empirical results for softmax-based attention also show in-context overfitting qualitatively similar with the theoretical prediction for the two-parameter model (with similar values of $L_{\\text{train}}^\\ast,L_{\\text{test}}^\\ast,\\Delta L^\\ast$).\n\n[c] Saxe et al., Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (ICLR 2014)\n\n[d] Zhang et al., Training Dynamics of In-Context Learning in Linear Attention (ICML 2025)"}]

Author rebuttal:

Meta-review: (a) summary: The paper investigates the phenomenon of in-context learning (ICL). Instead of the previously addressed test-time analysis, it focuses on the pre-training phases and uses a simplified linear-attention based transformer and a set of linear-regression tasks to enable theoretical analysis in mathematically tractable form. The analysis looks at the dynamics of task recognition and task learning abilities and reveals two key aspects of the ICL named in the paper prior forgetting and in-context overfitting.

(b) strength: The paper investigates an important phenomenon and puts forward fresh insights with possible impact on future developments in LLMs. The theoretical analysis is solid and thorough and is supported by relevant experiments. The paper is well written and structured.

(c) weaknesses: The simplifying assumptions (linear transformer, linear regression tasks) raise doubts about its validity and generalization to more complex and more realistic set-ups used in real-world LLMs.

(d) reasons to accept: ICL is interesting and important topic for better understanding of LLM behavoiur and abilities. The presented ICL analysis is novel, technically strong and though relying on unrealistically simplified setup provides insights that may serve as a useful stepping stone towards more complex analysis in the future. For these reasons I am leaning to accepting the paper.
At the time some doubts about the generalizability and practical relevance of the presented results to real-life and more complex LLM prevail. I would therefore not oppose if the paper was dumped down. 

(e) review discussion: Reviewers appreciated the novel approach to the analysis (pre-training instead of test-time only) and the general importance and hence interest of the in-context learning phenomenon. In addition to asking for some technical clarifications, they expressed their concerns about validity and practical relevance of the simplified set-up in realistic and far more complex LLM pipelines. In addition to the simplified models, the whole pre-training setup of the \emph{demostration-query task irrelevance} has been questioned as inconsequential for any real-life LLM settings. Authors answered the technical questions in the rebuttal to the satisfaction of the reviewers and where appropriate suggested minor clarification updates for the camera-ready. To address the generalization question, they have conducted additional experiment on more complex soft-max based transformer architecture, claiming similar observed behaviours and proposed to cover this in the final version. They also clarified the pre-training setup and its motivation and relevance for actual LLM pre-training. Though some doubts about the generalization of the simplified approach and hence its validity and usefulness for real-life LLMs prevailed, the reviewers were generally satisfied with the rebuttal answers and appreciated the theoretical strengths of the analysis.
Decision: Accept (poster)