Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
Authors: Blake Bordelon
Mary Letey
Cengiz Pehlevan
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Learning
scaling laws
in-context learning
transformers
attention
Primary area: learning theory
TL;DR: A theory of scaling laws for ICL regression that predicts optimal width/depth shapes.

Abstract: We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends on various computational and statistical resources (width, depth, number of training steps, batch size and data per context). In a joint limit where data dimension, context length, and residual stream width scale proportionally, we analyze the limiting asymptotics for three ICL settings: (1) isotropic covariates and tasks (ISO), (2) fixed and structured covariance (FS), and (3) where covariances are randomly rotated and structured (RRS). For ISO and FS settings, we find that depth only aids ICL performance if context length is limited. Alternatively, in the RRS setting where covariances change across contexts, increasing the depth leads to significant improvements in ICL, even at infinite context length. This provides a new solvable toy model of neural scaling laws which depends on both width and depth of a transformer and predicts an optimal transformer shapes as a function of compute.

Published as a conference paper at ICLR 2026
THEORY OF SCALING LAWS FOR IN-CONTEXT RE- GRESSION: DEPTH, WIDTH, CONTEXT AND TIME
Blake Bordelon Mary L Cengiiz Pehlevan oo
Center for Mathematical Sciences and Applications
John A. Paulson School of Engineering and Applied Sciences ?
Kempner Institute for the Study of Natural and Artificial Intelligence ? Harvard University
ABSTRACT
We study in-context learning (ICL) of linear regression in deep linear self- attention model, characterizing how performance depends on various computa- tional and statistical resources (width, depth, number of training steps, batch size and data per context). In a joint limit where data dimension, context length, and residual stream width scale proportionally, we analyze the limiting asymp- totics for three ICL settings: (1) isotropic covariates and tasks (ISO), (2) fixed and structured covariance (FS), and (3) where covariances are randomly rotated and structured (RRS). For ISO and FS settings, we find that depth only aids ICL performance if context length limited. Alternatively, in the RRS setting where covariances change across contexts, increasing the depth leads to significant im- provements in ICL, even at infinite context length. This provides a new solvable toy model of neural scaling laws which depends on both width and depth of transformer and predicts an optimal transformer shape as a function of compute. This toy model enables computation of exact asymptotics for the risk well derivation of powerlaws under source/capacity conditions for the ICL tasks.
INTRODUCTION
In recent years, transformer models have become the backbone architecture for modern machine learning systems (Vaswani et al., 2017; Dosovitskiy et al., 2020). The transformer architecture consists of alternating self attention and multi-layer perceptron blocks in a deep residual network. For variety of tasks, increasing the size of transformers by increasing width and depth of the model leads to empirically predictable improvements in model performance (Hestness et al., 2017; Kaplan al., 2020; Hoffmann al., 2022; Achiam al., 2023). Various protocols for scaling up the width and depth of transformers have been developed that provide stable training limits (Yang et al., 2021; Bordelon al., b; 2024b; Chizat & Netrapalli. 2024). One common strategy is to scale width and depth L linearly with fixed aspect ratio L/N (Noci et al., 2023; Dey et al., 2025), however existing theory cannot justify such a practice as compute optimal or distinguish relative performance gain from width and depth from total parameter count. This leads us to our first open question:
Q1: What sets optimal Transformer shapes and scaling laws?
How should width and depth scale under compute budget in a transformer? Do transformer scaling laws only depend on total parameters or have different width and depth dependence?
We explore this for in-context learning (ICL) problems, specifically in-context linear regression problems. In-context learning refers to the ability of models to condition their outputs on the past sequence of inputs provided by the user without updating model parameters (Brown et al., 2020). This contrasts with in-weight learning (IWL) where information about the dataset is encoded in the model parameters during pre-training. Whereas prior theoretical works on neural scaling laws es- sentially analyze the role of width, pretraining data, pretraining time (Bahri et al., 2021; Maloney et al., 2022; Simon et al., 2023; Bordelon et al., 2024a; Paquette et al., 2024: Lin et al., 2024; Bor- delon et al., a), our work explores how depth, width, context length, and pretraining time influence the quality of ICL. We find that the architectural requirements to perform ICL regression depend significantly the statistics of the pretraining ICL tasks. We are thus forced investigate:

---

Published as a conference paper at ICLR 2026
Q2: What task properties influence ICI. solution
How does the statistical structure of ICL. tasks influence the learned solution?
To address these questions, we develop solvable model of deep linear attention for three distinct ICL data covariance structures. Our concrete novel contributions and findings are as follows:
1.1 CONTRIBUTIONS
1. We identify a new asymptotic scaling for in-context regression with linear attention where context length P, number of masked points per context K, and contexts per step B all scale linearly with dimension This reduces the amount of compute and total data (by factor of D) to converge than prior works (Lu et al., 2025) and speeds up simulations.
2. We introduce three distinct ICL data models of increasing generality. We study isotropic input data and task vectors (ISO). We then proceed to analyze fixed and structured (FS) covariances for data and task vectors. Lastly, we examine an ICL distribution where data covariances are randomly rotated and structured (RRS) across contexts.
3. We show with asymptotic theory, that depth I, benefits the first two data settings (ISO + FS) only if a = context length/ / dimension is finite. In the limit of a + oo, there is no benefit from depth for these ICL tasks. Further, the FS setting is brittle with respect to variations in changes to the data covariance at test time. However, for the RRS setting, increasing depth is beneficial for any distribution and a non non-trivial width & depth scaling.
4. We introduce model width N bottleneck our ICL model and study the pretraining scaling law for model trained for steps on a width N and depth L linear attention model in the RRS setting. For powerlaw data, the scaling law takes the form of a Chinchilla scaling law but with width and depth contributing separate terms
L), = att t + eN N + cr, L + cp P-Pr.
5. From this scaling law, we consider compute optimal joint scaling of width and depth. De- pending on the structure of the ICL covariates, we obtain different scalings of L N" where y depends on properties of the data.
1.2 RELATED WORKS
Infinite Neural Network Width and Depth Limits and Commonly Used Joint Scalings. The empirical fact that larger networks tend to perform better on natural data (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022) has led to development of scaling procedures to sta- bly increase the size of model to allow optimization. The mean-field or μuP scaling theory (Geiger et al., 2020; Mei et al., 2018; Chizat & Bach, 2018; Yang & Hu, 2021; Bordelon & Pehlevan, 2022b) allows one to scale up the width of a model in a way that admits feature learning infinite limit. Fur- ther, this scaling protocol provides consistent optimal hyperparameters, while delivering monotonic improvements in performance (Yang et al., 2021). The same program has been carried out for deep residual models, such as transformers (Bordelon et al., b; Yang et al., 2023; Bordelon et al., 2024b; Dey et al., 2025). However, while these infinite width and infinite depth scaling limits have been established to exist and perform better than finite models, no theory currently captures the relative gains in performance from scaling up width or depth at fixed compute. Understanding compute optimal shapes could help guide architectural choices when training large transformer models.
Theories of Compute Optimal Neural Scaling Laws. Following the empirical scaling law results of Kaplan et al. (2020) and Hoffmann et al. (2022), many theoretical works have examined the generalization theory for fully trained kernel methods under power law features (Bordelon et al., 2020: Spigler et al.. 2020: Cui al., 2021; Bahri al., 2021; Maloney al., 2022: Atanasov al., 2023; 2022; Defilippis et al., 2024). More recently, several efforts have begun to incorporate SGD dynamics into these models to gain a notion of compute (and compute optimal tradeoff's between parameters and training time) (Bordelon & Pehlevan, 2022a; Bordelon et al., 2024a; Paquette et al., 2024; Lin et al., 2024; Kunstner & Bach, 2025; Bordelon et al., a). Lyu et al. (2025) recently extended scaling law theory to ICL by analyzing single layer linear attention model performing ICL on a sparse multitask sparse feature regression problem, resulting in a neural scaling law in time and context length. In these works, the model is essentially one two layers and the notion of finite model size is introduced with a random projection of the features to an N dimensional space. In this way, existing theories more closely resemble scaling laws for width rather than comparison
2

---

Published as a conference paper at ICLR 2026
where depth and width serve different functions. Recent empirical works have pointed out the utilltyy of increasing depth (or virtual depth through looping) for tasks requiring reasoning such as solving Sudoku puzzles (Wang et al., 2025) and solving math problems (Zhu et al., 2025; Geiping et al., 2025). Merrill & Sabharwal (2025) study regular language recognition where a clear computational advantage scaling depth instead of width is established and experimentally verified.
Empirical Studies of ICL. Brown et al. (2020) demonstrated that large pretrained language mod- els such GPT-3 exhibit remarkable in-context learning capabilities for natural language tasks. Following this finding, Garg et al. (2022) empirically investigated the ability of transformers to learn simple function classes such linear regression, sparse regression, and two-layer neural net- works. Various works have studied emergence of ICL beyond language tasks, focusing on what data structures (from dynamical systems classification problems) can be learned in-context (Liu al., 2024b; Chan et al., 2022; He et al., 2024; McCracken et al., 2025; Park et al., 2024; Chen et al., 2024). Several works have offered experimental evidence that ICL implements Bayesian inference (Xie et al., 2021; Wurgaft et al., 2025; Liu et al., 2024a; Panwar et al., 2024; Zhang et al., 2023b). Further investigations questioned whether attention even strictly necessary for ICL by studying the performance of MLPs on these tasks (Tong& & Pehlevan, 2025; Kratsios & Furuya, 2025).
Theoretical Studies of ICL. Inspired by empirical studies of ICL. many works have theoreti- cally investigated how ICL algorithms such as in-context gradient descent can be implemented in transformers (Zhang et al., 2023a; Ahn et al., 2023; Von Oswald et al., 2023; Li et al., 2023; Kim et al., 2024; Akyürek et al., 2022; Pathak et al., 2023; Mahankali et al., 2023). Some works have demonstrated ficxibility of the transformer to adapt to changing statistics (Vladymyrov et al., 2024) or performing model selection (Bai et al., 2023). Studies have pointed out the need for sufficient pretraining task diversity for ICL generalization (Raventós et al., 2023; Wu et al., 2024) which was theoretically analyzed in asymptotic scaling limit for a shallow linear attention model by Lu et al. (2025). Extensions to structured covariances and distribution shifts revealed the importance of train-test task alignment Letey et al. (2025). Gatmiry et al. (2024) investigate the need for a suffi- cient number of residual stream steps (either by increasing true depth or loops of attention layers) to solve ICL distributions with multiple condition numbers with larger condition numbers requiring more depth. While many of these works study final construction of weights, recent theory has described the training dynamics of one layer linear multi-head attention (Zhang et al., 2025).
2 DATA, ARCHITECTURE AND REDUCED MODEL
Deep Linear Attention Architecture The most general model we study is a depth L, residual linear attention model f that maps inputs contexts to output predictions. The data are formed as P input-output pairs {Tr,, yn |p= and K evaluation points {(rr,, *)] P+K target outputs) into a data matrix D
p=r++ (which do not carry
D
Ø1 31
EP
EP+1 *
IP+K
UP
where * indicates masked target values on the K evaluation points which are provided as 0 entries. The evaluation tokens μ E (P+ I,... P K} prevented from updating the model with a positional masking matrix Mad (Appendix B). The model fy is computed from
P
1
h1 W. Wyglur
het1 -h
Σ
Mn ((Kt) (..). of E / E [P K]
LP
d- = Wh p ky = Wi on Wi u
J w., = h
where W E NND Wy wo E RN and W, E N for j E kq v}. The loss function for context D is C(D) F Ep=p+1 (fr yn and the full population loss is L = (C(D)) where the average over D represents the distribution of context matrices. We stress that the operation q k corresponds to linear attention rather than commonly used in soft-max attention. For the regression tasks consider, this model is sufficient to solve the ICL task (Ahn et al., 2023; Von Oswald et al., 2023) and aids theoretical tractability (Lu et al., 2025: Zhang et al., 2025).
Recurrent Reduced-l' Model Following prior works on ICL in linear regression (Zhang et al., 2023a; Lu et al., 2025: Wu et al., 2024; Zhang et al., 2025), we examine the minimal (simplest) reparameterization of the above model which can solve this task where the residual stream encodes x information in a subspace orthogonal to w. = w, so that W Wy = 0 and W,, & w Instead of optimizing separate hidden weight matrices, we consider looped / universal transformers where
3

---

Published as a conference paper at ICLR 2026
each attention block is identical Wi W for E (k.q. (Dehghani al., 2018; Gatmiry al., 2024). We relax this assumption in Section 5 and Appendix G.1 and show that untying layers does not alter the dynamics in settings of interest. The reduced model defines the predictor f(x)) in terms of a single matrix P e Dx
T= (w, W. W W. W.W.. f(x)) 1 LP
Σ 6=
x X (I-L-1ET)
(4)
where -Xx is the empirical covariance for the data matrix X E DDX constructed from the first P context vectors {= with the first P target values y R", Due the simplicity of this model and loss landscape structure, we will first focus on the gradient dynamics when we directly perform gradient descent on the matrix r. Then in Section 5 we consider learning dynamies for untied parameters across depth (non-recurrent models) and in Section 5.1 analyze gradient flow on decoupled attention blocks containing the full collection of parameters { see.ka....
LEARNING CURVES IN REDUCED LINEAR ATTENTION MODEL
3.1 ISOTROPIC COVARIATES AND TASKS (ISO)
To start our investigation, we begin by considering D dimensional isotropic data and isotropic task distribution. For each context De and each data point μ E [P]. the distribution of x and y are
u. N I0
Be NOO, Upe
vD Be wh + ael ee (0,
(5)
In Appendix C.I. we analyze SGD with batch of B contexts sampled each step in the proportional asymptotics P. K, B, D + 0o. P/D = a K/D = K, B/D = T, establishing that successful pretraining requires total of Bt ((D) contexts, each of size P (D).
3.0
Theory
0.8
0.6
0.6
0.4
62
02
0.2
1000 2000
3000
4000
5000
6000
3000
2000
3003
4000
5000
6000
10a
20a
10
L 16, Linear Transformer
(b) 1, Linear Transformer
(c) Final Loss of The Transformer
Figure 1: Deep linear self attention models trained with SGD the ICL task with isorooicc covari- ales with 32. (a) Training dynamics for varying a. (b) Increasing depth can improve ICL predictions, especially for x 1. (c) The final loss compared to theory of L steps of GD.
These stochastic gradient descent fluctuation effects, while potentially interesting, are not the pri- mary focus of the paper. Moving forward, we will instead focus on gradient flow deeper models L 221 1. For isotropic covariates, we have the following result.
Result 1:
Gradient Flow for Isotropic Covariates (ISO)
Gradient flow ICL pretraining isotropic data any depth L > 1 from zero init yields
T(t) It)
(6)
where the dynamics obey a gradient flow for this scalar 11(1) = -0, aalice) for
L(7,a)= f p(),a) [ (1-
26
(1--11----
0²
(7)
ad
The density p(), ai) is the Marchenko-Pare Pastur law with P/D.
The final loss (a) min, L(y, a) can be interpreted as the loss for L steps of GD with optimal step size. We illustrate this effective loss function in Figure 2. Consider o² O. For L. 1, the
4

---

Published as a conference paper at ICLR 2026
loss saturates to L. (1 + a)- while for L 3o, the L. -1--a al_, illustrating gap in performance between shallow model and deep models at finite a. Loss curves varying L compared to linear transformers (dots)
shown in Figure I (c) and the [(7) in Figure (see App. C.2). For larger noise x² (Figure 2 c), the optimal is smaller since early stopping acts as an effective regularization (Advani al., 2020; Sonthalia al., 2024).
10
10
0-00
0.8
0.6
0.0
0.
1.
0.6
Y/L
Y/L
Y/L
(a) Loss Landscape 0.5
(b) Loss Landscape 2.0
(c) Effect of Noise T. = 16
Figure 2: The loss landscape for the reduced I model with r = I corresponding to the gradient flow limit. This limit is equivalent to optimal step size selection for in-context GD. (a)-(b) The effect of depth L and context length on the loss. (c) Larger noise decreases the optimal y.
Result 2: Depth is Unnecessary for Long Contexts Isotropic Covariates
a P/D + oo then L 1 achieves the minimal ICL loss (among any depth models). Any larger depth L 2 2 achieves the same (zero if o² 0) loss in the co limit.
3.2 FIXED STRUCTURED COVARIANCE (FS) + PRECONDITIONED GRADIENT DESCENT
Next, we let the population covariance be arbitrary (xx" E across all contexts and task corre- lations be given by the matrix (BBT) Ω. The ICL population loss takes the form
L - tr s)) (((((r---))]])
E -II-
(8)
Result 3: Depth Unnecessary for Long Contexts with Fixed Covariance
When the ICL distribution involves fixed covariance across contexts, there is benefit to increasing depth L beyond L = 1 in the large context a - co limit. For any L 2 1, zero ICI. loss can be achieved in the a - oo limit by setting r -E
We support this finding in Figure 3 (b) where we show that small depth models are not outperformed by deeper models even after very long training horizons. When the ICL pretraining distribution for Σ is fixed, the model will memorize statistical information about the covariance the inputs from the pretraining distribution. By preconditioning with the inverse of the data covariance Σ --1 the model is capable of achieving zero loss after even a single step of in-context GD. The gradient flow dynamics of the I matrix can be decomposed further in the case where Z and Σ commute.
Result 4: Decoupled Eigenvalue Dynamics During Pretraining on Fixed Covariances
Suppose that Ω and Σ are codiagonalizable with respective eigenvalues {wy} and } Then, when training from zero initialization, is diagonal in the same basis with eigenvalues (t) that obey the following dynamics (see Appendix D)
d Jl (1 wa L- (c7r(t)) 26-1
For L oo these dynamics have solution (x(t) = 1 In(1 + 4wx1it) generating the loss dynamics limı,-soe L(1, I.) Σe
2dx
wedt 7 Under powerlaw conditions A -- and
Eesh dews ~ k-v the ICL loss scales powerlaw L(t) tt --SNTT
5

---

Published as a conference paper at ICLR 2026
102
109
Edoo
200
10a4
Thoory
150
10a
122
107
10f
10a
10r
12r
30"
10a
3.50
0.05
0.10
2.35
0.20
0.25
(a) Eigenvalue Evolution
(b) Depth unnecessary
(c) Brittleness Distribution Shift
Pretraining FS ICL covariates leads to solution that does require depth but is brittle to distribution shift. (a) Evolution of the eigenvalues (x(() the (t) matrix for depth function of pretraining time compared with infinite depth L - theory (dashed black). (b) For powerlaw covariates, all depth models converge as a power law in There is no asymptotic benefit increasing depth beyond L 1. (c) The ICL solution obtained when training from fixed covariance is brittle to changes in the covariance Σ - exp(-0S). exp(OS).
Figure 3:
While pretraining the linear transformer in this setting can achieve zero loss for long contexts P + oo even in shallow networks, the solution the model finds from gradient descent is brittle to properties of the pretraining data. Rather than learning a generic algorithm that solves ICL re- gression for any covariance Σ, solution is specialized to the pretraining covariance.
Result 5: Brittleness of Fully Trained Model to Distribution Shift (OOD Loss)
depth I, model is pretrained on ICL tasks with fixed covariances E and Ω for inputs and task vectors, but evaluated on new covariances E', It'. The out-of-distribttion loss is
E' LOOD = =tr -t((22 [1- - E'
(10)
We illustrate this brittleness in Figure 3 (c) where we define family of new covariance matrices E' (pp(0))eepp( (-S) where
random skew-symmetric matrix. As increases Σ becomes more dissimilar to E, the OOD loss increases for all depths L.
3.3 RANDOM ROTATED AND STRUCTURED COVARIANCES = IN-CONTEXT GD
Next, we attempt to enhance the covariance diversity across contexts. To do so, we now allow that each context has random data and task covariances which are randomly rotated
xx N (0, E.) Σ. O.AOT Ω 0.90
(11)
where O is a random x orthogonal matrix sampled from the Haar measure. The idea to pretrain with diverse set of covariances Σ. across contexts is encourage the model to learn generic in-context learning algorithm that is not specifically tailored to a particular data covariance Σ. By introducing the random rotation across contexts, the model cannot encode whitening transform of the data in the matrix I which prevents zero loss solution in a shallow model with depth L = 1. Therefore, even the P/D
limit has the potential to exhibit nontrivial scaling law in
Result 6: Gradient Flow Generates Isotropic
Gradient flow on the I-reduced model maintains the isotropy condition r(t) = 7(t) with
= (
24-1)
(12)
This indicates that, provided the covariance is randomly rotated across contexts, the behavior of the learned solution is unconditioned in-context GD (see Appendix E). In the next section, we explore the consequences of this finding for optimal shapes.

---

Published as a conference paper at ICLR 2026
4 MODEL OF COMPUTE OPTIMAL NEURAL SCALING LAWS
We consider the third setting with power law features and also introduce a notion of width through a projection matrix A NND. Rather than training on D dimensional inputs æ, the model has access to N dimensional features I Ax, which leads to the following condition of ((t)
A.
(()) = 7() (AAT) E RRN
(13)
In this section, we consider arbitrary (non- pranppoooooooa) but large values of P, D. which requires an approximate non-proportional mean field theory (see Appendix F.I). This theory allows for the possibility of trace class covariates Σx=1 A DO for D) - co. As before, the loss can again be viewed as a function of the scale parameter 7t)). We provide recipe to compute it below by computing the average over the random orthgonal matrix, resulting in a two-point deterministic equivalent for free products in the same spirit of the results of (Bordelon et al., 2024a; Atanasov et al., 2025), using saddle point method (Appendix A, F.I). As the matrix M O(A A) Σ driving the dynamics is asymmetric, we characterize the loss landscape using dynamical mean field theory, a technique from the physics of spin glasses which allows asymptotic descriptions of high dimensional disordered dynamical systems (Sompolinsky & Zippelius, 1981).
10P
1314
5-0.5
210
0- 75
1.00
2.00
10
301
227
107
10
20-1
10a
207
19
107
20
127
207
10'
(a) Pretraining time & scaling
(b) Loss landscape vs Depth
(c) Learning Curves Varying
Figure 4: Loss dynamics for powerlaw RRS data. (a) Varying the source exponent B, that
the scaling with pretraining time has exponent 2+B (b) The loss landscape across depths L for the
B
scalar 7 parameter exhibits minima at 7 L. (c) The training dynamics of the reduced-L' model
exhibit t (2+/(2))) decay before hitting an asymptote which scales as L L-
Result 7: Two-Point Deterministic Equivalent (DMFT Correlation) for the Loss Landscape
The loss function L (1 (IA 1 (I<AAAAIT->XL0000(ATT A) O " can be explicitly eraged over the random orthogonal matrix O and expressed as deterministic function
C(1, N,,N,L, L) /
(2TT)2 (1+1.-'7(1)ww) (1 (1+10 1+++=* I (())a') C(w)w')
C(w,w') ~ Tr (iwl + (ox(w)A] -1 S Vxx( w'), Iiw' V (ox(w'))
(w')A
(14)
where Voy (w) and V Vxx (w, w') are deterministic functions that depend on the spectra of Σ and AT A (Appendix A, F.1). For example to obtain Vi (w)(w) we first solve
iw ())
in = Vox(w) ( (-1+
D
ww
(15)
where T and wM for matrix M is defined T Tr M (M R.M)
In Appendix F.5 provide formulas for V V (w) and v Voo (w, w/) in the case that A A D is a rank N projection (has N eigenvalues equal to 1) and Σ = } |XX E DxD is a structured
"Too-point refers to the correlation function of two resolvents evaluated at different arguments, rather than the one point function which is single resolvent and determines only the spectrum of the random matrix.
7

---

Published as a conference paper at ICLR 2026
Wishart matrix. We can further extract Chinchilla scaling laws for powerlaw data. We demonstrate the need for both width and depth in Figures and 5. We verify the scaling exponents for time and depth in Figure 4 and exhibit the compute optimal scaling law for width and depth in Figure 5.
Result 8: Scaling Law for Power Law Features and Optimal Shapes
Assume source and capacity conditions for the cigenvalues and target coefficients wy 94k
Σ de wk kB d C.
(16)
with source/capacity exponents (3, v). Let be rank and frozen. Then the loss follows neural scaling law in the resources of time t, width N, depth L, and context length P
C,,, N, L. + eN N cr. - + p-va.
(17)
As consequence, fixed compute C tP L the optimal width and depth scale as x N
12a
6x10-
LOSS
M4-54
4x20-
4x30-
3x20
3x10-
28:0
107
2x30
30
301
10'
10t
10
Compute
Compute
Compute
(a) Fixed Depth J. =
(h) Fixed Width N
Optimal Joint Scaling Vo
Figure 5: Increasing width and depth alone is insufficient to obtain monotonic improvements powerlaw data with random covariance across contexts. (a) Scaling only width leads to a depth bottleneck (dashed red line). (b) Scaling only depth leads to width bottleneck (dashed red line). (c) Increasing N and L simultancously achieves monotomic improvement with compute.
5 MORE REALISTIC SELF ATTENTION MODELS
In this section, discuss realistic models which exhibit similar training dynamics and depth dependence. First, we describe the dynamics when the I matrices are not tied. Second, we provide theory of training when each of the weight matrices is optimized separately. Lastly, we provide experiments with softmax attention models trained with Adam.
Gradient Flow with Untied Matrices In this section, we consider the effect of untied I ma- trices across layers f E [L]. When each of the weights are optimized separately with learning rate that is upscaled by depth r) = To,,, J., the dynamics are actually equivalent to the recurrent model that we presented previously under the RRS noise-free setting.
Result 9: Gradient Flow with Untied Layers on RRS Covariates
Let the model prediction on evaluation point x. dataset D after gradient flow time for pretraining on RRS covariates with L decoupled matrices {T') and learning rate 7 = T be written as (antica( t). Let the predictor resulting from gradient flow on tied matrices r r (looped transformer) with learning rate tlu be faed( t). If the initial condition for the matrices I I=== = I are isotropic and equal for all E, then
fumiee x.. t) = fuea (x+,, t) Vt O,
(18)
and consequently the ICL losses are also equivalent through time Luntiaa (t) Lrica(t).
8

---

Published as a conference paper at ICLR 2026
We prove this result in Appendix G.2, which relies on the permutation symmetry in the predictor function f with respeet to the variables (/ Since under RRS covariates, the I matrices remain isotropic through all gradient flow training (t)) = 7 all the matrices arising in the predictor dynamics commute, resulting in the permutation symmetry.
In Figure we provide a numerical demonstration that under equal identical conditions (such as I = 0 for all E). that the y scalars remain equal for all training times t when pretraining RRS covariates. However, we also provide evidence that this symmetry requires a symmetric initial con- dition. When starting from asymmetric initial condition 7 # 7*, the model does not necessarily converge to a symmetric configuration.
5.1 GRADIENT FLOW FOR FULL LINEAR ATTENTION
In this section we consider gradient flow on all of the attention weights {We, Wg. W.} separately rather than gradient flow on the P matrix. This corresponds to the dynamical system
W = aw; c, jE E fx, y, k.
(19)
dt
The dynamics of this model from small initialization are theoretically tractable as a set of low di- mensional ODEs (see Appendix G.2), but suffers some defects due to transient blowup and recovery in the scale of wy and w,, However, if we fix these weights to unit norm, the dynamics of the above model reduces to a one-dimensional ODE much like the reduced-I model.
Result 10: Gradient Flow on Full Linear Attention Module
Consider pretraining linear transformer randomly rotated covariance ICL data distribu- tion. Fix read-in weights w. and readout weights w. with wy = w. and w,yl = 1. Initialize the other weights to be small 2 [W W 2 = W,, = w,, = o² where o < 1. The gradient dynamics will maintain balanced condition where [W) WW w(t) for j.j E (x, k q, ( where the scalar (t) evolves as
d dt u(( w(1)² ir AA (I ()()))))(-)1
(20)
This can be interpreted as gradient flow on the loss function for the reduced I model under the reparameterization y(t) - w(t) For powerlaw covariates with source exponent 3, this gives a powerlaw scaling with pretraining time t and depth L
C(t, L) c s + CL D
(21)
We show the learning curves for this full attention module for isotropic covariates and powerlaw covariates in Figure 6. The theoretical dynamics and predicted powerlaw exponent for this new dynamical system closely match the predictions.
5.2 NONLINEAR (SOFTMAX) ATTENTION, MULTIPLE HEADS, MLPs
We also provide experiments showing that similar scaling behavior holds in softmax attention models trained with Adam. In this setting, the attention block has the form h++1 h + L E'' Aur W.h,, where A z, exp(h, W, W,W,h with Z, as the normalization factor. We train with Adam and show depth is still beneficial to ICL perfor- mance in Figure 6 (c) and 9, consistent with the phenomenology of our linear attention model. We also provide additional experiments varying heads, introducing MLP blocks in softmax attention models in Figure 10.
exp(h W.W,W,
6 DISCUSSION
Conclusion In this work we analyzed solvable model of in-context learning with deep linear attention. We showed that the pretraining statistics strongly determine the type of solution the model converges to and under what conditions scaling depth is necessary at large context lengths. When training on fixed covariance structures, large depth is not necessary (at infinite context length) as the model learns a preconditioner that whitens the data. However, this learned solution brittle to changes in the data covariance. When training tasks where the data covariance is randomly

---

Published as a conference paper at ICLR 2026
15
13-1
2
6.5
0.4
12"4
03
12a
Teeary
p-p0pp211
15
(a) Decoupled Att. on ISO a = (b) Decoupled Att. Powerlaw
(c) Softmax Attention Model
Figure 6: (a)-(b) Dynamics of the full decoupled linear attention (decoupled att.) module with separate (W,, 2, Wa, Wk. W.} ) as function of depth L agrec with theory generated through repa- rameterization y - wo as described in Result 10. (c) Softmax attention dynamics across depths indicate similar improvements to performance from depth. We provide additional experiments varying attention heads, depth, and the addition of MLP blocks in Figure 9 and 10.
rotated across contexts, the model learns general purpose in-context gradient descent al algorithm and exhibits a separable Chinchilla neural scaling law where limited width and depth can both bottleneck performance. Lastly, show these results robust to reparameterization of the attention blocks.
Limitations and Future Directions While our work presents an advance the solvable models of neural scaling laws and the structure of ICL in linear attention, there are many current limitations. The primary limitation that we focus on linear regression tasks solved with linear attention. Char- acterization of more general ICL problems such nonlinear function approximation and nonlinear attention could provide more insights into realistic in-context function approximation. Further, our analysis is focused on online learning in the present work. Future work could investigate overfitting effects caused by repeating tasks or context matrices during training, perhaps with dynamical mean field theory techniques (Mignacco al., 2020; Bordelon al., 2024a; Montanari & Urbani, 2025). While this current work identifies heterogeneity in data covariances as necessary for self-attention models to discover ICL solutions that utilize depth to generalize across covariances. future work could explore diverse covariances beyond the RRS construction that our work focuses on including data domain shift, changes label noise level, and hierarchical data structure. Future work could also examine the role of large learning rate effects during pretraining dynamics. Another direction that could interesting to explore in future works is scaling up the number of loops dynamically as the network trains to significantly reduce the total compute required.
ACKNOWLEDGEMEENS
The authors would like to thank Jacob Zavatone- Veth,, Alex Atanasov, Alexandru Meterez, Clarissa Lauditi, William Tong, Jamie Simon, Boris Hanin, Zhouran Yang, and Jascha Sohl-Dickstein for insightful discussions. B.B. acknowledges support from the Center Mathematical Sciences and Applications(CMSSA. of Harvard University. C.P. is supported by NSF CAREER Award (IIS- 2239780), DARPA grants DIAL-FP-038 and H.--HHO011220044 the Physics of Learning and Neural Computation, and the William F. Milton Fund from Harvard University. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
the Simons Collaboration on
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 087744 2023.
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Newwork, 132:428-44 446,, 2020.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn implement
preconditioned gradient descent for in-context learning, 2023. URL https://arxiv..r
00297.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm in-context learning? investigations with linear models. arXiv preprint arXiv:221 1. 151611, 2022.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernll learners: The silent aligmment effect. In International Conference on Learning Representations, 2022.
URL httpst://pppeeeeeeeeeeee/eee http///pppeevviieeeeeeeeeee n forum? 1d=1NvflqAdoom
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In The Eleventh Interna- tional Conference Learning Representations, 2023. URL
forum? wt.....II......YV.......
Alexander Atanasov, Blake Bordelon, Jacob Zavatone Veth, Courtney Paquette, and Cengiz Pehlevan. Two-point deterministic equivalence for stochastic gradient dynamics in linear models. arXiv preprint arXiv:2502.05074, 2025.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06711, 2021.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 11:57125--7225 2023.
Carl M Bender and Steven A Orszag. Advanced mathematical methods for scientists and engineers Asymptotic methods and perturbation theory. Springer Science & Business Media, 2013.
Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features. arXiv preprint arXiv:2106.02711. 2021.
Blake Bordelon and Cengiz Pehlevan. Learning curves for SGD on structured features. In Interna- tional Conference on Learning Representations, 2022a. URL https: forum? 1ddWP I 12vkkA13
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35:32240-32256
2022b.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. In The Thirteenth International Conference on Learning Representations, a.
Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyper- parameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, b.
Blake Bordclon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMLR, 2020.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, pp. -345-4382, 2024a.
11

---

Published as a conference paper at ICLR 2026
Blake Bordelon, Hamza Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head transformer dynamics. Advances in Neural Information Processing Systems, 37:35824- 2024b.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang. Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in- context learning in transformers, 2022. URL https:ttp..........t. : arxiv. www.vooooo worg//rggooo. abs 2200. 05055.
Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-heal softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint WrXiv:2402.19442, 2024.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. Advances in neural information processing sys- tems, 31, 2018.
Lénaïc Chizat and Praneeth Netrapalli. The feature speed formula: flexible approach to seale typer-parameters of deep neural networks. Advances in Neural Information Processing Systems, (166362--23883 2024.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131-101433 2021.
Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. Dimension-fiee deterministic equivalents and scaling laws for random feature regression. Advances in Neural Information Processing Systems, 111104460--46999 2024.
Mostafa Dehghami, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, Universal transformers. arXiv preprint arXiv: 003809 2018
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618. 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Shivam Garg. Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in neural information processing systems, 35:30583-30598, 2022.
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, and Sanjiv Kumar. On the role of depth and looping for in-context learning with task diversity, 2024. URL https:
//arxivv .org abs / 22112.111 2.44..21660 21698.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 (11):113301, 2020.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint amXiv:2502.05171,200 2025.
Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 15244-13273 Curran Associates, Inc., wtw.iinnnnnnuuuunnn neuri cc 2024. URL https: /ttp///pooocc httpe////occcecccceceooo.
12

---

Published as a conference paper at ICLR 2026
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: arXiv:1712.00400 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. /5556, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffire Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Juno Kim, Tai Nakamaki, and Taiji Suzuki. Transformers are minimax optimal nonparametric in- context learners, 2024. URL https: /aa.aaiiivv.../a///aaaaaaaa
12186.
Anastasis Kratsios and Takashi Furuya. in-context universality enough? mips are also universal in-context, 2025, URL https: / /arxivivvvvv..///////aaaa/aaaav/ otgg 03327.
Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under zipf's law. arXiv preprint arXiv:2505. 19227, 2025.
Mary I. Letey, Jacob A. Zavatone-Veth Yue M. Lu, and Cengiz Pehlevan. Pretrain-test task align- ment governs generalization in in-context learning, 2025. URL 2509.2 2651.
/bs/
Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning, 2023. URL htttt:////axxiv org/abs/2300 07067.
Licong Lin, Jingfeng Wu, Sham M Kakade. Peter Bartlett, and Jason Lee. Scaling laws in linear regression: Compute, parameters, and data, arXiv preprint 24X0::24066084666, 2024.
Shang Liu, Zhongze Cai, Guanting Chen, and Xiaocheng Li. Towards better understanding of in-context learning ability from in-context uncertainty quantification, 2024a. URL https:// arxiv. org. abs 220244055.. 15115.
Toni JB Liu, Nicolas Boullé, Raphaël Sarfati, and Christopher Earls. Lims learn governing
principles of dynamical systems, revealing an in-context neural scaling law. arXiv preprint 00X77:2402.00795. 2024b.
Yue M. Lu, Mary Letey, Jacob A. Zavatone-Vert Anindita Maiti, and Cengiz Pehlevan. Asymp- totic theory in-context learning by linear attention. Proceedings of the National Academy doi: 10t.://a/........./mm/// /paa///aaaaaaa/aaa 2ee.222122522222
of Sciences, 122(29):c250259991) 2025.
URL https:
www. www. pnws.pnss.a org/doi
Bochen Lyu, Di Wang, and Zhanxing Zhu. A solvable attention for neural scal- ing laws. In Y. Yue, A. Garg, N. Peng. F. Sha, and R. Yu (eds.), In- ternational Conference on Representation Learning, volume 2025, pp. 53372-53411, 2025. URL htttps://proceeeeee 0.2282770c3366622222222222 Sacceaccddd--PP-PP-Peee Conference.
iclr. wwwlr.cc//a cc/paper filef//l//pppe (alles/paper/2222 /2oer///2//ffe/////
Arvind Mahankali, Tatsunori B. Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention, 2023. URL https://arxiv.
Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint urXiv:2210. 16859, 2022.
Gavin McCracken, Gabriela Moisescu- Pareja. Vincent Letoumeau, Doina Precup, and Jonathan Love. Uncovering a universal abstract algorithm for modular addition neural networks, 2025.
URL https:ttp///c /ttes:///axiiii v. .rgg htsgg/bb//5000 18266.
13

---

Published as a conference paper at ICLR 2026
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academyy of Sciences, (15 115(33):E7665-7777,, 2018.
William Merrill and Ashish Sabharwal. A little depth goes long way: The expressive power of log-depth transformers. arXiv preprint urXiv:2503.3396/ 2025.
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540- 9550, 2020.
Andrea Montanari and Pierfrancesco Urbani Dynamical decoupling of generalization and overfit- ting in large two-layer networks. arXiv preprint arXiv:2502.2/266 2025.
Lorenzo Noci, Chuming Li, Mufan Li, Bobby He, Thomas Hofmann, Chris J Maddison, and Dan Roy. The shaped transformer: Attention models the infinite depth-and-win and-width limit. Advances in Neural Information Processing Systems, 36:54250-544 -4281, 2023.
Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism, 2024. URL /ttps::/arxiv. www.aiiwwwwwg...a..aaoeeeaee wwt/ 2306. 04891.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ phases of compute- optimal neural scaling laws. arXiv preprint 2r 2024.
Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana. Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. Iclr: In-context learning of representations. arXiv preprint arXiv:2501.00077 2024.
Reese Pathak, Rajat Sen, Weihao Kong, and Abhimanyu Das. Transformers optimally learn regression mixture models, 2023. URL stttps://arxii org/ abs 2311.083--
Mare Potters and Jean-Phine ShnnPhilipppe Bouchaud. first course in random matrix theory: for physicists, engineers and data scientists. Cambridge University Press, 2020.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non- bayesia in-context learning for regression, 2023. URL https: sttps://arxiv. org/abs 2306. 15063.
James B Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. More is better in modern machine learning: when infinite overparameterizztion is optimal and overfitting is obligatory. arXiv preprint 2r
2023.
Haim Sompolinsky and Annette Zippelius. Dynamic theory of the spin-glass phase. Physical Review Letters, 47(5))359, 1981.
Rishi Sonthalia, Jackie Lok, and Elizaveta Rebrova. On regularization via early stopping for least squares regression. arXiv preprint arXiv:2406.0442 04425, 2024.
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus studeet paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12): 12200111::44001, 2020.
William L. Tong and Cengiz Pehlevan. Mips learn in-context regression and classification tasks, 2025. URL https::/// arxiv. orq/abs/2405. 15618.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Max Vladymyrov, Johannes von Oswald, Mark Sandler, and Rong Ge. Linear transformers are versatile in-context learners, 2024. URL https: /ttx/v.vvvvv./////// www.arwwww....eaeeee.e.ee.ooeeee www./bsg////....o.oooo.....o...e abwp/4/2.22.. 141.14100 4180.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers lear in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174, PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint 4rXiv:2506.21734, 2025.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression?, 2024. URL https://arxiv. org/ abs 08391.
Daniel Wurgaft, Ekdeep Singh Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, and Noah D Goodman. In-context learning strategies emerge rationally. arXiv preprint 17X8592506.17899, 2025.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.020009 2021.
Greg Yang and Edward Hu. Tensor programs iv: Feature learning in infinite-widhh neural networks. In International Conference on Machine Learning, pp. 11727-11737, PMLR, 2021.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / //operreview httt:/fooummm..xxeeeeeeee
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023.
Ruiqi Zhang, Spencer Frei, and Peter L. Bartlett. Trained transformers learn linear models in- context, 2023a. StRp://p::/ann wwwp://www/w//////........ 099277.7.
Yedi Zhang, Aaditya K. Singh, Peter E. Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention, 2025. URL https://arxiv org/ abs/2501 16265.
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305. 19420, 2023b.
Ruike Zhu, Hanwen Zhang, Tianyu Shi, Chi Wang, Tianyi Zhou, and Zengyi Qin. The 4th dimension for scaling model size. arXiv preprint arXiv:2506. 18233, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A TWO POINT DETERMINISTIC EQUIVALENT FOR GENERAL FREE PRODUCT FROM DYNAMICAL MEAN FIELD THEORY
In this section, we describe a method to handle two-point correlation properties of free products of random matrices. Similar results were provided in a simple random feature model of (Bordelon et al., 2024a; Atanasov et al., 2025). This computation provides the necessary technical results to establish the behavior of ICL in the randomly rotated context setting. Note throughout that will use trM for normalized trace of the matrix M so that for N x N matrix trM instead study arbitrary (possibly trace class) covariates in Appendix F.1.
N MER- We
Intuitive Motivation for this Section Before working out this random matrix result, we first want to motivate it. To do so, we start by considering the infinite depth limit L - oo of the model in Section before describing how our results will also cover finite depth In the infinite depth L + oo limit, the loss can be expressed as the as a dynamical system in layer-time r (not gradient descent time) evaluated at 7
(()) trA (p()))(n)T) v,)(T) = -Mo(T) + 8(T)
(22)
where the average () is over both the matrix M and the initial condition 3 which has correlation (3 = Ω. We note that for the RRS setting, the matrix M is generally an asymmetric free product of the form M = OBO A where B and have known spectrum. A key idea to examine the Fourier transform of the correlation matrix for v(T)
(o())0(6))) = f dwdw''
(2r)2 cnlwtta)) (iw + M] 52 [iw + M !
(23)
We therefore that the loss depends on two-point interaction of the resolvent matrices [iw M evaluated different frequencies w,w', For finite depth L the result can again be expressed in terms of this two two-point resolvent correlation using Cauchy's 's Integral formula
(o'(o"))) /
dwod''
(2m)² (1 -L-LLL-- "www) - - ' /m (lio + M| Ω Www' + M -
(24)
Thus for any depth L the ICL loss is governed by the correlation function in frequency space
(ww,w)) = (v(w))((')))> = - + M s [iw' + +M
(25)
a valuable summary statistic to compute the test loss when averaging over the RRS model's ran- dom rotation matrix O. We can therefore consider simple linear dynamical system in continuous time and taking a Fourier transform to obtain (w,,w)).
To compute the pretraining dynamics for the general random M, one must compute gradients of the effective loss with respect to 7. For example, at infinite depth, the gradient flow pretraining dynamics are
d di 70())
dwdd'' - i(w + w')e ((w+t))) A C(w,w") (2m)²
-
(26)
Now that we have motivated computing the C(w,w") object, we will now give very simple Gaus- sian example before deriving the general free product result necessary for the RRS covariates.
Dynamical Mean Field Theory Illustrative Example To illustrate what Dynamical Mean Field Theory computes, we will start by deriving a deterministic equivalent result for a simple N x N Gaussian Orthogonal Ensemble (GOE) matrix Mg MG;- ~
d v(1) M 7())))
(27)
dt
2 Because this matrix is symmetric the two-point equivalent is not necessary since exp(-2Mt, allowing for a Fourier transform in single time variable. We still compute the two point function illustrative example.
exp( (ep(-Mt) exp(-M f)
16

---

Published as a conference paper at ICLR 2026
As N + oc, this dynamical system can be replaced with an uncoupled dynamical system which depends on some summary variables known as correlation C(t, t') and response function R(T)
d dt v(t) E(t) + f at' R(t - (')o(t))))))))))))))))))) + o())oo
fat
(28)
(E())E(T)T) C(t,t'),, C(t,t) tr (o(())u()) R(T) = tr /du(t+ -
(29)
@E(()T
Taking a Fourier transform of the above equation v(w) dte iwt (t),, we find ((w) = liw + R(w)] meu + {(w)]
(30)
where R(w) is the Fourier transform of the response function. This function iw]((w) = 1 + R(w)²
(31)
The correlation function ((w,w") (o(w))(w") thus satisfies the equation C(w,w = liw R(w)) liw' + R(W()] [ + C(w,w/)].
(32)
We see that the DMFT description enabled an average over the random matrix to compute C(w, w"). This result exact as the matrix size + oo. We would now like to perform a similar analysis for a general free product matrix (to deal with our RRS ensemble) which involves averaging over orthogonal rather than Gaussian matrices.
Tracking Linear Dynamics Generated by Free Product Taking this dynamical approach, consider the following dynamical system which depends on a random matrix M E RNNN
dev(t) = Mu)( + 8(t)oo M = OBO A
(33)
where O is a random N N orthogonal matrix drawn from the Haar measure for the orthogonal
group. Our starting point is to express the above dynamical system with an integral representation of Dirac-Delt functions 1 dzō(2) runs along the imaginary axis. Performing this for each time of the dynamics yields
dzdx e- where the integration variable (-ico, ico)
7
/
DuDx
exp (-f ddx(() (0.((1) + M mo)(())
(34)
where DoDX lim I uous time limit öt + O. The average (-) computed over the random orthogonal matrix O.
6t)(n-6))d is the flat measure over (t), x(t) in the contin- 2m
To simplity the calculation, we transform our dynamics into Fourier space
dw
dw
v (t) =
2n exp (iwt) v(w), x(t) =
exp (iwt) x(w),
(35)
2n
which transforms the original integral over into an integral over as dwx(w) [[iw))(w) + ( - vp).
Disorder Average Averaging the resulting expression over the orthogonal matrix O, we obtain the following representation of Z as an integral over order parameters {Σ, }
z= DED exp (-NS(E, 1
(36)
In the above expression, the action S has the form
S|E, v = TrE In (xP) - -TEE. In Ie S & I+ EE.IIVVOR Int Σ
(37)
- + tr I -1 and the single as ØE,
where E. solves the implicit equation site measure Z has the form
Zv(D)= = [DUDXexpt (-J dux(ww) (kw))(()) vo - /dddd ded w') ( Au(o)))
exp (-2/ desidw'' [Veolwe )ow)) APo(w") + V (xx(w, x(w')) (38)
17

---

Published as a conference paper at ICLR 2026
Taking the Large System Size Limit over Σ and v give
To study the limit of N - oc, the saddle point equations
as
Ev w') Eox(w,
(b(w) T TAZ)(w)) (x(w) Av(w'))
(w(w)) (T(w))AX(u"))) (x(w) x(w"))
N
0 (w,w") OS (w),ww")
E Exolw, Ex)lw,
V (w,w') - E. w/) + (W,")",'") =0
(39)
The average over ()) represents an average over the Gaussian measure defined by Zл. The solution to these equations has the following block structure
[or(w) Eux Eux(ws, (w),
Eex Bex(waa")
E(w,w') =
(40)
0
0
Exx(we,
[ww,w)) = (u,(w)
(41)
Exx wi))
V 1x (ox(Ww, w') V xx (ox(44,, w²)
V (w,w')
(42)
V Voxlw,
Off Diagonal Blocks The off diagonal blocks decouple over different frequencies Eux (w, w²) = 5(w w/ (x)x((()))) and satisfy the following equations
Eux (w) = trA (iw + L ux (W) = (Eex(w)+ (B)
(43)
ux (w)) (w)) = - Eue (w) (w))nx(w) (w)
(44)
Introduce the r-transform TM of matrix M as
TAs (ww(ww) = tr (iw + M)
(45)
as well as its inverse function (T) 3. Then our saddle point equations give us T= T=Eo (w) ux (w) = Ta (iwia) Tm (iwn)
(46)
where i Tox
we and iw (r - 1) ux (w). Putting these equations together, we find
(T))(
- (iw)
1
(47)
This equation is to be solved for r(w). Once this function is determined we can use it to compute the diagonal blocks of E, as we describe in the next section.
Diagonal Blocks Now, we can determine the diagonal blocks, which determine the covariance structure of the v(w) variables
1 N v0 (iw + (ox(w)A) A (iw' Vox (w)))) 00
Eno(w, N
1 xxx (w,w") trA² (iw + L ()y(w)A) (iwe' + 1 ())) -1
Alwoaa")
Enllw,w)) = - _xx) tr (ox(w)) B
Sox( (w') + B
L (w, = - -xx Evolw, L')'e'' (w) -''''xell Exx(w')~1 where we introduced functions A and B which can be determined from the (already obtained) off- diagonal blocks. Combining these equations
(48)
Ene w²) = N DT (iw + V V (w)(w)))-) A (iw' + Mor (w') 2r0 [Ex((w) - E (w)) - Blw w/) 1] A(w,w') w") Blw, w/ (w w') and v (w,w') = (ixx(w,w)
1
(49)
From this expression, both Eip (w,w') Evp (w, w/) ('))nx(()))))))))) Enx((()) are determined.
UUp to change in signs and iw + - this is equivalent to the t-transform of a random matrix (Potters & Bouchaud, 2020).
18

---

Published as a conference paper at ICLR 2026
Deterministic Equivalent From the functions M (w) and (wx(w,,") we obtain the following determinstic equivalent for the outer product of a variables
((w)o(w')) 2iw + Vax (w) A) (000 Vxx( (() (lwl (ox(w)AA)
(50)
where the = expression holds under trace against a test matrix (i.e. M1 = M2 = ttC = 22) (Potters & Bouchaud, 2020). This function will be used in subsequent expressions to give an exact result for the loss landscape of our randomly rotated ICL loss function. Approximations of discuss in Appendix F.5.
this will give rise our scaling law results
B MODEL DEFINITION, MASKING MECHANICS, REDUCED-T MODEL
B.1 LINEAR ATTENTION AND POSITIONAL MASKING FOR OUR TASK
In this section, we provide more detail on the structure of the masking used in our task. First, we note that the target values Vn only provided for the P labeled examples within each context matrix. Second, we note that the residual attention operations are masked differently for the P labeled examples and the K evaluation points. To define our masking operation precisely, we introduce the notation 1, E R as a vector consisting of all k entries equal to one. We introduce the masking matrix M E ((P+K)x(P-K) for the residual stream which has the following block structure
- 1p 1 1x 1 p 0
0
M =
(51)
Now that this positional masking matrix is introduced, for the residual stream
conveniently express our update rule
P+K
h2+1
h
Σ
Mar (Ke (.) or
PT
v=1
= n1 +
Σ
Myu (Ke ( v. u ME [P[KK]]
(52)
PL
where k = Wx h of
h and we dropped the sum over evaluation points (P+ + 1,... P + K} due to the structure of the positional mask M. We thus see that the masked update rule has two properties
= W je of =
1. It prevents the test points a for E stream updates. Only the first training points are utilized.
P K} from being used in the residual
2. It provides an opposite sign for the updates on training points and on testing points. We will see that this will enable the model to implement in-context gradient descent rule. In such rule, a subspace of the residual variables will encode residual errors yn f for the training predictions and ff for the test points μ {P+1, P K}. Instead, one could use the same signs for training points and test points in M and simply negate the output of the model at the end f
B.2 DERIVATION OF REDUCED GAMMA MODEL
In this section, we describe the conditions under which linear attention transformer model can be reparameterized as the recurrent reduced-I model we discuss in the main text.
Alignment Assumptions Following Von Oswald et al. (2023); Zhang et al. (2025); Lu et al. (2025), we study configurations of weights that encode information about inputs æ and targets in orthogonal subspaces of the residual stream. Concretely, the following assumptions made on the input weights, which implement in-context preconditioned gradient descent
W wy 0, W w. 0,, Wy wy |2s,] |
(53)
19

---

Published as a conference paper at ICLR 2026
Collectively these assumptions imply that read-in weights for the targets wy and readout weights w. perfectly align and that the read-in weights for a Wa project imput data to an orthgonal subspace. Next study the following set of alignment conditions for the key, query and value matrices
WH (W W x I' E RXD W, W. 0, W. & wy
(54)
(55)
Under these assumptions, we can define collection of D matrices r
T ww W w,y W. (WL) (W;) W E DxD
(56)
which gives rise the following residual stream dynamics for A w. h
A -+1 = = A
1
Σ Maccc M
(57)
LP
We note that this can be separated into two distinct dynamical systems, one for the first P training
points
=+1= - A 1 LP
1 a = μ [I....,P]
(58)
which form closed dynamical system the P labeled training points. From these P variables {A petpee we can describe how the remaining K points evolve
A 4 A
Σ .
T'
ΣE x r HE
P K..
LP
L,
v=
(59)
Recurrence Instead of defining separate I matrices for each layer É, we instead can examine recurrent model where the weights are tied r = P for all & E [L]. We relax this assumption in Ap- pendix G.1 and in many of the settings we analyze recurrence has no impact compared to decoupling layers. Under this constraint, the residual stream of the model has the following dynamics
4+1
A
1 LP
T Ix A. M. [I.
A+1 = 1
a Tæ A M (P P+K
(60)
L.P
Let A TR. represent the residual stream variables restricted to the P unmaskedd training points in the context. This vector has the residual stream dynamics
-- x--XXXX- y.
I
=
(61)
LP
However, the recursion is different for the since these points only receive attention signals from the P unmasked training tokens. For one of the test points æ., the prediction of the model f. satisfies
1 LP
PX Σ h'
L-1 LP I Σ
(1-IX
(1-1XXTX)'
(62)
Equivalence to Preconditioned In-Context Gradient Descent
This update rule is equivalent to implementing preconditioned in-context gradient descent steps for a linear regression model
3 æ. To see this, define an in-context training loss 2(B D) on the P labeled training points for context D
1 2P
L(D)
D x 3 y
(63)
20

---

Published as a conference paper at ICLR 2026
Preconditioned gradient descent with learning rate D/L and a preconditioner matrix T generates the following dynamics on the learned ICL weights B
B = - TCCSS' -) - (
VD LP
(64)
Defining a variable 1 y- D-12211 B², we note that this variable satisfies the same recursion the residual stream variables he the training points described above which gives the identical solution ht
=11- x y. The function f. on a test point æ, takes the form
L-1
1 xXx y. (65)
f+ = VT 3 II= LP
rX
h 1= LP T. IX
LP
t=0
Test Error Formula for Linear Models We now desire to compute the expected test loss (aver- aged over the dataset X and noise E) for (noisy) linear target function y())
v(x) 1
vD B. x + (axx)) Σ,
(6²)
(66)
where the (population) covariance of the imputs. The expected loss under this assumption
[ <=) ((8 B.) E (B4- B.)) -8))
(
+ o²
(67)
x.c
Letting e E R²" represent the noise on the P labeled training points, the target weights have the form
VD
3r.
rx
L, xTX)
+
LP
VD
t=
L-1
oVD
L-1
T
Σ
[ -- I I---
B.+
r
Σ
(I-L-1ET) Xc
[
LP
1=0
t=0
3. (1-L- L
B.+
OVD
Σ
L-1
(I-L-1E))
Xe
(68)
LP
where we defined Ε = p X E ERDxD Then we note that for a = P/D that the reducible loss o² has the form
C C-
( (B, - -²))² E (3.--33) 31))
37 (11- LT
E (1-11-13)
4 4 3.
(1-1-TT) 's
(I---11s)T
Tr r
(69)
aL?
8,6=0
In the next sections, we will compute this quantity for various distributions for E and Ba.
C ISOTROPIC (ISO) SETTING THEORY
In this section, we consider isotropic covariates and tasks so that the data covariance Σ and the task vector covariance Ω are both identity
Σ (xx =I=1 Ω (0.3.) =
(70)
where the average for target vectors 3. over different contexts. We further operate in the high dimensional asymptotic limit where both context length P and input dimension D diverge with fixed ratio
+ oo P/D =
(71)
In this setting, the spectrum of the empirical covariance matrix E x follows the well- known Marchenko-PB Pastur law, where the eigenvalue density p()) depends explicitly on the aspect ratio (Potters& & Bouchaud, 2020; Advani et al., 2020). We will first describe an asymptotic limit of SGD for the shallow = 1 case where the dynamics on the matrix I are linear. We will then pursue description of gradient flow dynamics in arbitrary depth L. model.
21

---

Published as a conference paper at ICLR 2026
C.I SHALLOW SGD THEORY
In this section, we consider the SGD dynamics of the shallow architecture L = 1. This excercise will establish what rate our dynamics will converge gradient flow and how finite batchsize B, context length P and number of masked evaluation points K impact the SGD dynamics. In this case, the updates have the form
1((+1) = r()) + G())
G(t) = Σ
1
Σ
(I [TEE),),, -o
+ OoP
B
1 Σn p X Xn E.,
X X
(72)
where X E RRD is the set of training points and X, E KxD represents the set of K evaluation points and Rf represents the label noise on the provided targets y. We remind the reader that we will operate in the joint scaling limit
P, D 00, P/D a, K/D = k, B/D =
(73)
The population ICL loss has the form
((1) D (T(()) 113)
(74)
where the average is performed over all possible draws of data and tasks. This function admits the recursion
C(1+1) 1) = [() - 2))r(T()) - 1JJ (G(O) + 72trr (G()
(75)
We have the following mean for the gradient
(G( -I- (1 +1+a + a²xx )r
(76)
Thus if we only updated using mean gradients the model would relax to fixed point r.
(1 + a- + ) - I. However, we will see momentarily that SGD noise in the gradients im- pact the that SGD converges to
tr (G(1)
1 B²
Um Σ tr um.
(77)
mm
where u, Einm (I TE,, - op V -ix and n₁ Enfr, + a Doing
the averages
D Uun Um) = 8m tr I - 2T + (1+ - 11)=2 -020-112 (1+
(78)
a a o2) (1 a - (1+
= 0 (1 + a 11 =1+ - ) (T (T=
(79)
D (On vm) = 6m + a (1
(80)
From the above results, we get the following second moment structure
tr (G(())TG(()) = (G((t) (G(t))
+ (1 a1 (T- (T-1.)²
(1+
(1+ o2))3
(81)
Thus get recursion
T(t+1)=T(t) (1 -1 (1+)9)) T. [(t] tnt
(82)
22

---

Published as a conference paper at ICLR 2026
where E IR Dx is the noise process with variance given above. We define quantity C(t) which measures the relaxation of I to T.
1
C(I) 5 (T()) - r. r. - (1+
(83)
This function exhibits the following linear dynamics
C((+1) 1) = - ( + (1 (1++a c(1
1/ - + + + (1+ + + C(I) (1+a (1 02))2
a (1 +
(84)
This recursion again takes the form
C(++ a(n, a N. T)C(t) b(7
a, = [1- (1+a ( [2]]] + n
+ (1+ 02)2²
2 b(,, _ x (1 (1+
(85)
Now we need to compute the ICI. loss
((1) = (r) - -r + - 1]2 ((1) + 2u((T - ((O) + tr(TT - 1)2
2a (1 + x
(1 (1+02)2
C(t) + (1+ a (1 (
(1 + a [
(86)
(a + 1 02)2
10
1 0.10
10
0.10
029
0.40
0.40
000
0.00
0.7
T 180
Theory
2.66
0.7
Theory
1.0
0.3
0.3
20
60
100
20
40
100
20
100
Vary Batchsize
Vary Context Length
(c) Varying Evaluation Size
Figure 7: Pretraining SGD loss dynamics in the shallow L = reduced I model for (a) a, and varying (b) varying a with 7 = ^ = 1 (c) varying x with o, = 1. The loss monotonically improves with all three quantities a, and is well predicted by the asymptotic theory.
Putting this all together, we summarize our findings below.
Result 10: Online SGD Dynamies for Shallow ICL with Isotropic Covariates
Consider the reduced model with single layer L 1 and proportional asymptotics
P,K,B.D + do P/D K/D B/D=7
(87)
Then the ICL test loss L(t) [tt) 1 after SGD iterations is
C(t) = C(t) + -1 o2)
(1 + 03)2
[1 +1
(88)
(1 a (1+
(a + (
1- a(1) T, a(n,
C(t) = ann, a, k))
(89)
1 1- a k.
where a(1,
k) k,,,),
+1 (1
(1++ and
b(7,0, 7) = '² (1 x -====a---0 -1 0²) capture the dependence on batchsize through T.
23

---

Published as a conference paper at ICLR 2026
We verify the validity of these learning curves for shallow pretraining dynamics in Figure 7.
We thus see that deviations from gradient flow in the L model come in at a scale of m//T. The purpose of this result is stress that only O total tokens required (unlike the prior work of Lu et al. (2025) which required ( tokens) for an ICL learner in this regime" 4. We verify these theoretical learning curves in Figure 7.
We also indicate the following facts about SGD effects
The number of masked evaluation points per batch = K/D only alters the SGD noise terms (the terms involving 1/r) and its effect vanishes in the gradient flow limit 7// - 0.
Unlike offline training, in this online SGD setting the model regularizes the final value of based on label noise o² to reduce overfitting.
(t) =
SGD fluctuations can impact the final loss through lim;-+ C(t) b(7,a,T.8)
1=a(1.0,7,))
One nice consequence of this result is that the gradient flow limit can be accessed simply by con- trolling the scale of n/T a limit where batch size is linear the dimension B ==7/) D.
C.2 GRADIENT FLOWW IN DEEP MODELS
From the previous section, we saw that the absence of SGD noise, gradient dynamies generated an isotropic I matrix. This fact is true for larger depth T. 2 well. The gradient flow dynamics generate is r(t) = I. To see this, we note that the gradient has the form
0 ar
2
r (
E (1-1-1)))) 41g
(1-1-11))*
C -
T. t=
20
(1-L-113) Σ (1-11-11)))
r
al? 1.
2x²
r
Σ
((-L--1s)
E II- (rS)
1-1-6 E
(1-L--rE) )
(90)
aL3 k
Now, evaluating this at r == 7I we find
i ar
= -
i
(3-----13))
26-11 (1-1-11))2--)+
2002
L
al?
Et'
20
i ( (1-1-1198) Σ F,8'
(11'-1)
+
x
(91)
at?
where recognize through rotational invariance, that the above averages are proportional to the identity. Thus if I currently in an isotropic configuration, it will remain in one throughout gradient flow. Further, we can compute the scalar 7(1) trr(() under gradient flow d, C(T).
d
a tr (I-
2L
27
dt 7t)
Σ
I (I-
(92)
Oy
a
Now, we note that this expression can rewritten in terms of the bulk Marchenko- -Pattr eigenvalue
density p(A) = 1/2 Eb= (8(1- - >xo)) = 2 V(2 - A)(A - -)(A-
- where x = (1 -1/2j2 as
म x00==-00 oy 2 ()x())) [1- 21 + ax 1- [-
d
0²
(93)
which shows that we can view the dynamics as a gradient flow on a one dimensional loss landscape.
'The gap is caused by suhoptimal scaling of the number of evaluation points K per context (usually K = in prior works). Note realistic L.Lss get multiple emror signals per context (one per token).
24

---

Published as a conference paper at ICLR 2026
D FIXED AND STRUCTURED COVARIANCE (FS) SETTING
In this section we discuss the case where the data covariance is structured but potentially structured. We consider the noise free setting and P/D for simplicity where the loss has the form
C= tr 52 ([II-L
Σ EI -1-L-
(94)
In this case, gradient flow will generate dynamics that cause to pick up anisotropy from the structure of 52 and Σ. We analyze the case where I2 and Σ are simultaneously diagonalizable with eigenvalues wi and Ag. In this case, I will share the same eigenbasis. Let the eigenvalues be %
θ Ot (t) 2wx (1
L -- (x)) 21-1
(95)
The fixed point of the above dynamics is 7/k LAx In the large depth limit L + co, we can solve the dynamics exactly
d
ai (1) 2wx 2Ax7a(() = 22 drn- = wn wnA2det
(t) = In(1 AwxAitt
(96)
Plugging this solution into the loss function we find
1 E wRRRe 20 C(t) D
D Σ
wxAx[l + w
(97)
Under powerlaw (source/capacity) assumptions on the structure of the data
As k- wk -
(98)
the loss scales in powerlaw as
C(t) 100=>>>>
k 16-03-111 k 2 ~ t ut00+1
(99)
k
This powerlaw provides decent approximation to large but finite depth models as we show in Figure 3.
E
RANDOMLY ROTATED AND STRUCTURED COVARIANCE (RRS) SETTING
E EXPLOITING SYMMETRY IN THE GRADIENT UPDATES
Utilizing the definition of the RRS setting, we can massage the placement of the orthogonal matrices so that the loss function can be expressed as
A 1/2 (I-L-'O 1. B.P)
C
(100)
From this expression, it is immediately clear that this function is rotationally invariant with respect to I since the transformation r + VIV for orthogonal V leaves the Haar average unchanged. The form of the loss gradients also reveals symmetry
① C - -T. Σ [( (M A (M--1-9) - rr
(101)
Note that starting from zero initialization, we have M = O which implies isotropic gradients ar Crr= I. To see this note that (OCOT) I for any matrix C' that is independent of
O. Further, suppose that we evaluated the loss gradient at any isotropic r = II, then M - and the gradients remain isotropic
OC
0r = -2 "
OOMMB.. 3.8. (M') A 1
(102)
=-21-1 Σ ir LEM' B.3. (M (M x1
(103)
25

---

Published as a conference paper at ICLR 2026
where we explicitly performed the average the only remaining factors of O since M = -I- independent of O. Further, we can express the dynamics of y(t) as gradient flow on the reduced one-dimensional loss landscape
[(>) = ir 1 A (I--Z--18)
(
(104)
F SCALING LAW THEORY
In this section, consider the scaling law theory with finite projection matrix A, following Bordelon et al. (2024a). In this case, our RRS setting requires utilizing the free product result in Appendix A. However, will first describe modification of the two point deterministic equivalent outside of the proportional scaling limit.
F.1 NON-PROPORTIONAL. SCALING LIMITS
To make contact with more realistic scaling law theory, we are interested in varying large but finite N and P for an arbitrary (trace class) spectrum
A 00.
(105)
This is the setting, for instance, where kernel methods and power-law random feature models have been analyzed successfully Bordelon al. (2020; 2024a); Defilippis al. (2024). In this case, the order of limits is to first take D + co at finite (but large) N and P. We stress that (unlike the proportional limit), this result gives an approximation rather than an exact description, as traces remain random variables for finite rank A Σ. However, as demonstrated empirically in Bordelon et al. (2024a) this provides an excellent agreement at even very modest values for N, P on powerlaw features.
We now work out deterministic equivalent for M = OBO A E D This requires slight modification of the two-point deterministic equivalent equations from Appendix A. The action in this case has the form
S v = -TEE In 2, V TrΣΣ, IIn det
Σ. E.&I+V SI+VV & B) + Dln Σ (106)
The saddle point of this action satisfies the following equations
< [((W)A(((w)) v(w) Ax(w)) ( v(w)) x(w) x(w')
== -ー1 - EE., E(w,w')
(107)
Σ = Tr Σ. +
(108)
The off-diagona blocks give
Euy (w) TrA (iw + V Aw) = (iww)
(109)
V Vnx(w) Eny (w) - (w)
(110)
Introducing the notation TM TrM (WWM +M) we have
(2- - 1) iw
(WA(T)iWB(T)
(111)
We note the saddle point value for v (w) An Similarly, we can express (w, w²) in terms of iws, iw alone. The key deterministic equivalent is thus the same structure as before
be expressed entirely in terms of iw.
0( Tww)"(w)) = (iw + V (w)(w) (aaal
v (x, w') (iw' + T Voy (w'))
(112)
'In this section, we 1) for the dimension of A and B avoid confusion with the projection dimension (width) N.
26

---

Published as a conference paper at ICLR 2026
F.2 DYNAMICS AT INFINITE N, L, P
The gradient flow in the limit of N, L, oc has the form
d 7())
② E Xuwce 27(1)Ax
oy - -
(113)
dt By
This is separable ODE with solution (under the assumption that 7(0) 0) up constants
(()) tōoz.
(114)
This implies a powerlaw loss scaling with exponent set by the rate of growth of y())
(t) ~ (() ~ ++x
(115)
We expect these dynamics to hold in the limit of N, L, P co. At finite N, I, P the model will saturate at some finite maximal value of 7.
F.3 DEPTH SCALING LAW AT INFINITE TIME, WIDTH, AND CONTEXT LENGTH
We explore the scaling law in depth L. In this case, we consider taking all other resources to infinity 1, N, P + co. The value of yy will stabilize, resulting in a final loss
[(r) = Σ PRau (1-7I-1Xi
(116)
While 7()) diverges (==3 in the infinite depth limit, we see that at large but finite depth, there is maximal which enables stability along the top eigendirection. Specifically we need y < X approximate the optimal value s
24 and
7 = L./A1.
(117)
where Å is the top (maximal eigenvalue). At The 7r. the error takes the form
f exp(- (opp-Lk-") exp
L Σ Au (z An/Di)
(118)
which matches the scaling of L steps of gradient descent problem with source exponent B Bordelon Pehlevan (2021).
F.4 MAPPING THE LOss FUNCTION TO Two POINT DETERMINISTIC EQUIVAL
We utilize dynamical mean field theory (DMFT) result of Appendix A to compute the effective loss as a function N, L and y. We start with a representation of the relevant polynomial M with the Cauchy integral formula
(I- 1. =
[
dw liw MJ MJ (1 71+
(119)
2T
where the contour C encloses the positive imaginary axis in complex plane (Bender & Orszag, 2013).
Next, we define v(7) = 7L "MM B,
L L= (w(7) Av(r)) S [1- A A(I-7L-IM)
(120)
The loss can thus be expressed as a double integral involving the resolvents
C L= | dwdd''
(2x)² (1 + (i)) + +TI.-iiu) Tr Ω [iw M
A liw' + M (121)
The main result needed is the following deterministic equivalent from Appendix A.
Si [iw + M MM A Tiw' + M M
Tr + V (wx(w) [ T Xxx w/)] (iw' + (w)) A where V (w) iw/iWAAT A)2 can be determined from the equation
(122)
iA A (r) = T (iw)
(123)
Once is identified as function of w, we compute Nux and V Xxx from the formulae in A.
27

---

Published as a conference paper at ICLR 2026
F.5 ORTHOGONAL PROJECTION AND STRUCTURED GAUSSIAN EMPIRICAL COVARIANCE
In this section, we describe what these relations imply for product matrix that arises in our RRS setting, which is free product of the width projection matrix (A A)² where A E RNxDD has rank most N and the empirical covariance for data in the context Σ XX E DRD
= O (A A) O Σ,
(124)
where O is random orthogonal matrix sampled from the Haar measure. The data X E PxD comprised of random iid vectors with covariance A, which we take to diagonal without loss of generality. Further, we let A A be rank N projection matrix
A exel E DRD
(125)
where eμ E Rº are Cartesian (one-hot) unit vectors. To utilize the result in Appendices A and F.1, we first need to compute the necessary resolvents for these two matrices.
[AT A)2 +1 (AAATA = -11 N
N
Tr(A"A 2) (iu AY= +
(126)
The other matrix can be easily worked out for structured empirical covariance with P samples following the techniques of Bordelon et al. (2024a) which reveals that
T = Tr
iiss + Σ
TrA P + A) = TA (wws(1- (1--)--)--).. (127) (127)
Using our result for products find
(-1
N 7) (iw)(T)) -1- D
iw
T TrA (wws(1 - 7)1 +A)
= TrA (iw (-1+ + Dr 1) Nr (1-
(128)
This final equation provides 7 (w) which can be inverted to derive the final deterministic equivalent as we outline F.I.
Width Bottleneck Now we obtain the scaling of the loss with width N in the regime where t, P. D. + co. To do, we examining the structure of the equations at P - oo and is - 0, which correspond taking gradient flow time y + oo (which is the correct solution for noise free problems at infinite depth). In this limit J = N so we have that
T N TrA (iss + +A)+
(129)
For powerlaw features Aς ~ k- we find an approximate solution for iw
k
iws N"
(130)
Thus the loss at width N and context length P can be approximated as
lim (,,,,,.D)- Σ (iwss) Au (3t)2 P
1- (1+k-uNy N
(131)
(twy + )x
investigate the regime where performance limited by context P. To access an approximation for this regime, we take N + D and iw + 0. In this case, have iu iwis. Since 1 1 as iw O. introduce the leading order behavior for small iw
Context Bottleneck Following the same logic,
ww iu 0
(132)
P
28

---

Published as a conference paper at ICLR 2026
where is (currently) unknown value. Plugging this into the self consistency equation, identify the value of r
P = P + Σ A = 2r=
(133)
which gives the scaling
P
We can now use the final value of the resolvent
li iw (iw + A) = (r A)-1
L = Σ 2 (32)) (r Ab)
k- (1+ ~ P-v3.
Σ
(134)
Rank Deficiency/Null Space Interpretation These last two scaling laws express the simple fact that rank N or rank P matrices only allow the top N or P eigendirections to be learned (Bordelon et al., 2024a).
G ENHANCING REALISM OF THE MODEL
G.1 DIFFERENT PARAMETERS FOR EACH LAYER
In this section, we analyze the case where the model is allowed distinct attention layers along the residual stream (r')2 instead of a single attention layer matrix T which is applied I, times recurrently. We focus our attention on the RRS setting at large width and large context length where the model still exhibits nontrivial depth and pretraining time scaling laws. We first consider the noise free setting x² 0 before considering the role target noise.
G.1.1 RRS COVARIATES -> ISOTROPIC I LAYERS REMAIN IDENTICAL
First, we note that by the rotational invariance, each of the matrices is isotropic I' 70 I they are initialized as isotropic (such as zero initialization). As consequence, the predictor on dataset X, and test point x, has the form
((x
T TP
(135)
E=1
A=U
We note through simple inductive argument, that the matrix product can be rearranged into form that is clearly permutation symmetric in the variables {
Σ Te k=
II
I
1 X
Σ (-1)--1
1 L.P X X
Σ
Th (136)
n=1
Thus because the predictor is permutation symmetric, the loss function is also permutation sym- metric in the (nr} variables. We therefore expect permutation symmetric dynamics and solution where 7t = ? provided the initial condition is symmetric 7²(0) = 7(0). Indeed, analyzing the gradient flow from 7d for all leads balanced solution where all of these are equal since
i 2(7 a-e
à 0.p' ((T ve, fI,....).
(137)
Thus gradient flow will maintain balance in these parameters. Further, this flow will exactly match dynamics of the recurrent model if the learning rate upscaled by We plot these dynamics and show balancing of the (l) variables in Figure 8.
G.2 DECOUPLING WEIGHTS IN ATTENTION LAYERS
We note that for the ISO and RRS settings, we can exploit similar symmetry arguments as C.2 and E to argue that the gradient updates for W.. We. W, are isotropic and that W. gets update in the w. direction. The gradient flow can be expressed in terms of the original gradients on I
dL dr
0, W.
i (x, u,
(138)
Ør ow,
29

---

Published as a conference paper at ICLR 2026
0.020
10
0.020
1-5
0.015
100
t=10
0.010
t=50
0.005
0.000
200
0.000
40
60
80
100
120
40
60
80
100
120
(a) Equal Initial Condition (0) =
(b) Unequal Initial Condition 7((0)
7 (0)
100
110
10P
1.a
10a
10
1000
2000
3C00
4000
5000
(c) RRS Powerlaw Scaling
(d) Effective Step Sizes Across E and L
Figure 8: Pretraining on population gradient flow with RRS covariates under decoupled layers [I') is equivalent to coupled layers (T' = T) if the initial condition symmetric. We plot for depth L = 120 trained on D) = 120 dimensions with P = 100 and a = 0.5. (a) If the ICL model initialized with all 7 y and pretrained with RRS covariates, then the symmetry is maintained throughout pretraining. (b) In the absence of symmetry in the initial condition, there is not symmetry the final configuration. (c) Dynamics for layer-decoupled reduced I' model. The loss dynamics under powerlaw RRS covariates exhibit the same powerlaws. (d) The different scale factors remain balanced throughout gradient flow * (t)) = y(t) for all l E {l,..., L} due to permutation symmetry.
30

---

Published as a conference paper at ICLR 2026
Lx2
L-31 PIKSEErOD
Satvaccry
MIASETruL
Varying depth softmax, softmax+MMLP, multihead attention and three. Softmax attention networks exhibit clear depth separation. The dynamics become more consistent across depths for increasing number of heads with softmax attention.
Figure 10:
As before, under the assumption of small initial conditions, we can reduce the loss to collection of ODEs on scalars representing the scale of each weight matrix
[WW,, Why Wag Wk: Wz wyy S2A WoWly (I-Li-L'' (
-1--
(139)
Gradient flow dynamics on this loss function can reproduce the dynamics of the decoupled self attention model. Under the further assumption that w/ wy 1, we can simplify the loss further to
[(w,s Way Wk. w ΩA [I - w,,, (wx)) AA] 22
(140)
We note that ue will be updated twice quickly as the other weights under gradient flow. To achieve balance, we can initialize it to have w. (0) = /2wx(() and have wy(0) wx (0) (0) a. In this case, the loss can be further reduced function of single variable
[(w,, Wag Wk, wr) ΩA [ I.-
w1) 2L
(141)
Under source and capacity assumptions, this will generate the following dynamics at large depth L
59 5742
w(t) = t
L ~
(142)
G.3 SOFTMAX ATTENTION
We also provide numerical experiments with non- recurrent softmax attention, decoupled layers, and Adam. this context we use CompleteP scaling for the learning rate (1( (Dey et al., 2025). The results are provided Figure (c).
10-1
Supplementing this figure is an equivalent figure for non-recurrent architecture with alternating softmax at- tention and MLP with GELU activation, given by Fig- ure
Figure 9: Softmax attention model with MLP Blocks on the residual stream also benefit from increasing the depth.
31


Reviews:
[{"review_id": "1eU5fIM1gn", "reviewer": "Reviewer_KSDs", "summary": "This paper studies deep linear self-attention by analyzing the corresponding solvable model. The authors systematically investigate how depth of the model, width, context length, and training steps affect the solution of the model. Specifically, the authors focus on three distinct types of data (termed as ISO, FS, and RRS), where the authors reveal that depth is unnecessary for long contexts on ISO and FS, along with a series of other results that characterize the gradient flow dynamics. Furthermore, the authors derive a separable scaling law for the RSS setting.", "questions": "1. As I'm mostly concerned about the role of the depth, which plays an important role in the novelty of this work, can the authors justify the validity of assuming equal weight matrices across layers and the corresponding generality?\n\n2. Furthermore, can the authors discuss the difficulty brought by varying weight matrices w.r.t layers and how the current framework can still be applied in that case?\n\n3. Is taking the joint limit of $P, K, B, D, N$ necessary for the results presented in this paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well written and well organized. The presentation is very clear and the flow of this paper is consistent, which I think can allow the readers to easily appreciate the core contributions of this work (the summarized theoretical results along with immediate numerical experimental results). \n\n   The proof is also easy to follow (though I did not check all the details): first transforming the gradient learning dynamics to an equivalent linear model, which has a simpler dynamics, then diving to this new dynamics under different conditions of the covariance. Although this general idea is not new, incorporating the depth to the analysis is novel.\n\n2. The derived theoretical results indeed consider attention in the multi-layer case, which I believe is an improvement over prior works. The authors demonstrate when and why the depth can be necessary. The summarized results are indeed novel and interesting.", "weaknesses": "While the results are interesting by considering the depth, I think the setting is still not significantly novel compared to prior works. In particular:\n\n - While equation (3) indicates a dependence on the layer depth $l$, the induced parameter $\\Gamma$ in fact does not depend on $l$, because the matrices $W_i$'s are treated equally for different layers given one specific $i$. Instead, this dependence on $l$ is replaced by a simple summation over $l$ in $\\Gamma$. As a result, the corresponding analysis in fact does not provide significantly novel analysis compared to prior works in this line of research, i.e., studying the gradient flow dynamics of the induced parameter $\\Gamma$ (which is still a linear regression) as a proxy of the true weight matrices of the attention model. It remains unclear whether doing so can really capture the effects of depth.\n\n- The RSS is positioned as the most general case, but the randomly rotated and structured across contexts covariance still cannot effectively capture  the essence of task diversity. In addition, the theoretical framework is built on taking a very specific join limit where $P, K, B, D, N \\to \\infty$ with fixed ratios. While convenient for analytical tractability, this obscures efects that are relevant at finite scales. \n\n- Due to the aforementioned limitations, the generality of the derived results remains unclear.\n\n\nMinor: As this paper considers solvable models and scaling laws of attention, I think the related work [1], which also studies a solvable model for attention and its scaling laws, could be discussed a bit in the related work.\n\n[1]. Lyu et al.  A Solvable Attention for Neural Scaling Laws. ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXkXZj12OH", "reviewer": "Reviewer_B4Ac", "summary": "The authors tackle the problem of how should we allocate depth, width, context length, and training compute when scaling transformers for in-context learning (ICL) on regression tasks. The paper analyzes a deep loop linear attention transformer trained (via SGD) to do linear regression in context, without finetuning at test time. It studies three task regimes: (i) isotropic data, (ii) fixed but structured covariance, and (iii) randomly rotated structured covariance (task distribution shifts every context), and derives dynamical equations and asymptotic scaling laws linking pretraining time, width, depth, and context length. The contribution lies in a unified, theoretically grounded scaling law that distinguishes the roles of depth, width, context length, and training time in ICL.", "questions": "1. Your theory and experiments focus on linear regression tasks with (mostly) linear attention, Gaussian feature distributions, and controlled covariance structure. How confident should we be that the same depth–width scaling conclusions hold for nonlinear Transformers trained on natural language, vision, or multimodal data? \n\n2. You mostly analyze single-head linear attention plus residual depth. How do you expect multi-head structure and MLP blocks (i.e., actual transformer blocks) to affect the depth vs. width story? For example, could significantly more attention heads (possibly also scales with D) also benefit the learning process?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I appreciate that the paper gives a concrete, theoretically grounded answer to a question that is widely discussed in practice: how should depth vs. width vs. context length scale for in-context learning? Instead of treating “bigger model = better,” it isolates when depth specifically matters, and ties that to properties of the task distribution (shared vs. varying covariance across contexts). Specifically, the model decouples network width $N$ from the problem dimension $D$ when studying the scaling law, and the use of loop transformer ensures that the total number of parameters does not go up with the number of computes---which I believe provides a decoupled and generic test bench that adds to similar work studying scaling with linear model.\n\n2. A demonstration that the usefulness of depth is task-distribution dependent: If all tasks share the same covariance, depth is asymptotically unnecessary (long enough context suffices, as the model weights can \"encode\" the covariate information). If covariances vary across contexts, depth is fundamentally valuable, even with infinite context (reflecting the philosophy of test-time compute). \n\n3. A good match between theories and experiments.", "weaknesses": "1. The entire analysis is built around linear regression tasks solved via in-context learning with (mostly) loop linear attention. I didn't find much discussion surrounding the use of loop attention block. One benefit I could imagine is the decoupling between total model weights and the depth. However, the use of loop could potentially restrict the model's expressiveness, where model could possibly implemented higher-order optimization algorithm, e.g., Newton's step rather than gradient descent [1], and the model might demonstrate different scaling behavior, especially in depth. \n\n2. The key conceptual result is that depth becomes essential when task covariances vary across contexts (“randomly rotated structured,” RRS). But the diversity they model is very specific: random orthogonal rotations of a shared spectrum. That’s mathematically nice but arguably still a stylized shift. Real heterogeneity looks more like mixture of domains, sparsity structure, nonstationary label noise, hierarchical latent factors, etc. It’s not obvious that random rotations is the right stand-in for natural distribution shift.\n\n3. The paper argues its results are relevant to large-scale LLM design, but the experiments cap out at synthetic regression and relatively small controlled transformers. There’s no ablation on modern-scale architectures (residual blocks with MLPs, nonlinear attention heads, long-context finetuning) to show even qualitative alignment. So the significance for frontier models is still somewhat speculative.\n\n[1] Giannou, A., Yang, L., Wang, T., Papailiopoulos, D., & Lee, J. D. (2024). How Well Can Transformers Emulate In-context Newton’s Method? arXiv preprint arXiv:2403.03183.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wR2Gozthtc", "reviewer": "Reviewer_qeR8", "summary": "This paper studies deep linear self-attention trained on the in-context linear regression task, characterizing how performance depends on width, depth, number of training steps, batch size and data per context.", "questions": "The problem considered in this paper is interesting and potentially important. However, recurring issues with presentation and clarity make it difficult to fully assess the contributions. Improving the clarity would make the results more accessible for proper evaluation.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper studies an interesting and relevant problem of training a deep linear self-attention model on in-context linear regression tasks, considering different in-context task structures. Extensive prior work has examined one-layer linear self-attention trained on in-context linear regression with isotropic task vectors. Analyzing deep models and non-isotropic task vectors represents meaningful progress.", "weaknesses": "- My main concern is the looped transformer assumption, $W_i^l=W_i^{l'}, i\\in \\{k,q,v\\}$, which means that all layers have the same weights. The expressivity and optimization properties of a deep self-attention model can differ significantly with or without this assumption. Why would the scaling limits derived under this constrain reflect those of a real multi-layer attention model, which typically learns different weights across layers?\n\n- In the reduced model in Equation (4), all weight matrices within a layer appear to be merged into a single trainable matrix $\\Gamma$, effectively making the self-attention layer \"shallow\". Since gradient descent dynamics and the loss landscape are sensitive to such reparameterization, the true loss landscape probably differs from those shown in Figure 2 and Figure 4b. If this is indeed the case, it would be helpful to explicitly highlight this distinction.\n\n- I suggest that the authors perform another round of proofreading and polishing. There are presentation issues that make the paper unnecessarily difficult to read smoothly. I list some below.\n\n  It appears that manual vertical spacing commands have been used in several places of the paper. The formatting on page 14 seems irregular.\n\n  The clarity of Equation (3) could be improved by specifying the dimensionality of weight matrices. \n\n  The authors use inner-product and transpose notations interchangeably. The expectation notation $\\mathbb E(\\cdot)$ and the angle brackets notation $\\langle \\cdot \\rangle$ are also used interchangeably. Adopting consistent notations throughout would enhance readability.\n\n  The symbols $\\boldsymbol X, \\boldsymbol y$ in Equation (4) seem to be undefined.\n\n  The symbol $i$ is used inconsistently: sometimes as an index and sometimes as the imaginary number. In particular, it is undefined in Result 7. Clarifying its meaning in each context would avoid confusion.\n\n  In Equations (13), (15), it appears that a scalar is being added to a matrix, e.g., $(1-L^{-1}\\gamma\\Lambda), (i\\omega+\\Psi\\Lambda)$. Please check these terms for dimensional consistency. Additionally, the trace operator $\\text{tr}$ is used without parentheses, which could be ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cGFWBa69Tw", "reviewer": "Reviewer_NdPw", "summary": "This paper theoretically investigates in-context learning for linear regression using deep linear self-attention models, analyzing performance based on width (N), depth (L), context length (P), pretraining time (t), and data structure. Examining isotropic (ISO), fixed structured (FS), and randomly rotated structured (RRS) covariance settings, the authors find that depth primarily benefits ICL in the ISO and FS settings only when context length is limited; for infinite context length in these settings, increasing depth beyond L=1 offers no advantage. However, in the more complex RRS setting where covariances vary, depth significantly improves performance even at infinite context length. For this RRS case, the paper derives a Chinchilla-like scaling law and predicts a compute-optimal shape scaling, linking optimal architecture to task data properties.", "questions": "- Could the authors elaborate on the necessity of employing DMFT to derive the closed-form loss expression in Result 7? Is this approach required because directly analyzing the gradient flow dynamics in (13) is intractable, perhaps due to the lack of a known closed-form solution for the ODE governing $\\gamma(t)$ in the randomly rotated setting with finite width N?\n- The current analysis focuses heavily on the proportional asymptotic regime. Are the techniques employed amenable to deriving non-asymptotic results that might provide insights into the behavior of the system with sizes?\n- The derivation of the two-point deterministic equivalent using DMFT in Appendix A introduces notation that appears distinct from the parameters used in the main text to describe the Transformer model and its dynamics and it's hard to directly match (22) with the main result. Could the authors provide a clearer mapping between the DMFT variables/order parameters and the model parameters/dynamics described earlier in the paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The paper provides a comprehensive theoretical analysis of multi-layer linear self-attention models for in-context linear regression across three distinct covariate settings (ISO, FS, RRS).\n- This paper rigorously characterizes the training dynamics using gradient flow analysis, revealing how the model learns under different data structures and providing an interpretation of the learned estimator as implementing multi-step gradient descent with optimal step sizes.\n- The derivation of a Chinchilla-like neural scaling law incorporating time, width, depth, and context length for the RRS setting in the context of linear regression with power-law features is a significant theoretical contribution.\n- The application of Dynamical Mean Field Theory (DMFT) to derive a two-point deterministic equivalent for the loss landscape under random rotations represents a novel technical approach for analyzing complex learning dynamics in this asymptotic regime.", "weaknesses": "-The presentation of detailed proofs and derivations within the appendix could be improved for clarity and accessibility, making it challenging to fully verify the technical steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)