Transfer Learning in Infinite Width Feature Learning Networks
Authors: Clarissa Lauditi
Blake Bordelon
Cengiz Pehlevan
Venue: ICLR 2026 Poster (2026)
Keywords: Transfer Learning
Infinite Width
Kernel Methods;
Primary area: learning theory

Abstract: We develop a theory of transfer learning in infinitely wide neural networks under gradient flow that quantifies when pretraining on a source task improves generalization on a target task. We analyze both (i) fine-tuning, when the downstream predictor is trained on top of source-induced features and (ii) a jointly rich setting, where both pretraining and downstream tasks can operate in a feature learning regime, but the downstream model is initialized with the features obtained after pre-training. In this setup, the summary statistics of randomly initialized networks after a rich pre-training are adaptive kernels which depend on both source data and labels. For (i), we analyze the performance of a readout for different pretraining data regimes. For (ii), the summary statistics after learning the target task are still adaptive kernels with features from both source and target tasks. We test our theory on linear and polynomial regression tasks as well as real datasets. Our theory allows interpretable conclusions on performance, which depend on the amount of data on both tasks, the alignment between tasks, and the feature learning strength.

Published as a conference paper at ICLR 2026
TRANSFER LEARNING IN INFINITE WIDTH FEATURE LEARNING NETWORKS
Clarissa Lauditi', Blake Bordelon', Cengiz Pehlevan 1.3 John A. Paulson School of Engineering and Applied Sciences* Center for Mathematical Sciences and Applications Kempner Institute for the Study of Natural and Artificial Intelligence
Harvard University
Cambridge, MA 02138, USA
{cl {clauditi, opehlevan @seas htw.aas....aa..eeeeeeeeeoooee harvard.edu, blake@cmsa. faw..arvard.ee..
ABSTRACT
We develop a theory of transfer learning in infinitely wide neural networks under gradient flow that quantifies when pretraining on source task improves gener- alization on a target task. We analyze both (i) fine-tuning, when the downstream predictor trained on top of source-induced features and (ii) a jointly rich setting, where both pretraining and downstream tasks can operate in feature learning regime, but the downstream model is initialized with the features obtained after pre-training. In this setup, the summary statistics of randomly initialized networks after a rich pre- tre-traaning are adaptive kernels which depend on both source data and labels. For (i), analyze the performance of readout for different pretraining data regimes. For (ii), the summary statistics after learning the target task are still adaptive kernels with features from both source and target tasks. We our the- ory on linear and polynomial regression tasks as well as real datasets. Our theory allows interpretable conclusions on performance, which depend the amount of data on both tasks, the alignment between tasks, and the feature learning strength.
INTRODUCTION
Modern deep-learning models achieve remarkable accuracy by scaling parameters, computation, and data (Hestness al., 2017; Kaplan et al., 2020: Hoffmann et al., 2022). Yet collecting such large volumes of data is prohibitively expensive outright impossible in many settings. Transfer learning offers a principled escape from this data bottleneck: by repurposing representations learned on data- rich source tasks, it reduces sample complexity while improving generalization (Tan et al., 2018; Brown et al., 2020; Li et al., 2020; Isik et al., 2025). Therefore, understanding which properties of the pretraining and downstream data distributions enable effective transfer is critical for modern deep learning. Despite its empirical success, transfer learning still lacks a principled theory that predicts when will succeed. In this paper, we present a novel theory of transfer learning in multi-layer neural networks that elucidate the rich phenomenology of transfer learning.
Mathematically analyzing transfer learning is challenging, in part because representation learning in generic neural networks remains poorly understood. To overcome this difficulty, we focus on transfer after representation learning in infinite-width neural networks in the uPPmean-fiell pa- rameterization (Song et al., 2018a; Chizat & Bach, 2018; Yang & Hu, 2021; Bordelon & Pehlevan, 2023). In this parameterization, feature learning preserved even as the width of the network goes to infinity. We focus on supervised Icarning for both source and target tasks and derive results for the network performance after each phase of transfer learning. particular, we analyze (I) linear toy models of fine-tuning with adaptive kernels after feature learning source task and (2) non- linear models of transfer learning when both source and target tasks can operate in a feature learning regime. Our theory enables accurate predictions of the resulting network models for wide but finite neural networks.
Concretely, the contributions of this work are the following:

---

Published as a conference paper at ICLR 2026
We develop theory of transfer learning for randomly initialized infinite width multilayer perceptrons (MLPs). This theory, in its most general form, allows for arbitrary laziness on task-1 (pre) task-2 (post) training. In general (for models with more than one hidden layer), this theory is quite complex and involves non-markovian history dependence during both phases of optimization.
To gain more analytical tractability we specialize our theory two layer neural networks and investigate transfer learning in this setting. We analyze both fine tuning, where training on the second task is lazy, and rich learning where training on the second task can cause large changes in the hidden features. In the regime of finetuning, we can utilize results for the final feature kernels to characterize the predictors on the second task.
We develop linear toy models of finetuning where we can explicitly compute typical test losses on the second task when sampling random pre and post training sets. These linear toy models reveal many aspects of the phase diagram of (un)successful transfer learning. If the pretraining (source) task is data rich, fine-tuning strictly improves over two-layer linear model trained from random initialization. With limited data during pretraining, noise due to finite sample-size effects can cause negative transfer. For very rich pre-training, fine-tuning is sample efficient if and only if the target has significant projection on the pre-training source feature.
We extend this investigation beyond linear tasks to polynomial source/target tasks and on real computer vision datasets. Consistently with our theoretical predictions, when the pre- training task is data-rich, fine-tuning on the second task after rich pretraining improves performance and sample-efficinnyy. With limited source data, rich pretraining can induce representation overfitting by causing negative transfer. In this setting, rich learning on the second task is often favorable.
1.1
RELATED WORKS
Theory of Transfer Learning in Linear Models. Several works have studied how properties of a representation support generalization from few examples on a downstream task (Bordelon et al., 2020; Canatar et al., 2021a; Sorscher et al., 2022; Dhifallah & Lu, 2021; Gerace et al., 2022). A general result is that the geometry of the neural representation (kernel-task aligmment) controls the ability to learn new supervised task from limited data (Canatar et al., 2021b). However, these the- ories at infinite width would predict a fixed representation at initialization, not allowing for features to adapt during learning, for either the source or the downstream tasks.
Training Dynamics in Wide Networks. Recent years have seen significant research on the learn- ing dynamics of wide, randomly initialized neural networks. In standard / neural tangent parame- terization, wide neural networks are described by kemee methods (Jacot et al., 2020; Arora et al., 2019; Lee et al., 2020). In this same parameterization, corrections to this limit at large but finite width reveal weak (perturbative) feature learning corrections to this limit, linearizing the dynamics of hidden representations around their static infinite width value (Roberts et al., 2022: Zavatone- Veth et al., 2021). Alternatively, other works have explored parameterizations that allow infinite width networks to learn features, known as mean-field or μPP scaling, resulting in fundamentally nonlinear predictor dynamics. These works developed tools to study the representation learning dy- namics during gradient descent training in infinite width neural networks, which require adoption of the meea-field/zup scaling of network width (Song et al., 2018b; Chizat & Bach, 2018 Yang & Hu, 2021; Bordelon & Pehlevan, 2023; Bordelon et al., 2024c; Bordelon & Pehlevan, 2022). In this infinite limit, the dynamics for kernels cannot be lincarized around the lazy learning solution.
Learning in Wide Bayesian Networks. In contrast to gradient descent training, some works have pursued theory of networks sampled from a Bayesian posterior (Welling Teh, 2011). the in- finite width N - co limit with neural tangent kernel (NTK) parameterization and dataset size P held constant, networks converge to neural network Gaussian process (NNGP) models, which lacks representation learning (Lee et al., 2018). Beyond this kernel limit, extensions of deep Bayesian MLPs in NTK parameterization under the proportional limit P,N - so with P/N = a reveal ccalerrenormaliedd kernels after training (Li & Sompolinsky. 2021; Pacelli et al., 2023; Baglioni et al., 2024), with extensions to convolutional architectures (Aiudi et al., 2023; Bassetti et al., 2024). Large-deviation analyses in NTK parameterization further show kernel adaptation in finite-
2

---

Published as a conference paper at ICLR 2026
width/proportional limits (Fischer et al., 2024; Rubin et al., 2024b; Seroussi et al., 2023; Andreis et al., 2025). An alternative strategy is to adopt a fildd parameterization where even the N + limit at fixed P give rise significant changes in the kernels and predictor statistics compared to NNGP regression (Aitchison, 2020; Lauditi et al., 2025). Proportional limits in deep Bayesian networks have also been analyzed under the mean field scaling (Rubin et al., 2024a; van Meegen & Sompolinsky, 2024).
Transfer Learning in Wide Networks. Bayesian networks have been studied in a general multi- task framework in NTK parameterization in both lazy and proportional (P/N = aa) limits (Ingrosso al., 2025; Shan et al., 2025). The works (Ingrosso et al., 2025; Shan al., 2025) first introduced Bayesian transfer-learning framework in which the target model is regularized to remain in the vicinity of the pre-trained source weights (which are treated as fixed realizations of the source pos- terior). In (Tahir et al., 2024) the authors analyze deep linear models of fine-tuning on synthetic data, in the special case when the source task has infinite data and the kernel low rank, by show- ing that positive transfer learning depends feature similarity between source and target tasks, A recent work analyzes fine-tuning for two-layer mean-field models under KL-regularized empirical risk minimization (Aminian al., 2024). Here, we develop theory for fine-tuning using adaptive kernels from source task, and in a finite-data regime where sample fluctuations can hurt generaliza- tion. Plus, we extend the theory for non-linear networks and in the jointly rich setting where feature learning can also happen on target task.
Continual Learning Dynamics. Gradient descent training under continual learning in large-width networks under mean-fiell scaling has been studied in Graldi al. (2024). This analysis revealed that richer training dynamics could lead to more catastrophic forgetting in sequential multi-task learning, where the task distribution shifts over training time. Average accuracy across tasks was often maximized at an intermediate feature learning strength. However, these results have not yet been studied within a theoretical framework.
MODEL AND TRANSFER LEARNING DEFINITIONS
Before specializing specific transfer learning settings (such as fine tuning or linear networks), we first provide general framework where we subsumes all of our analysis. Our width N and depth L MLP architecture has the form
f(x) = N wh ((h)(x)), h 1+1
W (h) h VN
VD W
(1)
where æ E D an input to the model and the variables h E R represent the hidden preactivation features in the forward pass. During pretraining, the model parameters (W*) are optimized with
(S)GD on the source or task-1 dataset Ti ((x))) training points in Ti takes the form
(1) y/
) where the loss function on the Pi
CT,(0) EV.ET (Mi f(x, 0).
where l is the per-data-point loss function (e.g. MSE or cross-entropy). The parameter ? represents the richness/non-iieeric of optimization for task-1 pretraining with 711 - corresponding lazy / kernel learning (Chizat et al., 2020; Geiger et al., 2020; Bordelon & Pehlevan, 2022). This generates final set of parameters 01. Using the final parameters from pretraining 01 starting point for
transfer, we then run (S)GD on a second task To second richness parameter 72-
{(x}]) up
(2)
)) on a loss function using a k=1
Crp ( Ex.yET. l(15 f(x,0).)
(3)
We are ultimately interested in the solutions (and generalization performance) of the model that was post-trained task-2. We will refer the case where lazy learning on task-2 is performed 72 as fine-tuning'
This general setting be extended for Bayesian networks (see Appendix F), by considering the source task weights as quenched disordered variables for the target task Tz. Here, an elastic weight coupling controls the reuse of features during transfer learning.
Technically, control initialization variance, take N first before taking 72 +
3

---

Published as a conference paper at ICLR 2026
2.1 UTILIZING INFINITE WIDTH FEATURE LEARNING LIMITS
To make analytical progress on this problem, we focus our attention on infinite width neural networks N + oo trained with gradient flow. Because the networks are in the mean-field/pP parameterization, this infinite limit preserves feature learning for 7/ 72 > O Bordelon & Pehlevan (2022). If the weights are initialized i..d. with unit variance, and the model trained with SGD with learning rate = No-Nin for E {1, 2}. then the final predictor f(x) after post-training on task 2 can be expressed in terms of collection of kernels that include
Φ'xx, (xx,x',,, x',,tt) 1 (1h(()))) (x',")(x)))) N
(4)
where (t, 1') are distinct time values for training across both gradient flow time in task-1 1 and task- 2 (Yang& & Hu, 2022; Bordelon & Pehlevan, 2022: Lauditi al., 2025; Graldi et al., 2024). In the infinite width N - op limit, these functions become deterministic in their evolution and the neurons become statistically independent over the random initialization of weights. While this (in principle) provides a closed set of equations for the evolution of the network predictions f(z), the resulting dynamics are quite complex (see Appendix B). To gain more insight into the mechanisms of transfer learning we will next specialize to simpler settings.
2.2 TWo STAGE GRADIENT FLOW DYNAMICS FOR Two LAYER NETWORKS
First, we will examine the training dynamics for two layer networks where the dynamics in feature space are Markovian.
Result 1 (In data-poor downstream regimes, feature learning on target task helps) Consider two-layer (L 1) MLP trained with gradient flow on Tj for times (0,4)) with "n and then subsequently trained on task Tz for times t E (f1, 122 with richness parameter "22. The infinite width
N - so dynamics of the second model under gradient flow and with weight decay converges after training time & >> li to predictor [(x, 1) on test point x
f1 t) = (z(t))(h(z, f)))
(5)
where the average (-) represents an average over the measure of hidden neuron activations. The preactivations (1) JD (L)a and the readout variables z(1) = = (t) evolve as single-site stochastic processes (neuron decoupled) under the dynamical mean fied theory (DMFT) equations
h(z,t) x(()) +71 f ds Σ
(s) + 72 L ds Σ (3) WETz
NETI
=(t) + + 1/' ds Σ (s(s)((h((s))
+ [ ds Σ (s(s)((h((s)) METa
NETI
(6) x', while (, = - (rpr represents error signals for the training points in Ti and Tz. The predictor on the second task can be computed as f2(x,t %/2 -1 (z(t) (a,tt)(t)))))))))))))))))))))))) t))) for any t >> t..
(4) ((h,
and the average (-) is over both ab ~ NO,0 1) and X(x) GP(O, (o,) K2) where K (z
At a high level, this mean-field theory states that interactions between neurons asymptotically de- couple and all macroscopic properties of the network, including the predictors f1, f2 obey determin- istic equations (Sompolinsky & Zippelius, 1981). Intuitively, averages over the neural population N Em (h;;) should converge to population averages (g(h)) over a limiting density by a law of large numbers. This two-stage learning result indicates that there history dependence of the dynamics on the downstream task Ta that is inherited from the dynamics of pretraining on task Tı, consistent with prior works on mean field continual/transfer learning (Graldi et al., 2024; Aminian et al., 2024). In this two layer setting, this dependence only enters through the random variables (h(tt), 2(1)) which set the initial condition for the downstream task Tz due to the above Markov structure. This property does not hold in deeper models (see Appendix B). To validate the DMFT predictions, we numerically simulate the single-site stochastic processes using a Monte-Carlo ap- proximation of the population averages (see Appendix D). At each time step evolve the fields via
1
4

---

Published as a conference paper at ICLR 2026
Euler discretization and compute the induced feature kernel and losses Ti and Tz. We provide simulations of transfer learning using the above stochastic processes in Figures 3, 4 revealing that (1,, 72) can both impact the impact of pretraining on transfer learning.
One finding that we consistently see is that the amount of data P2 on T2 is small, that transfer learning confers greater benefits. Since the above model implicitly depends the dataset size, but does not explicitly quantify how transfer learning depends on Pl, P2, 71, 72 we next investigate the even simpler setting of linear networks.
2.3 TOY MODELS OF FINE-TUNING IN TWO-LAYER LINEAR NETWORKS
While the previous section described nonlinear two layer networks on arbitrary data, they did not admit an average-case analysis of generalization through averaging over the random datasets Ti, T2- In this section, we analyze tractable model which enables average case analysis of generalization for transfer, specifically deep linear models where @hh) = h. The source task T1 is generated by a linear target function Ys vDB, 3. Iz, random isotropic data 3z -N(0, ID). The same is valid for the target task To with yr,,, = 3 2t, and Tt,y NO ID). We pre-train on Ti with gradient flow on a squared loss. This induces an adaptive feature kernel (t) = (h()))(!)T) where the average is over the distribution of hidden neurons under Eq. 5. At the end of pre-training (4 + oo), we freeze this kernel and treat it as a fixed feature map for the downstream task Tz2 Fine- tuning then reduces kernel regression via gradient flow. The predictor on the downstream task follows the dynamics d f2) k(x)² ( f2())) where [k(z)]μ = K (a, xT,) for E [Pa] and K E PxPx is the kernel the target task data and (t) R is the predictor on task 2.
Our goal is to quantify how the amount of data and the strength of feature learning 7/ during pre- truining affect performance on downstream task.
To do so, we study three regimes.
1. First, look the population limit (Pi - oo) in Result 2. In this regime, feature learning during pre- pre-training on Ti is always beneficial for transfer.
2. Then we consider how finite data on T1 influences fine-tuning generalization in Result 3.
3. Finally, we look at the ultra-rich regime 7 oo where the adaptive NTK kernel becomes low rank. In this regime, finite data effects severely limit performance.
In the following, we give a sketch of the results for each case, clarifying both (i) the adaptive feature kernel after pre-training on Ti and (ii) the final test loss performance on To at convergence. This has to be compared with the test loss of an uninformed linear predictor, trained directly on Tz.
Result 2 (Data-rich pre-training consistently improves transfer) Consider the deep linear MLP of Eq. with d(x) = x. Train by gradient flow on Ti and feature-lerrning strength ?n > In the infinite-width limit N + cx, and then in the population limit P1 - oo at fixed D, the adaptive feature kernel after pre-training converges to (see for instance Bordelon & Pehlevan (2022))
Kt x(XXx)= = x I+ x
3,3 D
X
i.e, rank-one spike along B.3 Moreover, xe increuses strictly with n
With this adaptive kernel from frezze the features and fine-tune the readou on Tz. In the propor- tional limit P2, D + with P2 = MzD and for a fixed source/target alignment 02. 13 Br, the downstream test loss at convergence is
2x 2x22z + (29)2323 1+x (1+x
L(v2, =1
=1 (1-12)
(1 - V2).
(8)
Thus fine-tuning with the adaptive kernel always better than the baseline would obtain from random initialization, whenever X > 0 and + O.
V2, which one
For gradient flow from random initialization in L 1 linear network (Bordelon & Pehlevan, 2022) showed that the kernel has this spiked form with X /1 + 7 1 (see Appendix C.1). Similarly,
5

---

Published as a conference paper at ICLR 2026
for Bayesian networks in the feature learning regime at infinite width, the x" at the end of training can be solved for exactly in terms of 7/1 as the solution degree L polynomial Lauditi et al. (2025).
Under gradient flow finetuning T2. the enror vector which defines the generalization error is vu(() Br ((t while the instantaneous training errors are A D- 0-1/2X, vo(t). The key quantities which determine the generalization dynamics on T/ are the correlation functions
1 Ca(1,)) = (1) A(!), (t, = = mo(()) vo Can (1) D vi
(9)
P2
with v1( to
vh x From these, train loss E(() and test losses [(I)) correspond respectively
E(t) Ca(tt)) ((t) = (t.t).
(10)
In the join limit P2. - oo with P2/D, each entry of the vectors (4) (t), on (1)) be- comes i.idd. and described by stochastic process known as single-site process. This is summarized by the DMFT fixed point equations, from which one can derive correlation functions as in Eq. 10, which are deterministic in the limit.
In the results that follow, the derivation for To test loss is similar in spirit, with the addition of correlation and response functions that depend specifically on the adaptive kernel after Tı. We restrict two-layer setting, even though we believe that the adaptive kernels after feature learning on Ti in the deep case have the same functional form as the one we study here.
Result 3 (Finite-sample size effects can harm fine-tuning gains) Consider the two-layer MLP of Eq. with L = 1 and 6(x) = x at infinite width. In the proportional limit where PI, D - oo with Pi = D, rescale " = h 2v/VD for feature learning to happen at infinite width. After pre-training on Tı, the adaptive feature kernel at convergence is
K K(X,X')
C2 3.3. - (93. +B ( D X"T, ( 1
X
I+
C3
(11)
i.e., low-rank deformation of the isotropic baseline: signal spike 3a3 noise spike gg and a crosstalk term _ + B.g We establish this novel result for the trained kernel in Appendix C.2 which incorporates finite data effects. The Gaussian vector g captures these finite-sample fluctua- tions of the Ti dataset and it is uncorrelated with Bg. Its covariance Cov(g) C is set by the train loss at convergence on Tn given by
1
A 1 X
VD W(t) wtt)).
C = lim
A(t) (t),
(12)
++ P1
VD N
The coefficients C1, C22 C3 are deterministic functions of (51, given by the DMFT saddle point equations.
With this adaptive feature kernel from Tr, freeze the features and fine-tune the readout on Tz- Call a, 3.3 Bi, 0g D B, the alignments of the target direction with the source and noise respectively. The downstream test loss at convergence (for a, 1, =0)
(1 2zU2)2 + EqU3
Cl,1, (1 ((1 22 (1+ c3V2) -22x2)²
(13)
which might can be higher than the baseline C (1 v2) depending on C12, C2 respective values.
With finite data, pre-training on Ti leads to an adaptive feature kernel as in Eq. after a short path integral derivation (see Appendix C.2). Computing the constants C₁ C22 C3 is in principle hard, because it requires solving for correlations and response functions from DMFT at limiting time. We leave them as constants and derive conclusions for some interpretable cases. We do not expect, in general, transfer learning to have a positive effect when crosstalk and noise components C1, <3 grow large compared to c2. In the population limit where V1 co, we expect instead Cov(g) - 0, thus recovering the pure signal spike when there are no sample size fluctuations.
With this kernel, similarly to the sketch of Result 2, we study the limiting dynamics of the error field (t) 3r k1/2 This time, together with the correlation functions (1(,,))) Cop (1, (') that

---

Published as a conference paper at ICLR 2026
define train and test losses, we get contributions from (t. B. and Cg bg. 21 (t) which we need to study at limiting time.
Because of the dependency many variables (i.c., V2, ag. ag. C1, C2, in Eq. 13 report the loss in the special case where a, = and ag = (see general expression in the Appendix C.2).
Notice that this reduces the linear-probe baseline & monotonically with C2; and worsens with increasing crosstalk ej in this special case.
122 for c C2 = c3 0; improves
Result 4 (Ultra-rich pretraining undermines fine-tuning performance) Consider the two-layer MLP of Eq. I with L 1, 6(x) and "1 ////DD at infinite width. On Tı. consider the balance condition @.WW www = O. When in - co, or equivalently for small weight initialization Wo W
wowlg then W wov is low-rank with v RRD. In the proportional regime P1 = U1 D, solve for v at limiting time through DMFT. The adaptive feature kernel after pre-training on TI is ( x') x X ( )X",,
" ,i.e
v D
(1 - vn) 3.3.
MI MIV (1 (B.9 + (99t X"T, (14)
K(X,X) - X
D 99" +
D
which is a rank-one kernel with signal Bg and noise g ~ (O, such that g L Bz. A noiseless linear target X collapses the features, and only the projection of Be onto this collapsed subspace is learnable. The asymptotic test loss for V1 E [0,1]
_ Br is exactly solvable iff Br E vp.n( Ultra-rich pre-training therefore
(15) with a, = 3.33. Bi, 0g = 1g Be the alignments with the source and noise respectively. In the data-rich limit 1/1 - 1, the learned feature collapses to the signal (v - B,) and the downstream loss to L = 1 which is the residual (unexplained) variance of yt-
C(v1, Og,, ag) ay) -1-
vI v1ay)
This result can be considered adaptive NTK after learning Ti (see Eq. 14 and Appendix C.3 for details). The loss of Eq. 15 does not depend on the amount of data 1/2 in T2, since any dependency P2 comes from how well it is possible to estimate single scalar coefficient in this rank-1 feature, which vanishes as P2 + co.
special case of Result 3. when there is no bulk component in the
3 TRANSFER LEARNING PHENOMENOLOGY
In the following, we illustrate the interplay between transfer learning, feature learning strength, sample size and task similarity leveraging our theoretical results in Section 2. We start with the fine-tuning setting, where data on both Tj and Tz tasks are generated by linear target functions, and then proceed to the jointly rich setting. allowing feature learning both tasks. By increasing the task complexity, we derive conclusions on the benefit of transfer learning from polynomial to real datasets.
3.1 -UNNTTNING
Infinite data on Ti In the population risk limit from Result 2, when V1 co, the test loss is monotonically decreasing function of source/task alignment a (see Fig. I(a)) and thus fine-tuning has always a positive gain from feature learning on Tí.
Finite data on Ti By contrast, when v is finite the features learned on Ti are noisy because of finite sample size fluctuations: the adaptive NTK (see Eq. 11) acquires, in addition to the useful sig- nal spike (controlled by c2), both noise spike (controlled by 03) and a crosstalk term (proportional to c1), as shown in Result 3. As consequence, the test loss is no longer decreasing function of source/task alignment Or (see Fig ((b).. If we suppose the target task having non-zero alignment with the noise 0g * 0, then transfer is most helpful in the low 1/2 regime and when source/target similarity a, is high; although, with enough data on Tz, both noise and crosstalk terms can corrupt the signal direction, making it convenient to learn from scratch instead of using transfer learning.
The simple alignment case a -0)) of Eq. 13 shows that there (i) larger 02 always helps, while (ii) e₁ always hurt, since it rotates the high-gain direction towards the noise. Instead (iii) c3 when the noise is uncorrelated with the target (ag, = 0) act as ridge (regularization effect) high dimension (see Appendix C.2).

---

Published as a conference paper at ICLR 2026
5,055
0y45.79
04
6,m2
09
00
0.5
0.4
P2
(a) 11 7 00
(b) D1 finite
(c) -
Figure 1: Fine-tuning from an adaptive kernel from TI. Dashed black: no pre- pre-training (linear probe).
(a) Loss is strictly decreasing with source/target alignment Os, (Resuli 2). (b) Non-zero alignment with the noise direction (ag = 0.1) can cause negative transfer high v2 = P2/ D (Result 3). (c) Test loss on Tz depends only on source data P1/ D and the alignments {dgg 0e)) (Resuli 4. In the panel, 0g = 0).
Large "o on Ti Consistent with Eq. 15 of Result when 0g (Fig I(c)), since a, [-1,1], then with this rank-one feature can only learn up C happens only when target task is perfectly aligned with the source task (i.e., a, = 1). This suggests that it is in principle harmful to have infinitely rich pre-training. We show in Appendix C.4 that this is consistent with what happens when fine-tuning a non-linear model on polynomial tasks.
a and the perfect interpolation
055
0.
X05
3.0
11
20.0
0.4
0.3
0.00
1.0
2.5
3.43
5.0
13.0
310
00
0.2
Pa
Pg
(a) Optimal
(b) ~ 0.1
1 0.1
Figure 2: Fine-tuning with adaptive kernels from Ti Losses vs vs and for different 7 values on Ti. (a) Linear model from Result 3 when = /PI(1 v1)x, = vix, C3 = p1 -1)xx with X イ 1 has optimal 7/1 at large 1/2. (b)(()) Two- ReLU MLP CIFARIO: source task is regression on {0, 1 classes; target task is regression on {0, 9} classes. Theory obtained by performing kernel regression on T2 from the adaptive kernel after Ti.
Real datasets To concretely show that most of the conclusions can derive from theoreti- cal models of fine-tuning are still applicable to non-linear models, we make some phenomenolog- ical comparisons. As anticipated for finite 11, theory from Result 3 predicts that the constants C1, C3 are functions of feature strength 71 and Vi. We make an ansatz for these functions at large Yi inspired by model in Result 4. The test loss of Eq. 13 will be then a function L(7,, V1, v2). When is finite and so the alignment between noise and target tasks is non-zero (i.e., 0gg # 0), our theory in Fig. 2(a) predicts that the optimal -eature-learning strength > (x2) large when 22 is small (variance reduction dominates), and it decrease as 1/2 grows (bias from feature drift starts to hurt). large 22, there exists an optimal value of feature learning strength ? that lowers the loss with respect to the baseline (see Fig. 2(a)). Similarly, after training non-linear model on CIFAR-10 with different 71 on Ti Figs. 2(b)/c) (which refers to performing kernel regression on To with fixed kernel from Ti, or equivalently to the lazy 72 - dynamics of Result I) show that larger ? yields lower test loss at small P2 (x 1/2), but the advantage shrinks and the curves collapse as Pa increases; with enough target data, pre-training feature strength matters less. Again, consistently with our theory (lazy fine-tuning 7/2 - O from Result 1), we also show in Fig, that on polynomial task high 72 can be detrimental when Pg is large.
8

---

Published as a conference paper at ICLR 2026
3.2 TRANSFER LEARNING OF POLYNOMIAL TASKS WITH NONLINEAR ACTIVATIONS
Low to High Degree Polynomials Kernel limits of neural networks strongly biased to fit their data with low degree polynomials when data is high dimensional and isotropic. This spectral bias (Rahaman et al., 2019; Bordelon al., 2020; Canatar al., 2021b) reflects the fact that kemnll methods learn eigenfunctions in order of decreasing eigenvalue (Novak et al., 2018; Belkin et al,, 2019; Zhi-Qin John Xu et al., 2020). By contrast, networks trained in the feature-learning regime can learn sparse polynomials from much fewer data and training steps (Mei et al., 2018; Dandi al., 2023b; Troiani al., 2024; Dandi et al., 2024). The staircase property (Abbe et al., 2021; 2023; 2024; Yang et al., 2025) explored by Dandi et al. (2023b) makes this hierarchy explicit in multi-index polynomial settings.
Inspired by the utility of feature learning sparse polynomials of Gaussian data æ ~ NO. I), we study transfer from linear source task to a quadratic target by employing the two-layer MI.P model of Result 1 in the jointly rich setting. Figure 3(a) shows that pretraining on the linear task (right panel) lowers the test loss on the quadratic target compared to training from scratch (left panel). The feature-learning strength "/2 T2 here accelerates early gains but it also induces stronger forgetting of the source features during transfer learning, as pointed out in (Graldi et al., 2024). Eventually, there is an intermediate value of 22 that minimizes both target loss on To and catastrophic forgetting on Ti
pre-training
Trassfr learning
No pre-training
Tansfer keaming
DNFT
Loos
Test LOSS
0.25
0.25
0.5
0.0
v-110
YeeLD
0.8
17
Steps
Steps
Steps
Steps
(a) Easy - hard
(b) Hard + easy
Figure 3: Test losses of a two-layer ReLU MLP vs steps for different feature learning strength 72
on Tz. (a) Low degree polynomial source task y1 (ar) D-1122 x with P1 = 1000, = 100 and
'1 1.0. Target task is (2(x)) = 1/-1/2 -))² with P: 100. (b) Source task Ho; 2) with Pi 1000 and 1.0. Target task: Hez (B2 x) with P2 600 and B1 -B 0.8. Solid lines:
gradient-descent on an N 20000 two-layer ReLU network. Dashed lines: DMFT theory from 1.
High to Low Degree Polynomials In Figure 3(b), we compare the model performances when learning a low degree Hermite polynomial target function from either a random initial condition or the features learned from high degree Hermite source task. In both cases, learning the target is speeded up by feature learning strength 72. Similarly to a grokking phenomena (Power et al., 2022; Liu et al., 2022; Kumar al., 2024; Fan et al., 2024), we conjecture that in this initial training phase the network begins memorizing its training set and slightly overfits, then after adapts features to the data, leading improved test loss late times. This adaptations of features happens faster when training with higher 72 (rich feature learning from Result 1). However, in this setting, because the pre-training on Ti makes the target model at initialization to rely on spurious features components that are not needed by the simpler task Tz. transfer learning has no benefit in this scenario compared to no pre-training performance.
3.3 ROLE OF TRANSFER LEARNING ON REAL DATASETS
Moving beyond synthetic tasks, we consider simple image regression problems. We start with 10. where a model pre-trained on two source classes then fine-tuned on two disjoint target classes. We compare the performance of a target model trained on this second task Tz from ran- dom initialization (Fig. 4(a) with the performance from data-rich source Ti (Fig. 4(b)). Here, transfer learning leads to a lower test loss compared to no- pretraining for each value of feature learning strength 7/2 In both cases, there exists an optimal early stopping time which minimizes the loss before slightly overfitting. We show that our DMFT theory from Result is well weel-predietive of this jointly rich setting. In Fig. 4(c) the distribution preac- tivations p p(h) of the target model shows that, as 72 grows large, feature learning makes p(h) highly
the same model when using features learned

---

Published as a conference paper at ICLR 2026
non-Gaussian. In Appendix B we also show that, similarly fine-tuning setting (i.e., linear probe) on real datasets (Fig. 2(b)(c)), feature learning on TI is crucial when downstream task is data-poor (small P2); with large P2 the model is able to rely more supervision signals from the data itself and transfer learning offers little additional improvement.
pre-training
DMIT
020
Test LosE
07
0.0
Steps
Steps
038
-3
(a)
(b)
(c)
Figure 4: (a)/(b) Transfer learning is beneficial for real tasks any feature learning strength 72-
Source task: classes 1/2 of CIFAR-10 with Pi = =0K and m = 1.0. Target task: classes 8/9 of CIFAR-10 with P2= 200. (c) Preactivation distribution of the target model for different '22. Solid lines: GD at convergence (N- = 20000, two-layer ReLU MLP); black dashed lines: DMFT from 1.
4 DISCUSSION AND CONCLUSION
In this work, we develop a theory of transfer learning infinitely wide neural networks under gra- dient flow. First, we provide the theory for non-linear MLPs, in the general setting which enables feature learning on both pre-training and downstream tasks. Here, transfer learning on polynomial tasks outperforms pre-training when moving from easy (low degree) to hard (high degree) bench- marks. No such gain is observed from hard to easy objectives, since the pre-trained model eventually biases the representation toward high-degree components that are misaligned with the low-degree task. On real vision tasks, transfer learning speeds up performance, showing a consistent improve- ment in test loss. Consistently throughout these benchmarks, feature learning on downstream tasks enhances performance with a data-limited target. Second, we study fine-tuning with fixed features from a pre-trained rich source. Our results illustrate how the source/target similarity, the amount on data and feature learning strength control the relative benefits of transfer learning compared to learn- ing from scratch. Here, different pre-traiming regimes lead different conclusions on fine-tuning benefits. (i) If source task is data-rich, fine-tuning is always beneficial; (ii) for finite source data, noise from finite samples can corrupt fine-tuning gains; (iii) when source task is infinitely rich, the target task is exactly solvable if and only if it is perfectly aligned with the source.
This work is limited for many reasons. Our data-average case linear toy models rely on simpli- fying assumptions, such isotropic data, that enable closed-form analysis but limit the scope of quantitative predictions. Relaxing these assumptions, for instance by incorporating data-averaged study of structured or heavy-tailed data, would help bridge the gap between our theoretical insights and the behavior of large-scale neural networks. Future works could also explore how representation learning in deeper networks enable transfer learning. Specifically, it could be interesting to study what number of hidden layers should be preserved during transfer learning (Bansal et al., 2021). An- other possible future direction could be connect our framework with curriculum learning, where tasks are organized in structured sequence rather than treated independently; our theory could help clarify when and why such curricula improve generalization and feature reuse.
5 ACKNOWLEDGMENTS
The authors would like to thank Stefano Sarao Mannelli and Luca Saglietti, as well as the members of the Pehlevan Lab for insightful discussions. C.L. is supported by DARPA grants DIAL-FP-OSST and AIQ-HR001 12520041. B.B. acknowledges support from the Center of Mathematical Sciences and Applications (CMSA) of Harvard University. C.P. is supported by NSF CAREER Award (IIS-2239780,, DARPA grants DIAL-FP-033 and AIQ-HROO 12520441, the Simons Collaboration on the Physics of Learning and Neural Computation, and the William F. Milton Fund from Harvard University. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Emmanuel Abbe, Enric Boix-Adsera, Matthew Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning, 2021. URL https:
//taxxv.orgaaas///... 3.573 10573.
Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics, 2023. URL https:t/ptaatte org/absr//s.. /abs////////// 2302. 10002.11050 055.
Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. The merged-staireeen property: a necessary and nearly sufficient condition for sgd learning of sparse functions two-layer neural networks, 2024, URL https: wawaaaivvv..oo org/ abs 2202 08658.
Elisabeth Agoritsas, Giulio Biroli, Pierfrancesco Urbani, and Francesco Zamponi. Out-of- equilibrium dynamical mean-field equations for the perceptron model. Journal of Physics A: Mathematicaal and Theoretical, 51880008500, January 2018. ISSN 1751-8121. doi: 10.1088/ 1751-8121/aaa68d. URL http: : dxx doi . www..org//l.o 10..0.118811111 9121/aaa68d.
Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks, 2020.
URL http:: http:://rxvvvvoo .ogg abs 19109991111. 08013.
Riccardo Aiudi, Rosalba Pacelli, Alessandro Vezzani, Raffaella Burioni, and Pictro Rotondo. mechanism for feature learning in overparametrized con- volutional neural networks. Nature Communications, 16, 2023. URL attps://app. semanticscholadt wttp::sscccccoaa.oo/oouuupuuuu ID:
Local kernel renormalization
Gholamali Aminian, Lukasz Szpruch, and Samuel N Cohen. Understanding transfer learning via mean-field analysis. arXiv preprint arXiv:2410.17128 2024.
Luisa Andreis, Federico Bassetti, and Christian Hirsch. Ldp for the covariance process in fully connected neural networks, 2025. URL httts::/aaxxiii wrtpivi.org/aaaaaaa / h2w.:.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks, 2019. URL 02281.
wrg.:/vv.orrraooo www.or///////////////o.o/eeeee/e /1810.
P. Baglioni, R. Pacelli, R. Aiudi, F. Di Renzo, A. Vezzani, R. Burioni, and P. Rotondo. Predic- tive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit. Phys. Rev. Lett., 133:027301, Jul 2024. doi: 10. 10.1103/PhysRev..e 133.027301. URL https: httt:/liink.aar wwp.liinin.p.........mm org. htt///oo/1100/0 htw..111.////.eeeeeeett... 133. 027301.
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Sub- dominant dense clusters allow for simple learning and high computational performance neu- ral networks with discrete synapses. Physical Review Letters. 115(12), September 2015. ISSN 1079-7114. doi: http///psseeeet.1........ PhysRevLett 115. 111.1221 128101.
URL http: dx. doi
Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes. Proceedings of the Na- tional Academy of Sciences, 113558):E7655 E7662, 2016. doi: 10.1073/pnas. 1608 103113. URL https: www. wwws. org/ doi abs /10.///10..0 1 /pw.......... 6081 03113.
Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1): 161-170, December 2019. ISSN 1091-6490. doi: 1..1773/nas.19086361111 URL http:
//dx. wtt.:doi..rgg1... 1073 /paas.n... 1998833617..
Carlo Baldassi, Clarissa Lauditi, Enrico M. Malatesta, Gabriele Perugini, and Riccardo Zecchina.
Unveiling the structure of wide flat minima in neural networks. Phys. Rev. Lett., 127:278301, Dec 2021. doi: 10. 10.110///yssev 128.27.301. URL satps://lingg aps org/ dw// 10.
11 11 1103/PhysRwi.ette 127. 177.273311.
11

---

Published as a conference paper at ICLR 2026
Carlo Baldassi, Clarissa Lauditi, Enrico M. Malatesta, Rosalba Pacelli, Gabriele Perugini, and Riccardo Zecchina. Learning through atypical phase transitions overparameterized neural networks. Phys. Rev. E. 106:014116, Jul 2022. doi: 10.1103/PhysRcvE.006..... 1ttt://11.1111///eeeeee
URL
https: /tt///innkk. aps. org/ dor doi /10. 03 P233//hyysee..... 106. 01 11116.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances neural information processing systems, 34:225-236, 2021.
Federico Bassetti, Marco Gherardi, Alessandro Ingrosso, Mauro Pastore, and Pietro Rotondo. Fea- ture learning in finite-width bayesian deep linear networks with multiple outputs and convolu- tional layers, 2024. URL https: //w.i//aavvv/////////aa/.aaaaaaa org/ abs 2406 03260.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-
learning practice and the classical birs-variance trade-off. Proceedings of the National Academy of Sciences, 116632) URL http:
July 2019. ISSN 1091-6490, doi: (0.1073/pnass19003771.1. doi org/ 10. 1ttp:/11.111//////
1903070116.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynammi field theory of kernel evolution in wide neural networks, 2022. URL https::////axii /abs//////// //222055 2205.... 09653.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2023(11): 114009, 2023.
Blake Bordelon and Cengiz Pehlevan. Deep linear network training dynamics from random ini- tialization: Data, width, depth, and hyperparameter transfer. In Forty-second International Conference on Machine Learning. 2025. URL https://openrev/ net forum?id= SEj9uopOWP.
Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMLR, 2020. URL https:////// http///xxvvvrrr/ee/e /
/abs 20020 02561.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01099, 2024a.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws, 2024b. URL
wwwpwwwwwwwww.w./o.o//oo.oooaa/t 2atbb///44...... 17858..
Blake Bordelon, Hamza Tahir Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head trans- former dynamics, 2024c. URL
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distrrbutiion generalization in kernel regression. Advances in Neural Information Processing Systems, 34:12600-12612, 2021a. 2021,
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communi- cations, 12(1):2914, 2021b.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. Advances in neural information processing sys- fems, 31, 2018
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming, 2020. URL https: //wxxivaavvvv///////////aa www.riiiw....oooooooooo..ooooooe abs /1///1111111 07956.
Crisanti and H Sompolinsky. Path integral approach to random neural networks. Physical Review E, 0826),,, 2018.
12

---

Published as a conference paper at ICLR 2026
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. arXiv preprint vrXiv:2305.18270, 2023a.
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time, 2023b. URL https://arxiv. httaxaiivvvo..//. wwwpxiio.orr/...oo abs/ 2305. 18270.
Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. A ran- dom matrix theory perspective the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint arXiv:2410. 188888 2024.
Oussama Dhifallah and Yue M Lu. Phase transitions in transfer learning for high- dithesionnsi per- ceptrons. Entropy, 2304):400, 2021.
Simin Fan, Razvan Pascanu, and Martin Jaggi. Deep grokking: Would deep neural networks gener- alize better?, 2024. URL https: http://aaxvvv... orgg a240///s.s//......///// 19454.
Kirsten Fischer, Javed Lindner, David Dahmen, Zohar Ringel, Michael Krämer, and Moritz Helias. Critical feature learning deep neural networks. arXiv preprint urXiv:2405. 10761, 2024. URL
org/ abs 2405 2405.10765 10761.
Silvio Franz and Giorgio Parisi. Recipes for metastable states spin glasses. Journal de Physique 1, 1,5011):141111455, November 1995. ISSN 1286-4862. doi: 10.1051/pp111995201 URL http: dx doi
//dx/
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Staristical Mechanics: Theory and Experiment, 2020(11): November 2020. ISSN 1742-5468. doi: 10.1088//142255468abbhhhkk URL http://dss doi. http:///..do..ooor1.1111111./ 10742 5468/ / abc4de.
Federica Gerace. Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe, and Lenka Zdeborová. Probing transfer learning with a model of synthetic correlated datasets. Machine Learning: Sci- and Technology, 3(1):015030, 2022.
Cédric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborová. Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM Journal on Mathematics of Data Science, 6(2):400-42) 2024a. doi: 00.1137/23M1594388. URL https:
/ //doi. (iddii.or/////oooooooooooooooeo. 10. / 2 3M1 594388.
Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM Journal Mathematics of Data Science, 6(2):400 427, 2024b.
Jacopo Graldi, Giulia Lanzillotta, Lorenzo Noci, Benjamin F Grewe, and Thomas Hofmann. To learn or not to learn: Exploring the limits feature learning in continual learning. In NeurIPS 2024 Workshop Scalable Continual Learning for Lifelong Foundation Models, 2024. URL
/ttp://///eeeeeeeeeeeeeeeeeeeeee net http:///aaaaeeeeeeeeeeeeeooeeeaa Forumiid--YPBBYYHyee
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Moritz Helias and David Dahmen. Statistical Field Theory for Neural Networks. Springer In- URL
ternational Publishing, 2020. ISBN 9783030464448. doi: httpp://xx doi. www.oo..//ooo 10000000920000 0110:/9999-----4-----
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint
2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
13

---

Published as a conference paper at ICLR 2026
Alessandro Ingrosso. Rosalba Pacelli, Pietro Rotondo, and Federica Gerace. Statistical mechanics of transfer learning in fully connected networks in the proportional limit. Physical Review Letters, 134(17):17730L. 2025.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance in machine translation. In The Thirteenth International Conference on Learning Representations, 2025.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks, 2020. URL ht http://axxvvvooooo wwwpwaiwwww./////////// abs/ 07577.0777.. 07572.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXi::2001.0836/ 2020.
Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the tran- ats.:////aaab/////a/a///////
sition from lazy to rich training dynamics, 2024. URL 06110.
Clarissa Lauditi, Blake Bordelon, and Cengiz Pehlevan. Adaptive kernel predictors from feature- learning infinite limits of neural networks. In Forty-second International Conference on Machine Learming. 2025. URL https: http///ppppeeeeeeeeeeeeeeeeeeeee .et. / 1d=NLiWENWaa..
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz. Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes, 2018. URL https: //arxiv. org/abs 1aaabs/1711 1/1///111111 00165.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jaseha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent Journal of Statistical Mechanics: Theory and Experiment, 2020(12): doi: 10. 00.188//742-5466/abbeee
124002, December 2020. ISSN 1742-5468. //dx. doi. org/ 1088 24222/546//1 abc62b.
URL http:
Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X, 11(3), September 2021. ISSN 2160-3308. doi: 10. 11t3/physeev..1..330000 URL http://dst dei org/10.....o.oo 1103/
PhysRevX. 11.00.......11.....
Xuhong Li, Yves Grandvalet, Franck Davoine, Jingchun Cheng, Yin Cui, Hang Zhang, Serge Be- longie, Yi-Hsuan Tsai, and Ming- -suan Yang. Transfer learning in computer vision tasks: Re- member where you come from. Image and Vision Computing, 93:103853, 2020.
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, 2022. URL attps://arxee
org/ abs 2205.1033.
P. C. Martin, E. D. Siggia, and H. A. Rose. Statistical dynamics of classical systems. Phys. Rev. A, 8:423-437, Jul 1973. doi: 10. 10.1103/hysRevA...422 URL httph:://///ni aps org. doi/ 10. 1...103/Phyysie
8.4...4422 423.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33);E7665- E7671, 2018. doi: 10. 10.1003/pnas.18000 (0.1073/pnas 1806579115.
1at.mmas..065991111 1806579115. URL https: wwwpppna..oor//oo///ee.....e
Francesca Mignacco and Pierfrancesco Urbani. The effective noise of stochastic gradient descent.
Journal of Statisticul Mechanics: Theory and Experiment, 2022(8):083405, 2022.
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 11.9540-9550 2020.
14

---

Published as a conference paper at ICLR 2026
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeftrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study, 2018. URL https:// arxiv.org/ abs aww/aas///2 11000 08760.
R. Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo. A statistical me- chanics framework for bayesian deep neural networks beyond the wndinite-width limit. Nature Machine Intelligence, 5(12):1497-1507, December 2023. ISSN 2522-5839. doi: 10.1038/
$42256-023-00767-6 URL http: /wwp/xxx.ooooo/o//ooo//////..o.d wtt.:ddooooro..oo.ooooo 0100110000000/22225-0000-00-1010
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting on small algorithmic datasets, 2022. URL Btl.https://aaae htttt::://axxiv.orgg. abs/2201 02177.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL wrgg 806.066600 08734.
https: http:://arxiv.rrrr
Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, volume 46.
Cambridge University Press Cambridge, MA, USA, 2022.
Noa Rubin, Zohar Ringel, Inbar Seroussi, and Moritz Helias. A unified approach to feature learn- ing in bayesian neural networks. In High-dimensional Learning Dynamics 2024: The Emer- gence of Structure and Reasoning, 2024a. URL https:/ /opp/reveewww...c////// net. forum?id- 2mOSJ2MV2R.
Noa Rubin, Inbar Seroussi, and Zohar Ringel. Grokking as a first order phase transition in two layer networks. In The Twelfth International Conference Learning Representations, 2024b. URL
https://openreete
net/ ott.:e/wwweeieee..eeooooooaaaaaa
Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Marvels and pitfalls of the langevin algorithm in noisy nigh-dimensional inference. Physical Review X, 10(1), March 2020. ISSN 2160-3308. doi: 10.1 10.pp00//yysr 10.
01 011057. URL http: /x dx doi. oog.ioorr/ioooooooooooooooo org/ 10..10000000 1103 1103//hys/eexxx..... 10. 011057.
Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and thermodynamic descrip- tion of feature learning in some cnns. Nature Communications, 14(1):908, 2023.
Haozhe Shan, Qianyi Li, and Haim Sompolinsky. Order parameters and phase transitions of con- tinual learning in deep neural networks, 2025. URL 10315.
H. Sompolinsky and Annette Zippelius. Dynamic theory of the spin-glass phase. Phys. Rev. Lett., 47:359-362, Aug 1981, doi: 10.1103/hhyReewLett4..3. URL httpt:t////ii aps org/ 1tss/10.1003/Physeeeeeeee 3533
Mei Song, Andrea Montanari, and P Nguyen. A mean field view of the landscape of twy-rs neu- ral networks. Proceedings of the National Academy of Sciences, 115(33):E7665-E777,, 201 8a.
Mei Song, Andrea Montanari, and P Nguyen. A mean field view of the landscape of two-layers ral networks. Proceedings of the National Academy of Sciences, (15(33):E7665--7777) 2018b.
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry under- lies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43): e22008001 19, 2022.
Javan Tahir, Surya Ganguli, and Grant M. Rotskoff. Features are fate: a theory of transfer learning in dimensional regression, 2024. URL https: // wawii.iiiiv..//.i org/
08194.
Chuanqi Tan, Fuchun Sun, Tao Kong. Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey deep transfer learning. In Artificial Neural Networks and Machine CcarningIICANe 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 270-279. Springer, 2018.
15

---

Published as a conference paper at ICLR 2026
Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, and Flo- rent Krzakala. Fundamental computational limits of weak learnability in high-dimensional multi- index models. arXiv preprint arXiv:2405. 2024.
Alexander van Meegen and Haim Sompolinsky. Coding schemes in neural networks learning clas- sification tasks, 2024. URL /tttps://axxiv.oot wwwpriiww..o..vo.ooooooooooooooe abs/ / 2406. 16689.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference Machine Learning, ICML'I M 11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 978 7781450306199
Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks, 2022. URL https: / arxiv. org/ 2ttos22011.20
Jiang Yang, Yuxiang Zhao, and Quanhui Zhu. Effective rank and the staircase phenomenon: New insights into neural network training dynamics, 2025. URL https: /ttp::://axxi..org/bbb// 2412. 05144.
Jacob Zavatone- Veth, Abdulkadir Canatar, Ben Ruben, and Cengiz. Pehlevan. Asymptotics of repre- sentation learning in finite bayesian neural networks. Advances neural information processing systems, 34:24765-24777, 2021.
Zhi-Qin John Xu Zhi-Qin John Xu, Yaoyu Zhang Yaoyu Zhang, Tao Luo Tao Luo, Yanyang Xiao Yanyang Xiao, and Zheng Ma Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 2865):1746-1777, January 2020. ISSN 1815-2406. doi: 01.4208/cicp.oa-22220 4208 http:/ccp....220......
URL http: dx. doi.org/1C.
OA
A PRIMER ON DMFT
Dynamical Mean Field Theory (DMFT) is a method from statistical physics for analyzing high- dimensional dynamical systems in the presence of quenched random disorder. The disorder is "quenched" in the sense that it remains effectively fixed over the time scale of the dynamics. The method was originally introduced for classical spin-glass systems, where the disorder takes the form of random couplings between spins Sompolinsky & Zippelius (1981); in neural-network models arise from random connectivity Helias & Dahmen (2020), random data Mignacco et al. (2020); Gerbelot et al. (2024a), or random initial conditions for the weights Bordelon & Pehlevan (2022).
In specific setting, the relevant source of disorder is the random initialization of the model pa- rameters, which shapes the subsequent feature-learning dynamics. The key idea behind DMFT is that, in the limit of large number of neurons (width N - oo), the high-dimensional coupled dy- namics of all neurons becomes statistically equivalent to that of a single neuron evolving under an effective stochastic process. This reduction is possible because, at infinite width, each neuron decou- ple statistically and the interaction with the rest of the network appears only through macroscopic quantities (population averages), which become deterministic by the law of large numbers.
These macroscopic quantities are called order parameters in statistical physics jargon, and are respectively:
correlation functions, which describe how the neuron's ' activity at different pairs of time t, co-varies, capturing the temporal structure of the dynamics, and
response functions, quantifying how sensitive the neuron's dynamies are small per- turbations in the input or the effective stochastic field. They measure how tiny change introduced at time L' influences the neuron activity at a later time l.
To make this paper self-contained, dynamics through DMFT for a linear regression problem with isotropic covariates, which is relevant for all the theoretical results present in the main text.
report explicit example on how study gradient flow
16

---

Published as a conference paper at ICLR 2026
A.1 LINEAR REGRESSION WITH ISOTROPIC COVARIANCE
Let X E RNXP denote the data matrix, whose columns 2t E RN are P i..dd samples of an N- dimensional isotropic covariate, i.e. 3p NO, Ix) with j ,..., P. We study the joint high dimensional limit P. + 2o with v fixed. A linear teacher generates the labels according to
1
x B.
(16)
VN
where B. E R&V is the fixed target vector. A student with parameter vector B(t) is trained by gradient flow on the squared loss
1 | (1)1
((t)
| T (B, - 3(0)
(17)
2P
2P
where we introduced the emror vector (t) on the training set
x vt()), vo(t) = 3. -.t
(18)
N
Thus (t) measures the current mismatch between student and teacher in parameter space. Gradient flow on (()) reads
_ 3(1)
d
- (B(t) B.).
(19)
In terms of the error vector vo()), (t), this becomes
d vt dt
VN
P X + 8(t)B
(20)
(t) + 8(1)
where we introduced an initial condition 5(t) and the field vn (t). The key quantities which determine the generalization dynamics are the correlation functions
1 - (t) (t), (t, N vo(t) (t P
Ct t)
(21)
which correspond to train and test losses respectively.
We aim to characterize the joint distribution of the fields (t)),, mo((), 0n (4) over draws of the random disorder D {X}. ). To do so, we can start by defining the moment generating function of those fields with a Martin Peatin--iiggiaRAee Rose integral over trajectories Martin et al. (1973)
Z [jaد ijo.joo J Jo:1 = exp Σ (ia(t) A(t) + +.m. vo(t) + ju, (t) - v1 (1(t)))) D
(22)
This object, once derived, enables computation of arbitrary moments from derivatives with respect to the j variables j = O. For example, the two-point correlation between the vo variables is given by
8²
(t). Ojv.. jt) (t) Z Juo
(23)
Following a similar derivation to Bordelon & Pehlevan (2023), in the asymptotic limit, the Marko- vian (deterministic) system reduces to low-dimensional stochastic non-Markovian system after the disorder average (neurons decouple statistically), i.e.
avvl(t) -ui(t) S: dt' (t, (t)(tt)))o(t'))) + 8(1)8., ui(t)~ GP(O, 1 -P))
(24)
A(t) uz(() / at tutt, us(t) ~ CP(O,C..)
(25)
Ca(t,') = (L)))
(t,(t, ( (vo(()oo(!'))
(26)
Rs(L()) = (1)) (
()())
(Ovo(t) Qvo(t)
Roi(1, Jun (1))
(27)
where the Gaussian Process in Eq. 24 is disorder dependent (notice that this vanishes in the popu- lation limit oo), while the second term dt' R. (')))(t)) is called memory term, since it depends on early stages of the system at times t' < t. The same is valid for the A(t) vector. The averages (.) over the random variables {un (4), uA (1)).
17

---

Published as a conference paper at ICLR 2026
A.1.1 TEST LOSS AT LIMITING TIME
Since the system linear, the response functions are T.T.I. By taking Fourier transform, the DMFT equations becomes
kwvo(w) = -ui (w) Ra(w)oo(w) + Be
(28)
RA (w: (1 (1-v-
(29)
Roi (w: -H(ww
(30)
by defining H(w) (=) Solving for the error to (w).. this gives
(w(w) iw + R& (w) 3+ un (w))
(31)
Remembering the test loss definition of Eq. 232, we then get
On Ow (we, w²) (ub(w))o(() (w()))
H(W)H((w))
(32)
1- wー (a(w) (w HW)))((W))
At limiting time, we take the w.w w - 0 limit of the loss Cuo (w: w')
lim Cao (t, ww"'-0" 4000
lim (Ree) C (w, w²)
(33)
Using the equation
RA = - RAH 7
lim R&H = v w-+0
(34)
V
and by noticing that
iw
lim kWH((w) = lim -00
=
(35)
--+ iw v\ (iwH)
we can combine all the results to get the loss at convergence
(iwH)
lim Cro. (1, t) 1 1-v 10----RHAA RHRH
(36)
10000
= (1-v). v).
(37)
B DEEP INFINITE WIDTH TRANSFER LEARNING DYNAMICS
Using the dynamical mean field theory techniques of Bordelon & Pehlevan (2022), we can track the dynamics of preactivations h (zx t) and pre-gradients z" (x, which are defined
(x,t)) = VN
(1) t))
g () - = ()) 2 zt (æx (), (xx VN
w g
(38)
ला (1, 1), y(x)), which for mean square error is simple y(x) - f(x)). On task one Ti and times t E (0, 12) we have
We also introduce the variables A(x,
n((x,) t) = u (xx, m / da' S' at'
fdx
(,t') pz (z'))(((())))))) (') - (2, ,t,
o
(x, (x, + m / da' [ dt' [B'(z, ,,,'' + +p)('') f (0, (t,,,,,,,,)), ((' t'))
(x)) = P Σ ō(æ =²), we ~ GP(O, ('-) , CP(0 G
(39)
T'ETi
18

---

Published as a conference paper at ICLR 2026
where the correlation functions Φº, G are defined
(x x', ,e',, tt) = (6(h' (z)t)) (x',t))) ((,,,',t,))) t)g
(40)
and the response functions are
(x, x',x,'',t, = 6r'(x',))
(
(x, ( (0g
(u'(x', (
(41)
On task-2 where & E (2), we have the following dynamics
(x, = (x, + 11 dæ' S(" dt' [A x' t,t) - p1(x') th t,t (a',t)
12 dx' (" de
x. of (2',1)
xx, 1) = =2(( +11 f da' (-' dt' o', +
f) - - o' t, t')] ((h'(( ,)))
+ I da' S dl [A- ,,d', pa( G) (x, 1, 1,,, (1C((e'
(42)
where p2(2r)= t/ Exress 8(ar x²). The (ax features for / (11, takes the form
d
di f(x,t)= Σ Ex G (x, x', ,t, t) (o, ae,, t,t))(a' 1) f(x, t2) = O.
(43)
1.00
- 0.25
V:
Y:
Test Loss
DMFT
0.75
0.79
P2
Figure 5: Test losses as a function of target data P2 for different feature learning strength 7/2 on downstream task. Source task is regression two classes (0/1) of CIFAR with P: 1000 labels y E (-1, 1 and richness = 1.0. Target task is a regression on two classes of CIFAR (0/9) with P2 data points and labels y {-1,1)
C TOY MODELS OF FINE-TUNING IN THE PROPORTIONAL REGIME
In the current section, we will develop theories of transfer learning in the proportional regime, i.e. by allowing the data on both source and target tasks to grow arbitrarily large P1, co, such that 1/2 D (1)) is fixed, with D input dimension. In the following, will make three distinctions regarding the source task Tr In general, Ti is defined by teacher model B, IR.
ya,u DD
B x.
(44)
for random isotropic data αn ~ NO, I) and labels 1y- = 1. The student instead a two-layer
model
f(xw) = VD
[1]
Nro a T
VD w)) x
(45)
19

---

Published as a conference paper at ICLR 2026
with W E RND E RNN whose dynamics we study at limiting time - 0o after learning with gradient flow (GF) and from random initial conditions (0), a, (,(0) ~ NN, 1). Depending on P1, pretraining learns either (i) single rank-one spike aligned with the signal direction B3 (population regime), or (ii) a finite rank deformation composed of the aligned spike plus several spikes correlated with noise direction g RD and independent on the source direction B.. For this reason, we make distinctions in pretraining with the following scenarios: infinite data on Ti (i.e., 11 + oo); limited data on Ti (i.e., finite v1),, and feature learning strength To + 30 on Ti. In each of these settings, we wonder if the NTK kernels after feature learning on Ti have either positive or a negative effect on transfer learning. For that, we consider a downstream task To defined by a target rule
3: T. yt.u VD
(46)
and a fixed 122 f. We study gradient flow (GF) with the final NTK kernels from Ti, and in each case the dependency of the loss of T2 on the amount of data {u1, 22), the source/target alignment 1 3. B3 = a, and the feature learning strength Yo-
with T E 1),,,
C.1 INFINITE DATA ON Ti
As pointed out in (Bordelon & Pehlevan, 2022), by sending width N + oo first at fixed P1, the dy- namics of model such as Eq. 45 with Vec W, a} can be studied through the lens of dynamical mean field theory (DMFT). If we choose a MSE loss on Ti, i.e. L = TP, > (Ua.p - f,)), and study gradient flow get that one of the summary statistics we can track is the feature kernel ((t) (t(t)
zt 0 V from random initial conditions (O), (0) ~N 1). we
Prxx xP,, with (() vit W (1) being the preactivation vector. With isotropic data o E (O,,)). 1), it is possible to show that the kernel (tt) only grows in the rank one source direction y.y. (see (Bordelon & Pehlevan, 2022) for a complete derivation). In particular, the limiting kernel has the form lim;- ((t)= = I + XU.U." T, with = v1 + n
which is an increasing function
of the feature learning strength yo-
Now, if we allow the Ti dataset P1 + 2o at fixed D, by averaging over the data distribution we get a kernel after feature learning on Ti which has the form
X K <xx,x')= = x [ D) 3.3. x'
(47)
where we recall Bg being the source task vector. After pretraining on Ti, the adaptive feature kernel K (æ, is fixed. Thus, at infinite width, fine-tuning on T/ is equivalent to kernel regression with this frozen kernel (see Jacot al. (2020)). The natural objective in this regime the mean-squared error
c(t) = 2P2 (lfa(X.) 3e.12
(48)
and using the feature representation from Ti this becomes
((t) = 2P X T
I +
3.B. D
X
1/2 3(11 - [00)-x,BB3 1,12
(49)
with X, E DxPs, This leads to
d
((t) dc
(50)
aB
from which, by defining vo = B, 1 [I [ + ]3 1/2 3(t), we get
d - - I 3,
XX
P wo + 8(t)
(51)
dt
D
where 5(t) is a Dirac Delta function. We can introduce the following auxiliary fields
X T vD
E E
(52)
VD
11
XA
R
(53)
P 1
C.n
D B. .0
(54)
20

---

Published as a conference paper at ICLR 2026
which are self- averagingg in the asymptotic limit P. - due to concentration of the high-dimensional indices. The above dynamics becomes
d
dt 20 vi -00=0(00xxxCC00++000))
(55)
with initial condition (0) 3r.
C.I.1 DATA AVERAGE
Our goal is track the statisties of the random field By at limiting time, from which will be able to recover the loss function L at convergence. Once we average over the random T2 dataset, we expect this to depend on the finite sample fluctuations Tz since 1/2 = 1% aligmment with the pretraining source which we is controlled by hyperparameter = 3 B..
D fixed, and on the
In order do that, we develop DMFT or path integral derivation (Agoritsas et al., 2018; Sarao Mannelli et al., 2020; Mignacco et al., 2020; Mignacco & Urbani, 2022; Gerbelot et al., 2024b; Dandi et al., 2023a; Bordelon & Pehlevan, 2022; Bordelon al., 2024a).
First, we enforce the definitions of the fields and the vo dynamics by functional 8-constraints with conjugate fields (ôo. Á, 01, C.v): The resulting moment generating function (MGF) 2 of DMFT depends linearly on the data matrix X
f
dC dCcodC.o
[ [denddo
/ dAdA
[ [ d exp i / dtto (0.00 + u1 + x33.Co(t)- -8()))))
2
2T
2n
2T
- fifate fddtt dt
(VD X xToo) -i dtùr (xxx))
exp
1 exp i ifall ( (AD + v10)) +i / (1 8.. -2.))) .
(56)
Since the entries Tni NO0, 1) are i.i.d., we can average over the data distribution
i f drTrX
1 AT
VD
)1)
exp
ú
VD
exp
2 dtdt'
1
D Doo(t) ((tt)-ooole o(()))) A(t) + ! A (()))1()) oi(e))
((0)01()-01(1)))
(57)
1 (f daar 1 A() ((())o()) -(10))
exp
By defining the correlation and response functions
Cor.en (t,t) vo(() vo(t')
(58)
Ca,alt,t') A(t) Att') i (t,''),t') = A(A) (t) A(t)
(59)
(60)
i we can enforce their definitions with the use of delta functions, for instance
Bug. (1,11) = vn (1)
(61)
1 = /
dC (t, (')dCro.n (t,
1 exp (2 C (t,) (t,t') Co (t, (t,t))en(t). vo (t'))
2m
(62)
21

---

Published as a conference paper at ICLR 2026
thus getting
dC d (1t)) / j (11 / d (t, (t,t) (tt,t)
/
dC (Csn(t)dC
2 =
2TT
2x
2T
2x
di 0i (t, (t,t) (t,6)
D
dtC 2.
/ /dtd'/Coo Conren (t, t'r Cooooooo (t,
x
cxp
2
2
V22 / Caltt (12 / ( (tlt)))aAl)
X exp
/ dtdt' Ru (t, (t,t') Boo (t,t') + DJ f dtdt' R Rs, Ruo
X exp
Σ In Zoi Ca,,a Csus Cov. BAA
In Zs Croivo Rooirs) Σ j=1
X exp
(63)
where we collect every single site action (factorized respectively over input neurons and patterns)
dundün
/ Dv,
/ (t)
Z01 Cosss, Cov, Cov, Cro. R BAS =
exp
2m
2m
1
/ aid'' Eng.no
0 (1)) 2 [
x exp
2
/ dtdt' R (A,))0(()0 + (mn(()610)))
x exp
+ / dtôo + xx, +
x exp
(64)
dAdÅ
2 ʃ dtdt (all))s(()))))(()((()
Za Cro.no: &e CA,A
exp
2
dtdt' Con, vn (t, dtdt'Con,
(tt))(())(()))
fdtdt Redt'Ren. Re ((t)
exp
1/2
+i (()())((())
exp
(65)
C.1.2 DMFT ACTION
We now group all of the correlation and response functions, well as their conjugate order param- eters into list named q. The MGF can be written in the compact form
/ dg ( (-DS(a))
2 =
(66)
where S is the O(1) DMFT action
= 1 / (ICCso()) - 1
f dtdd' Coo (1,
V2
f aadd' (ull")s.s
+ +f |ddtt' RA,
Σ In Zo1 [Cs.s. Cou, Ceur BAA
Σ j=1
In Za Ceniero Repo
D
(67)
22

---

Published as a conference paper at ICLR 2026
As D + oc, the moment-generatin function 2 is exponentially dominated by the saddle point of S. The equations that define this saddle point also define our DMFT. First of all, we realize that at the saddle point
1
R
A
&1
(68)
1/2
Reg R
(69)
The resulting equations 2q
give
D
Ce (1) +
Σ
(Be1(1)), =0
=0
(70)
2
2D
1=1
1 Coonn (t,t')
1
1 (((())))))
+
(71)
2
2D
i=1
12 (t,(t,t)
1
(A(A)(())) j=1
(72)
2
2D
Here, Q) represents average over the single site distribution defined by the moment generating function Zo1- Similarly, 0, is the average over the distribution defined by Za. Regarding the response functions we have
RAA Σ
(AIAA())
=0
(73)
+ (oo(t)) =0 Σ f=1
i
Reo 01
(74)
D
Lastly, we have a collection of saddle point equations that defines the conjugated order parameters, which must vanish at the saddle point (Crisanti & Sompolinsky, 2018; Bordelon & Pehlevan, 2022)
Coo(1) = Čro.en CA.(11,))
(75)
C.1.3 HUBBARD TRANSFORMATION
Since we know that the correlation and response functions must take deterministic values in the limit D + oo, we represent the quadratic terms in the log-density in 01 Á(t) linear averages over Gaussian variables un (t), u. (t)
1
didi' (i - (exp ai (esp(=ifass(0)l /
exp
(76)
2v
1/2 f
(t, (t (1)))((())()))) = (exp (ep(- ) us.
=<xxp
exp
(77)
solve the integrals over the conjugated fields 0o, 01, A, and obtain the defining equations for the random variables of interest
After introducing these Gaussian random variables,
"1 (t) un (t) / dt' R
(78)
0,60 = (t) - dt' R, (t, - (t)
(79)
1 A(t) (t( + 5/
/ { (t,
(80)
23

---

Published as a conference paper at ICLR 2026
C.1.4 SIMPLIFYING THE RESPONSE FUNCTIONS
From the saddle point equations, we notice that the response functions involve averages over the conjugated variables {å, }1}, which we now argue can be replaced as derivatives with respect the Hubbard variables. For instance
/ II
exD (1 (n ua(()) (1( (1) (()
1 . dt' Reg, on ((,))A(())) ua
R RA (1,t(
2
122
dS(D)A(()
(1s( exp ( - ua)
/ dt' dd' Rwo,on (1, (())))(1)))
2T
U2
f at" (t)) [Coo,vo]
(t',t"))ss
(1(
(1)) U2
(81)
which holds via integration by parts and Stein's i lemma. The same can be said for Reo. (t,
@vvo(t) (ōvr (tt) )tT))..
Rep, in (t,
(82)
C.1.5 LIMITING TIME DYNAMICS
We can recognize that the response functions the above system will have time- translation invariant structure so that R(t, t') = R(t t'). We can therefore take a Fourier transform of these equations, which gives
&n (w)
(83)
iw + RR, (w)
(w)((w) = (1 (1-1; 2/2 Roo.oh 01 (w)
(84)
being 1/2 D. The same for the random variables which define the DMFT equations
(wo((w) = U1 (w) RA (w))lw)) - xCso(w)3 + 3
(85)
vo(w)- iw + RA. (w)
3: xCso(w) - us (w)
(86)
For compactness, we introduce a shorthand H(ww)
Similarly for A(w) we have
おo A(w) = RA. (w) (w).
(87)
The loss is governed by the two-frequency correlation function Cen, (w, w²) = (ww
By calling B. Br = a the alignment between source and target task, Cro.no (w, w') can be derived being
(w,))) H(w))((w)) 1 Cm ax ax(Cro(w) Coolw()) + (ad)))aa
1/2
(88)
By collecting Co,olw,, we get
H(w))(ww)
Con.oo(w.l =
1+x -ox
V22 "
(89)
It is important to notice that, soon as we send Nu + 0, which is the feature strength on source
task Ti, then X + and we recover the test loss
H(w)H((w)
Cop.ro (w, (w.w") 1 1- Uz RA (w (WW)))
(90)
24

---

Published as a conference paper at ICLR 2026
which is the would expect in absence of any dependency the source vector Ba, meaning without any pretraining on T1. So, the interesting setting is the one for which > 0 for a given alignment value In particular, we would like to study the sign of the term in the brackets [-] of Eq. 89 when t + oo or, equivalently, when w.w' 0 O.
First, we compute what the correlation (w)(w) is
(w) (m) (w) = (wi(w) + (w) (vo(w)3.) = (a
(91)
aRlHH 1 x RRaH'
(92)
Now to get the final result, we take the w. w - 0 limit of the loss ro (we, w")
lim Cro.no (1, (') = lim (iw) ((o.")" (w, (w) 1000000
(93)
waw -0
Using the equation
1
RA = 1/2
RAH
lim RAH = 1/ w-+0
(94)
and by noticing that
lim (uH((w) lim iw (wwH)
=1 1/2
(95)
we can combine all the results to get the loss at convergence
(iwH)
lim Coo. (t, 1-000
[] 10-2xx + xx2
(96)
1-V2 1/2 RHRH
2xa2v2
Xaa223
= (1 - 12) x
|
(97)
1 1+ X22
(1 + X22)
Some key observations about this result:
The loss only depends a² rather than directly. This reflects the symmetry of the
problem B, -3.
The loss is always lower than the original loss for any feature learning strength X > 0, since
[ X22a 2
L c1
S -1- 11)
(98)
+ X
which means that transfer learning has positive effect in this setting, as soon as feature learning happens on Ti. This is because during pre-training we minimized population risk by allowing Pı - 0o on Ti. a consequence, the NTK kernel rank-one spiked kernel in the source direction 3,33,: there
spurious noise spikes, and as soon as a > 0 (nonzero source-target alignment), transfer learning cannot hurt.
When O, meaning the target vector of the downstream task B lies in the orthogonal space w.r.t. Gr,. we recover the usual L = 1 - 1/2 learning curve for linear probes (Hastie et al., 2022). This happens also when X O, meaning if we choose a lazy pretraining on T. In that case, indeed, the NTK at initialization would have just the bulk structure with no spike aligned with the source.
If x 0o, which happens if the feature learning strength on the pretraining To oc, then
L = (1 a1)a)).
(99)
C.2 FINITE DATA ON Ti
In the proportional limit, i.e. when 1/1 = P. is fixed, the pretraining on Ti learns a noisy version of the source vector B, due to finite sample size fluctuations, and modulated by the feature learning strength "n on Ti- As a consequence, we expect an interplay between signal and noise components on the benefits of transfer learning on Tz.
D
25

---

Published as a conference paper at ICLR 2026
First, let's recall the network definition, which is
f(x) VD a Noo
(
1 VD
w
(100)
This means that GD dynamics 01+1 = 02 17, V 0, &. for the parameters collection 0 = Vec (W, .}}
2
and loss function L 27% 2p=1 (3/r fu,)² be written layer-wise
( = (0) + mo
Σ
((t))((t)
(101)
VD
a(l) a(0) + TNo
VD W(()()
Σ
(102)
having defined the fields
1
A (t) = vD
Xx(t) ER
(103)
vtt) B. VD (() a E(t) 11 ccut.ss)h(s)
(104)
Noo
(t)) = VD P XTA())ER..
(105)
As consequence, the feature matrix H(I) E RRNPP is
H(I) = W(0) + To Σ a X
(106)
VD
P'
hence, the kernel
(ttt) N
H (t) (t))
x
W (O) (0)
7
Σ
(E)))(()T +(())(()))
12a Σ (a(s,)))(())(
XT
N
D
D
sct
8
(107)
with
VD W (0)a(s)
E(s)
(108)
Nvo
8a N (aass) a(s').
(109)
If we proceed by substitution, we get
E(t) = VD W T (0)a(0) + T Σ (s)h((s)
" N
No0
Sci
VD W WT(O)a(0) + 17 W (0) En(s) N
2 VD
h
h(s') h(s)
w WT (0)
10 als))
NYu
N
D
= " Σ h(s)
+ 12 hh())
(3,,)))E()))
SCt
(110)
where we realized that DD W (0)a(()) = oly R) vanishes if we send N + x at fixed D, No
since (()) and (0) are uncorrelated at initialization, and that + W ( + Ip for the same reason. Plus, we know that the correlations Ca(s,s") A(a A(d') concentrates in the limit
D oo; the same holds for sa) &a² (s)a(s') in the N oo limit.
26

---

Published as a conference paper at ICLR 2026
Now, collect the time indices of matrix variables, for instance & Txx and solve for ç. thus getting
[ - 1- n/th
(111)
ERTXT
being Cx = (,, 8¹) the lower-triangular matrix and (+)t., = 8). In the same way, for the htt) E RD field, which we can get from short path integral derivation similarly to what we have done above (see (Bordelon & Pehlevan, 2025)), we have
h(() - u11) + Σ
R& (1, 8)0(6)
(112)
= =u(t) + Σ (t B. (5-E(s)-n) - Ca(s,s)))(s)
<<t
s'<a
with u(t) ~ GP(O, CA) and D1 = B. Again, by collecting the time indices we can solve for TxD h =
I I+ II - 1-- + C.
(113)
ERTxI
having defined
(t,s) ((t-))
(114)
(115)
By staring at Eqs. 111, 113 we realize that, since time operators do not create new spatial direction, both (E(I), ht)) D fields only grow in either the source direction Bg in the uncorrelated noise direction u(t), which comes from finite sample fluctuations of X. Consequently, {E(t), (t)) admit the causal decomposition
h(1) c(I)B,+ +
Σ (1))(()))
(116)
E(t) d(t)),++ +
Σ oC1
Reu (t)
(117)
where we replaced time-dependent scalars {c(t), d(t)}, which are functions of {n) 7o, v1 These represent the projection of the fields along the fixed teacher direction B,, while the (Rru Reu are the usual casual-time response functions which map the drive u(.) to the features h(-) and E(-). Precisely
I 1 -1-1 -1
Rbu
(118)
Rşu = -
OR
(119)
In general, deriving the limiting time of the fields (h(t), E(t)) requires study the limit of correlation and response functions as they appear Eqs. 111, 113, which is in principle hard. Because of that, in the following derivation will assume the casual decomposition in Eqs. 116, 117, and recover the feature kernel from that.
C.2.1
ANSATZ THE KERNEL STRUCTURE
Given the above discussion, and going back the kernel expression as in Eq. 107, we can now assume the kernel at convergence (f - oc) having the functional form
K(X,))= = X I D (93, + +Bog
02 B.g_ +
99
X
(120)
D
M
27

---

Published as a conference paper at ICLR 2026
where RR is Gaussian vector g 1 G3 such that Cov(g) lim 1 (t) A (t') which concentrates as PI + 0o. As 11 + 0o, we expect Cx 0. P.
i1 CR. and with Cτ
Instead, {ci, C2, ca) J are constants which are functions of (7, 70, )-
Notice that, differently from before, the kernel depends constants (3). We do not expect, in general, transfer learning to have a positive effect as soon the niose component 03 grow large compared the signal spike tuned by c2.
the noise direction 9 tuned by the
Again, we do gradient flow with this final NTK and loss function [(t) 2 X M 313((() x 3.1² and X E DxPs
d
3(1 1/2
XX (2-M)/3(()) P2
(121)
dt
from which, by defining uo = B: - 12)2B(t) as usual, we get
di 00 -1+
d 9 993 + B.g g3g
+
- B.3.
62
" )
C3
XX
vo 8(1)B,
(122)
P2
We can introduce the following fields
1
A
X Uo E R.
(123)
VD
VD
V1
XA
Rd
(124)
P2
CNv
BU. U1
(125)
D
1
Cge D 9
11
(126)
and getting the dynamics
d no v1( (og (c1g+ + c2B.) (eol)c(l())) - (013, ((o(()(((()))))) + cgg (23)CcoO1) 5(1)3,
(127)
dt
By enforcing the fields definitions, and by averaging the To dataset with 1/2 Pa fixed, we get the usual MGF of DMFT Z f exp exp (DD((a) with g being the collection of correlation and response functions while S being the DMFT action.
do path integral derivation similar to the one in Sec. C.1,
C.2.2 DMFT ACTION
In this setting, the action takes the form
S -
(1))) - / didi' Cog 2 (CC...())C...1))) 1/2 2/2
1
no (1,
Co (1,
V/2
didd' R RA. (1,, (1) (1,1")
2
1DD
In Zo1
C.u, Cov, CA.A, Cav. Cgu, Cup.bo BA,
Σ j=1
In Zo Cop.vo Ron,e Ca.A]
(128)
28

---

Published as a conference paper at ICLR 2026
with single site functions
duodio
duidii
/
(0, dic ((C.
201 =
exp
2T
2x
2v
2 / gt)gg(t)g(((()
f R (())))) dtdt'
x exp
X exp
f dtōo (dr00 + 01 + G1 g +
V
22
Ba) Ce0
VD
( 3g + cgo))(()
Dj
x exp
(129)
and
dAdá
dtdt' on (tt)))))))))(())(( 1 2
dtdd' (ttdCCss(t:))AAAAAC
Za =
exp
2T
/
/did' B (L))))())))))))((() + (
x exp
- -iv1
(130)
Again, in the D >> limit, the saddle point equations which make S locally stationary give
Can (1)
1
BLv1 (1),
=0
(131)
2
2D
(1)
1
Σ
(g0i(0)) -0
(132)
2D
Crn.oo (t, +
1
Σ
(1o())oo(()) =0
(133)
2D
i=1
Pa
ct +
(A(1)((t))
(134)
2
2P
and the same for the response functions
((t))()))) =0
R +
(135)
P2
j=1
D
R + (10(0)0(())) =0
Σ
(136)
D
being the averages ()),, ( over the single site distributions 201 and Z (factorized over i (D) and j (Pz) respectively). At the same time, as usual, the conjugated fields vanish
Coo(t) = Croiee, Cs,a(t,t')
(137)
Since in the Pz, D - Do limit with 1/2 = f fixed all the correlation and response functions concentrate, Hubhard-Stratonovicc transformations to linearize the quadratic terms in
29

---

Published as a conference paper at ICLR 2026
Z01 and 3a by introducing some Gaussian fields
/ dtdt' dtdt'Ca Casn(((00 - -(oxp 2v/
( (-i (m Ca.a)
((ti(()ss())) u,/N
exp
(138)
1 2
Atdt' Cen.vo (t, (e (t,)))(()((())) =
= -(xxp) ( d(s(()u(()))) us-NN(0.Co.m))
exp
(139)
As consequence, the DMFT equations that describe the single site stochastic processes are
(1) = =1(()= + f de' (s,a(())oo un(() ~ GP(0, 1 Ca.s)
(140)
1/2
0,00 -w1 (t) / dt' RAA(( (g c2 (1183 + cag) 8()
(141)
(t)) = (t(+) + 1 f Ru A(t), ua(t) ~ gP (0, Con.
(142)
C.2.3 SIMPLIFYING THE RESPONSE FUNCTIONS
As we did Sec. C.I.4, via integration by parts and Stein's lemma we can simplify the saddle point equations for the correlation functions, which become
Ovo
Roo..
(143)
(t')
0A(t)
RA
(144)
dus (t')
U.A
C.2.4 LIMITING TIME DYNAMICS
We notice again that the loss can be obtained from the time-time diagonal the correlation function Ceg.2o (()))))) which we would like to study at limiting time. Because of that, and by noticing that the system time translational invariant, we can take Fourier transform of Eq. 141, thus getting
(w) -w (w) - RA (w(wooo(w) Can(w) (c1g cig + +22.) Carlw) (1 + cag) 3x
(145)
+ vb((w) = iw + (w( B, - uw(w) - Crolw) (ig + c2B.) - Cgu(w) (c.8, + cag)
where we call H(w) s (w) as before. The same can be done for A(w)
Alw) = RA (w)us( (w)
(146)
and for both the correlations of en with the signal Gg and the noise g directions of Ti, once we define the alignments
0,- D B B.
(147)
0g = D
3r g.
(148)
Recalling their definitions, we get
Cgu = (gmn(w)) = gRs (w) (wolu))))
Ra(w)H(w) [ - (oCol( c3 Cow)
(149)
RAHt
0g (c))
1 RAH
30

---

Published as a conference paper at ICLR 2026
and
(w)( = (3m1 (B. (oow))
-Bal) (as - c2C - ci (p) (w)
(150)
RAH (1+ -1 C.
(L - -G
which implies
Cog.vo Cop.oouu, (w,,"))= = (no(w))l(w)) (
[1- ((1ag 1 1 V/ RA (W) (- toaas) (Cool() + Coolw())
H(W)?((w)
(aus + caa,) (Cgo(w) + Cgolw')) + (ef (w)
+
(10 + 002)
(Cal + Coopl"")Coo()))
+ (c + Corl
(151)
Now to get the final result, we take the wi w/ -> O limits. Using the equation
1
RA 1
RAH = lim RAH = 1/
(152)
1/2
w
which implies also
lim H = 1-12 w-0
(153)
we can derive the limiting time of correlation functions
(0)
1/2
0g c1 c1 (0)
(154)
1 C31/2
1/ 12 + C3V2) as Ci 92
C.. (0)
(155)
(1 + 0202) (1 + 0z12) cu3
Because of the dependency of many variables, let's study the loss in the special case where and 0g 0. this case, one obtains the following loss
(1 2z + cG
L= (1 v2)
(156)
D2
with
D (1 + 0212) (1+ (1+c3)2) - cGp
(157)
1/2 (1 + C312)
C.u (0) (1 + (212)
(158)
(1 0312) 17
(g) (I (2+c2)))) (1 c3122 c7pz
(159)
now interesting to distinguish between some limiting cases in the overparameterized setting
where 12 E [0, 1J. First of all, for the kernel to be PSD it is sufficient to restrict to the span {B., 9 ). from which we get the conditions
(1 + c2) ca) > di
1 0; 1 c3 0.
(160)
Baseline (el = 02 = C3 0): we recover
L= 1 - 1/2
(161)
the reference loss of a linear probe with no pretraining on Ti.
31

---

Published as a conference paper at ICLR 2026
If the signal term C2 = 0, then
(1+ czug)2 + o7v
C (1 U2)
(162)
(1 C312 -G9U3)2
- No crosstalk (c) 0), then
C=1- 12, VC3
(163)
so the noise has no effect the baseline loss in this aligned setting (0g = 0, 1). In this setting, crosstalk proportional to e1 can never actually help because of PSD conditions the kernel, which means that C1 of has always negative effeett on transfer learning. One would need ag * 0 to get non empty range of values for which e₁ can actually help.
pre-training
Cy=0
C,11
Cy-5
0.4
No pre-training
044
===
017575
9-1
0.0
62
04
0.6
0.0
3.0
0.0
02
64
0.6
0.6
10
V2
V2
(a) (02 0, cs = 0.53
(b) {er 0.5, = 0}
Figure 6: Fine-tuning from adaptive kernel with limited data source task (F) finite): loss vs downstream data 1/2 = P2/D. Dashed black: no pre- pre-trannig (linear probe). absence signal from Ti (i.c., C 0) (a) crosstalk с₁ has negative effect transfer since ag O; (b) noise C3 uncorrelated with the target acts has a regularization effect on the loss, pushing it towards the baseline L = 1/2
If the crosstalk term e 0, then
1
L L= (1-1-14
(164)
(1 + 12c2)²
and the loss is independent on the noise c3, while the signal c2 > 0 strictly helps. If the noise term c3 = 0, then
1 + c2
C= V2)
(165)
(1 C202 - - u3
and the loss is a monotonically increasing function the crosstalk term ei of
C.3 FEATURE LEARNING STRENGTH to + 0o ON
If, initialization Wo, dg are small on Ti under gradient flow
0, WWW aa T
(166)
which, if we choose exactly W.W. 0
agag
T implies that W TvT Since f = J X with
X E xD then we can solve for v E D and studying the dynamics
0, = - [x ()(((t) B.))
(167)
32

---

Published as a conference paper at ICLR 2026
No prttanngg
No petranty
5,10
C.125 Call
04
50 P2
04
0/
(a) f2z 1, = 0}
(b) {cr = 0, C3 0.5}
(c) fcr = 0.5, C2 0.5}
Figure 7: Fine-tuning from an adaptive kernel with limited data on source task (21 finite): loss vs downstream data P2/ D. Dashed black: pre-training (linear probe). No crosstalk (cI 0): (a) positive signal 02 0 from Ti strictly lowers the loss compared to the baseline; (b) at fixed
signal, curves collapse for any noise c3, since it is uncorrelated with the target direction in this case (ag = 0).
4.0
35
0.8
50
2.0
0.4
1.5
1.0
85
0.0
0.2
0.4
0.6
0.5
1.0
Figure 8: Linear model from Result 3 when 6 = " - vixx.02 = -3x,,03 = 41 - v1)x
with X vI- 1. When source and target tasks sufficiently aligned (a, 0.7, -0.01), the test loss on To is monotone in 7/.
from which the feature kernel can be derived as M W T W G - B!)), we get
la/2 To.. By calling ea(t)
N
0,00(1) v1
(168)
1
A(t)
X Xvo(t) R
(169)
VD
VD
(1 P XTA(DEER..
(170)
With short path integral (or cavity) derivation similar what we did in previous sections, it is pos- sible to exploit translational invariance of the model, thus getting the DMFT equations that describe the single site stochastic processes. In the current setting, those
(t) (1)+) + / di ('ttt'''')) ui(t) ~ sp(0, _
(171)
(t) = -un((t) - A(t) (t(() + 1 { [d'Ra((t,))A(())) ( "1(t) GP (0, Co.0)
(172)
(173)
i/
33

---

Published as a conference paper at ICLR 2026
where, usual if P: D, then
Ca(t,t' 1
Σ (AI))))))
(174)
Pi
D
Σ to,olt,t')= D 1=1
(vo(t)oott))
(175)
(A(()
(t(t,t) = Dus (t')
(176)
(vo(t)
Roi (t, = Ouu (1")
(177)
being the averages respectively over
/ dAds
+i ( ((a(t)((()) - (t( - ʃ dt Rot (ottt)))))))
Za
exp
2π
(178)
and
= =f { dupdio
[ dvidōr
(t)r exp + atir (t) (t) uz()) -f at (R
Zo1
2T
2
u, Caa
x exp + / di
do(()) in(()
+0(())))(()))))))))))))
(179)
Taking a Fourier transform the DMFT equations simplify
vo(w) iw + RA RA(w) (w)
1
[8 18-2,p [w(
(180)
us (W)
A(w) 1+ H((W)
(181)
Roi (w)
iw + Ra(ww) -HW(W)
(182)
Ra 1+ 19 H(w)
(183)
and the loss function can be written as
Co,o(w,u') =1oov (vo(w)oo(w))
(184)
-H(W))(W') 1+ Ca
1 1i
while the correlation
Ca(w,w') (A(W) (a(ww) Rs (w') R (o,ou)),,
(185)
C.3.1 LIMITING TIME DYNAMICS ON T
If [0, 1J, then from the equation
RA =1 1 RA
(186)
V1 li + Ra
34

---

Published as a conference paper at ICLR 2026
and so 1n find that, at limiting time RA (0) 1-11 3. vt) we get
3. (1-2)) From the definition ot
lim G, (w))(w) (w) w-0
lim (1 iwH(w))) (WH(()) w-0
(187)
B1 + v v1 vi)g
by defining 9 NO0, 1) as Gaussian vector uncorrelated with the source B... As consequence, the kernel is
vv [111+++ vni(L -1)9 [1B.+ + vnn 2))g
(188)
With this kernel, we did above, we would now like to study fine-tuned model with fixed pre- trained features and a linear readout that has to align with the downstream task To identified by target vector Br R...
We call Uo 3 k()2B(() and get the dynamics
0,0u
1 (1) V1 -11) (Cogg())B++
+8(1)B, (189)
where
(1) = 1
Xeu(t) E E
(190)
VD
VD
81 XA RDD
(191)
P2
C
D 01 BB
(192)
Co 11 99
(193)
0, D 31 3. -3
(194)
0g = - 3r ... D
(195)
As a consequence
O,Cope(1) -n [ (r,Cr, + (1-111)00 ()) + aa,5(1)
(196)
(tCo = - - -i
( (1)) +
- v1) - (t) + o(t().
(197)
At this point, by realizing through DMFT that
vn (1) = w1 + (') and by taking Fourier transform of Eqs. 196, 197 get
(198)
(wCona(w) - (w) + 11 un (1)Cong(()) (w) + as
(199)
iwCrng (w) = ---Vvi vi) (n (w))(())) + (1l1ii)Crppler ag
(200)
with Rs = 1. By solving the above system at limiting time we get that
Cons(0)= 11, + (M1(1- - 12Ja,
(201)
11
(ou Vii(1 vn)
(1as + vn v1ja))
(202)
35

---

Published as a conference paper at ICLR 2026
From these, the loss function is
C lim wwii'' vo vo =1 (via, + (1 )
(203)
wese -0
V
We list some interesting conclusions that can be derived in this setting.
The loss, well as the correlation functions, do not depend on 1/2 in this setting. This is reasonable, since any dependence on the amount of P2 data only comes from how well you can estimate single scalar coefficient in this rank-1 feature, and that vanishes the sample size P2 grows.
As 11 + 0, then L -- dg.
In the limit where V1 1 find L = o which is what would expect when the learned feature after Ti is rank-1 1 along B.. In this case, indeed, the best predictor explains a fraction of y²'s variance, so the residual variance is exactly a²
If a, O, then increasing function of 1/1
vla decreasing function of Vi; if a, 0, then C is an
C.4 FINE-TUNING ON POLYNOMIAL TASKS
In this small section, we make comparison between the takes of our linear models of fine-tuning, and what actually happens when training non-linear model on polynomial tasks, from an easy source to hard target. In Fig. 9 we show that for data-rich source fine-tuning is always beneficial, while for a data-poor source feature learning on Ti and related unite-sample size fluctuations can harm performance on the downstream task.
No pre-traniing
No pre-training
y==0.1
3.0
y150.5
V5-5.0
Y2-1.0
Y2-10.0
101
1/2
12
(a) V1
(b) 0.1
Figure 9: Test loss vs target data for different pre-training richness levels ?1. Source task is Hez (3, æ²) target task is Hez(B, x)) with B, 3r 0.8. (a) When source task data-rich, fine- tuning is always beneficial and the higher 7/, the higher the gain. (b) When source task is data-poor, high feature learning on Ti can be harmful comparing to no-pretraining.
D NUMERICAL DETAILS ON DMFT SOLVER
In this section, provide more details regarding the numerical methods used for solving DMFT fixed point equations as reported in Eq. 6.
To generate the DMFT curves in Figures 3 and 4, simulate the single-site dynamical mean- mean-field equations as defined in Eq. 6, i.c. we capture the evolution of preactivations hueT,uT, (t) and readout variables =() via Monte Carlo sampling.
We start by generating
50K Gaussian Monte Carlo samples of the pre-activation fields at initialization h(0) E (P) being (Pi, the sample size of source and target tasks respectively. Given K2 fxx (PPP++A)x(PP+PP. the data Gram matrix
36

---

Published as a conference paper at ICLR 2026
Trarsfer
Tandfer
125t LOGS
087
Steps
Steps
Staps
Slers
(a) P2
50
(b) P2 = 500
Figure 10:
Linear regression on anisotropic data with power-law spectrum: comparison between training from scratch (left panels) and transfer learning (right panels). (a) Transfer learning provides an initial boost at early-time but the final performance bottlenecked by small sample on T2: (b) transfer learning produces a consistent improvement in test loss across training, with richer pre- training (71) corresponding to higher gains.
(Ti U T2). this is just (0),.(0)) init are given by (zx (0) (0,.
(,, In the same way, the readout fields at
From (h(0), z()), we can evaluate the activation field and the gradient signal initial- ization accordingly: &f ((h(0)) g(0) (as defined through Eq. 6). The initial error signal on T2 (0) E T PIvP2 and computed as A = 71/23 ()))))2(0).
At each DMFT time step we update the fields using an explicit Euler discretization of Eq. (6), approximating population averages (-) with empirical averages over the S Monte Carlo samples.
During pretraining, the updates depend only on the residuals Aμ for train samples in Ti: dur- ing transfer learning the updates use only the residuals T2. each time step we com- pute the training/test losses from the DMFT residuals and estimate the learned feature kernel as K(I, () ((h(())) ((h(1)).
E SETTING AND RELATED WORKS FOR BAYESIAN NNS
In this section, we would like to study the effect of transfer learning for infinitely wide Bayesian neural networks. Here, suppose that two layer NN with parameters O = Vec W w) has to learn target task T2 composed of Pg input-output pairs {æ ) where the input vector is Tu R.D. {D, P2] = (1) are fixed, and the network width is going to infinity. The case where the solution space is sampled from posterior that is a Gibbs distribution with generic log- likelihood C(O, T) and a Gaussian prior 1 0|00 [2 has been studied in (Lauditi al., 2025). Here, the purpose is to integrate the effect of transfer learning from source task Ti with the effect of feature learning on Tz.
We consider the weights 0 = Vee W, w of a pre-trained model on Ti = disorder variables for the target task T2, since these weights adapt only Tı. while the target task variables are annealed 0 Vee W, w). The quantity of interest we would like to compute the free energy
{w... Tn 1 as quenched
E E F(W] = lim w (P()). In [[]
No N
= lim
1
E Ew In [ exp
Σ
L(O
T2)
1
0
W
WIP]
No N
2
(204)
Here, the dependency on the source weights W NxD appears through an elastic coupling d that acts form of regularization for the target task weights W RND of Tz. To guarantee that the
37

---

Published as a conference paper at ICLR 2026
source configuration effectively solved T1, take the expectation over the posterior distribution of the source weights as sampled from the Gibbs measure
p 0 T1) = Z1
exp
BN7
Σ ==1
P1
C T1) - -
(205)
2
As clarified in the main text, both (70. 703 = (1( in the mean-field parameterization act as richness parameters that tune the level of feature learning strength, respectively on Ti and T2 (Bor- delon & Pehlevan, 2022; Bordelon et al., 2024b; Lauditi al., 2025). This is the reason why, in our theory, representation learning remains an @N((1) effect at infinite width even when P = O (1), contrary to what would happen in the theories of (Li & Sompolinsky, 2021; Pacelli et al., 2023), whose infinitely overparameterized limit P/N + 0 recovers the NNGP lazy kernel at infinite width.
The way on constraining the target weights to the source weights through an elastic coupling in Eq. equation 204 was first proposed by (Ingrosso et al., 2025) in the context of transfer learning and then studied by (Shan al., 2025) in the continual learning setting. This is common practice in the theory of spin glasses, where the form of Eq. equation 204 is known under the name of Pranz-Parrii potential (Franz& & Parisi, 1995), used to bias the posterior measure through metastable states in the energy landscape. In the context of machine learning theory, line of works (Baldassi et al., 2015; 2016; 2019; 2021; 2022) focused shallow architectures, made use of the Franz-Paris potential in order to target subdominant flat regions of solutions in the loss landscape of given task T. Here, we stress that our theory of transfer learning described by Eq. equation 204, leads to different results than the theory of (Ingrosso et al.. 2025). The authors of (Ingrosso et al., 2025) focused on a proportional limit where both the size of the training sets (P1. P3 in our notation) and the width N go to infinity with some fixed ratios 0₁- = Pi/N and 02 = P2/N. The network parameterization they study is the standard NTK parameterization. In order to be able to study the proportional limit, they make Gaussian Equivalence assumption for non-linear activation functions. Their theory predicts that, at finite a, the effect of transfer learning occurs due to a renormalization effect of a fixed source-target kernel, accordingly to the Bayesian theories of (Li & Sompolinsky, 2021; Pacelli et al., 2023). More importantly, in the a ++ O overparameterized limit we are considering here, their theory predicts that TL has effect on learning, since they recover the NNGP lazy kernel in this limit.
On the contrary, here we study the effect of mean-field (nP)) parameterization to transfer learning in the overparameterized limit. As clarified by Eq. equation 204, we scale the likelihood by N in order ensure we get a non-trivial contribution from the likelihood in the infinite width limit, and we scale the network readout with No The form of our posterior combined with the parame- terization we choose allows us to get theory of feature learning where kernels adapt to data in a non-trivial manner even when P = x (1). In fact, as clarified in (Lauditi et al., 2025), the posterior of Eq. equation 205 do not recover the NNGP lazy kernel, and the effect of transfer learning remains non-negligible in our theory finite P. Our theory do not require any Gaussian Equivalence sumptions on the pre-activation distribution. Indeed, the combined effect of feature and transfer learning leads to non-Gaussian pre-activations. We get set of saddle point equations for the ker- nels of both source (Ti) and downstream the kernels in our theory are not fixed but adapt to data, because representation learning shapes the pre-activation distribution.
(T2) tasks that have to be solved self coliisonsst....y. Thus,
38

---

Published as a conference paper at ICLR 2026
F THEORETICAL DERIVATION OF THE FREE ENERGY
Here, we proceed in reporting the actual computation of the free energy in Eq. equation 204. order to compute the average over the source posterior we use the replica trick In lim,-+0 2n-1 and introduce set of replicas {n} for the source weights {W", w²) As consequence, get
T1
N
n
EZ = dW dw II dwodw dwo ala df exp
Σ 152 - yn1 METS
2
NET
1
1
1
6 a
exp
W
100
2
W
W
2
2
2
/
II a.ETI
II
dh dh, dh exp
n
Σ
ho
h
W W
+ i Σ
hpp
1
Wx, x.,)) )
VD
vD
PET2
METS
NETI
| df exp
Σ a.
jn (Nrosf w (((h))
Σ PETi
Lu (NoJp v x(h,)))
(206)
Step I. The first step consists in integrating out over W and w". We will write these as averages over standard normal matrices (the prior)
E N10, exp
SW
W
h W.
MET
82
8
Σ hn
hy Cuu +
W
Σ h hu
exp
2(1 + +8)
2(1
+
1 5
H.NET2
HETo
Eng ((0.1) exp
ΣΣ UET2
(h)))))))
wa
= exp
Σ
Σ H.NET2
fa fo m
(207)
We that must introduce the kernels and their dual variables (oo Φ M T order parameters, but these are decoupled over replica index
h
((h)) p(h))
(208)
and enforce their definitions through some Dirac-delta functions
1 = / d nv 8 (
dda nu
dda
(hhh ((h)))(())))))))))))))))))))))))
i pe
(h2)) ((h))
N
2T
cxp
N
(209)
Step 2: integrate over W and w
8n 2
82n 2(1 + 8)
E Bw exp
W
Σ
Wap
VD
METIUT
1
Cuulen New
~n-0 exp
UuETIUT
E Ew exp
Σ
(uD(h)) 100 =exp
N
E
Σ
iui
(210)
WETI
MINET,
39

---

Published as a conference paper at ICLR 2026
Here, similarly as did in Eq. equation 208, enforce the definitions of the source task kernels (u ParJureT which do not carry any replica index.
Step 3: Factorize everything across the N hidden neurons
BN
BN10
(Z") x
dbddddf II
d a d df exp
Σ
ynl2
Σ
Σ
[5
ynl
2
NETI
MET2
exp
Noo
Noo
N
Σ
h
To
N
Σ
T
& pr
2
per
2
JAQ
ope
JAM
exp
N
JuJu
ja pa
4 1u
+
Fufu N
+ N In Zjoint
2
2
age
(211)
where Jooine is the joint single-site density that carries contributions from both Ti and T2. has the form
Zjoint ff dhy, di,g,dh,,di, exp
Σ
Cp 1
Σ ((N)))))))
2(1 2((1 +5)
2
opVETs
ape
exp
Σ
Σ
Ca
-2
1 Σ
o(h))o(hp
i 1 1 aj
2
PUETIUTS
100
iaga a + Eindtey
exp
ap
(212) Notice that, if 5 0 in Eq. equation 212, the single site densities on Tn and Tz perfectly decoupled as it should be, since no transfer learning effect would come into play. Instead, as soon we keep 5 > 0, there is interaction between the fields of the source task h and the dual fields of the target task h that will modify the p(h") distribution as we show the next section.
F.1 RS ANSATZ
Step 3: Staring at these equations the only solution that makes sense is the Deplica-Symmerte solution th and fo = f. Plugging this ansatz into the expressions and taking the n 0 limit,
get
In Zjoint In / dxp ( -
Σ MPETIUTA
Σ ANET
(hh)) pi +i
Σ METIUTS
exp (n In h1)
- + In + n (In [11i) In + (I 22(h))
where In 3g is the single site density for task T2
Z2(h]] = dha exp
Σ
Cy - Σ
(bh))o(he)e
(-
2( + σ)
TVETa
uuETs
8
x exp i >
1
1
14
= = dh exp
(1+
5 hu
1 Σ
Σ
hp
h
Cil
h2
((hw)
2
1 +
8
+ 8
2
nueTo
40

---

Published as a conference paper at ICLR 2026
Again, if 8 O, there would be dependency the source task T1 in Eq. equation 213. We stress that transfer learning has the effect of shifting and scaling all the moments of the distribution p(h) towards p(h) becomes larger and larger, while feature learning effect on Eq. equation 213 appear through the contribution of the Gaussin exponent proportional to the dual kernel sb
F.2 SADDLE POINT EQUATIONS
In the infinite width N + oo limit the replicated action of Eq. equation 211 is dominated by the set of kernels { } E Ti and {Φ, } E To that makes the action S locally stationary (ōS = 0)
(Z*) d exp (NSi(( $11 1/ dDd exp (NS2(( +1) ]
870
S1 Σ ns T
E
Sniv + 50 EinJ Σ gul²² In 21
120
21 = / dh dhu exp
(-2
Σ
$ Suo
Σ
Np + Σ huti
uu
870 Eie yul²+ Σ
S2 To Σ fufn Σ
fufu nM
1 ye L Pe (In 12(1))
22 /
dhyndhieee exp
Σ
(mm((()))))))))) + Σ
hmp)
2(1 + 8)
(213)
From these definitions, the saddle point equations give
as
_ Φ (o(h)o(()); + +(n)
1
a6 2
2
0s 4
1
((0(h)))((h))) 1
0 2
2
0s = fo. -3ilsp - -ypl =0
95,
0S = Σ Dw Po + +0ff= =
of.
0S p
1
2 Iaat
(214)
0
F.3 REGRESSION TASKS
These equations are generic for any loss function L(,,T) In the following, for simplicity, we will specialize to regression problems where L(,,T) = Ep= (Sp - yn)² for both source and target tasks. In this particular case, one can solve for both fur and (Sur fu explicitly, since the squared-error loss (SE) allows to integrate out the last layer readouts. From that, one gets for the dual source and target kernels
$ = -
I
I
+
+
8
8
I
I
$ -
yy
3 + $
(215)
3
Notice that the two equations are functionally equivalent, but what changes is the dependency different task labels } E T, vs { E Tz, different levels of feature learning strength in principle (70:70): and especially different adaptive kernels & T.
41

---

Published as a conference paper at ICLR 2026
F.4 GENERALIZATION ERROR
Knowing the form of the transfer free energy of Eq. equation 204, makes it easy to compute the test error of the target model
(unseen) example (00. yo) For generic loss, this is defined (Zo yo) Ew-p(0 (C(O; { 3o0)))))ooo 30))))0p000007..
(216) and can be easily computed by realizing that, if we introduce "test-point coupling" into the transfer free energy by adding a weighted loss for the unseen sample (æo. Vo). we get an extended free energy
F(c) = lim N
-p(0))) In f do exp
3N-6 2
Σ MET
C(O, eC(O; fxo. (oo)
N N N
10
8 W W](
x exp
from which the test loss can be easily computed as
2 aF(()
(217)
3
c=0
For regression task and SE loss, consistently with (Lauditi et al., 2025), this gives the kernel predictor
L
Cg (2 Mo)
1/o
Σ
Pop
+
3v
(218)
B
pau
being Por; the train-test kernel from the saddle point equation
Pop
(219)
(ho.h)
similarly to Eq. equation 214 for the train kernel. We explicitly derive the close form of the train-test. kernel for linear networks in the following 'F.5.
7-636
5.5 12.5 25.
Figure 11: Langevin sampling from the energy function given in Eq. 204. Two-layer ReLU network with width N 20000 function of 8 and for different feature learning strength values noo Test loss at convergence: the network is trained for 10 and averaged after f x 10 every 10 steps. Lazy learning are smallest benefit from transfer learning. Optimal intermediate value of 7o-
F.5 LINEAR NETWORKS
If specialize to linear networks where ((h) h and regression tasks, the target action can be solved explicitly. Indeed, this is given by
S - - Σ
202
I
-1
2
y
+
+
3
y (In Zz [h)
(220)
42

---

Published as a conference paper at ICLR 2026
where the single-site site remains
Gaussian even after feature learning, being
(- (- 2(1 + 8) h Ry Σ
22 = / dh exp
Sunhthy + > 1h= - 6(1 6) 6)-
pau
(221)
Here, we can think h, h as jointly Gaussian with
h
N
(H, E)
h
(1+ (1+ C -if
i0(1 + +00
(1+6)-16 -C
-if
E
-1
0
-il
$
The mean and covariance are equal to
(h).1a ih 5 (1 (1+ (1+ $
h Cov.ja(h) (1 (1+5)0-11 +
(222)
We can thus compute the correlation of hilh as (hh = (h) (h) + Cov(h)
(hh) ) 1/ = [1++)) + Φb +
h- (1 (1 +-000+-1+4
(223)
Now, we must perform the covariance of using 21. Note that this is technically h restricted the second dataset Tg. The full covariance of h for both Ti U T2 has the structure
hh ThhT> C.UUU
a
0
CTUUT
0
CTUUT
(224)
0
We are interested in the lower (2 block of this matrix, which gives the Schur complement
(hr,h = |0- 112 -C (Io +②)
[C 1122
(225)
Thus we are left with the final equations for the target kernels
+ +8 (1+0)
C-1 CC (c- (Io-)), +6) | 3)-1001111
C (1+ 8) +
(226)
$
-0 1 ($ + 3 11 -1 yy ( (8)
(227)
being the action
S2 = TT(
2
+
b x +
20-
2
7(
3
1) -1
y +
2
det
I ( (CO, {5)
(1+ 5)
8²
Tr(
(CT))- (),)))) + $ ( (hr,,
2
The saddle point equations for the source kernels were firstly derived in (Lauditi et al., 2025) and are instead
CT + ②
b= - 1 1-11)
ブy ( +---1) -1
(228)
43

---

Published as a conference paper at ICLR 2026
F.5.1 TRAIN-TEST ADAPTIVE KERNELS
In order to compute the test-train kernel get the network predictor in the linear case, we need to compute Dor (hoh = (ho) (h + Cov(ho, (ho, h ). The covariance is computed by resorting to the single site extended the test point with index
P2 hp
| x / II
dh exp
1+1 (1t
hm)
(1) (Cre))
)
-1 7(h
11 hw
h 4 uv
1+
==0
=MV=1
(229)
from which
A = (1 (1+ - 0
()+))
(230)
and Cov(ha, h Aor. It remains to compute
(ho) In (h).pp
) =n(( Aoo + or h Aor CTa ho + CTth) A (Ar(CCC ho + CoT h) = ArT(CTr ho + CTTh
=n
(231)
where the subscript O refers to the test point while to the training points P2 Tz. From the above equation, we get
(po) (h 7,11 =1 Auo C ho
Coo hoh CrO + Co C + Co CTO) Aur
2 Aoo
Coo 00 h CoT + Coo hoh CTT + O7 hhg
Cor + Cir hh TT
ATT
2 Aor
Cru hohg
+ CTO hah CTu C Goo
Crihh
CTu
Agr
n" AoT CTo hoho Cor +
C
Crihhg
CoT + CTthh CTT ATT.
(232)
As we did for the train kernels in the previous section, we are now interested in the lower (2, 2) block of each kernel matrix (hh ThhT TT, in Eq. equation 232, which would give the source kernel predictions of train and test kernels on T2, having learned the source task Ti.
Test Loss
mi-
20 0.5
0.75
(a) Alignment and Elastic Term Improve Transfer
(b) Adaptive Feature Kernels
The benefit of transfer learning increases with the similarity between source and target tasks. (a) Test losses of a two-layer linear model as a function of the elastic coupling å for different levels of task-similarity. Data are generated from an isotropic Gaussian distribution ~ NN(0, I). Target vector is given by linear model y w æ with || 12 1. Here, the target depends on the source task vector / (such that || = 1) by the relation w a3 + v1 a²wu where w w O. Solid lines taken from Langevin sampling on N 20000 network, black dashed lines from Bayesian theory. (b) Target kernels as function of task similarity m ÿ y.
Figure 12:
In this setting, studying the test loss given by Sec. F.4 as function of requires to iteratively solve the saddle point equations equation 228 after having the adaptive source kemel values (Φ, Φ } E T₁. Fig. 11 shows that, depending the feature strength "o value Te, transfer learning advantage
44

---

Published as a conference paper at ICLR 2026
0.170
0.5
0.165
Vs
Test Loss
0.160
0.155
0.150
0.145
0.140
20
40
80
6
Figure 13: Test losses as function the elastic constraint 7/. Source task regression on two classes (0/1) of MNIST with Pi 400 labels {-1, 1) and richness no 0.5. Target task is regression on two classes of Fashion MNIST (2/5) with P2 = 50 data points and labels
y E 1, 1 for different T0-
Theory
Expt
Init
Theory
Expt
(a) 8 = 0, 70 0.1
(b) 8 = 10, = 2.0
Figure 14: Kernels clustered by labels = [=1] (P2 = 50 Fashion-MNIST data from classes
2/5) improve their task alignment with 8 > and high %o- "Init" represents the Gram matrix of data, "Theory" and "Expt" refers to the adaptive feature kernels o.
and the dependency of test loss to å may vary. When "o is small and the target network is almost lazy on Tz, transfer learning has minor effect improving the test performance. There exists some optimal values of feature learning strength To and ō (which tunes how much the target network relies on source task features) which optimizes the network performance. In Fig. 14 we clearly show how the clustering of data points by labels pops in the kernel appearance as soon we both tune Yo and 8.
F.5.2 DECOUPLED CTUUT
special case we can study is the one in which data are whitened, and uncorrelated across both source and target tasks, meaning
I 0 0 I
CTTUTA =
(233)
In this case, we have
(hh")
(234)
which simplifies the kernel saddle points on target task as
+ (1 [1++)II 8)I + + 82 (1+5)I
[1+
$ - - ( + 3 yy (0)
(235)
As mentioned in the main text, since in this case the kernel only grow in the rank-one yy direction, by solving for the overlaps + yy and & p we get
Φ + -1 + (1
(236)
45

---

Published as a conference paper at ICLR 2026
and similarly that
8 - + +0)
(237)
In the same way, the saddle point equations for the source task Ti be simplified in the source direction yy giving
(1 + 8)
(
(238)
Interestingly, here, when 8 oc, since source and target tasks uncorrelated, then Φ which means that the source kernel 4 is the identity along the target direction as expected.
F.5.3 SAME DATA ON BOTH TASKS
Another relevant case the one where both source and target tasks share the same data and labels. If data whitened, then
CTUTA
I
I I
I
(hh)
=
I
I
I
I
I Φ
0
$
I
(239)
which means
(h2h2) = =- (I+ -1
$ + I (I+ -1
(240)
giving
+ (1+ 11++8II ①
+ 52 (1+ 01+ + -1
(I+ (I+
-1 " 8) $ 61.
(241)
Again, we can solve for the overlaps, knowing that for Ti
お - (1 6)
(242)
= 3 + Φの)-
(243)
For Tz we get
(1 + (
(244)
$ 3= の
(245)
Contrary to the previous uncorrelated case, here, when the elastic constraint -> oc, then = and the target kernel converges to the source kernel expected.
F.5.4 SAME DATA, DIFFERENT LABELS
Suppose again that
I 1
CTrUTA I I
(246)
but that in principle, this case, From the saddle point equations for Tı, we know that
--I+ 1) Ti
(247)
and since the saddle point equations for To are
= [1 (1++)) + (1
7) +
(I+ (o -6- 106-11200 1))rnl)) (1
n) + $)
Φ= -=-2((0) -'
Уンリク + -1
(248)
46

---

Published as a conference paper at ICLR 2026
8.2
12
10.0
25.0
1x
2x127
50 0.
1.100
EX307
91 5.0
10
10
(a) 7 2 7
(b) 7 S
7 S 7
(d) 7o 7
Figure 15: Transfer learning for linear networks trained on whitened data C = I increases the the source is richer than the target model. (a)/(b) Overlaps d vs elastic constraint ô for two-layer linear model trained on P 8 patterns with y = [= Source network is pre-trained on the same data as the target, with a richness parameter 7/8 = 5.0. Solid lines taken from Langevin dynamics on N = 20000 network, dashed lines from the Bayesian theory. (c)/(d) Examples of learned kernels as function of the clastic coupling d.
overlap ø with the label direction y by
Thoory
(a) 7/
Figure 16: Kernels (theory vs experiments) as a function of the clastic constraint 8 with the source task (Ti). When % Ya. there exists an optimal å value for alignment with T2, since in the target task you saw twice the data than in Tí.
one realizes that the only non- con-trivial contributions to & comes from the span (311 y2). in principle one can decompose
Φ - a= 1.IIbbbii + cc
your
dd y2u?
(249)
which means
$ = al + 1y
y2l
b
y
(250)
d
yz
from which
Φ = 0-1=(al +uCu uCu - = (C
- - -u uu) u2) u
(251)
and
② --1 a - y2 a -u (C-1 - u
y1 y22
(252)
being Ie 822 m. turns out, one can solve for (a, c, d} self consistently and for different values of m.
47


Reviews:
[{"review_id": "myUelFB1iM", "reviewer": "Reviewer_djNz", "summary": "Summary\n\nThe manuscript presents two different theories of transfer learning in\ntwo-layer neuronal networks in the limit of infinite width\nwhere feature learning is assured by muP scaling.\n\nThe first theory studies gradient flow and technically employs\ndynamical mean field theory to reduce the problem of transfer learning\nto an effective single neuron problem on the background of some\nself-consistently determined fields.\n\nThe second approach (mostly shown in the appendix) studies a more\nidealized version of transfer learning in a Bayesian setting, where an\nadditional \"elastic term\" is added to the energy function that aligns\nweights between the pre- and the post-training phase.\n\nThe main part of the manuscript explores the relevant parameters that\ncontrol transfer learning and explains qualitative properties, such as\nthe dependence of the amount of feature learning in the pre- and\npost-training phase as well as the alignment of the targets and their\nrelative complexities (e.g., linear vs higher order polynomial\nfunctions).\n\nThe general theoretical results are further illustrated in more\nidealized settings (linear activation functions and Gaussian i.i.d.\ndata) to explain fundamental mechanisms which are subsequently\ndemonstrated on more real-world data sets (e.g. CIFAR-10, Fashion MNIST).\n\nOverall the manuscript makes an important contribution to the theory\nof transfer learning. My main points below are with regard to the\npresentation that should still be improved so the work becomes\nmore accessible also for a broader set of readers.\n\n\nSoundness\n\nMost of the results appear to be sound and the authors support their\ntheoretical calculations by comparison to numerical results.\n\nA few points to my mind deserve more explanation:\n\n1.)\nIn the discussion of Figure 2 it would be important to explain\nwhy panel a) for the theory of the linear model differs qualitatively\nfrom panels b) and c) for the real dataset: Panel a) shows that there\nis an optimal feature learning scale \\gamma_1^\\star (as also stated in the text)\nwhereas in panel b) and c) that the test loss monotonically\ndecreases with gamma_1.\n\n2.)\nIn Figure 3a), the authors should comment on why the asymptotic\nvalue of the test loss is so different between the case of\nno pre-training and transfer learning. Should the pre-training\nnot ultimately become irrelevant at large step sizes of the\nsecond task, so that the two curves are expected to reach the\nsame asymptotic value?\n\nA similar question for Figure 3b): Here the initial transient\nof the test loss seems to be almost identical until the\npeak but ultimately at large step sizes, transfer learning\nseems to lead to worse generalization than training from scratch.\nWhy is this?\n\n\n\nPresentation\n\nThere are a couple of points that the authors should try to improve.\nThis is to the most extent due to the very dense presentation of\nthe main text due to the 8 page limit. Here are points I felt\ndeserve a some care:\n\n1.)\nIt is confusing to use subscripts _t and _s in section 2.3 to denote\nsource and target, as the same letters appear as time points.\nThe authors may want to consider some alternative notation.\nWhy not use _1 and _2, as also for \\mathcal{T}?\n\n2.)\nwith regard to Result 4:\nPlease explain the idea of the balance condition in words in the\nmain text. The appendix is more explicit here.\n\n3.)\nCaption of Figure 1:\nRemind reader that \\alpha is source-target alingment.\nWhy are you using \\alpha_s and \\alpha, both of which are defined\nin the same way?\nSame for \\nu_1 \\nu_2: remind reader that these are the ratios of P/D.\nPanel c: What is the value of \\alpha_g for the shown plot?\n\n4.)\nA remark would be good explaining why the test loss in Fig 1a\nis non-zero at \\nu_2 = 0, even if \\alpha=1, so source and target task\nare identical. Should the loss then not be identical to the test\nloss on task 1 alone, so rather small?\n\n5.)\nNeither at Eq 13 nor at the discussion of the results in relation\nto Fig 1 it is mentioned what is the meaning of the constants c_1 and c_2.\nThey are explained in the appendix, but a few words in the main text\nare needed to understand their meaning, in particular here:\n\n\"The simple alignment case (αs =1, αg =0) of Eq. 13 shows that there (i) larger c2 always helps, while (ii) c1 always hurt, since it rotates the high-gain direction towards the noise.\"\n\n6.)\nFigure 5: misalignment of symbol \\gamma_2 in caption and \\gamma_0\nin figure legend.\n\nAppendix B: Here the feature learning strength is defined as \\gamma_0,\nin the main text as \\gamma_1 or \\gamma_2, respectively. This should\nbe made consistent to minize confusion for the reader.\n\n7.)\nEq. (25): not clear how right hand side depends on x' (what is x' btw?);\nplease clarify.\n\n8.)\nPlease explain between Eq. (25) and Eq. (26) what is the motivation to\nstudy the particular loss. Also please add some additional steps to\nclarify how training the parameters a and W on task T_2 relates\nto the dynamics of \\hat{beta}.\n\n9.)\nWhen introducing the auxiliary fields in eq. (28) - (30) it would be\ngood to motivate this by their anticipated self-averaging due to the\nsummations over high-dimensional index spaces.\n\n10.)\nAre the results Eq. (84) - Eq. (95) used subsequently?\nIt seems you make an ansatz in Eq. (96) that is directly motivated\nfrom Eq. (83) where effective parameters c_1, c_2, c_3 are determined\nad hoc. If this is the case, I would recommend removing Eqs. (84) - (95)\nfor clarity.\n\n11.)\nFor the results shown in appendix B it does not become clear how\nthe numerical results were obtained. It is stated that they are\ngiven by Langevin training.\nMy guess is that they are obtained by Langevin sampling from the\nenergy function given in Eq. (180), including the term \\propto \\delta.\nIs this right? If so, it would be important to be stated, for example\nin the figure caption.\n\nIn particular it should be clarified to the reader that the\nresults for the Bayesian case cannot be obtained by pre-training\nthe weights on task one first and subsequently switch to task two\n(as is the case for the DMFT), because the posterior only captures the\nstationary distribution which I believe should be agnostic to the\ninitialization. This qualitative difference should be explained\nmore clearly.\n\n\nContribution\n\nThe main contribution of the work is to present a theoretical treatment\nof transfer learning. This is an important topic and the authors\npresent important theoretical results that explain qualitative\nobservation in networks trained on real data.", "questions": "Please see itemized list above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Transfer learning is an important topic of practical relevance\nand theoretical progress is needed. The manuscript condenses this\ncomplex question down to nicely analytically or semi-analytically\ntractable settings.", "weaknesses": "The conciseness of the main part makes it hard to follow the main\ntext. The appendix, however, supplies all details as far as I see\n(apart from points marked above).\n\nSome connections between the idealized settings and the real-world\nsettings are still a bit loose and should be mentioned honestly\nin the text (see points under \"soundness\" above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ML8gIaa00g", "reviewer": "Reviewer_FaLX", "summary": "The authors theoretically analyze a wide variety of transfer learning setups in a feature-learning regime. The analysis relies on DMFT calculations; the resulting learning trajectories are typically history-dependent and expensive to numerically evaluate. In more tractable settings (shallow models, linear models) the authors analytically predict various measurables of the trained estimator, revealing conditions under which transfer learning helps or harms. The theory holds qualitatively in practical empirical settings.", "questions": "1) Out of curiosity -- do you think there exist approximations that aren't as restrictive as linear models, but which might give more analytical insight than Result 1? Would it help at all to impose a lot of mutual structure on $\\mathcal{T}_1$ and $\\mathcal{T}_2$?\n2) Fine-tuning can often be made parameter-efficient by using LoRA. Can DMFT-style results extend to the case where the updates on T2 are forced to be low-rank?\n3) In result 2, do you assume that $\\|\\beta_{(\\cdot)}\\|^2=D$? That seems necessary to ensure $\\alpha \\in [-1, 1]$.\n4) It would be interesting to extend this analysis to prescribe optimal curricula. Do you think it's possible?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a strong paper which uses powerful technical machinery to extract insights about learning trajectories in a transfer learning setup. It's really nice to have a predictive toy model of this phenomenon -- this is the first result (afaik) that establishes average-case predictions for this feature-learning behavior. The DMFT formalism is quite opaque (to me) but the takeaway messages are clearly communicated.", "weaknesses": "1) The authors don't treat the case where fine-tuning lazily updates all weights in the network (rather than just the readout). Do we expect quantitatively similar behavior in this regime?\n2) The analytically tractable results rely on an isotropic data assumption which does not hold in many interesting settings. Data anisotropy can qualitatively change the behavior of learning algorithms, especially if $\\Sigma_{xx}$ and $\\Sigma_{xy}$ do not commute. I think it'd be really interesting to understand how the data geometry interacts with the task geometry, and how the transferability of learned features depends on this interaction! I wish the results touched on this.\n3) It's unclear how to think about $\\nu_1$. The data-rich limit appears to be $\\nu_1\\to\\infty$, as stated in line 288 and Fig 1a. But the post-training NTK in Result 4 diverges in this limit, and the limit in line 313 seems to contradict the previously-defined data-rich regime.\n4) I wish there was more empirics measuring task overlap on real targets :) can't have it all, I guess...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j7HLXEF3mj", "reviewer": "Reviewer_RgUn", "summary": "This paper develops a first-principles theory of transfer learning for infinitely wide neural networks operating in the feature-learning (mean-field/µP) regime. The analysis covers two scenarios: fine-tuning with frozen features and a \"jointly rich\" setting with feature adaptation on the target task. The core of the theoretical work involves deriving analytical results for increasingly simplified models (nonlinear $\\rightarrow$ linear $\\rightarrow$ linear two layer).The authors leverage the intuition gained from these simplified models to explain the broader phenomenology of transfer learning, identifying the conditions under which it succeeds or fails as a function of dataset sizes, task similarity, and the strength of feature learning.", "questions": "1. Regarding **Result 1**: \n(a) The result is stated, but no derivation or methodological sketch is provided in the main text. It would be helpful to include a brief description of the key methods used to obtain this result to make the paper more self-contained.\n(b) Can the authors explicitly explain what this result implies beyond the already established fact that the downstream task has a history dependence on the pre-training dynamics? The mathematical formulation is quite involved for what appears to be a known concept.\n\n2. How are the DMFT curves in Figures 3 and 4 obtained? The text states this is from Result 1, but these are non-trivial high-order integral equations. An explanation in the main text of the numerical methods used to solve these equations and generate the curves would be very helpful for reproducibility and understanding.\n\n3. In the phenomenology section (Section 3), it was often not clear which theoretical results were being used to support the empirical findings and simulations. For example, when discussing the CIFAR-10 experiments, it would be beneficial to explicitly state which result (e.g., Result 1, 3, or an ansatz based on them) provides the theoretical basis for the claims being made. This should be clearly emphasized throughout the section.\n\n4. Minor point: In **Result 3**, the text states: “With this kernel, similarly to the sketch of Result 1…”. This seems to be a typo, as the preceding discussion in Result 2 is more relevant. Is it possible the authors were referring to Result 2? If not, it is unclear what sketch from Result 1 is being referenced.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "This paper considers transfer learning, which is both an important problem in the field and difficult to analyze analytically. The approach of simplifying a complex system to a tractable toy model to build intuition is a valuable and necessary process. As deep learning models become increasingly complex, developing a single theoretical framework that explains their behavior in its entirety becomes improbable. For this reason, this type of work, which provides rigorous insights into a simplified but representative model, is an excellent and crucial step in the right direction. The theoretical results appear promising and provide a new lens through which to understand the interplay of factors governing transfer success.\n\nOverall, this is a very interesting work. With improvements to the overall writing and clarity, particularly in the presentation of the main results, I would be happy to raise my score.", "weaknesses": "The primary weakness of the paper lies in its presentation, which, especially in Section 2, is exceptionally dense. This density hinders the paper's accessibility and, more importantly, obscures the key findings and novel contributions of the work. The following are concrete examples where the writing and structure could be improved:\n\n1.The authors assume the audience is familiar with DMFT (to the point that the acronym is not defined in the main text), which may not be the case for many readers. The paper could greatly benefit from a short primer on DMFT in the main text or a longer appendix dedicated to a brief overview of the key principles relevant to this work. \n\n2. The italicized \"result\" paragraphs in Section 2 are long and convoluted, making it difficult for the reader to distill the actual new result, understand its specific implications, and separate it from established background knowledge. For example, in Result 2, Eq. 7 is an established result on infinite networks trained on infinite data, not a new finding specific to transfer learning, and should not have been presented as such (separately) . A similar issue occurs in Result 3, but this time it is further complicated with the introduction of a noise term that is not contextualized properly with respect to previous results.\n\n3. The results are presented sequentially in a technically dense section with minimal guiding principle connecting them beyond the change in setting. A short paragraph giving an overview of these results and how they connect could be highly beneficial. Furthermore, while the titles of the results are meant to provide intuition, the link between the plain-English title and the complex mathematics is often not at all intuitive. A clearer bridge is needed to help the reader understand how the mathematical expressions support the high-level takeaway.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJWvEMrIvg", "reviewer": "Reviewer_xbwa", "summary": "This paper develops an analytical theory for transfer learning (TL) in infinitely wide neural networks trained under gradient flow. The authors extend the work on infinite-width feature learning (FL) to the TL setting, focusing on two main paradigms: (i) fine-tuning on top of source features, and (ii) a jointly rich setting where both pretraining and downstream tasks operate in the FL regime. The main contribution is quantifying the generalization benefit of pretraining as a function of the data and model \"richness\" parameters ($\\gamma$) for both source and target tasks. The model is well-motivated by explaining empirical observations, which I believe is the right approach.", "questions": "1.\tDMFT, MLP, NTK: Please define these acronyms upon first use (e.g., Line 051, 275).\n2.\tResult 1 ($\\langle \\cdot \\rangle$): What measure is average $\\langle \\cdot \\rangle$ taken over? Does it include averages over $\\phi \\sim N(0, 1)$ and $\\chi(x) \\sim GP(0,K_x)$?\n3.\tResult 2 ($\\chi^\\ell$): Please define $\\chi^\\ell$ and provide a clear interpretation. Also, justify why the baseline for $\\mathcal{L}_2$ is $1-\\nu_2$.\n4.\tError Definitions: Please define $\\mathcal{L}(t)$ (Line 217) and $\\hat{\\mathcal{L}}$ (Eq. 10) explicitly. I assume $\\mathcal{L}(t)$ is the loss at time $t$. Why does $\\mathcal{L}(t)$ in line 217 have a $K^{1/2}$ on only one term? Also, is $\\Delta(t)$ (Line 242) the train error residuals and not the train error? Finally, it could be helpful for readability to use a distinct letter for the limiting object (e.g. $\\mathcal{L}(t)$) to avoid confusion.\n5.\tAppendix A: What is $\\Delta(x,t)$? \n6.\tAppendix B.1: Could you explain Eq. 27 also what is $\\delta(t)$? Should it be $P_2$?\n7.\tLine 1068-1069: Is it clear that the response function is translation-invariant? Also what happens at the transition point? \n8.\tSimilar to 6, could you explain Eq. (98)? I’m probably missing something, but shouldn’t there be an additional $XX^\\top$ based on the definition of the Kernel? \n9.  Assumption of Input Isotropy: Given the use of Statistical Physics methods, please clearly state the assumptions made about the input data distribution, particularly regarding isotropy (e.g., Gaussian inputs), and discuss how the results might qualitatively change if these assumptions were violated (e.g., for real-world, highly anisotropic datasets).\n10. Generality of $\\gamma$ Dependence: The main takeaway is that pretraining improves generalization if $\\gamma_1 \\ge \\gamma_2$. Could the authors comment on whether this relationship is expected to hold qualitatively for deep, non-linear networks? For instance, in a deep non-linear network, what surrogate quantity would replace the analytical $\\gamma$ to describe the \"richness\" of the source task/model?\n11. Unbounded Feature Learning: Does the result imply that unbounded feature learning ($\\gamma_2 \\to \\infty$) in the target task ($T_2$) does not matter for the final transfer benefit, or is this simply a case not fully explored?\n12. Figure 1(c) and $\\gamma_0$: What is the parameter $\\gamma_0$ referenced in Figure 1(c) and Line 360?\n13. In Section 3.1 (Line 377), it states that large $\\gamma_2$ can be harmful. Should this be $\\gamma_1$? Please clarify the dependence.\n14. Optimality of $t_1, t_2$: Figure 3 is compelling. How do the results change if one varies the times $t_1, t_2$? Is there a theoretical way to determine the optimal time for transition to the new task? How do these times depend on the feature learning scales, $\\gamma_1, \\gamma_2$, and the overparameterization ratios, $\\nu_1, \\nu_2$?\n15: Harmful Large FL Size: In section 3.3. You state in Fig. 4(a) that there is an improvement in the test loss for any value of $\\gamma_2$, also in lines 444-445. However, it seems from the figure that for a large $\\gamma_2$, it can be harmful. Could you clarify?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strength is the successful analytical quantification of the transfer learning process in the Feature Learning (FL) regime of infinite-width networks. Unlike the Neural Tangent Kernel (NTK) regime, which predicts no feature learning, this work captures how the source features influence the target task generalization. The results provide explicit, closed-form expressions for metrics like generalization error and the overlap with the label direction. The results are supported by numerical simulations. This is a significant theoretical advance over existing NTK-based TL analyses.\nThe distinction between fine-tuning and the \"jointly rich\" setting is interesting. The theory provides concrete and testable conditions, such as the relationship between source and target richness, for when pretraining is beneficial, which aligns well with the experimental results also tested on more realistic data.", "weaknesses": "1.\tThe author assumes the reader is fully familiar with their previous work and neglects to define many key terms in the main text. This makes the paper not accessible to many readers. In addition, common abbreviations and critical analysis components are not defined: DMFT (Line 275), MLP, NTK (Line 051), and the distinction between the limiting objects and their finite counterparts is often blurred. In addition, some of the quantities are not always defined, which makes it hard to read. \n2.\tAlthough the theoretical analysis lacks rigor, I believe the method and analysis are robust in certain settings. The underlying model is highly constrained but captures the phenomenon nicely. I believe the analysis relies heavily on strong assumptions about input data isotropy (e.g., isotropic Gaussian inputs), which significantly simplifies real-world datasets. This reliance on simplified, isotropic features is somewhat hidden and must be explicitly stated and discussed as a key limitation, as it fundamentally affects the general applicability of the quantitative results. I believe that the structure of the features can significantly change the results.\n\nThe paper is technically sound and achieves a novel, closed-form result for transfer learning in the feature learning regime, which is a significant theoretical contribution, together with a detailed comparison to experiments. However, the highly inaccessible presentation, relying heavily on the reader's deep knowledge of the authors' previous work, is a major drawback. That being said, I’m willing to raise my score, provided the authors improve the presentation by addressing the numerous undefined terms, confusing notation, and clarify statements as I detailed in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)