Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization
Authors: Zixiang Chen
Greg Yang
Qingyue Zhao
Quanquan Gu
Venue: ICML 2025 poster (2025)
Keywords: Feature learning
Neural networks
$\mu$P
Tensor Program
Primary area: deep_learning->theory

Abstract: Despite deep neural networks' powerful representation learning capabilities, theoretical understanding of how networks can simultaneously achieve meaningful feature learning and global convergence remains elusive. Existing approaches like the neural tangent kernel (NTK) are limited because features stay close to their initialization in this parametrization, leaving open questions about feature properties during substantial evolution. In this paper, we investigate the training dynamics of infinitely wide, $L$-layer neural networks using the tensor program (TP) framework. Specifically, we show that, when trained with stochastic gradient descent (SGD) under the Maximal Update parametrization ($\mu$P) and mild conditions on the activation function, SGD enables these networks to learn linearly independent features that substantially deviate from their initial values. This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum. Our analysis leverages both the interactions among features across layers and the properties of Gaussian random variables, providing new insights into deep representation learning. We further validate our theoretical findings through experiments on real-world datasets.

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widie Neural Networks under μt Parametrization
Zixiang Chen
Greg Yang"* 22
Qingyue Zhao
Quanquan Gu'
Abstract
challenge: how can we explain the effectiveness of neural networks given their non-convex optimization landscape and over-parameterized nature? Traditional optimization and learning theory frameworks struggle to provide satis- factory explanations. breakthrough came with the study of infinite-wite neural networks, where the network be- havior can be precisely characterized in the limit of infinite width. This theoretical framework has spawned several im- portant approaches to understanding neural networks, with the Neural Tangent Kernel (NTK) emerging as a prominent example.
Despite deep neural networks' powerful repre- sentation learning capabilities, theoretical un- derstanding how networks can simultane- ously achieve meaningful feature learning and global convergence remains elusive. Existing ap- proaches like the neural tangent kernel (NTK) are limited because features stay close to their initial- ization in this parametrization, leaving open ques- tions about feature properties during substantial evolution. In this paper, we investigate the train- ing dynamics of infinitely wide, L-layer neural networks using the tensor program (TP) frame- work. Specifically, we show that, when trained with stochastic gradient descent (SGD) under the Maximal Update parametrization (μP) and mild conditions on the activation function, SGD en- ables these networks to learn linearly independent features that substantially deviate from their ini- tial values. This rich feature space captures rel- evant data information and ensures that any con- vergent point of the training process is global minimum. Our analysis leverages both the interac- tions among features across layers and the proper- ties Gaussian random variables, providing new insights into deep representation learning. We further validate our theoretical findings through experiments on real-world datasets.
Under the NTK parametrization (NTP) (Jacot et al.. 2018), neural network training behaves like linear model: the features learned during training in each layer remain essen- tially identical to those obtained from random initialization. Consequently, the training process of over over-varameee parameterized deep neural networks can be characterized by training linear models with random feature (Lee et al., 2019; Arora et al., 2019b; Cao & Gu, 2019; Chen al., 2021). Since random features are linearly independent, global convergence can be proved for wide neural networks trained using (stochastic) gradient descent (GD/SGD) (Du et al., 2019c; Allen-Zhu et al., 2019; Du et al., 2019a; Zou et al., 2019; Zou & Gu, 2019). However, the NTK parametrization has significant limitations, such as its inability to perform feature learning and transfer learning, which involve pretraining and fine- tuning. While NTK theory provides convergence results under infinite width, its inability to explain feature learning motivates to ask:
1. Introduction
Can deep neural networks simultaneously learn meaningful features and achieve global convergence?
Deep learning has achieved remarkable success var- ious machine learning tasks. from image classification (Krizhevsky al., 2012) and speech recognition (Hinton et al., 2012) game playing (Silver et al., 2016). Yet this empirical success has posed significant theoretical
In this paper, we show that deep neural networks can achieve both objectives through proper parametrization. While pre- vious approaches like NTK and standard parametrization fail to perform meaningful feature learning, and mean field parametrization suffers from feature collapse in deep net- works, we demonstrate that the μ parametrization (Yang & Hu, 2020; 2021; Yang et al., 2021; Yang, 2019a) enables both feature learning and global convergence. Specifically, working with L-layer neural networks under juP scaling, we prove that despite substantial feature evolution during train-
"gqual contribution 'Department of Computer Science, Univer- sity Califomia,, Angeles, USA *xAI, USA. Correspondence to: Quanquan Gu <ggu@cs.uclaan u.......duu
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widthh Neural Networks under μPP Parametrization
ing, the networks maintain linearly independent features in each layer when trained with stochastic gradient descent. As a consequence, if the training converges, it must converge to global minimum. Our contributions are summarized as follows:
sequence of probability spaces F,, P) with Fo F S S Fr. An event & Fr occurs almost surely (denoted as a.s.) if P(E) = 1. The functions f and x denote the indinite-width limits of network outputs and error signals induced by f respectively.
We establish that multilayer perceptrons (MLPs) under Maximal Update Parametrization (μP) learn linearly in- dependent features that capture task-relevant information. The learned features substantially deviate from their ini- tialization, demonstrating true feature learning rather than random feature approximation. This resolves funda- mental challenge in deep learning theory: characterizing feature properties that ensure global convergence while allowing meaningful feature learning.
2. Related Work
Neural Tangent Kernel Parametrization Jacot et al. (2018) first introduced the neural tangent kemel (NTK) by studying the training dynamics of multi-layer perceptrons (MLPs) with Lipschitz and smooth activation functions un- der square loss. Based on NTK, Allen-Zhu et al. (2019); Du et al. (2019a); Zou et al. (2019); Arora et al. (2019a) proved the global convergence of (stochastic) gradient de- scent for various neural architectures with general activation and loss functions. Standard parametrization (SP) and NTK parametrization (NTP) share the same weight initialization scheme but with different learning schedules. As network width increases, SP requires learning rates to decrease as ((1/width) for all layers to maintain stability (Yang & Hu, 2020). When considering the infinite-widhh limit, neither SP nor NTK parametrization can learn features the fea- tures remain essentially the same as those from random initialization. Both theoretical studies and empirical evi- dence demonstrated that these parametrizations failed to capture the feature learning behavior observed in practi- cal neural networks (Woodworth et al., 2020; Geiger et al., 2020; Bordelon & Pehlevan, 2022; Yang et al., 2023a). Ev- erett et al. (2024) shows that SP and NTP can empirically exhibit feature learning when using large per-layer learn- ing rate exponents. However, the existing NTK analysis framework cannot directly analyze feature learning in SP and NTP under this setting, leaving open questions about convergence guarantees during such feature learning.
Our proof technique analyzes neural network Gaussian processes by exploiting their second-order invariants across adjacent layers. These structural properties per- sist during training. which allows us to track the evolution of feature correlations. Through a careful inductive ar- gument over network layers and iterations, we establish that when training converges, the linear independence global minimum. The proof reveals deep connection between the feature learning dynamics and the structural properties of infinite- width neural networks.
of features ensures convergence
Through experiments on classification tasks, we validate our theoretical findings by demonstrating that features maintain linear independence through analysis of covari- ance matrix properties. Our empirical results demonstrate uP's unique capability to simultaneously achieve mean- ingful feature learning while preserving feature richness, supported by non-vanishing eigenvalues as network widths increase. Through comparative analysis against other parametrization schemes, we show that this behav- ior robustly persists across different choices of activation functions, illustrating the practical implications of our theoretical results.
Mean Field Analysis The mean field limit emerged when networks and learning rates were scaled appropriately as width approached infinity, yielding nonlinear parameter evo- lution (Mei et al., 2018; Chizat & Bach, 2018; Rotskoff & Ganden-Fijnden 2018; Sirignano & Spiliopoulos, 201 8). Early analysis of two-layer networks showed promising re- sults, proving convergence to global optima with explicit convergence rates established through both direct analy- sis (Chen et al., 2020) and mean field Langevin dynamies (Nitanda et al., 2022). Progress extended three-layer net- works with Pham & Nguyen (2021)'s global convergence results. However, studies of deeper architectures revealed significant limitations: for networks deeper than 4 layers, both feature vectors and gradients degenerated to zero vec- tors (Nguyen & Pham, 2020; Fang et al., 2021). While Hajjar et al. (2021) introduced Integrable Parameterization (IP) to address this, networks with more than four layers still started at a stationary point in the infinite-wide limit,
Notation. For any positive integer N, we use [N] to denote the index set [1....,N} N]. We use Øi R + IR to denote the activation function. For an I.-layer network, we use super- script 1 E [L] to index layers, with zd and z= denoting pre-activation and post-activation features respectively.
For matrices and vectors, Wot1 = W denotes a scaled last layer weights. For any matrix W and vector x, ZWa denotes the Gaussian component of ZWz! We use E|-] to denote expectation.
We consider filtration (F. where F, is the olalgebra generated by all random variables up time t. This gives
1 W= = W + WW, which is detailed Appendix B.
2

---

Global Convergence and Rich Feature Learning in L-Layer Unfinite-Widie Neural Networks under μPP Parametrization
hard to achieve rich feature learning.
computation is formally defined as
Tensor Programs Tensor Programs (TPs) emerged as unified framework for understanding infinite-width lim- its across neural architectures (Yang, 2019b; 2020a:b). This approach generalized previous architecture-specfii parametrizations (Du et al., 2018; 2019b; Hron et al., 2020; Alemohammad et al., 2020). Yang & Hu (2020) charac- terized two distinct behaviors in wndinite-width MLPs: one where initialization dominated the training dynamics (the kernel regime), and another where training data substantially influenced the learned weights (the feature learning regime). ) Within this framework, the j parametrization was identified as enabling maximal feature learning across all layers and ar- chitectures (Yang& & Hu, 2020; Yang et al., 2021; Littwin & Yang, 2022). The framework has continued to expand with analysis of depth-dependent scaling (Yang et al., 2023c). Recent work by Yang et al. (2023a) refined the understand- ing through spectral analysis and input dimension scaling, which we adopt in our experiments.
h² = W E E RR" x' = ((h') E IR",
h h/41 = w x' eR"
f(5) = Wh R.
(3.1)
where is any positive integer and E {l...., L- 17. Among these schemes, the Maximal Update Parametriza- tion (μP) shown in Table 2 achieves maximal parame- ter updates at initialization. As + 0o, we con- sider the following indtnite-width feature learning process: (E) " f((5) (Yang & Hu, 2020, Theorem 6.4). The neu- ral network is assumed to be trained using a differentiable loss C by stochastic gradient descent, where the s-th sam- pled batch is denoted by {(E: y1) S S where B, is the index set and S is the training dataset. For simplicity, we present the full-batch gradient descent result in the main paper, i.e., Bg = |S| [m].
Our experimental results reveal distinct feature learning behaviors across different parametrization schemes. As shown in Figure 1, Standard Parametrization (SP) keeps fea- tures close to initialization (demonstrated by small feature change in the left panel), while Integrable Parametrization (IP) achieves feature learning but suffers from feature col- lapse (shown by decreasing feature diversity in the right panel). In contrast, μP achieves both substantial feature change and maintains feature diversity. We summarize these key characteristics in Table 1. Additional experiments with different activation functions, further illustrating these trends, are provided in Appendix A.
Represent Hidden States via Z Random Variables: Following Yang & Hu (2020), we represent network's 's hid- den states using Z random variables. This representation generalizes the spirit of two-layer mean field analysis: even with multiple hidden layers (L 2 2). the entries of preacti- vation h and activation vectors x in (3.1) become approxi- mately i.i.d. width n approaches infinity. This allows us to characterize their asymptotic behavior using scalar ran- dom variables that reflect their elementwise distributions.
Specifically, for a vector x E TR. we track it using z", where x's entries behave like i.i.d. copies of z When x is properly scaled such that ||>|| = ((n) (i.e., its typical mag- nitude is independent of nt), then 2T becomes independent of n. For any two such normalized vectors x; y E R" their corresponding random variables Z and 2v are correlated via limm,-so x: y/n n = RZ* 7u. Our goal is to characterize these Z in (3.1) throughout the training process.
Table 1. Feature Properties Under Different Parametrizations Parametrization
Feature Learning
Feature Richness
Standard (SP)
Rich
Neural Tangent (NTP)
Rich
Meanfield (IP)2
Low
Maximal Update (pP)
Rich
Definition 3.1. [Yang & Hu 2020] During training, we de- fine the error signal Xe,5 at time step / for the i-th sample. When training with SGD to minimize the loss function C, this error signal is computed as λc.- = C' 36) 1i E B.). where fo is the model output at time t, (Es, 34) is the i-th training sample pair, and B, denotes the mini-batch at time step t. The indicator function 1 {-} ensures that the error sig- nal is only computed for samples the current mini-batch.
3. Preliminaries
Different parametrization schemes for MLPs are shown in Table 2 3. Given a general MLP with L, hidden layers spec- ified by weight matrices W E nxd {W'H/ J E g8*x" WL+1 E R", and activation o RR- + R. the network
This error signal captures how much the model's prediction deviates from the true label for each sample in the current mini-batch, and serves as the driving force for parameter updates during SGD training. For instance, in the case of mean squared error loss, the error signal takes the form λe = 2(f((E)) - y,) l(i E B,). Having defined the error signal, we now describe how the Z-variables characterize the
IP (Integrable Parametrization) refers to parametrizations with a 1/n scaling factor for layers except the first one, which leads to absolute convergence of weighted sums in the mean-field limit. Init. Var. denotes initialization variance, LR denotes learning rate scaling. 7) the base learning and is the layer width. For notational simplicity, we omit the constant in the table.
3

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Neural Networks under μPP Parametrization
Layer 2: Feature Change vs. Width
Layer 2: Current Feature Min Eigenvalue
gr
[lbbd) 1000l///11 (x0ll bxlllz
Min Eigenvalue h(x)
Layer 2
Layer 2
30'
Width d
Width
Figure Different parametrization schemes exhibit distinct feature learning behaviors as width increases in 3 hidden hidden-layer MLPs. We train it on the CIFAR- 10 dataset and measure pre activation feature properties in the second hidden layer (Layer 2). Left: Feature change ([I[(=) A A shows only yP maintain stable feature representations, Right: Feature diversity measured by the minimum eigenvalue of the feature gram matrix Kij- (h(2:), (a(z))), where larger eigenvalue indicates the features span a higher dimensional space. results reveal that Meanfield parametrization suffers from feature collapse while SP, NTP and μP preserve rich feature representations, Notably, only μP achieves both feature learmngg capability and feature richness. Sec Appendix A for experimental details.
Table 2. Initialization variance and learning rate scaling under different parametrization schemes for MLP networks.
SP
NTP
IP
PP
Layer
Init. Var.
LR
Init.
Var.
LR
Init. Var.
LR
Init.
Var.
LR
Input (W1) Hidden (Wi)
1
17 n-1
"
?) 7).71 n
" n
n-1
17 -1
n-1
17
n
17
n=
1] 1).
Output (WL+1)
n-1
17- 17-
17
n
7) 72
n=
network's computation in the infinite-widd width limit fe).) The forward pass tracks how network features propagate through layers, while the backward pass characterizes gradient flow. For clarity of presentation, we next introduce simplified version of f that includes the key properties needed for our theoretical analysis. The complete derivation and technical details can be found in Appendix B.
pendix B for detail), and 2Woazt I(() are zero cen- tered jointly Gaussian with covariance matrix Cov Cov(? Wheet ((0), 2W352t"' Eizōs! '(0) 26x)--111
2. For last layer weight, we have Wi = WWt + zow +... + ZoW where
ZoW!+t!
77 Σ (
Forward Pass
(3.3)
sE/m]
1. For 2 E (fl,h') h, we have Z=((E) -==(E)) + 552((E) +
3. The output deltas have limits f((e) 6ff(E) + f where
... (62 where
(a) for 1 [L], (dr1(()= ((Za((0) ((2
8f((E) EZOW! zot EZ (t) (3.4)
(b) for 1 1, we have
Σ 1 TTzaliii (()) (5 kE/m]
Backward Pass
1. For gradients:
for 5 1 L, we have
dz (E) W.
(3.5)
(6 Zwoodl (0) F.(E),
(3.2)
an((E) -Z o' (Z))(
(3.6)
where F, is a function that is determined by the random variable (E:), (e)))))))) (see Ap-
ddot-tie (() -W! TW + G.(E)
(3.7)

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widie Neural Networks under μPP Parametrization
where Ge is function that is determined by the random variable {2-- ((E)) sE(t-1 (see Appendix B for detail), and where (2w6 dii(s)) Jg, are zero centered jointly Gaussian with covariance matrix
Assumption 4.3. We assume that the activation function satisfies the following properties.
1. Φ is twice continuously differentiable.
Cov(ZW6 dni(t) z d(((()) El (0h)(()))
2. øf and ø are bounded.
Remark 3.2. The error signal generalizes to different opti- mization objectives. For binary classification problems, it can be expressed as Xe, = -// (1 + explys f((E)))).
3. Φ is a GOOD function.
4. {i E R ((x) = y) and {x E d(()) = y} are countable for all y E R.
-
4. Main Results
Remark 4.4. Assumption 4.3 imposes regularity and smoothness conditions on the activation function, ensur- ing that ø' is pseudo-Lipschite? a requirement for Yang & Hu (2020, Theorem 7.4). These conditions are met by many commonly used activation functions, including the sigmoid function (xx = 1/(1 + (tanh), which rescaled version of sigmoid.
In this section, we present main theoretical results, which rely on the following assumptions regarding the train- ing data and activation function. Specifically, we will first state a mild geometric condition on the inputs, and then dis- cuss the regularity requirements on the activation function.
and hyperbolic tangent
Assumption 4.1. Consider input vectors & drawn from the training data set S C Rd satisfying that for any three differ- ent points E,, Ej. &x E S, the following property holds,
Modern activation functions such as the SiLU (Sigmoid Linear Unit), defined as SiLU(x) = (xx (Hendrycks & Gimpel, 2016), also satisfy these assumptions. SiLU has been widely adopted in practice, including in several state- of-the-art open-source foundation models (Touvron et al., 2023a;b). A detailed discussion of activation functions that meet these criteria is provided in Appendix D.
KK:. 6, # K&:, E
[CE E + 0,
Assumption 4.1 rules out the possibility of identical or zero inner products among different data points, which could otherwise lead to degenerate analyses. Although it may appear restrictive, it holds with probability 1 if the samples are drawn from any continuous distribution (e.g.. Gaussian). Indeed, the of points violating the above requirement-siie such as those with exactly matching inner products has Lebesgue measure zero. In practice, minor random perturbations to discrete data can also ensure the condition is satisfied.
With these assumptions in place, we can now state our main theoretical results regarding feature non-degeneracy and convergence. In particular, the following theorem estab- lishes that in wide neural networks, feature representations evolve while maintaining their diversity and avoiding col- lapse throughout training.
Theorem 4.5. Consider an infinite-width L-layer MLP trained with gradient descent. Under Assumptions 4.1 and 4.3, the features in each layer are non-degenerate at any time during training. Specifically, for each layer I E [Z]:
Definition 4.2 (GOOD Function). A function o: R - IR is called GOOD if it prevents degeneracy in neural networks by ensuring non-trivial compositions. Specifically, for any finite set of parameters {a,), {b,), {c,} satisfying agbs * 0, 3h and [ba] * [byl, Vi + j we have that the composite mapping
1. The pre-activation features (24((0) JeES are linearly in- dependent.
2. The post-activation features (2+1(9) are linearly independent.
f(x)
Σ a,olb + ci),
E
This non-degeneracy property has important implications for the convergence behavior of the model. In particular, it allows characterize the state of the model at conver- gence, described in the following corollary.
is not a constant function. Moreover, for any real numbers T1, 72, the function (r, + ((z))(r2 + ('xx)) is not almost everywhere constant.
We next introduce an assumption on the activation function that ensures it is both sufficiently smooth and GOOD:
Corollary 4.6. Consider an infinite-width L-layer MLP un- der the conditions of Theorem 4.5. If the model converges time T, meaning that the model weights remain unchanged
' is only required to be continuously differentiable with re- spect its first argument (Yang& & Hu, 2020), which we omit in subsequent presentations.
Seee Yang & Hu (2020, Definition E.3) for the definition of Lisscht-ipsschizz functions.
5

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
for all & 2 T, then the error signal vanishes for all subse- quent mini-batches:
through forward and backward propagation. For forward propagation in layer and backward propagation in layer (+ 1, we have by (3.2) and (3.7):
XT. = O, ViE WEU..
201((5) = -WW6oa (6) + Ft(() z = 2 TT (t) + G.(E),
127
(5.1)
where B, denotes the mini-batch at time 6.
(5.2)
This corollary establishes that feature non-degeneracy forces convergence to occur only at critical points where the error signal vanishes. More precisely, when the network con- verges, the error signals must vanish across all samples in subsequent mini-batches, implying convergence to global minimum of the training objective. This is consequence of the feature non-degeneracy established in Theorem 4.5, as non-degenerate features ensure that weight updates can only stop when the network has effectively minimized the error signals.
where Fi and G. capture the historical dependencies through previous features (2d))((E.) he[m)) and gradients (2=5- "e]____ )ifm]),e](t- respectively. This in- tricate coupling between forward and backward passes makes it challenging to ensure that features remain well- behaved as they propagate through the network.
Our key insight in addressing these challenges lies in ana- lyzing structural invariants preserved by the induced Gaus- sian processes during training. While features evolve substantially, we find that certain second-order proper- ties- specificci,,lllly non-degeneracy- saagenerrcy--eemiin invariant across layers and time steps. This invariance ensures rich feature learning while preventing the network from getting stuck in local minima.
5. Key Techniques and A Analysis
In this section, we first identify the key technical challenges in establishing our main results, and then present the tech- niques and insights to address them. We begin by discussing two fundamental challenges: the tension between feature evolution and Structural stability, and the intricate coupling across network layers. We then develop systematic frame- work based Gaussian processes to overcome these chal- lenges. The complete proof is presented in Appendix C.
5.2. The Gaussian Process View
In the infinite- limit, neural network training induces two families of Gaussian processes that capture forward and backward propagation:
5.1. Technical Challenges
(2 (k))) ) (ie
(5.3)
Establishing global convergence while allowing meaningful feature learning presents two fundamental technical chal- lenges that must be addressed simultancously:
W an)()s)) (ee
(5.4)
The forward process ((5.3)) tracks how features evolve across layers, while the backward process ((5.4) describes gradient flow. Unlike prior work that studies these processes in isolation, we discover fundamental connections between their structural properties that enable both feature learning and convergence.
1. Feature Evolution vs. Structural Stability: In con- trast to the NTK parameterization (where features stay near their initialization), juP enables features to evolve substantially during training. Specifically, for any feature z E {z', ,t), in the Forward Pass (a) of Section 3, we have:
Covariance Structure of Gaussian Processes Our key technical insight is that these Gaussian processes (5.3) and (5.4) exhibit invariant covariance properties that persist throughout training. Recall from (5.1) and (5.2) that both forward and backward propagation can be decomposed into a Gaussian term and a history-dependerr term:
ze((t) =Zzo(t) + 551 +... + (3)
feature learning term
The presence of the feature learning term makes it chal- lenging to track and characterize features' properties throughout optimization. This contrasts sharply with the setting under NTK parametrization, where (E) stays equal to its initialization ==o(E) (Yang Hu, 2020) a mathematically simpler but limited case where the net- work behavior is fully determined by the initial kernel.
(ot(()) Whstt (E)
+ Fi(E)
Gaussian term
history term
zdzl(() = Z (E)
Gx(E)
Gaussian term
history term
2. Cross-Layer Coupling: In deep networks, changes in one layer's features affect both earlier and later layers
We notice that these Gaussian terms can preserve covariance
6

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Width Neural Networks under μPP Parametrization
relationships across layers throughout training:
Layer 2: Combined Feature Min Eigenvvauue
Cov( Cov(ZWooo)+or " Zwoool (()) = -E|Z (5) 700
Cov(Zwó (e) zwo ant(e)) = (BI ((n)))))))
Layer 2 Min Eigenvalue
These covariance relationships reveal that feature correla- tions between adjacent layers follow consistent patterns, even as individual features evolve. They link the feature spaces of adjacent layers through their second-order statis-
tics, providing a structural bridge that persists throughout the training process. While previous work has investigated co- variance structures neural networks (Pandey et al., 2022: Guth et al., ine the interaction of the covariance matrix across layers during training dynamics. Our analysis reveals how these structural invariants enable both feature learning and global convergence under juP..
studies do not explicitly exam-
Width d
Figure 2. Layer 2: Minimum eigenvalue analysis of pre-activation features by concatenating initial and final states across different parametrizations in 3 hidden-layer MLPs trained CTFAR- The μP parametrization maintains higher eigenvalues width increases, demonstrating better preservation feature richness. In contrast, other parametrizations (NTP, IP, SP) show substantial decay in eigenvalues with increasing width, indicating feature degeneration. This empirically validates our theoretical analysis that μP uniquely preserves non-degeneracy across both spatial and temporal dimensions, while NTP and SP fail to capture new information during training.
5.3. From Covariance Structure to Non-degeneracy
The preservation of covariance relationships across layers ensures the non-degeneracy of the induced Gaussian pro- cesses throughout training. In the proof of Theorem 4.5, we consider any linear combination of the Gaussian processes:
(E.),
Σ
X1, Ww W.TTNNE,EE.
(E(m],+E((
xE(m]
To empirically validate this theoretical finding, we analyze the minimum eigenvalue of the feature matrix constructed from joint space-time features at Layer 2, complementing our analysis of feature diversity in Figure 1. Under the same experimental setup with 3 hidden-layer MLPs trained on CIFAR-10, Figure 2 constructs joint space-time represen- tations: for each input E., we concatenate initial features (E,E)) and final features hT (E:) to form the combined ma- trix [h AT. h2. hT ...., h&, h. then compute the mini- mum eigenvalue of its Gram matrix. The results show that jP maintains higher eigenvalues across different network widths compared to other parametrizations.
Through our covariance preservation property. we show that if these linear combinations degenerate (i.e., equal to zero almost surely), ) then the corresponding linear combinations of original features and gradients must also degenerate:
di ba ((E) 0,
sE|! Σ
Nr, 20hh a.s. 0.
(E(m)..E(t]
This connection through linear combinations allows us to transfer the non-degeneracy property from feature space to the induced Gaussian processes across layers, establish- ing that both forward and backward processes remain non- degenerate throughout training. This result reveals funda- mental connection between covariance structure and feature richness: the preservation of covariance relationships en- sures that linear independence propagates through layers.
This aligns with our theoretical prediction and further strengthens the findings in Figure where we observed jP's unique ability to achieve both feature learning and feature richness. Additional comprehensive eigenvalue spectrum analysis across all layers and both pre- and post-activation features is provided in Appendix A. A.2, and A.3.
This directly contrasts with other parametrizations. In the NTK parametrization, since features stay close to ini- tialization with Z= (E) the process necessarily be- comes degenerate as it fails to capture new information dur- ing training. Our analysis can demonstrate that μPP uniquely maintains the non-degeneracy of features across both space and time dimensions, enabling the network to learning rich and meaningful features throughout training.
5.4. Evolution Framework
To rigorously track how these structural properties evolve throughout training, we need to carefully handle the natural flow of information in neural networks: forward propagation followed by backward propagation. In each iteration, the network first computes forward features through all layers, then calculates gradients backwards for parameter updates.
7

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
This computational pattern naturally leads to a two-level filtration framework. We introduce sequence of o-algebras to track the evolution of random variables during training. Let Fu denote the initial condition:
3. Non-degeneracy Preservation:
By leveraging GOOD activation functions as introduced in Assumption 4.3 (e.g., Sigmoid, Tanh, SiLU) and the covariance structure, we show that non non- -ogeeeenyry propagates forward in time. Specifically, at time we have:
Fo - (0=0(12%/5)) 216650 HE()),E(L)) Z 2
11
Σ
Xe A..22Woz) (5.)
Then we define Fz to track all completed iterations up to time t, and an extended filtration Ge to capture the forward pass of the (t + 1- iteration:
ve [e(]).e(()
Σ
2 w (n!(Ei)
R.g.
f 0
Fi = oFu, (2Wod (())) [tt
:Eims].aE|1]
Then, we show that these sums remain nonzero at time ++ based on two key properties: (1) the non-degeneracy of Gaussian processes preserved when they share the same covariance structure, and (2) GOOD activation functions, such as Sigmoid and SiLU, exhibit a erucial "non- chon-collapping"" distinct outputs unless all combining coefficients are zero. Together, these properties ensure that features can evolve significantly during training while preserv- ing their diversity, a fundamental distinction from the NTK parametrization, where features remain near their initialization.
12 an) (n)()))) Helm)
(5.5)
G. = 12 Whood he(m)
(5.6)
This filtration structure allows us to precisely track how information flows through the network: F, contains all information up to time I, while G. extends this to include the forward pass information at time t + 1 before its backward pass begins. This framework enables us to:
distinet inputs to
1. Inductive Proof Structure:
The filtration framework enables structured inductive proof that follows the natu- ral flow of computation in neural networks. We establish non-degeneracy in four steps, motivated by how informa- tion propagates through the network:
Now, we revisit the key technical challenges introduced at the beginning of this section and demonstrate how our framework addresses them.
Step I: Features in first hidden layer (E). This forms the base case as only depends on the input,
Feature Evolution vs. Structural Stability: Unlike NTK, where features remain close their initialization, jP en- ables substantial feature evolution. Our framework ensures that new randomness, represented by Gaussian features such as W enters the system with a well-defined struc- ture that preserves non-degeneracy. This structured evolu- tion prevents feature collapse while allowing representations to adapt dynamically, ensuring both expressivity and stabil- ity throughout training.
Step 2: Features in remaining layers ZWode (E.).
Using the non-degeneracy of previous layers, dh) Built
Step 3: Gradients in last layer ZW!Tahh!
upon the established feature properties,
Step Gradients in remaining layers 2n0 (e.). Completing the backward pass analysis.
Each step leverages the non-degeneracy established in previous steps, creating a chain of dependency that mir- rors the network's 's computation graph.
Cross-Layer Coupling: The interplay between layers in- troduces dependencies that can destabilize training. By leveraging two-level filtration structure F, and Gr, our framework tracks both forward propagation 2ºd and back- ward propagation Zdz!, ensuring that updates in one layer do not collapse the feature space of others. This structure maintains well-defined covariance relationships across lay- ers, allowing juP to support both deep feature learning and global convergence, distinguishing it from NTK and stan- dard parametrizations.
2. Conditional Analysis:
The filtration enables precise decomposition of feature and gradient updates into new and historical information:
2ii = 2Wode! (S)
randomaess
history
Zw!+ (igh+iTantttee (E))
(G)
new randomness
history
where (EA) E F captures the accumulated feature history, and (G) represents previous gradient infor- mation. This decomposition is crucial for our inductive proof: by focusing on the new randomness in each step, we can show that non non-degeny is preserved when conditioned on all historical information.
6. Conclusion and Future Work
In this work, we establish a fundamental theoretical result: deep neural networks under jPP parametrization can simulta- neously achieve meaningful feature learning while preserv- ing feature non-degeneracy. Through a rigorous analysis of
8

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μoP Parametrization
Gaussian processes and their covariance structures, we show that features not only remain linearly independent through- out training but also undergo substantial evolution from their initialization. This provides insight into a fundamental question in deep learning theory: how neural networks can simultaneously learn expressive representations and achieve global convergence.
how different parametrization schemes affect the learning dynamics. While primarily theoretical in nature, our find- ings may inform the design of more efficient training algo- rithms and architectures. As our contribution focuses on mathematical foundations, its societal impact aligns with the well-established consequences of advancing machine learning research.
Our analysis establishes fundamental connections between covariance preservation and feature richness. By preventing feature degeneracy, our framework provides a rigorous foun- dation for understanding how overparameterized networks learn expressive representations. Moreover, our results high- light the crucial role of parametrization in enabling both stable training and meaningful feature evolution. These in- sights into how μP enables both feature learning and global convergence suggest promising directions for bridging the gap between theory and practical deep learning success.
References
Adler, R. J. and Taylor, J. E. Random fields and geometry. Springer Science & Business Media, 2009.
Alemohammad, S., Wang, Z., Balestriero, R., and Baraniuk, R. The recurrent neural tangent kernel. arXiv preprint arXiv:2006. 10246, 2020.
Allen-Zhu, Z., Li, Y., and Song, Z. A convergence theory for deep learning via over-parameetriization In International Conference on Machine Learning, pp. 242-252, 2019.
Several promising directions for future work emerge from our analysis. First, extending our theoretical framework to transformer architectures, particularly the attention mech- anism, would be valuable for understanding feature learn- ing in modern language models. Second, our analysis of structural invariants could provide new perspectives con- vergence rates beyond just global convergence, potentially informing optimization strategies in deep learning. Third, studying how our insights on feature non-degeneracy in- fluence generalization bounds may yield deeper theoretical foundations for understanding the generalization properties of deep neural networks. Finally, exploring how μP interacts with more complex training paradigms, such as fine-tuning and self-supervised learning, could further enhance our un- derstanding of deep network training dynamics in practical settings. Additionally, extending our convergence analysis to continuous-deph depth architectures such as neural ODEs and residual networks (Yang et al., 2023b; Marion et al., 2023; Bordelon al., 2023; Gao et al.) represents an interesting direction for future research.
Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-graincs analysis of optimization and generalization for overpa- rameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332, 2019a.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, 2019b.
Bordelon, B. and Pehlevan, C. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35: 32240-32256, 2022.
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehle- van, C. Depthwise hyperparameter transfer in residual networks: Dynamies and scaling limit. arXiv preprint arXiv:2309. 16620, 2023.
Cao, Y. and Gu, Q. Generalization bounds of stochastic gra- dient descent for wide and deep neural networks. In Ad- vances in Neural Information Processing Systems, 2019.
Acknowledgements
We thank the anonymous reviewers and area chair for their helpful comments. ZC is supported by the UCLA Disserta- tion Year Fellowship. QZ and QG are supported in part by the National Science Foundation CAREER Award 1906168 and IIS 100990, as well the Sloan Research Fellowship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
Chen, Z., Cao, Y., Gu, Q., and Zhang, T. A generalized neu- ral tangent kernel analysis for two-layer neural networks. Advances in Neural Information Processing Systems, 33: 13363-13373, 2020.
Chen, Z., Cao, Y., Zou, D., and Gu, Q. How much over- parameterization is sufficient to learn deep relu networks? In International Conference on Learning Representation (ICLR), 2021.
Impact Statement
Chizat, L. and Bach, F. On the global convergence of gradi- ent descent for over-parameterizer models using optimal transport. In Advances in neural information processing systems, pp. 3036-3046, 2018
This work develops theoretical foundations for understand- ing feature learning in deep neural networks, focusing

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widie Neural Networks under μcP Parametrization
Du, S., Lee, J., Li, H., Wang, L.,, and Zhai, X. Gradient descent finds global minima of deep neural networks. In International Conference Machine Learning, pp. 1675 1685, 2019a.
Hron, J., Bahri, Y., Sohl--DccstteinJJ... and Novak, R. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pp. 4376- 4386. PMLR, 2020.
Du. S. S., Lee, J. D., and Tian, Y. When is convolutional filter easy to learn? In International Conference on Learning Reprexentations, 2018.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. -571- 2018.
Du, S. S., Hou. K., Salakhutdinov, R. R., Poczos, B., Wang, R., and Xu, K. Graph neural tangent kernel: Fusing graph neural networks with graph kemels. Advances in neural information processing systems, 32, 2019b.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, pp. 1097-1105, 2012.
Du, S. S., Zhai, X., Poczos, B., and Singh, A. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Rep- resentations, 2019c.
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Sohl- Dickstein, J., and Pennington, Wide neural networks of any depth evolve linear models under gradient descent. In Advances in Neural Information Processing Systems, 2019.
Everett, K., Xiao, L., Wortsman, M., Alemi, A. A., Novak, R., Liu, J., Gur, I., Sohl-Dickstein, J., Kaelbling, L. P., Lee, J., et al. Scaling exponents across parameterizations and optimizers. arXiv preprint wxXiv:2407.05872, 2024.
Littwin, E. and Yang, G. Adaptive optimization in the infinit- width limit. In The Eleventh International Conference on Learning Representations, 2022.
Fang, C., Lee, J., Yang. P., and Zhang, T. Modeling from features: mean-field framework for over-parameterized deep neural networks. In Conference on learning theory, pp. 1887-1936. PMLR, 2021.
Marion, P., Wu, Y.-H., Sander, M. E., and Biau, G. Implicit regularization of deep residual networks towards neural odes. arXiv preprint arXiv:2309.01213, 2023.
Gao, T., Sun, S., Liu, H., and Gao, H. Global convergence in neural odes: Impact of activation functions. In The Thirteenth International Conference on Learning Repre- sentations.
Mei, S., Montanari, A., and Nguyen, P.-M. A mean field view of the landscape of two-layer neural networks. Pro- ceedings of the National Academy of Sciences, 115(33): E7665-E7671, 2018.
Geiger, M., Spigler, S., Jacot, A., and Wyart, M. Disentan- gling feature and lazy training deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11) 1): 13301, 2020.
Nguyen, P.-M. and Pham, H. T. A rigorous framework for the mean field limit of multilayer neural networks. arXiv preprint arXiv:2001. 11433 2020.
Nitanda, A., Wu, D.,, and Suzuki, T. Convex analysis the mean field langevin dynamics. International Confer- ence Artificial Intelligence and Statistics, pp. 9741 9757. PMLR, 2022.
Guth, F., Ménard, B., Rochette, G., and Mallat, A rain- bow in deep network black boxes. Journal of Machine Learning Research, 25(350): 2024.
Hajjar, K., Chizat, I.., and Giraud, C. Training integrable parameterizations of deep neural networks in the infinite- width limit. arXiv preprint 1rXiv:2110.155%0 2021.
Pandey, B., Pachitariu, M., Brunton, B. W., and Harris, K. D. Structured random receptive fields enable informative sensory encodings. PLdS Computational Biology, 18(10): c1010484, 2022.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606vv66066088 08415, 2016.
Pham, H. T. and Nguyen, P.-M. Global convergence of three- layer neural networks in the mean field regime. arXiv preprint arXiv:2
Hinton, G., Deng, L.,, Yu, D., Dahl, G. E., Mohamed, A.-r,, Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):2-97, 2012.
2021.
Rotskoff, G. M. and Vanden-Eijndee E. Neural networks as interacting particle systems: Asymptotic convexity the loss landscape and universal scaling of the approximation error. arXiv preprint arXiv: 1805 005.00915, 2018.
10

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Neural Networks under μcP Parametrization
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I,, Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
Yang, G., Yu, D., Zhu, C., and Hayou, S. Feature leam- ing in infinite-depth neural networks. In NeurIPS 2023 Workshop Mathematics of Modern Machine Learning, 2023b.
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.0222
Sirignano, J. and Spiliopoulos, K. Mean field analysis of neural networks: A central limit theorem. arXiv preprint /80.v 2018.
2023c.
Zou, D. and Gu, Q. An improved analysis of training over- parameterized deep neural networks. Advances in Neural Information Processing Systems, 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
Zou, D., Cao, Y., Zhou, D., and Gu, Q. Gradient descent optimizes over-parameterized deep ReLU networks. Ma- chine Learning, Oct 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Woodworth, B., Gunasekar, S., Lee, D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro. N. Ker- nel and rich regimes in overparametrized models. In Conference Learning Theory, pp. 3635-3673. PMLR, 2020.
Yang. G. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient in- dependence, and neural tangent kernel derivation. arXiv preprint wwtivv:/02..44700 2019a.
Yang, G. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019b.
Yang, G. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. /4548, 2020a.
Yang. G. Tensor programs ii:: Neural matrix laws. arXiv preprint arXiv:2009.1066 2020b.
Yang. G. and Hu, E. J. Feature learning in infinite-wide width neural networks. arXiv preprint arXiv:20/1.1455 2020.
Yang, G. and Hu, F. J. Tensor programs iv: Feature learning in infinite-width neural networks. In International Con- ference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Yang. G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D.. Ryder, N., Pachocki, J., Chen, W., and Gao. J. Tuning large neural networks via zero-shot hyperparameter trans- fer. Advances in Newral Information Processing Systems, 34:1 7098447777777 2021.
Yang, G., Simon, B., and Bernstein, J. A spec- tral condition for feature learning. arXiv preprint 17.113,, 2023a.

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
A. Experimental Details
Experimental details for Figures and 2. We conduct experiments using MLPs with three hidden layers and input dimension 74 3072 (flattened CIFAR-10 images), three hidden layers of equal width n2 n2 fg varying from 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, and output dimension In the experiment, we only use 10 samples randomly selected from the airplane and automobile classes in CIFAR-10. All networks use SiLU activation functions and are trained on a binary classification task with +1 targets for 1000 steps. We use global learning rate 1/ 0.1 across all parametrization schemes with 10 runs for each width setting. The learning rate = 0.1 is chosen to ensure stable training across parametrizations.
We implement the following parametrization schemes:
Standard Parametrization (SP):
2
=
ng-1
at all layers, with v 0.1.
Neural Tangent Parametrization (NTP):
2
of V 11g-1
1/8 = 1)
116-1
at all layers, with 17 = 0.1.
Integrable Parametrization (IP): For initialization variances:
2 1 d'
V2
01 =
00
82 2.
n
For learning rates:
1
111 = 1) d
1/2 = 1/3 = m
1/4 = 1)
n
Maximal Update Parametrization (μıP): For initialization variances:
2 V
2
V2
d'
02
For learning rates:
71 d'
=
171 = 17
Networks are trained with batch size 10 for 1000 steps, sufficient for all widths to achieve stable feature representations with training loss smaller than 0.05. For each width configuration, we conduct 10 independent trials with different random seeds (42 ~ 53) and report the mean values in Figures 1 and 2. To quantify feature properties, measure two metries:
Feature change: [xz)) Ao (2)11) where h0 represents features at initialization
Feature diversity: minimum eigenvalue of the Gram matrix Kij = (M<,;), ((x,)) computed over batch samples
These measurements allow us to track both the evolution of features from their initialization state and the maintenance of feature richness throughout training.
12

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widd Neural Networks under μaP Parametrization
Layer heature Change va Wadhh
Layer Current Feature Mn Bipemakee
Layer 2: Combined Festure Min Bigenoke
Lajye 2 Edjentatue
Lagerr
Lajprr Elgeniie
W
With
Figure 3. Feature learning behavior for Tanh activation. Left: Feature change (1lb(z) ho (z) || (a) [l2). Middle: Feature diversity
(minimum cigenvalue of Gram matrix). Right: Minimum cigenvalue analysis by concatenating initial and final features.
Layer
Layve Eipenvatue
Layar
Lagger
Width
Whathd
Figure 4. Feature learning behavior for ReLU activation. Left: Feature change (1)(=)) (z) (x))))) Middle: Feature diversity
(minimum eigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
A.1. Additional Results Activation Functions
To further examine the impact of activation functions, we conduct experiments with Tanh and ReL.U under the same settings as described in Section A. Below, present the feature evolution and diversity results for these activations.
Our theoretical analysis fully explains the feature learning behavior of Tanh networks, as confirmed by our experimental results in Figure 3. Tanh enables meaningful feature learning while leading to a gradual decrease in feature diversity as width goes up. This is consistent with our theoretical predictions, which account for the smooth and bounded nature of the Tanh activation.
While our theoretical analysis does not directly apply to ReLU due its non-smooth nature, our experimental results in Figure 4 indicate that ReLU-trained networks still exhibit feature learning and maintain meaningful representations under Maximal Update Parametrization (μPP). One possible explanation is that, although ReLU lacks explicit smoothness assumptions required in our analysis, its piecewise linear structure still allows for non- trivial feature evolution in practice. Moreover, μa ensures that weight updates are appropriately scaled across layers, preventing degenerate training dynamics that could otherwise hinder learning in deep networks. Understanding the precise mechanisms behind ReLU's feature learning in the infinite- width setting remains an important direction for future theoretical work.
13

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Neural Networks under μPP Parametrization
A.2. Additional Results Eigenvalue Spectrum Analysis
A.2.1. POST-ACTIVATION EIGENVALUE SPECTRUM
acttaattiaion (W6dh
Pestactiation Boenvauu with (eddhw1006)
kP
(a) Post-activation Eigenvalue Spectrum (Layer 1)
(b) Post-activation Eigenvalue Spectrum with Initial (Layer I)
Figure 5. Layer Post-activation Eigenvalue Spectrum Analysis. The x-axis shows percentiles of the sorted eigenvalues of the feature Gram matrix, where 0% represents the largest eigenvalue and 100% the smallest. The y-axis (log scale) shows the magnitude of these cigenvalues. yP and TP maintain higher eigenvalues throughout the spectrum compared to NTP and SP, with the right plot including initialization values (dashed lines) comparison.
Pestractivatinn geoua be Spectrum (Wich -Layer?
Pestractiaaiion Egenoalue Spectrum Int (Witth Layer?
18
13
(96
Poccustoo 362
(a) Post-activation Eigenvalue Spectrum (Layer 2)
(b) Post-activation Eigenvalue Spectrum with Initial (Layer 2)
Figure 6. Layer 2 Post-activation Eigenvalue Spectrum Analysis. The plots display eigenvalues of the feature Gram matrix ranked by percentile (0% = largest, 100% smallest). IP shows dramatic cigenvalue collapse at higher percentiles (smaller cigenvalues, right side of plot), while /P maintains substantially higher cigenvalues across percentiles, indicating preserved feature diversity even among the least significant dimensions.
14

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
adivvain (Widhw4/99) Laver
adivaain Egenolue Specrum tiki (eighwdd3s) Layer
(a) Post-activation Eigenvalue Spectrum (Layer 3)
(b) Post-activation Eigenvalue Spectrum with Initial (Layer 3)
Figure 7. Layer 3 Post-activation Eigenvalue Spectrum Analysis. The x-axis shows percentiles of ranked eigenvalues, with 100% representing the smallest cigenvalue. In this deepest layer, IP exhibits catastrophic collapse, with eigenvalues higher percentiles (60-100%) plummeting to 10-T. In contrast, μP maintains eigenvalues orders of magnitude larger across all percentiles, demonstrating superior feature independence throughout the entire spectrum.
A.2.2. PRE-ACTIVATION EIGENVALUE SPECTRUM
Ireaacivation Bienaaue (01)
Elennable inkial 0860thw0096)
01
(a) Pre-activation Figenvalue Spectrum (Layer 1)
(b) Pre-activation Figenvalue Spectrum with Initial (Layer 1)
Figure 8. Layer I Pre-activation Eigenvalue Spectrum Analysis. The cigenvalues of the feature Gram matrix are plotted from largest (0% percentile) to smallest (100% percentile). Higher eigenvalues, particularly higher percentiles (right side), indicate greater feature diversity and less redundancy. ,P maintains strong fcature independence throughout training compared to other parameterizations.
15

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
Ire-aclvation Bjonvaue Spedium 40500 Lover
Ieeaacvtton Ejenaluue Socdium in (iid
XI h5a
087
081
(a) Pre-activation Eigenvalue Spectrum (Layer 2)
(h) Pre-activation Figenvalue Spectrum with Initial (layer 2)
Figure 9, Layer 2 Pre-activation Eigenvalue Spectrum Analysis. Bigenvalues of the feature Gram matrix are ranked by percentile on the x-axis. The smallest cigenvalues (highest percentiles, right side) are particularly important as they indicate linear independence among features. uP maintains significantly higher eigenvalues at all percentiles compared to NTP and SP, while IP experiences severe feature collapse above the 20th percentile.
me-activatin Elpemsauue Sprchuu (With: Layer
Ir-sativaton Elpenaslue Spedium intal (With 108g layer
144
Percentle 24
Peccurt D62
(a) Pre-activation Eigenvalue Spectrum (Layer 3)
(b) Pre-activation Eigenvalue Spectrum with Initial (Layer 3)
Figure 10. Layer Pre-activation Eigenvalue Spectrum Analysis. Eigenvalues are arranged by percentile rank, from largest (0%) smallest (100%). At this deepest layer, the percentile-base analysis reveals the most dramatic differentiation between parameterizations. uP maintains substantial eigenvalues even the highest percentiles (80-100%) while IP experiences catastrophic collapse beyond the 20th percentile. The right plot shows initialization values (dashed lines) for comparison with final trained features.
16

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widd Neural Networks under μPP Parametrization
A.3. Feature Learning and Independence Metrics
A.3.1. POST-ACTIVATION FEATURES
Lager 1: Poetskethval on Festure Zhenge vs. Wath
Loyer L: Acttactiation Foture Mis Ejenalue
Liver 1: Post Activation Corb nod Fecture Mis Ejenvalue
Larer AAlllllAA
Mo Sggomelee....
Men Egrivau
Laver's
Wldth d
Wisth
Whsth d
(a) Feature Change (I|== x 22/12/11>012
(b) Feature Diversity (minimum eigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure/ I1. Post-Activation feature learning hchavior for SILU activation in Layer 1. Left:
Feature change (| - x 2//22// (|2)) Middle: Feature diversity (minimum eigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
2 Pestvrttival on Feature Change With
Actiecctitionn Bignvaluee
Activation Comb Hygnvaluv
Min pervoaue
Laver Etpniiile
Layer
Whath
Wisth
Wetth
(a) Feature Change (I|== - x 20122/11>-11>>
(b) Feature Diversity (minimum eigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure 12. Post-Activation feature learning behavior for SILU activation in Layer 2. Left:
Feature change (|r - x0 1/2/|==9l22) Middle: Feature diversity (minimum eigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
Layer Change
Layer Actiactiiiionn Festure Higeralue
Liver 3: Poot Activatiionn Comb nnd Feuture Higeralue
Layer
Lavorr 3 syywehie
(a) Feature Change (||= - x ///22/112011
(b) Feature Diversity (minimum cigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure 13. Post-Activation fcature lcarning behavior for SILU activation Layer 3. Left:
Feature change (|r x /|2/ (12)) Middle: Feature diversity (minimum cigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
17

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Wiin Neural Networks under μPP Parametrization
A.3.2. PRE-ACTIVATION FEATURES
Layer 2: frm Ine-hctiration Feature Change Widh
Prevlctivaionn Feature Min Eipenvalue
Layer Pre-cctvatann Combined Teature Min Eigencauu
Laver Etyoraale of
1000
Min Equevabee
Widhh
Widh
W
(a) Feature Change (Iih- h 112221111 ll2) l12)
(b) Feature Diversity (minimum eigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure 14. Pre-Activation feature learning hehavior for SILU activation in Layer 1. Left: Feature change (/ - h |2/1||>>)>>> Middle:
Feature diversity (minimum eigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
Layer: 2: Tre Activation Feature Change vs. Weth
Layer 2: Fre Actoadtooo Feature Min E ganmea
Layer 2: Pre-ccivatoon Combinne feature Min Eggnnaaus
Lapers To
Lawera stpe Siperialle
Whith
Withh
With
(a) Feature Change (1l A /1/12/111111
(b) Feature Diversity (minimum eigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure 15. Pre- Activatiit feature learning behavior for SILU activation in Layer 2. Left:
Feature change (I/ - ht [///||/ ([a)). Middle: Feature diversity (minimum eigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
Layer Fre AcDogdon festure getteal
Layer Pre-Aciiation Comblnned festure Min Egensa
Dou inpouuaee tus
Layae
(a) Feature Change (1]h- - A 11112/110011 (12)
(b) Feature Diversity (minimum cigenvalue of Gram matrix)
(c) Minimum eigenvalue analysis by con- catenating initial and final features
Figure 16. Pre-Activation feature learning behavior for SILU activation in Layer Left:
Feature change h h | Middle: Feature diversity (minimum cigenvalue of Gram matrix). Right: Minimum eigenvalue analysis by concatenating initial and final features.
18

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
B. More Details for j.P Parametrization
Formally, the MLP definition (Yang Hu, 2020, Table 1) in this section
h' W IR, x' =(h') R", M W E IR, ((E) W x",
(B.1)
where I. > is any positive integer and {I,.....u L 1}. Then the yP for this I-hidden-laaer ML. is defined as follows (Yang & Hu, 2020).
1. Initial weight matrices in the middle layer: W Wo, with each coordinates (W))a ~ 1/0,)/))
2. Initial weight matrix in the input and output layers: input layer matrix Wo E R and output layer matrix WL+1 w Rixa, with each coordinate (Wi)a.. (Wo++1) )ag (,,
3. Initial model outputs: we define the scalars fo( = W
2 W for any input &.
Assuming the same Assumption 4.3 for c. can characterize the Z variables in the infinite-width training dynamics of SGD for this L-hidden-layer MLP similarly as follows (Yang & Hu, 2020).
1. For a E {ar, 16)) Jr, we have
2 220l()) + 20z1(6) + ... 2)z(())
(B.2)
2. For 1 E [L], 2t h = it, have
75, (b(/v(1)) (0
(B.3)
3. For h 2,2, we have
70h.(5) Σ
1x (())
ie(m)
4. For 2 11SL, 110 ====x= 111-111 W = W we have
4-1
She((E) (5) + (odv
11 Σ
Σ
te
==0 1E ml
where
a (m)
z (E) Σ
z (E:)
ô azw, de.
16 1E/m] ==0
5. For last layer weight
t-1 Z W141 W/
Σ
Σ (E)
(B.4)
==0 5c|m]
6. The output deltas have limits
8)((3) = WZ 2z) BZZ 202/ (6)
(B.5)
and
19

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Neural Networks under μPP Parametrization
7. For gradients:
(s) = z
(B.6)
Z = Z d' (2))())
(B.7)
(E) 2 an((E) wwo dn)(E)
Σ
Ta
((:)) 2 (E:) z(n
(B.8)
==0 m
where
t-1
dzdi((
w ds/ 3>
(5)) E
W2
xE(m] a=0
8. Loss derivative:
Xe. = 1 {.).
Intuition behind the entanglement term for the two-hidden-layer case The inclusion of W in the backward pass largely increases the system complexity by introducing multiplications between W and certain nonlinear transformations of W in the forward pass, which necessitates involved definitions of (E)((E)) and Zw (E) Since they are all "conditioned out" in our analysis, we only showcase the definition of 2W Σ Er=o 0ry (5,) to give sense of entanglement between W and W where 0, is calculated like so: 2=, by definition is constructed as
z 4 (2ww dho(Es)
ZW dh1-) 2 dhh.. ZU0)
for some function $ pmxt-1 R. Then
8r.j E(00(2" T dho(E1)) Z dhhhhhsmm) Zw dh: 1(51) ...., "..,,.... ZW dhe 1(5m), 220))/20 (
dh.(s:)).
C. Proof of Theorem 4.5
We begin by describing three key lemmas, each highlighting crucial aspect of our subsequent proof. Lemma C.1. Suppose random variables [=]]] and (ux ]R=[K] satisfy B[uau,] = E [u:,y), Vi,j, then
0 +
Σ
a. 0
kE[K]
kE[K]
Proof. ΣkE(K) 0g d- implies B[( B[(Ekejk] 0g ug variance matrix, we have that E[(E<E[K] axez)² }| = O.
Because (ux [u=]k=]K] and {u=Tk=jk] share the same co-
Lemma C.2. Suppose any level set of Φ R + R is countable and 91.... 9K are jointly non-degenerate Gaussian. If E (,o(g) CC) > 0 where C' is a constant, then = 0 for all i and C' = 0.
Proof. (P,E, (, Cla2..... ... gk) 0} has positive probability only if a, 0, because 91 ... .K non-degenerate Gaussian random variable. We conclude that Hi)((] a; = following similar reasoning inductively. O
9K is
Lemma C.3. Suppose ó satisfies Assumption 4.3. Moreover, suppose ....... 9K are jointly nen-degeee Gaussian. If (1 (gr()))) (c2 + Σ, b, (9:)) C where C1, C22 C' is a constant, then C 0 and either 0 for all i E [K] or b, 0 all i E [K].
Remark C.4. Considering the function tail, it is easy to prove that the Sigmoid function o(x) 1+)xp(-a) the smoothed ReLU function ReLU(x) = log(1 + exp(x)) = f (x)dd, and the SiLU (Sigmoid Linear Unit) function, defined as SiLU(z)= = x (xx) all satisfy these assumptions. Notably, SiLU is employed in state-of-the-art open-source foundation models (Touvron al., 2023a;b).
1
20

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μoP Parametrization
Proof. We first prove that C Condition on the random variables 92.-..99K and denote g = 91 (g2 ... .......9'''' 9K). Then g is non-degenerate univariate Gaussian.
Case 1: Suppose C' * 0. - In this scenario, by cannot be zero; otherwise Φ (or Φ') would have an uncountable level set, contradicting our assumptions. Given that (c' + ald(9))(es + (1)((g)) = C almost surely, we may rewrite:
( + + @1
_e +'(())))
C
01b1
Here di: C2 are constants (absorbing the conditioning on 92, .2..... 9k) But this implies that for almost all IR ( +
d(x)) & + (x)) دgبı contradiction the assumption on the activation function. Therefore, contradiction arises, implying C must be zero.
Case 2: Now consider C' = 0. Then at least one of the following holds with positive probability:
C1 + Σ ago())) = 0 or C2 + Σ (..) = O.
In either case, applying Lemma C.2 (which crucially uses the fact that é has at most countable level sets, forcing the sum to avoid being constant on any uncountable domain with positive probability unless all involved coefficients vanish) completes the proof of zeroing out the corresponding coefficients. Concretely, if
P Σ (.o())) = C. 92....... ........99.. .9 > 0
then for those realizations we view 9i (conditioned on 92...., 9K) as a non-degenerate univariate Gaussian. Holding 92.....9Z .K fixed, the only way Σ, ag((g;) can remain a constant over a positive-measury set of 91 values is if a 0. Repeating this argument inductively for 02-93...- shows that [[EE[K] ag 0. Therefore, either all ag vanish all by vanish,
completing the proof of this lemma.
With these lemmas at hand, now prove main theorem by inductive argument. In particular, we show that the following two families of Gaussian processes, introduced in Section 4, remain non-degenerate throughout training:
{ WWwoz! ((.) (1 (2 an))).) He(
(C.1)
(C.2)
Recall that a Gaussian process is non-degenerate if its covariance matrix C at any finite collection of points satisfies (Ct # 0 (Adler & Taylor, 2009). Using the filtration framework introduced in Scction 4, our proof follows the natural flow of computation in the network, proceeding layer by layer and separately handling forward and backward passes. We break this into four key steps, each building upon the results of previous steps:
prove -oneeegenyy for the features in the first hidden layer (13))) This forms our base case as it only depends on the input data and network initialization, providing the foundation for our inductive argument.
Step 1:
Step 2: prove non-degeneracy for the features in remaining layers Zw(s2!, ((,)), 3 < l < L. This step leverages the non-degeneracy established in Step and shows how it propagates through deeper layers of the network.
Step 3: prove negedegeyyy for the gradients in the last layer TwoTTan((s)) Here we transition from analyzing forward features to backward gradients, showing how the established feature properties ensure meaningful gradient flow.
Step 4: prove non-degeneracy for the gradients in remaining layers Z (s.) 2 < S -- Finally, we complete our analysis by showing how gradient non-degeneracy propagates backward through the network, ensuring effective training dynamics all layers.
The proof proceeds by induction on the time step f, where at each step we verify these properties hold across all layers. This structure allows us to carefully track how the non-degeneracy property is maintained as information flows both forward and backward through the network during training. This systematic proof structure allows us to establish the global property of non-degeneracy by carefully tracking local changes at each layer and time step. We now proceed with the detailed proof.
21

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μcP Parametrization
Proof of Theorem 4.5. Considering Trajectory Until Error Signals Vanish. Throughout this proof, we focus the training trajectory up to the time when all error signals Xt. become zero. This is because the emor signals vanish, there are no further parameter updates, and the training dynamics remain static thereafter. Our analysis ensures that up to this point, the Gaussian processes governing the feature and gradient updates remain non-degenerate, thereby maintaining the linear independence features across all layers.
Connecting Zw éz h and a'. Recall from Section that each pre-activation (C) and post-activation (E) can be decomposed into a primary Gaussian increment plus lower-order
terms in the infinite-width limit:
Because these additional terms do not alter the essential covariance structure when conditioned past information (they vanish become deterministic in the limit), the linear (in)dependence of (h)(E)) or (x'(E)) is governed by the non- degeneracy of { (2w) 8x. (()). Hence, showing that (2w) 8z., *(E) remain non-degenerate under conditioning on historical variables directly implies that (h'(E) and (E)) cannot collapse into linearly dependent set.
Below, we provide inductive argument to establish precisely this non-degeneracy each step.
By definition when ( i [m]] 2 < / S are independent and therefore non-degenerate Gaussian.
Now assume that the random Gaussian Process features defined in (C.1) and (C.2) non-degenerate at time t, specifically for
{ Wooas *(E:) JEEm)..e)))))) ww (E()(E)) /iEm
where layer 2 2 L.
Step 1: We first prove (2W36))(E)) [elmm) is non-degenerate. Suppose there exists not all zero (Ai, (Jis)emmm)
such that
Σ
X 2 6x - 0.
[E]m]
Since (a he([]..e[(] (at are non-degenerate, we conclude that (Aru41hie(m) are not all zero. Consider the second moment, we have that
2 Xr 2W3dr)(6s)
)9
Σ
=0.
ke(m].
Because (2W362((5)) shares the same co- co-variance matrix with (252) By Lemma C.I, we have that
E
Σ
)
Xr, (..2ox(6)))
-0=
37
Σ
Zzz(0)) a 0.
Therefore by definition we have that
Σ
di,o (d(zh)(3.) - ((2h11((.)) 0
(C.3)
where ()) satisfies that
Σ
Σ 186-1157 d.7
jE(m)
jE(m
Plugging (3.6) and (3.7) into the above equation further gives the following reformulation of (
(E))))
Σ
TiaJEJ E.o (1)
al?(Es)
(C.4)
5E(m)
where As(E)) E G.-1 a random variable. Notice that (Es) E -.-1 and AM(EE) G..
22

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Neural Networks under μPP Parametrization
At least one of the (Ee.))Eim) is zero, W.L.O.G assume Na, + By induction hypothesis the non-degenerate property time holds. Therefore, Z th((tx) condition on G {2 dl (ef(e)))ne jyyk is non-degenerate Gaussian.
Plugging (C.4) into (C.3) and then condition on G U u d))(()))))) jiyk gives that
Σ A EiZU + +a)= CC 1m
where & and are constant and T is a non-degenerate uni-variate Gaussian random variable. Since & meets the conditions in Assumption 4.3 and the dataset fulfills Assumption 4.1, ensuring the inner products and level sets behave as required. We can conclude that A1,0+1 for all E [m]] A contradiction! Therefore, {Z (2W he(m]],e[((_]] indeed non-degenerate.
Step 2: We prove the following is non-degenerate.
( (WWost (em 3.
Suppose there exists not all zero {Ar xetm such that
(E(m].+E(t+1]
Since (2 ((E:) he(m),wet() are non-degenerate, we conclude that {Ar,411eej are not all zero. Consider the second moment, we have that
27 ) =0. O.
Σ
Xi.o Ai. Wooo, ((E))
E
1E|m],EE(44]]
Because (2Wdde shares the same co-variance matrix with (2 By Lemma C.I, we have that
)
E
Σ
= =0
Σ
A A (E:) as 0.
xE(m))
(E|m).E((++1]
Therefore by definition we have that
Σ
Ai,o (ool - ((2()_1(3))) =
(C.5)
<c(m].xc[(+1]
where nd!
satisfies that
nz! (E:) 24-²1(E) (E.) Z 1 +... z ((e.).
A reformulation of the above update rule further gives that,
~ Ao(E)) +
(x0--xxx 2d
(C.6)
where (.)))) E F--1 a random variable. Notice that -w (Ex)) Z E F.. Arbitrary pick index k. Because in induction hypothesis assume the non-degenerate property at time & for all layers and already proved the non-degenerate property at time + layer I 1, condition (C.3) on a (Fr 1 1 ) gives that
A +cx)= C²è
( where Up is a non-degenerate uni-variate Gaussian random variable z to (Ex) (Fi U (2 (ond-tan77?E0) jozk), and C2 are constants. By Assumption 4.3 of activation function, we know that At,441 0 for arbitrary t: E [m]] A contradiction! Therefore, ((.))) [e((m) indeed non-degenerate.
23

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Width Neural Networks under μPP Parametrization
Step 3: We prove the following gradients non-degenerate.
{ (Z db (E.)
Suppose there exists not all zero (e such that
Σ
A A.226 (E:) as
<c[m]]eE[[+1]
Since (2 (5) [ee[m) Ne(m) are non-degenerate, we conclude that (Ai,+1 Jejmg are not all zero. Consider the second moment, we have that
9
27 =
E
Σ
Xi, (.oTTaah
iE|m]]sE[t+1]
Because ( shares the same co-variance matrix with (Eih) By Lemma C.I, we have that
)
Σ
=0=
Σ
Xi.s 7ul (5t) 0.
iE(m]
Therefore by definition we have that
Σ
(ds (Z)((()))) -
(C.7)
where zdzt satisfies that
8-1
de) -ZWA.1
Σ Σ
Xer Z
s'=0ie(m]
A reformulation of the above update rule further gives that,
de = A. -7 17 Σ (
{E|ml
(E) A. + Wo (E,)
where A., A F. and (i) is due to (C.6). Notice that Zdx! ( E F.. In (C.7), only W (Ed) E Fe+1 provides new randomness. Because induction hypothesis we assume the non non-degenerate property at time for all layers and already proved the non-degenerate property of W sx 1 (E.). condition (C.7) on Fs gives that
(c-n Σ xu.o(z + b.))) (E(m).aE((+1] km)
Σ
\ +b))== =
where U, 1 (8i) Fr and bi, C² are all constants. Since λe, are not all zero, by Lemma C.3, have that Ai,++1 = for all E [m]. A contradiction! Therefore, (2W6" an (E:) he(m]) is indeed non-degenerate.
Step We prove the following gradients are non-degenerate.
2 WT dh) 11
Suppose there exists not all zero aA hc(()) such that
> 2.. (n) as 0.
24

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widie Neural Networks under μPP Parametrization
Since (2w6" (E.) [e[m]].e]]] are non-degenerate, we conclude that (AE,e-1hr((m)) are not all zero. Consider the second moment, have that
2 (Wo"dd((es)) di((E:)) )
27 0.
Σ
E
[e(m])ae((+1]
Because (2w) ds)(s shares the same co-variance matrix with { (Et)). J. By Lemma C.I, we have that
2LoZZl)(EE) ) :E[m]],EE(++]]
Σ
=0
Σ
X N (G) oz
sE(m]]sE((+11
Therefore by definition we have that
(Z - 0,
(C.8)
iE sE(+t1)
where (5) satisfies that
2d2) = z dil) + GE()))
(C.9)
Similar to Steps 1 and 2, have that (Si) Go, (e) 99-1- Therefore, condition (C.8) Ir only Z (S)
gives new randomness. Arbitrarily pick an index j. Because in induction hypothesis we assume the non-degenerate property at time for all layers and already proved the non-degenerate property at time t + layer + 1 condition (C.8) on
G. w dn dhi4i(s1) (5:) Jegj gives that
X 2 =G
where Uj is non-degenerate uni-variate Gaussian random variable ej and Cj are constants. By Assumption 4.3 of activation function, we know that ej * which induces Aj,++1 = 0 for all j [m]. A contradiction! Therefore, (2w6" an), ) Ne(m]) is indeed non-degenerate.
Proof of Corollary 4.6
Proof. As stated in the main text, if the training parameters stop updating time T, then the training loss must be zero.
By Theorem 4.5, the training trajectory remains non-degenerate throughout training. Suppose, for contradiction, that at time T the training loss is still nonzero for some sample (fx, y4). This implies that the error signal XT, is nonzero. However, the non-degenerate trajectory ensures that a nonzero error signal λr, would necessitate further parameter updates.
Specifically, we establish this through a detailed contradiction argument. Suppose at time T, there exists some sample with non-zero error signal Ar,; A 0, yet the parameters no longer update after time T.
According to parameter update rule in (3.3), for the parameters remain unchanged from time T to T 1,we have:
Z sw L 17 Σ 10 Ze
sE|ml
where Z + represents the weight update step f, [m] (1,2,...,,))) m) denotes the set of indices for the training samples, and as stated in Section 3, the weights evolve as:
Zw = Z Whi+1 + Z5w + ..... Zswet1.
For the parameters to remain unchanged from time to T + 1, we must have: Zzw2+1 7+1 this implies:
0. Substituting the update rule,
24 (E,) 0.
KEBT
25

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Widle Neural Networks under μPP Parametrization
Activation Functions
Derivatives
Sigmaid Tanhh
Signoid
Tann SUU
SILU Gelu
18-
010
-2
Figure 17. Different activation functions (left) and their derivatives (right). Note the exponential decay behavior in the tails of a, tanh, SiLU, and GeLU, which crucial for the GOOD property.
Since the learning rate 1 > O, this simplifies to: EieB, XT (E:) However, by Theorem 4.5, we have established that the post-activation features Z7" (E:) are linearly independent at any time T. Since By [m]] and the features {Zi (E:) are linearly independent, any subset of these features is also linearly independent. Therefore, the only way for the equation Ligg, Zf (5i) = 0 to hold is if XT,5 = O for all i E By. This contradicts our assumption that XT, * O for some i. Therefore, if any eπrоr signal is non-zero at time T, the parameters must continue to update.
Moreover, since this argument applies to any batch B, for 2 T, we can conclude that the model converges at time (meaning parameters no longer update), then all error signals must zero: ÅT,+ = 0 for all E U12T B,, implying convergence global minimum.
D. Activation Functions with the GOOD Property
We now verify that many practical activation functions, especially those with exponential tails, satisfy the GOOD property introduced Definition 4.2. By "exponential tail," we mean that as |x]] - co, the function and/or its derivatives decay at least as fast as e -d|a] for some e 0. Representative examples include the sigmoid, tanh, SiLU, and GeLU. Below, we
restate the full definition of GOOD and then show how each requirement met by these exponential-tai activations.
Definition D.1 (Restatement of Definition 4.2). An activation function two conditions:
IR - called GOOD if satisfies the following
(a) Non-constant decomposition. For any finite set of parameters {a;), {b;), {c) such that 3k with ag F 0 and [b:| + [by] for all # j, the function
m Σ
f(x)
abbr, + +c;)
(D.1)
1=1
not a constant function.
(b) Non-degenerate product with derivative. For any real numbers F1, r2, the product
(ra + ((x)) + o'xx)
(D.2)
is almost everywhere (a.e.) constant on R.
Before analyzing each activation function in detail, visualize these functions and their derivatives in Figure 17. These plots illustrate the key characteristics we will exploit in proofs, particularly the exponential decay behavior in the tails. Note how most activation functions and their derívatives exhibit rapid decay as |ar] -> with ReLU serving as a contrasting example that grows linearly.
In the following subsections, we formally prove that these exponential -tail activations satisfy both conditions Defini- tion D.I.
26

---

Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μPP Parametrization
D.I. Sigmoid and Tanh
Proposition D.2. The sigmoid function o(x) #Fexp(-a) satisfies both (a) and (b) in Definition D.I, hence is GOOD,
Proof. We first prove condition (a). Without loss of generality, set c; 0, as they will not affect the tail of the activation function. Define f2 = {ila; # 0}, A = {i E 22 > 0f and A = fi E Ω < 0}. Let i" = argmin |b. If by- = 0, we can redefine f2 +- 52 (2 and f +- f - ax- ar-/2 2 and reenter this proof. Thus we assume b # 0 without loss of generality.
have:
f(x) = Σ (,o((; + Σ (b,((;
= Ee, a,
E EEA-
a111 o(b,z)] Σ a,o(b,x))
TEA+
SEA-
iEAt
iEΛ-
For bi- < 0, we have:
[f(x)
Σ
ail =
Σ =
a - (b;,)]]
Σ (bar) = (0)-l((b;- + O(exp(-Bxx)
EEA-
where B [b:-
This dominant term cannot be cancelled unless ar- O.
For bi. 0:
1f(z) Σ 2al= aa1= SEA+
Σ SEA+
aa|1- a(b,))]
Σ a,,o(b,))) = aaa,- [[1 (b) + O(exp(-Bz))
SEA
where |b,- This shows f(x) cannot be constant unless ar- bi- contradicting our assumption.
For condition (b), need to show (r1 + (r2) + o'x))) is not constant. Note o'(x) oxx) o(x)) has exponential decay as |x| + co. A direct computation shows:
(n ( (z)) o' (zr) rire + (2) + (1 - (1)))))) mir (r2 r1)o (x)²
(D.3)
Consider the tail in (D.3). If this expression were constant, then by examining the coefficients of different powers of (x)) must have Ti 0 and ri r2 0, which impossible. Thus (ri + (x)))(22 o'x)) cannot be constant almost everywhere.
Remark D.3. Since Lanh(x) is a linear transformation of Sigmoind function a, it inherits the same exponential-tail property and similarly meets both (a) and (b).
D.2. SiLU and GeLU
Proposition D.4. The SiLU function SiLU(x) = (z) is GOOD.
Proof. Define Ω = {i 03. A+ = (i E 926b > 0} and A = {i E 0/6, < 0J. Let 1* = argmin [b... Using similar
reasoning as the sigmoid case, we assume b,. * without loss of generality.
We have:
f(x) = Σ a(o(b;x)+ +
Σ 1EA
a,0(bxx) = Σ abbbx Σ
ag/bex -b(bx)) +
Σ az((bx)).
iEAt
1EAt
iEA+
iEA
For b.s < 0, we have:
ffxx) - Σ
asbiod =
Σ
az axbbat - ((brar))
Σ
azb(br))) = (az-1o(bbr.2) + (e((eexp(BBB
NEA!
1EA1
iEA
Majortail
where Ibz.1.
27

---

Global Convergence and Rich Feature Learning in L-Layer Infinite- Width Neural Networks under μPP Parametrization
For by- O, we have:
1f(x) Σ
asbsol
Σ NEA
as as(b ((b(x))) Σ
ago(bez) = 1EA-
=aa,- 10x- - (bbx+))) +O(xexp( Bx)),
TEA+
Majortail
where B> 10:-1 Note that Majortail bounded by some constant and asymptotically Majortail = ((xepp(-10x-1))) Therefore, f(x) is constant only if EvE a&b; = and a,- O, which contradicts our assumption.
For condition (b), we need show (r1 xz) + o(x) + zo' (x)) is not a.e. constant. Note that:
0'(x) = o()) + (a) zz( o()))
(D.4)
Then we have:
(ri (o))() + ta() + (z) miz + (1)()) + +r ol))
+12c0(s) + xx) + (1) (1- o(x))
(D.5)
Consider the tail in (D.5). If this expression were constant, the coefficient of x² term must vanish, which requires az=))(1-o(a) = 0. However, this is impossible as (xx E (0,1) for all æ. Thus this product cannot be constant almost everywhere.
Remark D.5. GeLU, defined by (x) where & is the Gaussian CDF, similarly satisfies (a) and (b) because of its strong exponential decay. Specifically, as |æ]| + co, GeLU and its derivatives exhibit Gaussian-like decay O(e-=²/3) which is even stronger than the exponential decay of sigmoid and SiLU.
Conclusion. We have shown that key exponential-tai activations (a, tanh, SiLU, GeLU) fulfill both (a) and (b) in Definition D. and hence are GOOD. These results rely crucially on the exponential decay properties of these functions, which ensure that scaled copies cannot combine to yield constant functions. This ensures rich, non-degenerate behavior in widhh analysis under μP scaling.
28


Reviews:
[{"review_id": "8Tm14vcTas", "reviewer": "Reviewer_guAg", "summary": "The paper aims to investigate rich feature learning and convergence to a global minimum via a Maximal Update Parametrization. Networks learn linearly independent features which are different from features at initialization, and due to covariance structure over layers, this implies a convergence to a global minimum under the Maximal Update Parametrization. This holds over the choice of activation function.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and valuable suggestions. \n\n---\n\n**Q1**. Lack of experiments across different architectures, layers, and seeds.\n\n**A1**.  We focused on MLPs as the conventional building blocks widely used in theoretical studies.  While a full investigation across diverse architectures is beyond the scope of this theoretical paper, our core contributions establish the mathematical principles governing feature learning under different parameterizations. To improve experimental breadth, we've expanded our analysis with comprehensive eigenvalue spectrum plots across all three layers for both pre-activation and post-activation features [https://anonymous.4open.science/r/mup_rebuttal-0E0B/Rebuttal.pdf]. These visualizations show μP's advantageous properties becoming more significant in deeper layers.\n\n**Layers**: While our current plots focus on the second layer, we add results demonstrating the minimum eigenvalue trend across different depths to show that the effect persists. [https://anonymous.4open.science/r/mup_rebuttal-0E0B/Rebuttal.pdf]\n\n**Seeds**: Robustness across initializations was already considered in our experiments. As detailed in Appendix A (Experimental Details), the presented results are mean values computed over 10 independent trials using different random seeds (seeds 42-51).\n\n--- \n\n**Q2**. Questions on empirical verification of linear independence and choice of minimum eigenvalue vs. entire spectrum analysis.\n\n**A2**. Regarding the crucial empirical verification of linear independence among the feature vectors corresponding to different input data points, this is precisely achieved in our work by analyzing the minimum eigenvalue ($\\lambda_{min}$) of the $N \\times N$ Gram matrix computed from the centered features. This matrix, capturing the inner products between feature representations, is fundamentally related to kernel methods and the Neural Tangent Kernel (NTK).\nWe compute this Gram matrix $G$. If $X$ is the $n \\times N$ matrix whose columns are the centered feature vectors $h(x_i) - \\bar{h}$, then the Gram matrix is $G = X^{\\top} X$. (Its element $G_{ij} = (h(x_i) - \\bar{h})^{\\top} (h(x_j) - \\bar{h})$represents the similarity between features of data points $i$and $j$). Crucially, the set of $N$ centered feature vectors $\\\\{h(x_i) - \\bar{h}\\\\}\\_{i=1}^N$ is linearly independent if and only if the minimum eigenvalue $\\lambda{min}(G) > 0$. Linear dependence among these$N$vectors occurs precisely when $\\lambda_{min}(G) = 0$. Therefore, observing $\\lambda_{min}(G)$ provides a definitive and direct verification of whether the features representing different inputs maintain linear independence.\n\nWhile focusing on the minimum eigenvalue provides the most direct test for linear independence, we have now included comprehensive eigenvalue spectrum analysis [https://anonymous.4open.science/r/mup_rebuttal-0E0B/Rebuttal.pdf] that examines the entire distribution of eigenvalues. These plots reveal that $\\mu$P maintains significantly higher eigenvalues throughout the entire spectrum compared to other parameterizations. Notably, IP exhibits catastrophic eigenvalue collapse at higher percentiles (e.g., eigenvalues dropping to $10^{-7}$ in layer 3), while $\\mu$P maintains eigenvalue orders of magnitude larger across all percentiles. This full-spectrum evidence further strengthens our claims about feature diversity under $\\mu$P.\n\n---\n\n**Q3**. Definition of GOOD function arbitrary? How about $r_1=r_2=0$?:\n\n**A3**.  Regarding Assumption 4.3 / Condition 4: Here, we require that for any real numbers $r_1$ and $r_2$, the function  $(r_1 + \\phi(x))(r_2 + \\phi'(x))$ is not almost everywhere constant. It is worth noting that this condition involves \"any real number,\" not the existence of some real numbers satisfying a property.\n\nThe GOOD function definition, including this condition, serves as a sufficient condition that guarantees our theoretical results. For an activation function that satisfies our definition, we can show that feature learning maintains linear independence. This condition allows for a broad class of activation functions, including most commonly used ones (e.g., tanh, sigmoid).\n\n---\n\n**Q4**. Previous works investigate covariance structure of neural networks and differentiate them from the current study i.e. they don't explicitly study cross-layer interactions.\n\n**A4**. Thank you for suggesting these valuable references. As recommended, we will add the related works \"A Rainbow in Deep Network Black Boxes\" and \"Structured random receptive fields enable informative sensory encodings\" in our revision. We will also highlight, as you noted, that these works don't explicitly study cross-layer interactions, which is one of the key differentiating aspects of our contribution.", "claims_and_evidence": "I can see the story that the authors tell through the figures, however, I find there to be a lack of experiments. I would appreciate seeing experiments investigate this across different architectures (especially for more conventional networks) and layers. I would also like to verify that these results hold across multiple seeds. \n\nMuch of the claims behind the paper relies on theorems, which are given extensive proofs. From what I can understand from looking at the appendix of the main proof claim, the logic appears to hold up. I would appreciate more clear exposition on when the GOOD function definition is used in the proofs. \n\nI am unsure if the features are empirically shown to be linearly independent. \n\nI am also confused on why minimum eigenvalue is investigated? shouldn't the entire spectrum be looked at in order to measure feature diversity?", "methods_and_evaluation_criteria": "Most of the paper makes a theoretical claim and extensive proofs are provided, so the approach makes sense. I wish more detail was given to explaining the graphs.", "theoretical_claims": "I looked at the main theorem, and the steps appeared clear and correct.\n\nI question if the definition of a good function is somewhat arbitrary (for example, why can't I set r_1 = r_2 = 0 and ensure $\\phi$ is different for each input x?)", "experimental_designs_or_analyses": "I would like more explanation behind why the authors just investigated minimum eigenvalue. I would like to see more experiment over architecture, layers, and seeds of initialization. \n\nI would also appreciate empirical investigations of the linear independence of the features.", "relation_to_broader_scientific_literature": "The paper provides a novel investigation of global convergence properties and rich feature learning", "essential_references_not_discussed": "I would mention previous works that investigate covariance structure of neural networks and differentiate them from the current study i.e. they don't explicitly study cross-layer interactions\n\nA Rainbow in Deep Network Black Boxes\n\nStructured random receptive fields enable informative sensory encodings", "other_strengths_and_weaknesses": "The paper is easy to read.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "I question if the definition of a good function is somewhat arbitrary (for example, why can't I set r_1 = r_2 = 0 and ensure $\\phi$ is different for each input x?)\n\nI am unsure if the features are empirically shown to be linearly independent. \n\nI am also confused on why minimum eigenvalue is investigated? shouldn't the entire spectrum be looked at in order to measure feature diversity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Dj3YVVWr9", "reviewer": "Reviewer_k6NX", "summary": "This paper studies the training of infinite-width $L$-layer FFN under the $\\mu P$ parametrization. The authors establish that features evolve significantly during training while remaining linearly independent, ensuring convergence to a global minimum.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. Due to space constraints, our responses are necessarily concise while addressing all key points:\n\n--- \n\n**Q1**. How does your SGD analysis account for mini-batch randomness when your theoretical focus is on full-batch GD?\n\n**A1**.  Our analysis based on the Tensor Program framework already accounts for mini-batch randomness in SGD. Definition 3.1 models this with the indicator function $\\mathbf{1}\\\\{i ∈ \\mathcal{B}_t\\\\}$, and our derivations incorporate this sampling process. Under SGD, the induced Gaussian processes maintain the same covariance structure as in GD, with the additional randomness from batch selection not affecting the feature independence property central to our global convergence proof. \n\n---\n\n**Q2**. Why aren't your experiments more comprehensive in validating theory and testing assumption necessity?\n\n**A2**. CIFAR-10 provides an ideal balance between complexity and tractability for testing our theoretical claims across multiple parameterizations. In the revised manuscript, we expanded our experimental section with comprehensive evaluations across varying network widths and depths.\n\n---\n\n**Q3**. Which layer features do your plots show, $h_2$? Could you add x2 results to validate both pre/post-activation feature independence?\n\n**A3**. Yes, it is $h_2$. We've analyzed both pre/post-activation features across all layers. Our supplementary analysis [https://anonymous.4open.science/r/mup_rebuttal-0E0B/Rebuttal.pdf] includes complete eigenvalue spectrum plots demonstrating linear independence is maintained throughout training in all layers. These results validate Theorem 4.5 across feature types and network depth. The contrast between $\\mu$P and IP becomes more significant in deeper layers, where IP shows catastrophic eigenvalue collapse while $\\mu$P maintains robust feature diversity.\n\n---\n\n**Q4**. Why do Tanh and ReLU show similar results despite not meeting your theoretical assumptions?\n\n**A4**: Tanh satisfies the GOOD function properties in Assumption 4.3. ReLU was included for completeness despite its non-smoothness. As we already discussed in Appendix A.1, extending our analysis to non-smooth activations is a promising future direction. ReLU's similar empirical behavior suggests our theory's core mechanisms may be more general than the specific assumptions in our proofs.\n\n---\n\n**Q5**. The authors claim to follow [1], but their setup is actually closer to [2]. \n\n**A5**. Both schemes give the same dynamic. [1] appears to scale learning rate uniformly. But this difference arises because [1] expresses weights as $w_{\\ell}$ with scaling $a_{\\ell}$ in $W^{\\ell}=n^{-a_{\\ell}}w^{\\ell}$, while [2] directly uses $W_{\\ell}$. Such equivalence is justified in Tensor Program V (arXiv:2203.03466).\n\n---\n\n**Q6**. No explicit constraints on learning rate for global convergence. Paper only specifies $\\eta = \\Theta(1)$ without clear convergence conditions\n\n**A6**.   We prove that when convergence occurs under $\\mu$P parameterization, the non-degeneracy of features ensures it will be at a global minimum. We didn't provide explicit convergence conditions because our focus was on characterizing the convergence point, not the convergence process itself. Determining explicit convergence condition of $\\eta$ for $\\mu$P  remains an important open question.\n\n---\n\n**Q7**. The paper omits some recent theoretical works on $\\mu$P and NTK for infinite-depth neural networks.\n\n**A7**. Thank you for pointing this out. We already discussed Tensor Program 6 in our paper and will discuss the other suggested works in our revision.\n\n---\n\n**Q8**. Definition of $\\alpha_k$ in Lemma C.1\n\n**A8**. They are arbitrary real numbers representing linear combinations in our independence proof.\n\n---\n\n**Q9**. Does a finite-width FFN always converge to this infinite-width case? ... Can the results be extended to the finite-width FFN case? What's the challenge?\n\n**A9**. Yes - convergence is formally established by Theorem 7.4 in Tensor Program IV. Our empirical results show networks with widths larger than 128 nearly align with infinite-width predictions (Figures 1-4). The main challenge is determining precise convergence rates due to the complexity of tracking higher-order interactions in finite-width networks.\n\n---\n\n**Q10**. Can you explicitly write out the covariance expressions for equations (5.3) and (5.4)? \n\n**A10**. The expressions are provided in Definition 3.1 (Line 187) and Line 212 and restated after Equations (5.3) and (5.4) in Lines 310-311.\n\n---\n\n**Q11**. What's the intuition behind GOOD functions, and why is Condition 4 necessary?\n\n**A11**. GOOD functions maintain feature diversity and ensure non-trivial gradient flow by preventing constant decomposition. Condition 4 helps prevent feature collapse, allowing sigmoid, tanh, and SILU to satisfy our analysis.\n\n---\n\n**Q12**. Does your paper assume W and $W^\\top$ are independent during training?\n\n**A12**. No, we consider standard backpropagation.", "claims_and_evidence": "The theoretical claims in the paper may be correct, as I did not verify all details of the proofs. However, the insights may not generalize beyond the specific setting studied, as the paper makes restrictive assumptions and does not provide comprehensive experimental validation.\n\nAdditionally, while the authors claim their results apply to SGD, their theoretical analysis focuses on **full-batch** GD rather than SGD. In SGD, there is an additional source of randomness from sampling the training data, which may not accounted for in their analysis. As a result, the conclusions drawn for GD may not directly extend to standard SGD.", "methods_and_evaluation_criteria": "The results seem correct but the paper does not provide comprehensive experiments to support its theoretical results or to verify whether the assumptions are necessary. Without empirical validation, it remains unclear whether the proposed theoretical framework accurately captures real-world training dynamics or if the restrictive assumptions limit its practical applicability.", "theoretical_claims": "I did not check the proofs in detail, but since they follow results from the Tensor Program framework, the claims are likely correct. However, there are some important concerns.\n1. The authors claim to follow [1], but their setup is actually closer to [2]. In [1], the learning rate is scaled as  $\\eta n^{-c}$  uniformly across all layers, whereas [2] allows different layers to use different scaling factors. This difference raises concerns about whether their training dynamics assumptions are fully aligned with prior works.\n2. The paper does not impose explicit constraints on the learning rate for global convergence. This implies that as long as  $\\eta$  is constant with respect to width $n$ , arbitrarily large learning rates (e.g.,  $\\eta = 100$) could be used theoretically, which is possible in some machine learning models [3]. In practice, learning rates are much smaller ( $\\eta \\ll 1$), suggesting that there might be missing stability constraints in the theoretical framework.\n\n[1] Tensor Programs IV: Feature learning in infinite-width neural networks. ICML 2021\n[2] Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. NeurIPS 2021\n[3] Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability. NeurIPS 2023", "experimental_designs_or_analyses": "The experimental results are not comprehensive, and it is unclear what features are being demonstrated in the plots. Based on the authors’ description, I assume the features $h$ correspond to  $h^2$  in the second layer after $1000$ epoch, but this is not explicitly stated. The authors may also consider adding results for  $x^2$, as they claim that both pre- and post-activation features remain linearly independent.\n\nAdditionally, the paper includes results for Tanh and ReLU, which do not meet their theoretical assumptions, yet the trends in the plots remain similar to those in Figures 1 and 2. This suggests that the assumptions made in the paper may serve primarily for proof convenience rather than being strictly necessary in practice.", "relation_to_broader_scientific_literature": "This paper extends the theoretical understanding of infinite-width neural networks by demonstrating that the $\\mu P$ enables both independent feature learning and global convergence.", "essential_references_not_discussed": "The paper omits some recent theoretical works on $\\mu P$ and NTK for infinite-depth neural networks, which are relevant to its contributions. Notably, the following works should be considered:\n[1] TP6: Feature Learning in Infinite-Depth Neural Networks\n[2] Implicit regularization of deep residual networks towards neural ODEs\n[3] Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit\n[4] Global Convergence in Neural ODEs: Impact of Activation Functions", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Equation 3.1:  should be $h^1 = W^1 \\xi$, not $W\\xi$", "questions_for_authors": "1. What is the precise definition of  \\alpha_k  in Lemma C.1? How does it relate to the covariance structure of the random variables involved?\n2. The paper studies the training dynamics of infinite-width feedforward networks (FFNs). Does a finite-width FFN always converge to this infinite-width case? If so, under what conditions does this hold, and what is the convergence rate?\n3. Intuition Behind GOOD Functions: What is the intuitive reasoning behind the definition of GOOD functions? Additionally, why is Condition 4 in Assumption 4.3 necessary?\n4. Gradient Independence Assumption: Does the paper assume the independent gradient assumption, meaning that  W  and  W^\\top  are independent during training? If so, how does this assumption affect the generality of the results?\n5. Gaussian Process Covariance: Can the authors explicitly write out the specific covariance expressions for the Gaussian processes in Equations (5.3) and (5.4)?\n6. Can the results on independent feature learning and global convergence be extended to the finite-width FFN case? If not, what are the key challenges in making such an extension? If so, how large does the width need to be in your experiments for the results to align with theoretical predictions?\n7. The paper does not seem to impose explicit constraints on the learning rate to ensure convergence, except $\\eta=\\Theta(1)$. Under what conditions on the learning rate does convergence hold?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOelerefxR", "reviewer": "Reviewer_vRka", "summary": "The submitted paper analyzes the global convergence of MLPs in feature learning parameterization. By demonstrating that features remain independent during training, they prove global convergence.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support and constructive feedback. We address each question below:\n\n--- \n\n**Q1**: Can the authors directly measure correlations between different features during training to support their analysis?\n\n**A1**: Yes, we have directly measured the correlations between different features during training. Our analysis confirms that features remain largely independent throughout the training process. Specifically, we compute the Gram matrix G of the centered features, where $G_{ij} = (h(x_i) - \\bar{h})^{\\top} (h(x_j) - \\bar{h})$ represents the similarity between features of data points i and j. The set of N centered feature vectors $\\\\{h(x_i) - \\bar{h}\\\\}\\_{i=1}^N$ is linearly independent if and only if the minimum eigenvalue λ_min(G) > 0. Our experiments track this minimum eigenvalue throughout training, demonstrating that $\\mu$P maintains feature diversity while IP shows catastrophic eigenvalue collapse. These results directly validate our theoretical claims. We have additional experimental results here: https://anonymous.4open.science/r/mup_rebuttal-0E0B/Rebuttal.pdf.\n\n--- \n\n**Q2**. Regarding SP and NTP at large learning rates and feature learning\n\n**A2**. Thank you for pointing out the paper \"Scaling Exponents Across Parameterizations and Optimizers.\" It is relevant and we will add a citation and brief discussion about it in the revision. This work empirically characterizes when feature learning occurs.\n\nOur work provides a rigorous mathematical framework guaranteeing global convergence during feature learning. The referenced paper shows that SP and NTP can empirically exhibit feature learning at large learning rates but lack theoretical guarantees. The existing NTK analysis framework cannot directly analyze feature learning in SP and NTP at large learning rates, creating a theoretical gap.\n\nWhile the precise theoretical characterization of SP and NTP at large learning rates remains an open problem in the community, our analysis provides new insights that could potentially be extended to study this challenge in future work. \n\n--- \n\n**Q3**. Construction of feature matrix from joint space-time features in Figure 2. \n\n**A3**.In Figure 2, we analyze the combined space-time feature diversity by constructing a feature matrix that simultaneously captures both initial and final representations. This approach provides a comprehensive assessment of how features evolve during training while maintaining linear independence.\n\nFor each parameterization scheme, we construct a combined representation matrix as follows:\n\nLet $h_0(x_i) \\in \\mathbb{R}^n$ represent the feature vector at initialization for input $x_i$, and $h_T(x_i) \\in \\mathbb{R}^n$ represent the feature vector after training completion. For our dataset with $N$ samples, we form the combined feature matrix $H_{\\text{combined}} \\in \\mathbb{R}^{n \\times 2N}$ by concatenating both initial and final representations:\n\n$$H_{\\text{combined}} = \\begin{bmatrix} h_0(x_1) & h_0(x_2) & \\cdots & h_0(x_N) & h_T(x_1) & h_T(x_2) & \\cdots & h_T(x_N) \\end{bmatrix}$$\n\nWe then compute the Gram matrix $G = H_{\\text{combined}}^T H_{\\text{combined}}$, where each element $G_{ij}$ captures the similarity between combined representations. The minimum eigenvalue of this Gram matrix quantifies the linear independence across both spatial dimensions (different inputs) and temporal dimensions (initialization versus final state).\n\nThis combined analysis provides a stronger test of feature diversity than analyzing features in isolation. As demonstrated in Figure 2, μP maintains significantly higher minimum eigenvalues compared to other parameterizations as width increases, confirming its unique capability to preserve feature diversity throughout training while enabling substantial feature learning.\n\nFor example, if a network doesn't learn meaningful features (e.g., NTP at large widths) and $h_T(x_i) \\approx h_0(x_i)$ for all inputs, the combined matrix would have linearly dependent columns, yielding a minimum eigenvalue near zero. The higher minimum eigenvalues for μP confirm it learns new features that are linearly independent from initialization.\n\n--- \n\n**Q4.** Realistic nature of Assumption 4.1 for practical settings beyond toy models\n\n**A4:** Thank you for this question. Assumption 4.1 requires distinct inner product magnitudes between data points. Our empirical verification (sampling 5,000 random triplets from each dataset) shows:\n\n| Dataset | % Triplets Satisfying | Min Dist Between Inner Products |\n|---------|----------------------|------------------------------|\n| MNIST   | 100% | 2.74e-06 |\n| CIFAR-10| 100% | 3.08e-05 |\n| CIFAR-100| 100% | 1.51e-05 |\n\nThese results definitively confirm that Assumption 4.1 is not merely a theoretical convenience but reflects geometric properties inherent to real-world datasets. The perfect compliance across standard benchmarks validates that our theoretical framework directly applies to practical settings beyond toy models.", "claims_and_evidence": "Yes. The theorems support the claims.", "methods_and_evaluation_criteria": "I am not sure if the experiments support the assumptions considered in the paper. I would like the authors to verify these assumptions (see Questions).", "theoretical_claims": "I have checked the Theorems and arguments presented in the main text but not the Proofs.", "experimental_designs_or_analyses": "I am not convinced why the Figure 2 experiments support the results in the paper. I would request the authors to help me understand this result better.", "relation_to_broader_scientific_literature": "This paper contributed to understanding convergence of feature learning limits of neural networks. Prior literature has mostly focused on the kernel limits.", "essential_references_not_discussed": "Relevant references are discussed to the best of my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n\nThis work analyzes the global convergence of feature learning parameterizations, which is of significant interest to the community.\n\nWeaknesses:\n\nI am not sure if the assumptions made in this paper generalize to practical settings beyond toy models (see Questions).", "other_comments_or_suggestions": "Suggestions:\n\n* Can the authors directly measure correlations between different features during training to support their analysis? \n\nComments:\n\n* SP and NTP don't learn features at small learning rates only. By comparison, at large learning rates, all parameterizations learn features and can perform equally well or better [1]\n\n[1] Scaling Exponents Across Parameterizations and Optimizers\nhttps://arxiv.org/abs/2407.05872", "questions_for_authors": "Questions:\n\n* Can the authors clearly state how they constructed the feature matrix from joint space-time features in Figure 2? I am not sure how this experiment supports the theoretical results. \n* I am unsure about how realistic Assumption 4.1 is for practical settings beyond toy models (MLPs trained on CIFAR; MLPs only achieve around 50% accuracy on CIFAR-10 and may not require feature learning to achieve this performance). Can the authors check the assumption on a few datasets? \n* A recent paper has shown that both SP and NTP can exhibit hyperparameter transfer (and feature learning) [1]. Is it straightforward to show this in the framework introduced?\n\n[1] Scaling Exponents Across Parameterizations and Optimizers\nhttps://arxiv.org/abs/2407.05872", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCL9WXariU", "reviewer": "Reviewer_DKQW", "summary": "This paper investigates the training dynamics of infinitely wide neural networks with mup and SGD. They show that these neural networks can learn rich feature spaces and enjoy global convergence, which is better than other mainstream parameterizations such as NTK, MF, and SP. They also validate the theoretical findings through real-world experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for taking the time to review and give feedback on our manuscript. We appreciate the positive comments regarding the paper's clarity, the topic's significance, the originality of our ideas, and the supportive experimental results. We are particularly grateful for the reviewer's recognition of Theorem 4.5's contribution regarding feature learning under $\\mu$P.\n\nRegarding the reviewer's main concern about the global convergence result, we believe there is a significant misunderstanding about what our result delivered and why it is non-trivial. The reviewer states (in the comments on \"Theoretical Claims\" and in Question 1) that\n\n **Any neural network enjoys zero training loss and global convergence if it will no longer be updated after some training step.** \n\nWe respectfully but strongly disagree with this premise as a general statement and wish to clarify this crucial point, as it seems to underlie the reviewer's doubt.\n\n---\n\n**Clarifying the Core Misunderstanding Regarding Global Convergence**\n\nWhile it is true that training algorithms like SGD stop updating when they reach a point where the gradient is zero (a stationary point), this absolutely does not guarantee that this point is a global minimum. In the complex, non-convex loss landscapes of neural networks, such stationary points can very often be suboptimal local minima or saddle points, where the training loss is significantly higher than the lowest possible value (the global minimum) and the model has not achieved the best possible fit to the training data.\n\nThe core contribution of our Corollary 4.6 is precisely to demonstrate rigorously that, for the specific setting we study (infinitely wide networks with $\\mu$P trained using SGD), the optimization dynamics avoid getting trapped in these suboptimal stationary points. Our proof establishes that the training process is guaranteed to converge to a global minimum of the training loss function. \n\n**Therefore, proving global convergence is far from trivial**; it requires showing that suboptimal stationary points (such as local minima with loss higher than the global minimum and relevant saddle points) are escaped or avoided, leading specifically to the desired global minimum (the lowest possible training loss). This property is not inherent to any network training process that stops; it relies critically on the theoretical properties of $\\mu$P in the infinite-width limit that we analyze in our paper. Our work provides a theoretical foundation for how $\\mu$P facilitates finding the global optimum, which is a key aspect of understanding its effectiveness.\n\n---\n\nResponse to Specific Questions:\n\n**Q1**. In my opinion, any neural network enjoys zero training loss and global convergence if it will no longer be updated after some training step.\n\n**A1**. As detailed above, this statement reflects the core misunderstanding we wish to clarify. Halting updates only signifies reaching a stationary point (zero gradient), which is not necessarily a global minimum. It could be a suboptimal local minimum or a saddle point with higher loss. Our key result in Corollary 4.6 is the non-trivial proof that, under $\\mu$P and infinite width, SGD specifically converges to a global minimum of the training loss, overcoming the challenge of potentially getting stuck in other stationary points with higher loss values. This is our specific contribution.\n\n--- \n**Q2**. Is the t in Theorem 4.5 a finite constant w.r.t. width like that in the TP series (e.g., TP4)?\n\n**A2**. Our current analysis focuses on the behavior of infinitely wide neural networks. Therefore, investigating the dependence of constants like t on finite network widths is beyond the scope of this paper. This remains an interesting open question for exploring finite-width corrections.\n\n--- \n\nWe trust this clarification addresses the reviewer's primary concern by highlighting the crucial distinction between halting training updates and achieving mathematically proven global convergence. We believe our theoretical analysis provides a rigorous justification for the observed empirical success of µP and clarifies the optimization dynamics in this important regime.", "claims_and_evidence": "No. The proof of the global convergence property (Corollary 4.6) is too intuitive and seems wrong. Please see the details in \"Theoretical Claims\".", "methods_and_evaluation_criteria": "Yes. mup is a popular parameterization in the pretraining of large models. However, its mechanism is rarely explored. This paper tries to prove the advantages of mup over other parameterizations (both in kernel and feature learning regimes).", "theoretical_claims": "I appreciate the result in Theorem 4.5, which proves that the feature representations evolve while maintaining their diversity and avoiding collapse throughout training.\n\nHowever, the proof of the global convergence is intuitive and seems wrong. In my opinion, any neural network enjoys the zero training loss and global convergence if it will no longer be updated after some training step.", "experimental_designs_or_analyses": "Yes. The paper conducts experiments to verify that mup can perform feature learning (better than NTK and SP) and keep a rich feature space (better than IP). I think the empirical results can support their main results.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is written clearly.\n2. The paper tries to study a significant topic in the theory for mup.\n3. As far as I know, the proof idea in this paper is original.\n\nWeakness:\n1. I think the authors need to carefully address my concern about the global convergence.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "1. Is the $t$ in Theorem 4.5 a finite constant w.r.t. width like that in the TP series (e.g. TP4)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper makes a strong theoretical contribution by proving that infinite-width neural networks trained with the $\mu$-parametrization learn linearly independent features and globally converge to the minimum of the loss. It improves our understanding of feature learning dynamics in contrast to the kernel regime, using the Tensor Program framework to analyze training under SGD.

The reviewers agree that the theoretical insights are novel and technically sound. The authors well addressed concerns raised by reviewers by providing a comprehensive and technically rigorous rebuttal, including detailed derivations of the result, empirical results on full spectrum, and justifications for global convergence.

Overall, this paper offers a meaningful advance in the theoretical understanding of deep learning under the $\mu$-parametrization and is suitable for publication at ICML.
Decision: Accept (poster)