How Gradient descent balances features: A dynamical analysis for two-layer neural networks
Authors: Zhenyu Zhu
Fanghui Liu
Volkan Cevher
Venue: ICLR 2025 Poster (2025)
Keywords: learning theory
over-parameterization
learning dynamics
Primary area: learning theory

Abstract: This paper investigates the fundamental regression task of learning $k$ neurons (\emph{a.k.a.} teachers) from Gaussian input, using two-layer ReLU neural networks with width $m$ (\emph{a.k.a.} students) and $m, k= \mathcal{O}(1)$, trained via gradient descent under proper initialization and a small step-size. Our analysis follows a three-phase structure: \emph{alignment} after weak recovery, \emph{tangential growth}, and \emph{local convergence}, providing deeper insights into the learning dynamics of gradient descent (GD). We prove the global convergence at the rate of $\mathcal{O}(T^{-3})$ for the zero loss of excess risk. Additionally, our results show that GD automatically groups and balances student neurons, revealing an implicit bias toward achieving the minimum ``balanced'' $\ell_2$-norm in the solution. Our work extends beyond previous studies in exact-parameterization setting ($m = k = 1$, (Yehudai and Ohad, 2020)) and single-neuron setting ($m \geq k = 1$, (Xu and Du, 2023)). The key technical challenge lies in handling the interactions between multiple teachers and students during training, which we address by refining the alignment analysis in Phase 1 and introducing a new dynamic system analysis for tangential components in Phase 2. Our results pave the way for further research on optimizing neural network training dynamics and understanding implicit biases in more complex architectures.

Published as a conference paper at ICLR 2025
How GRADIENT DESCENT BALANCES FEATURES: A DYNAMICAL ANALYSIS FOR TWO-LAYER NEURAL NET-
WORKS
Fanghui Liui, i École Polytechnique Fédérale de Lausanne zhenyu. zhu, volkan. colker.cevher/ @epfl.
Zhenyu Zhu',
Volkan Cevher! University of Warwick anghui www.wawwwck........o uk
ABSTRACT
This paper investigates the fundamental regression task of learning neurons (a.k.a teachers) from Gaussian input, using two-layer ReLU neural networks with width m (a.ka. students) and m, k O(1), trained via gradient descent under proper initialization and small step-size. Our analysis follows a three- phase structure: alignment after weak recovery, tangential growth, and local convergence, providing deeper insights into the learning dynamics of gradient descent (GD). We prove the global convergence at the rate of ((T for the zero loss of excess risk. Additionally, our results show that GD automatically groups and balances student neurons, revealing an implicit bias toward achieving the minimum "balanced" 4--norm in the solution. Our work extends beyond previous studies in exact-paran caaut-pprammetrizzaiion setting (m = = 1, (Yehudai and Ohad (020)) and single-neuron setting (m k = 1, (Xu and Du 2023)). The key technical challenge lies in handling the interactions between multiple teachers and students during training, which we address by refining the alignment analysis in Phase and introducing new dynamic system analysis for tangential components in Phase 2. Our results pave the way for further research on optimizing neural network training dynamics and understanding implicit biases in more complex architectures.
INTRODUCTION
Learning target function R R via neural networks through gradient descent flow has received significant attention in machine learning theory. Research in this area primarily focuses on understanding the learnability and dynamies, aiming to theoretically explain the advantage of feature learning in neural networks. This problem is often studied under various assumptions about f* For instance, f* is frequently (implicitly) assumed be smooth in a kernel regime (Jacot al. 2018 Allen-Zhu et al., 2019) Arora et al. 2019). Additionally. f might possess further structures, such as being located on low-dimensional subspace Mouusav-Hosseenn
al., 2023) or a manifold (Arora
et al. 2022). A typical example is assuming f is a sparse polynomial (Abbe et al. (022).. In this setting, the separation between kernel methods and neural networks well studied through metrics like the information exponent (Arous et al. 202I), leap complexity (Abbe et al. 2023), and generative
exponent (Damian et al. 2024).
In contrast to smooth functions, another research direction focuses on non-smooth target functions, such as ReLU. This non-smoothness naturally highlights the difference between kernel methods and neural networks in terms of approximation ability (Bach) 2017). As result, researchers have turned their attention to studying the learning dynamics to gain a deeper understanding of convergence, For
instance, they investigate learning with single ReLU neuron (Wu al.) 2023] Xu and Du 2023)
multiple ReLU neurons (Zhou et al. 2021 Akiyama and Suzuki 2023)
We consider the problem of learning one-hidden-layer ReLU networks under the Gaussian measure.
The target function f* sum of multiple ReLU neurons fxx = Σ (((1) x)) with the
'ee assume that teacher neurons are positive. However. if both the student and teacher networks contain least one negative ReI.U neuron, proof techniques can address such cases with minor modifications, requiring no significant changes the overall analysis.

---

Published as a conference paper at ICLR 2025
parameters {vr which can be learned from n i..d. samples {(z,, (z: via two-layer neural network with m (student) neurons with random Gaussian initialization (w; {w ~ ON under the expected squared loss:
2
L(W) = ((o.L
Σ (( x) - ((xr)
which aims to find good approximation of f from the student network. To ensure learning performance, m 2 k is needed.
This problem is identified an additive model in statistics (Stone 1985: ; Hastie and Tibshirani 1987 and it receives great attention in theoretical computer science (Chen et al., 2023) and machine learning theory, especially on sample/time complexity well as training dynamics (Boursier and Flammarion 2024, Bietti et al. 2023). However, understanding how tradient-based training algorithms recover the teacher network and analyzing the entire training dynamics are still challenging. Therefore, most current analyses are limited to non-gradient-basel algorithms (Chen et al., (023), or local analysis for gradient-base algorithms, which assumes that the loss has already been minimized below a very small threshold, or the angles between teacher neurons and their nearest student neurons are already small (called strong recovery). e.g., (Zhou et al.] 202I). If we go beyond the local analysis sis, previous result
GD training can only handle specific cases, such (Yehudai and Ohad 2020) for m = k- = 1. WW et af. 2018) for m = k = 2, and Xu and Du 2023] Chistikov et al.] 2023) for m k- 1. In fact, studying more general cases, such as m. k ((1), remains unresolved, even in local analysis. Accordingly, we aim to address the following question:
How can gradient descent recover teacher neurons and balance features beyond local analysis?
To better understand the learning dynamics in the above question, we follow the "align then fit" framework (Maennel al. 2018 [Boursier and Flammarion 2024), which also helps to explain the implicit bias of the learned solution. In this study, we run the gradient descent (GD) over [q. (I) Since analyzing the entire training dynamics is still challenging and is an open problem, so we assume the weak recovery, where for each student neuron, exactly one teacher neuron exists that is not nearly perpendicular it. Note that the weak recovery condition still much weaker than the condition with local analysis and strong recovery that will be proved in our analysis. An informal version of our theoretical results is given as below.
Theorem 1 (Global Convergence after Weak Recovery: Informal). Under proper assumptions (e.g.. 3., teacher neurons are with same length |//l) and orthogonal to each other), sufficiently small initialization with o = o(poly(d &)), and trained via gradient descent with sufficiently small step-size 1 o(1) to minimize Eq. (1) after time T 5M (2)) ). for any T N, we have:
|| T3
A *||/ll) m
LT + +T) <
and [T + T)| = θ
Vi E [m], w.h.p-
Our result demonstrates that the Eq. (1) can be solved by GD in the polynomial time to find the global minima and achieves the global convergence rate at )1) We admit that the derived sample/time complexity is not optimal, but to our knowledge, this is the first polynomial-time result of GD training beyond the local analysis for Eq. (1) with m, k = (l)) Besides, our results also indicate that the obtained solution will converge to a minimum "balanced" &g solution, where the "balanced" is determined by student neurons and their respective nearest teacher neurons.
Technical challenges. We employ the similar proof framework Xu and Du (2023) on m 2 k= The main challenge of this paper is how to address the coupling of different teacher neurons" influences on the student neurons, even though the teacher neurons are orthogonal to each other. For instance:
In phase 1, single teacher neuron (k: = 1) (Xu and Du 2023) allows for monotonic convergence on the angular difference between the teacher and student neurons. However, this does not hold for & > 1. In this case, we use approximations of sine and cosine values for decoupling when the angle is very small or near perpendicular. Hence we can simplify the training dynamics and prove that the sine of the minimum angle converges linearly to a tiny neighborhood.
IIn our setting. the number of training iterations T' corresponds to the sample size n, and the sample complexity achieve an expected risk is O(e 1/3 p/poll(m, k)).
2

---

Published as a conference paper at ICLR 2025
In phase 2, during the analysis of neuron growth, the tangential components of the student neurons at each teacher neuron (and for more teacher neurons) are quite complex. Classical recursive not handle this. Instead, we develop new technical tool by
relationship in (Xu and Du, 2023) building dynamical system: we formulate the matrix iteration form, estimate the eigenvalues of the transition matrix, and establish the upper and lower bounds of such dynamical system.
RELATED WORK
Dynamics of gradient descent in the teacher-student setting: Li and Yuan (2017) studied the pauuttparameterized setting and proved convergence for SGD with initialization near identity. The separation between kernel methods and two-layer neural networks is further described in [i et al.) (2020). To further understand the convergence and generalization of regression tasks using non-linear networks, is essential to thoroughly analyze the dynamics throughout gradient- paddentt-based training, commonly described "align then fit" (Maennel al., [0188 Boursier and Flammarion (024) in three-phase analysis framework. Xu and Du (2023) provide global convergence of learning with single ReLU neuron, where the proof for the local convergence (i.e., the third phase) is given by Zhou et al. (021). This analysis framework is also used in various settings, e.g., binary classification (Min al. 2023) and matrix sensing (Xiong et al., 2024). Additionally, [Zhou and Ge (2024) propose an algorithm combining gradient descent with convex optimization, achieving strong recovery in single gradient step through specific initialization. Our work also analyzes the dynamics from weak recovery to strong recovery.
Besides, our problem can be cast as a special case learning with multi-index model (Bietti et al. 2023) where the link function (i.e., the activation function used in this work) is unknown. However, the techniques are different and our three-phase analysis framework allows for a better understanding of global convergence. Some statistical physics studies work have explored related topics but differ from our work (Goldt et al. 2020 Arnaboldi et al.) (203) by focusing on generalization errors without providing convergence rates detailed analyses of training dynamics and convergence phases.
Implicit bias: Recent studies suggest that gradient descent is implicitly biased towards a low-rank hidden weight matrix sparse number of directions represented by the neurons (Safran al.. 2022 Shevchenko et al. 2022: Chizat and Bach 2020). This implicit bias is often characterized by the minimal norm interpolator, which is closely related to sparsely represented directions (Lyu and
Li 2020). These findings indicate that the early alignment phase enforces the alignment of weights towards small number of directions, even with omnidirectional initialization, leading to implicit regularization at convergence (Boursier and Flammarion 2023).
NOTATIONS, PROBLEM SETTING, AND ASSUMPTIONS
In this section, give notations that needed in this paper and then introduce our problem setting as well as the required assumptions in our proof.
3.1 NOTATIONS
Basic notations: We use the shorthand [n] = 21, 2,......., ,,) for a positive integer n. We denote by a(n) 2 b(n): the inequality a(n) 2 cb(nt) that hides positive constant that is independent of n. Vectors (matrices) are denoted by boldface, lower-case (upper-case) letters. The used norm || || in this paper is lz norm if we do not specify. We follow the standard complexity theory c.g., O, o, Q, and θ for order notation.
Ciimmann-Landaa notation in
Notations on angle: The angle between any two non-zero vectors and v is denoted as L(w, v) cos -1 (w..) Then we use the following notations for any i, E [m], E [*:]
w [|]]
0g L(w;; vi): the angle between a student neuron w; and a teacher neuron V).
Pl /(w,, ) the angle between two neurons w; and wy for student model.
Ti - arg min, ((w, by (0)): the index of the teacher neuron with the smallest angle the w, initialization, which the smallest angle is denoted 0,. - f.., = L(w, vy.).
Σx..e wi - w: the difference of the teacher neuron 8, and the sum of the student neurons around 0%-
3

---

Published as a conference paper at ICLR 2025
For notational simplicity, by denoting a ||a] we denote the tangential part hu _ (w,, öt) the projection of w, along with the direction of and similar notation for hę- - (wi, (r,)) Besides, we denote (t) as the vector w, at time t, which also adapts to (t), etc.
Notations loss: The standard Gaussian distribution is (0, 1) with zero-mean and unit variance. We denote (o,1) by Ex for simplicity. By defining the residuals of the neural network as:
R(x)= = Σ M(w,, x)) ==1
Σ )(vv),
then the loss can be written as I(W)
2 B R(x)²
3.2 CLOSED FORM EXPRESSIONS OF GRADIENT OF LOSS: VL
To make our paper self-contained, we present the closed- form expressions for VL when the input data follows Gaussian distribution, given by Safran and Shamir (2018). see the details in [Appendix DJ We denote V, Sun, as the gradient of loss the ws, when w; o O. Then for any E [m]. the loss function is differentiable with gradient given by:
@L(W)
V
1
1 2
sin Ply w
sin Ou (m
Σ Pay j=1,jzi
0 0uri
|
1=1
j=1,j
1=1
We use random Gaussian initialization for neural network training. i.e. Vi E [m], w.( (O La) with the variance o². Then we can prove that ||w:]] has bounded norm with high probability if the dimension d is not small, see Lemma i Appendix D
3.3 ASSUMPTIONS
We state the used assumptions in this paper.
Assumption (Weak recovery). Regarding the angle (O) defined before for any E [m].j [k]] at initialization, denote 04- (0) as the smallest angle between and its closet teacher neuron. The
weak recovery assumes 0,- (0) < (0)(0) with j E [k] and j + Tx. We mathematically formulate this
as below.
0. (0) is acute: 0 << < 0₂ (0) A C, = 01), and S. (0,
. (0)(0)) is close orthogonal:
- 01)(0)) S <- o(1) with j [k] and j # тІ.
The weak recovery assumption requires that a student neuron is not orthogonal to its closet teacher neuron but is nearly orthogonal the remaining teacher neurons (Dandi et al. 2024). If we focus on the single ReLU case like Xu and Du (023), this assumption can be directly removed because there only one teacher neuron. With only one teacher neuron, there are no competing neurons for alignment, and thus the angle between the student and teacher neuron is naturally the smallest.
Remark:
Assumption (Orthogonal and same norm for teacher neurons). The teacher neurons given by (u,)* and are assumed to be orthogonal to each other with the same norm, i.e., (v,,,,) = 0 and [w.
| ||l|| Vi + j, i,j E [m). Clearly, we have k S due to the orthogonality
teacher neurons.
This assumption requires all teacher neurons pointing different (orthogonal) directions, which is important for identifiability or recovery. It aligns with practical considerations by allowing diverse tasks such that the target feature directions do not significantly overlap. This assumption as well as its variant (e.g., separation among teacher neurons) has been widely used in previous theoretical results.e.g. (Zhou al. 2021 Oko et al. 2024 Simsek et al. 2023). We can relax this assumption where the teacher neurons are nearly orthogonal and have similar norms. However, such relaxation would require additional computations our analysis. On the other hand, extending the analysis to the fully not-orthogonal case (arbitrary angles between teacher neurons) remains an open problem under our setting and would likely require new techniques. To avoid unnecessary complexity and focus on the core analysis, concentrate the basic assumptions.
Remark:
4

---

Published as a conference paper at ICLR 2025
Assumption 3 (Balance condition at initialization). Ar initialization, record the number of student neurons w; with TΙ = as my. Then we assume
34 S my
m
S 3m
14
E [k].
Remark: This a balance condition such that the number of merged student neurons among each teacher neuron is not extremely small or large. It is motivated by |Boursier et al. (2022) Assumption 3) and (Wojtowytsch) 2020) that requires the student neurons to cover all directions of the teacher neurons. Our assumption requires student neurons coincide with teacher neurons in a balanced way.
MAIN RESULTS
In this section, we will provide the main theoretical results, First, in Section 4.1, we provide the primary result global convergence. Then, in Section 4.2, we summarize the training dynamics across the three phases, highlighting their durations, key processes, and contributions to global convergence in a unified framework. Then, in the following subsections, we discuss the training dynamics of the three phases and provide proof sketches. In [Section 4.3.1] we provide the main dynamics and final state results of the alignment process in the first phase. In [Section 4.3.2 provide the main dynamics and final state results of the tangential growth process in the second phase. In Section 4.3.3] we provide the results of the local convergence in the third phase and then achieve the final global convergence result.
4.1 MAIN THEOREM
Theorem 2. Assume d = (log(m//)) (8)) with 5 E (0,1), under Assumptions
2 and 3
let
_d- +)) - oppoly(d })) and trained via gradient descent with step-size, 1) - o(poly(m dd
oppoly(m -n)) = (poly(()) to minimize Eq. (1) then there exists T 52( k log log m (((2)
vay
such that with probability at least 5 over the mitialization, for any T E N. we have:
k ||112 n
||||| 4m
||le|l
L(W(T* + T)) 0
and
S w. WT + +TII I
Vi [m].
my
Remark: Theorem 2 provides a convergence rate of 3 which is consistent with previous results (Xu and 20/33. Moreover, indicates that the more teacher neurons and the larger their norms, the slower the convergence rate. This aligns with our intuition that when the initialization is very small, larger norm and more teacher neurons require student neurons to take more time to align and converge to the teacher neurons. Unlike (Xu and Du 2023), results appear to be independent of the number of student neurons m, but rely on the choice of the initialization that implicitly depends on m. Furthermore,
results indicate that the student neurons will implicitly converge specific teacher neuron and maintain a balance among themselves.
4.2 SUMMARY OF MAIN RESULTS WITH PHASE ANALYSIS
We summarize the three training phases and associated dynamics in a unified framework to provide clear overview of our findings. This structure highlights the processes that occur in each phase, their respective contributions to the global convergence result, and key observations. For a concise summary of the duration and key results, see Table at the end of this section:
Phase 1- Alignment: In this phase, each student neuron aligns with a specific teacher neuron by minimizing the angle between their vectors. This process depends the initialization scale (a) and the learning rate (7)). By the end of this phase, the angle between student and teacher neurons is reduced to OEE)), ensuring proper alignment while maintaining bounded norms.
Phase 2- Tangential Growth: Following alignment, the student neurons grow tangentially along the direction of their aligned teacher neurons. During this phase, the projection strength between student neurons and teacher neurons balances, leading to a linear decrease in the loss. This phase ensures that the norms of student neurons become comparable to those of teacher neurons.
Phase Local Convergence: In the final phase, the training achieves global convergence the loss becomes sufficiently small. The loss decreases rate of (TT-3), and the
5

---

Published as a conference paper at ICLR 2025
student neurons converge implicitly to the directions of their corresponding teacher neurons. This phase solidifies the final convergence result.
Table 1: Summary of Training Phases and Dynamics
Phase
Duration
Key Resalts
Phase 1: Alignment
Ti
e
Angle reduced ( norms remain bounded.
Phase
langential Growth
in O(T-3)
Projection strangth balanced, decrenses
Phose 3: Lcal Convergenie
Loss decncases ((T naurons converge teacher divections.
4.3 PROOF OVERVIEW
In this section, we provide a sketch the Theorem 2
The complete proof can found in the appendix. Our proof is primarily divided into three phases: alignment (Section 1. tangential growth (Section 4.3.2), and global convergence (Section 4.3.3). Finally we can summarize the results in these three phases for the main result.
4.3.1 PHASE ALIGNMENT
During this phase, each student neuron individually aligns with specific teacher neuron. The outcomes this section are divided into two main parts: i) the upper and lower bounds on the lengths of the student neurons, as well as the angle between student and teacher neurons during the time Theorem? 3) ii) the upper bound on the angle between student and teacher neurons at the end of phase 1, as well as the balance of projection strength from different student neurons onto the teacher neuron Corollary The detailed derivation can be found in Appenddix F)
Theorem 3 (Phase 1:
Alignment Process). Assume d ((log(m/ (m/0) / with 5 (0, for any El
under Assumption with 10kς s o = o(1) and Assumptions 2 such that o = of polv(e))ll)
vd
in our random Gaussian initialization, and the stepsize satisfies v of (o ), then there exist
T = (. s Tı. the following statements hold with probability at least - 6:
81 < (1)] < 82 + 2kn [10l
Vi E [ma] with 81 == ==2oV Looa, :2 20vd,
1
(3)
and
(1-(1) sin?
- & < (1 ryk |00l
sin (1,(0) 2
- 22 Vi [m].
(4)
Remark: Our theorem implies that, during phase of the training, the norm of each student neuron always has an immutable lower bound, while the upper bound increases linearly over time. Additionally, for each student neuron, the angle with its nearest teacher neuron converges linearly within an error range of ej. Compared to the results of Xu and Du (023)), the upper bound of our neuron norm increases times faster because we have h different teacher neurons, which naturally leads to this outcome. Taking k 1, our convergence rate is faster by a constant factor compared to the results of Xu and Du 2023). and our condition for a is weaker. When the same a is selected, the total duration of phase 1, denoted as Tı, they are the same.
Then, we briefly introduce our proof technique, due to the presence of multiple teacher neurons, the gradient expression in Eq. contains 2(k 1) cross terms including 0n with detailed interactions, which do not exist in Xu and Du (023). To handle this challenge, provide additional analysis on alignment related to these cross terms in phase Specifically, we prove these results by induction.
Proof of Eq. (3); This formula provides the upper and lower bounds of ||w||| during the training. For the lower bound, based on the gradient expression V. in Eq. (2). we prove (w,(t), (t)) < O, which ensures that the norm ||w/| increases monotonically such that [tw:(t)] 2 81. For the upper bound, we need to bound the norm gradient using Eq. (2) Then, applying the triangle inequality, we can obtain the desired result.
Proof of Eq. (4); This formula illustrates the angle dynamics (i.c., the alignment process) between the student neuron and its closest teacher neuron during phase 1. For larger 0z,, is easy to prove

---

Published as a conference paper at ICLR 2025
that the 0-- decreases monotonically. However, when 8,- is very small, we cannot guarantee the monotonic decreasing property of fz.. To this end, we build the connection between sin (0,- (t) and the following angle difference
cos(O,, (t 1)) co,(0;-- (t))
+ 13, the first-order term 12 and the second-order term 13
By estimating 12 and I3, we can track the dynamics of the angle difference and then prove that sin O.- converges linearly to a very small neighborhood (i.e., e
At the end of phase 1, we conclude the following result:
Corollary 1 (Final State of Phase 1). Under the same conditions as Theorem 3 at time Tú, the following statements hold with probability at least 8:
4e1. and hy (Ti) - (Zi), Vi,j E [m].
(5)
Remark: By the end of phase 1, each student neuron will align with its nearest teacher neuron with the residual angle at the order of O(e)). Additionally, the projection lengths of these student neurons in the direction of their corresponding teacher neurons relatively balanced, with a rough upper bound of 2.
Proof of Eq. (5) For the first part, substituting the parameters from Theorem 3 into Eq. (4) will yield
the result. For the second part, firstly, we derive the upper and lower bounds for h4- (t + 1) - h4s
and then accumulate these bounds. Next, we prove that before certain time (e.g., : /505),
the upper bound of h; (t) is relatively small compared to this accumulated value. This allows us to establish the upper and lower bounds for all h2 (Ti) and thereby determine the maximum ratio of 1:p (Ti) among different student neurons.
4.3.2 PHASE 2 TANGENTIAL GROWTH
In this section, present the results of the second phase, in which each student neuron grows along the tangential direction of the teacher neuron aligned in phase as below. detailed derivation can be found in Appendix G
Theorem 4 (Phase 2: Tangential Growth Process). Assume d = ((lo(mm/8) with d E (0. 1).
for any 61 > 0, 62 > O, under Assumption with c = o(1), = o(poly(c2)), Assumptions 2
such that of ( in our random Gaussian initialization, and the stepsize satisfies 1j
aavi ), then by setting there exist T2 = Ti + O(2 In (2))), then VT; S t S Tz, we define
of [|l|
(t) || E E I, (t) E [Rt), the following statements hold with probability at least
1- -à:
hz)(1)) S (1))))h.(0))) (1); and
2|10|
2 h1.
(1)
81 2'
Vi,j E [m]] and T3 = T
(6)
my.
11-
t-Ti
2
(1- 33mm
t-
| |||||| 2 H()
|||||
STez ||a|| 2 24Tez || V/ E [k]]
9k
3
2k
(7)
and
(z- < 62, Vi E [m].
(8)
Remark:
This theorem tells us that during phase
1). The norm of student neurons close to the same teacher neuron remains relatively balanced, with each neuron having strict upper and lower bounds (Eq. (6)) It is worth noting that, unlike in phase one, see Eq. (5) this balance is not maintained for all neurons.
2). The projections of the student neurons near each teacher neuron will gradually increase, and the difference from ||aa| will approach zero a linear convergence rate (Eq. (7)). This result implies that as training progresses, the loss gradually decreases. We will further prove that by the end of phase 2, the loss has decreased to sufficiently small value.
3). The angle between each student neuron and its nearest teacher neuron stays within a small range (Fqq. (8). However, the angle is slightly larger than that of Phase because additional cost/movement
7

---

Published as a conference paper at ICLR 2025
is required to handle the convergence for tangential difference and the decrease of loss. For example, in Phase 2.
we have [|)ll < 2 in Phase but it changes to [[V)(t)] I 15k |||||
4). Taking k = our condition for e2 is similar to that Xu and Du (2023). but we have relaxed the learning rate condition by a factor m. And the total duration of phase 2, is reduced by a constant factor of 2-
Then, we briefly introduce our proof technique. Compared to one teacher setting (Xu and Du 2023), the tangential analysis requires new dynamical system analysis regarding the dynamics of (Hi())]== due to the coupling tangential components among student/teacher neurons. Besides, the loss function becomes more complex [Eq. (14) and we have control the loss below certain threshold in the presence of these interactions, which requires additional quantities to estimate. Specifically, we prove these results by induction.
Proof of Eq. (6) For the first part, we follow the proof of Eq. (5) to build the connection between h4- (t
heg (t) and H₁ in a weighted sum relationship. with an additional constant term Qi- For two different student neurons close to the same teacher neuron, these weights are the same. By studying the changes of 0.- and O2 during this phase, [Q)( will be bounded by a small quantity. Then we conclude the result by summing and combining the results with [q. (5) For the second
part, based on Eqq (7) we can derive H1 2 and finish the upper bound by combining the results from the first part. For the lower bound, we derive h,- (t + 1) h4- (t) > O, which implies that hy- is
monotonic increasing. Combining this with Eqq. (3) and (5) the proof is complete.
Proof of Eq. (7); Using the above analysis about hig- (t 1) h2- (t) > 0 and the relationship between his and H,,, we can establish a recursive relationship between H( + 1) and (t) as well. Note that there is coupling between different H and interference from small quantities Qi,
express the iterative formula in matrix form. To be specific, by denoting H == {Hi)/=1 (we write in matrix formulation), it admits the following recursive relationship:
H( + 1) = (H) + Q()) for a certain transition matrix A and (t) depends on Qi(t).
By analyzing the eigenvalues of the transition matrix A, we estimate the upper and lower bounds of such dynamic system. For the small quantities Qi.
adopt the same approach used in proving Eq. (4) Finally, we prove that H converges to small value at linear convergence rate.
Proof Eq. (8): The proof here is similar Eq. (4) as it also analyzes the dynamics of cos Oz.. However, the difficulty lies in that at this phase, the influence of w in the gradient is no longer negligible compared to v, making the iterative relationship between angles more complex. First, by proving
(w,(()) ((1), Vi.j E [m]. Ti S st T2 ww (t)
we are able to analyse the dynamics of cos 0;- (i.e., 12 and 13 in Eq. (4) based two cases T, = (T.. First, we use some properties of trigonometric functions to decouple this relationship so that it only involves the coupling between each student neuron and its nearest teacher neuron. Then,
we estimate the difference sin?
(++(1+1)
sin
(.-(t)
for the final sin 0,. (t). Unlike in phase 2
1, here we obtain an upper bound for the linear growth of the angle 0z.. However, we can still prove that within the range of T2, the angle remains small.
At the end of phase 2, we can draw the following results: Corollary 2 (Final state of Phase 2). Under the same conditions as Theorem 4 at time T2. the following statements hold with probability at least 1 1-6 8:
| 3m
3 ||ll m.
1 0.05 Vi E [m]) and L(W( < k2.2 ||0112
(9)
Remark: After phase 2, the norms of each student neuron have balanced, and the loss has decreased to very small value. This provides the foundation for proving local convergence in phase three.
Proof of first part Eq. (9) We use the results in Theorem 4 to prove this result. For the lower bound, we first observe from Eq. (7) that H, is very small at time T2, meaning the sum of h among
[e]l student neurons near each teacher neuron is close to ||l||, i.e., (T2) s 33
Using the balance
8

---

Published as a conference paper at ICLR 2025
of them in [Bq. (6) we can then establish lower bound for h4- </=[[| cos f.-), which further allows us to derive a lower bound for |/,,|l- Similarly, for the upper bound, we first observe from Eq. (7) that time T2, the sum of h among student neurons near each teacher neuron is close to but still
less than || Using the balance of them in Eq. (6) we can then establish an upper bound for
hy (llen cos 0,.) ). Given that the angle between each student neuron and its nearest teacher neuron
is very small (sccond part in Eq. (8)). we can further derive a lower bound for ||w:]|-
Proof of second part of Eq. (9); The key point of this proof involves introducing an auxiliary function to help decompose the L. The loss L can be expressed in the summation of g. see Apppendix for details. First, based on the upper bound of the angle in phase 2 (second part in Eq. (8). we know that there are two scenarios for the angle in the closed form of the loss: close to 0 and nearly orthogonal.
We discuss the upper and lower bounds of auxiliary function 9 in these two cases. Then, according
to Eq. (7) we find that at time 72, the sum of the norms of the student neurons near each teacher neuron close to the norm of teacher neurons, i.e. T / (72) 2 (1 o(1)) | Combining these two results, we can derive upper bound for the loss L.
4.3.3 PHASE 3- LOCAL CONVERGENCE
In this section, we present the results of phase 3 - local convergence. Specifically, we show that when the loss is already small enough, the loss function converges to zero at rate of O 7 Theorem Our results build upon the previous works of Xu and Du (2023); Zhou et al. (2021
Safran et al.
(202I). The detailed derivation can be found in Appendixx H
Theorem 5 (Local convergence). Suppose the initial condition in Lemma and Assumption p [and holds. If we set 62 = (pply(1)) and = o(1) in Theorem 4 then VT E N, the following statements hold with probability least 6:
L(W(T (T + T2)) S
(10)
((W Ω k 1|01
1/T
and
||l|| 4m,
4 ||l|||
S [T
T2)
Vi E [m]]
(11)
my,
Remark: This theorem shows that, under the condition that the loss is less than a very small value and the neurons remain balanced at the end of phase two, GD training can achieve the global minimum
with convergence rate of 7 This result is consistent with the previous result in Xu and Du (023) and is superior to 1 in Zhou et al. (2021). Furthermore, this result also indicates that, without using regularization during traunin, every student neuron will implicitly converge to the directions of specific teacher neurons, and there is balance among student neurons that converge the direction of the same teacher neuron.
Proof of Eq. (10); The proof framework of Theorem 5 is standard based on the local convergence analysis, e.g., (Zhou et al. [221] Xu and Du (023). The key point is utilizing the result of classic optimization in Appendix H.4 and the lower bound of the gradient to satisfy the conditions of (Xul and Du. 2023 Lemma 24). First, we follow (Zhou et al., 2021) to derive several lemmas related to the properties of the loss function. Based on these lemmas, we can obtain the lower bound of the gradient in terms of the loss. Then, similar to Safran et al. (02T)) we deduce that when the neurons maintain a certain balance, the loss locally smooth. This allows us to directly apply the classic optimization theory conclusion regarding the relationship between adjacent iterations of gradient descent Appenddx H.4 Finally, we build the final convergence result by Xu and Du (2023) Lemma 24). Additionally, our proof requires that the balance condition of the neurons is consistently maintained Eq. (11) which can be proven using induction and convergence results alternately.
Finally, by combining results from Sections 4.3. to 4.3.3 with the hyper- prrramper selection in Ap pendix B we obtain the global convergence result in Thcorem 2
When A = 1, compared to the results of Xu and Du 2023) our paper needs stronger requirements on σ. 17 and time. This is due to the upper bound of the loss after phase 2 in Eq. (9) and its relationship with 62. Due to multiple teacher neurons, the number of student neurons converging each teacher neuron directions are different. This leads to different norms for the student neurons, which makes a looser upper bound, However, in the case of l 1, such handling is not necessary. Therefore, our

---

Published as a conference paper at ICLR 2025
371
202
28
(a) 1: 2, 771
20
(b) 4, m =
(c) 1: 4, m 20
(d) m 40
(c) 1: = 2, 20
(f) 1: 4, m = 12
(g) 1: = 4, 20
(h) 1: 4, m 40
Figure 1: Convergence curves for different m and k.
results can cover the results Xu and Du (2023) with only minor modifications and have better constant factor in phase discussed before.
5 NUMERICAL VALIDATION
In this section, we empirically validate our theoretical results by plotting the convergence curves under the following setting: we set |!| = 5, data dimension d = 100, batch size of 512, and a total of 5000 batches. The total number of training samples (equivalent to the previously mentioned T T)) is 2.56 x 10 Besides, we have added 1/7 reference line in the log-log plot for better comparison. We selected four sets of parameters i and h 2, m 20, A 4, m 12, k 4.m 20, and k = 4, m = 40 with initialization variance o = 10-6 and learning rate = 5 x 10 The plots in Fig. I [sow the cosine of the angle and norm convergence during training (top row) and the log-log plot of the loss during training (bottom row) for different values of k and m. The results show that larger values lead longer t1 and f/2 and slower convergence rates, while larger m values result in shorter t and tz but have little effect on the convergence rate. This matches our theoretical results such that using more (student) neurons decreases the time for alignment. We admit that learning more (teacher) neurons generally requires more time but this is given under the same initialization strategy. Instead, our initialization strategy depends on m and k; leading to different learning dynamics.
Regarding the timescale experiments, we divided the training dynamics into three phases for analysis. We can observe the clear "align then fit" phenomena where in phases 1 and 2, the angle aligns and the tangential grows until the norm of neurons' weights is unchanged. In phase 3, the loss function decreases for fitting data. The phase transition from Phase 1 to 2 is not very clear in the experiments but can still be observed with distinct difference in that Phase 2 finishes later than Phase 1. Nonetheless, we have marked the figure's approximate endpoints of the first and second phases.
CONCLUSION
Our three-phase analysis framework provides comprehensive analysis on global convergence, i.e., 1) alignment: the angle decreases 0:- (Ti) < O(e1) satisfying the balance condition but the norm of student neuron gradually increases with Ti; 2) tangential growth: the projection of the student neurons near teacher neurons gradually increases. The angle is still small but slightly larger than that of phase 1 due to the additional cost of handling the convergence of tangential difference; 3) local convergence: the loss is close to zero and the neurons still rell-balanced thus achieving the global convergence at the rate of (T-3).
One potential drawback of this work is the weak recovery which simplifies the analysis. However, without weak recovery, the analysis will be quite complex, remaining unsolved, and thus we leave it as future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We are also thankful to the reviewers for providing constructive feedback. This work was supported by Hasler Foundation Program: Hasler Responsible Al (project number 21043). Research was sponsored by the Army Research Office and was accomplished under Grant Number W91 INF-24-1-0045 This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement n" 725594 time-data).
REFERENCES
E. Abbe, E. B. Adsera, and Misiakiewicz. The merged-stane -trrreest property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory (COLT), 2022.
E. Abbe, H. B. Adsera, and T. Misiakiewicz. Sgd learning neural networks: leap complexity and saddle-to-saddle dynamics. In Conference on Learning Theory (COLT), 2023.
S. Akiyama and T. Suzuki. Excess risk of two-layer rel.U neural networks in teacher-student settings and its superiority to kernel methods. International Conference on Learning Representations (ICLR), 2023.
Z. Allen-Zhu, Y. Li, and Z. Song. convergence theory for deep learning via over-parametteriztion.. In International Conference on Machine Learning (ICML), 2019.
L. Arnaboldi, L. Stephan, Krzakala, and B. Loureiro. From high-dimensional & mean-field dynamics to dimensionless odes: A unifying approach sgd in two-layers networks, 2023. URL
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In Advances in neural information processing systems (NeurIPS), 2019.
S. Arora, Z. Li, and A. Panigrahi. Understanding gradient descent the edge of stability in deep learning. In International Conference on Machine Learning (ICML), 2022.
G. B. Arous, Gheissari, and A. Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research, 22(106):1-51, 2021.
Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 2017.
A. Bietti, Bruna, and L. Pillaud-Vivis On learning gaussian multi-index models with gradient flow. 1rXiv:2310.19999 2023.
E. Boursier and N. Flammarion. Penalising the biases in norm regularisation enforces sparsity. In Advances in neural information processing systems (NeurIPS), 2023.
Boursier and N. Flammarion. Early alignment in two-layer networks training a two-edged sword. arXiv:2401.10791, 2024.
Boursier, Pillaud- Vivien, and Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. In Advances in neural information processing systems (NeurIPS), 2022.
S. Chen, Z. Dou, S. Goel, A. Klivans, and R. Meka. Learning narrow one-hidden-llyer relu networks. In G. Neu and L. Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 5580-5614, PMLR, 12-15 Jul 2023. URL /LEtp::/pproceeiingg.....: /tt.a//////......o..........a//a
D. Chistikov, M. Englert, and R. Lazic. Learning neuron by shallow reLU network: Dynamics and implicit bias for correlated inputs. In Advances in neural information processing systems (NeurIPS), 2023.
11

---

Published as a conference paper at ICLR 2025
L. Chizat and F. Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory (COLT), 2020.
A. Damian, L. Pillaud-Vivien, J. Lee, and J. Bruna. Computationallstatisticaa gaps in gaussian single- index models. In The Thirty Seventh Anmual Conference Learning Theory, pages 1262-1262. PMLR, 2024.
Y. Dandi, E. Troiani, L.. Arnaooli,, L. Pesce, L. Zdehorová, and F. Krzakala. The benefits of reusing batches for gradient descent two-layer networks: Breaking the curse of information and leap exponents, 2024, URL /R..ttpp:://arss..rrrr///ss
03220
S. Goldt, M. S. Advani, A. M. Saxe. F. Krzakala, and L. Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks the teacher-student setup". Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124010 Dec. 2020. ISSN 1742-5468. doi: 10. abc6le
(088/1742-5468/abcchh URL htl.bttp://dde doi http://ddddoooooolll.o/o/ 1088/
Hastie and R. Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-386 1987.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems (NeurIPS), 2018.
Y. Li and Yuan. Convergence analysis of two-layer neural networks with relu activation. In Advances in neural information processing systems (NeurIPS), 2017.
Y. Li, T. Ma, and H. R. Zhang. Learning pereparametrized two-layer relu neural networks beyond ntk. arXiv:2007.044 04,50, 2020.
K. Lyu and Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference Learning Representations (ICLR), 2020.
H. Maennel, O. Bousquet, and S. Gelly. Gradient descent quantizes relu network features. arXiv:1803.083661 2018.
H. Min, R. Vidal, and E. Mallada. Early neuron alignment in two-layer relu networks with small initialization. arXiv:2307.12851 2023.
A. Mosseni- S. Park, M. Girotti, I. Mitliagkas, and M. A. Erdogdu. Neural networks efficiently learn low-dimensional representations with SGD. In International Conference on Learning Representations (ICLR), 2023.
Y. Nesterov al. Lectures on convex optimization. Springer, 2018
K. Oko, Y. Song, Suzuki, and D. Wu. Learning sum of diverse features: computational hardness and efficient gradient- based training for ridge combinations, 2024. URL hattps://axxiv o barg/bb// 2406. 11828
I. Safran and Shamir. Spurious local minima are common two-layer relu neural networks. In International Conference on Machine Learning (ICML), 2018.
I. Safran, G. Vardi, and J. D. Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. arXiv:2205.0000 00002, 2022.
M. Safran, G. Yehudai, and O. Shamir. The effects mild ver-paraaeteriizatoon on the optimization landscape of shallow relu neural networks. In Conference Learning Theory (COLT), 2021.
A. Shevchenko, V. Kungurtsev, and M. Mondelli. Mean-field analysis of piecewise linear solutions for wide relu networks. Journal of Machine Learning Research, 2022.
B. Simsek, A. Bendjeddou, W. Gerstner, and Brea. Should under- parameterized student networks copy or average teacher weights? In seirth- Conference Neural Information Processing Systems, 2023. URL htttpps://ppeeeeeeeee....
d=MGOmY
C. J. Stone. Additive regression and other nonparametric models. The annals of Statistics, 13(2): 689-705, 1985.
12

---

Published as a conference paper at ICLR 2025
S. Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv:2005.135.00 2020.
C. Wu, J. Luo, and J. D. Lee. No spurious local minima in a two hidden unit reLU network, 2018.
J. Wu, D. Zou, Z. Chen, V. Braverman, Q. Gu, and S. M. Kakade. Finite-sample analysis of learning high-dimensional single relu neuron, In International Conference on Machine Learning (ICML), 2023.
N. Xiong, L. Ding, and S. S. Du. How over-parameteriizatonn slows down gradient descent in matrix sensing: The curses symmetry and initialization. In International Conference on Learning Representations (ICLR), 2024.
W. Xu and S. S. Du. Over-paaraettrrizationn exponentially slows down gradient descent for learning single neuron. In Conference on Learning Theory (COLT), 2023.
G. Yehudai and S. Ohad. Learning single neuron with gradient methods. In Conference on Learning Theory (COLT), 2020.
M. Zhou and R. Ge. How does gradient descent learn features a local analysis for regularized two-layer neural networks, 2024. https:////aax http://awxxvvvv..aa.ea. wttg/abs/240..... 01
M. Zhou, R. Ge, and C. Jin. A local convergence theory for mildly over-parameterized two-layer neural network. In Conference on Learning Theory (COLT), 2021.
13

---

Published as a conference paper at ICLR 2025
APPENDIX INTRODUCTION
The Appendix is organized as follows:
In Appendix A
we provide a comprehensive introduction to the notations used in this paper.
In Appendix B
we discuss the selection of hyperparameters in this paper.
Appendix C
we provide some additional numerical results.
In Appendix D
provide detailed explanation of the closed-form expression for the loss and its gradient as mentioned in the main text.
In Appendix E
we provide detailed analysis of Assumption
In Appendix
we present the main results of phase along with detailed proofs.
In Appendix G
we present the main results of phase 2 along with detailed proofs.
Appendix H
we present the main results of phase along with detailed proofs.
A
SYMBOLS AND NOTATION
Table 2: Core symbols and notations used in this paper.
Symbol
Dimension(s)
Definition
Num [n]]
Gaussian distribution with mean / and variance o²
Shorthand for (1, :
O, o. S2
Bachmann-Landau asymptotic notation
| 2
Euclidean norm of vector
. la
Spectral norm of matrix M Frobenius of matrix M
M F
v.
Rd
Weight vector of the i-th teacher neuron Weight vector the i-th student neuron Dot product of vectors u and
w.
Rdd
(u, v)
T, Ti,
Total training time durations of different phases
a
IRd
Normalized vector: ā Ta
hag
Projection of w; in the direction of uz
(())
ReLU activation function
V.
Rd
Gradient of loss with respect to w:
Index of teacher neuron closest to w, at initialization
Ci
Angular offset between and nearest teacher neuron Angular offset between w, and other teacher neurons Residual the network output
C
R(x)
a))
my
Number student neurons close to teacher neuron v
Ty
Rd
Difference between and sum of nearby student neurons Angle between and w;
0g
Angle between w, and wy
L(W)
Loss function: L(W) _ R(x)²
d
Input dimension of the data Number of student neurons Number of teacher neurons Learning rate for gradient descent Initialization variance of student neurons
m
k
1)
B SELECTION OF HYPER-PARAMENN PARAMETERS
We set 62 o(m 600 - 100) = (pol((1)) in Theorem as required by Theorem 5
We set of ol /m) o(poly(e2)) in Theorem required by Theorem 4
14

---

Published as a conference paper at ICLR 2025
20
(a) 07 1e 4.7 = 5e (b) 5 le 4,7 5e- (c) 07 1e 7,7 5c 5c-4(d) le 7, 17
(c) 0 16-5- 5,7 5e - (f) 0 le -0.55 = 5e (g) 0 le 8, (e-4(h) 0 = le 1e-8, 8, 5e -4
spocts
(1) 0 le 6, 5e (() o le 6, 5e 4 (k) σ = 1e-6..6600 1e-3 1e- 1e
Figure Convergence curves for different a and r7.
18k+2 |ill
ol poly(e vid
We set a
in Theorem 3 as required by Theorems and
10000mm/
We set
mefaj
f
32846
(poly(cf)) in Theorem 3 as required by Theo
|
mk
rem
We set Ti =
( in Theorem 3
100gkm
We set T2 = Ti +
k
0 1) in Theorem
21ym
48mez
C ADDITIONAL NUMERICAL VALIDATION
In this section, we selected four sets parameters a and ry: 10 5 x 10-4, and m = 20. The plots in Fig. 2 show the cosine of the angle and norm convergence during training (top row) and the log-log plot of the loss during training (bottom row) for different values of a and 7).
10 5, 77 5 x 10
10 10-7, 5 x 10 and 10 10-8, - 10 with k = 4
The results demonstrate that smaller initialization variances are crucial for ensuring the training dynamics in the first two phases align with theoretical predictions, thereby facilitating faster and more predictable convergence in the third phase. In contrast, larger initialization variances, while potentially shortening the duration of the first end of the second phase, leading to slower convergence rates the third phase. This underscores the importance of adopting smaller initialization variances and stepsizes maintain stable training dynamics across all phases and achieve consistency with the predicted 7-33 convergence rate.
phases, disrupt the balance of the student neuron norms by the
D EXPRESSION OF LOSS L AND ITS GRADIENT V
In this section, we provide a detailed explanation the closed-form expression for the loss and its gradient mentioned in the main text. The main content of this section follows (Safran and Shamir 2018, Section 4.1.1). Besides, the bounded norm of [w for any E [m]] is also given in this subsection. We include these results here just for completeness.
15

---

Published as a conference paper at ICLR 2025
For notational simplicity, introduce the following auxiliary function:
g(a. b) B. (o(a x))(bTz)) x)] ||a||
|b]|
sin L(a, b) (t Z(a,b)) cos <(a.))). (12)
2T
which implies that
* if a and b are orthogonal, i.e., (a, b) = 0, then ( b)
||a [|b]|
2m
If a then g(a, b) | | ||||| = 2 ||a|| |b]]2
Then we can derive that the gradient for g(a,b) b) w.r.t a as follows:
g( 0g(a, b) da
1 ( [Ibl]sin <aa, b) |
+ + (T,-)a,,))bb
(13)
2x
Using this auxiliary function, we can rewrite the loss function in Eq. (1) as the following form:
L(W)- 1 E (0.11
Σ
p(w)
x²)
(e at)
(14)
m
6 k g(e,, wn,)
m
k ge 1y))
g(u, 03)
i=1 1=1
1=1
Accordingly, when w; + O. for Vi E [rz]. the loss function is differentiable with gradient given by:
V, (L(W)
Ow;
m
Wg( w,)
0g (w w:)
dg(w,, vi)
Own
Ow,
W7
1
||w ||
sin so
wi
+ (TT
Psy psy)e2,
1
Σ
||0|||
sin Ogt |wwll]
+ (
02 011)01
2π
2T
1m1
k
k Oavi
U1
+
wi
sin
Pig
||w
sin On [|0|])
Pij
2
2
2T
||ess]
1=1
1
1=1
1=1
(15)
The bounded norm [| at initialization can be given as below.
Lemma 1 (Adapted from Lemma 3 in (Xu and Du 2023)). Let $1 := Lovdd 82 == 20/d, if ((log(m/0)) with probability at least
:= the Jollowing properties hold at the initialization: Vi [m].
81 - w. (0) < $2
Remark: This is a standard fact in high-dimensional statistics, and on this basis, our result only involves this randomness. In the rest of the analysis in this paper deterministic.
E DETAILED ANALYSIS FOR ASSUMPTION
Here prove the following lemma:
Lemma 2. When 52( (2( [og(mn//6)) with Ç o(poly(m k2 kk)))),, then with probability at least
8, the following property hold a the initialization:
2 - (2) < +
Vi E [m].j E [k].
16

---

Published as a conference paper at ICLR 2025
Proof. According Lemma
have for Vi [m].j [k]], we have:
(w, 0,) ooan [,)( 2 jová cos 0c)(0)
2 550
0:j(0)
S
+ 5.
4
By concentration incquality of Gaussian, we have:
4 Şova) S 2eep
(- (Sova)2
8
P (0,,
2
3mk
Then:
IP 0
π + +Vvv - π CVj [k]]
*
+
5
3mk
3m
3m
Applying the union bound for vi [m], which finishes the proof.
GLOBAL CONVERGENCE: PHASE 1 (ALIGNMENT)
In Phase 1, we are interested in the dynamics of 0,- well as the angle difference between the student neuron and its closest teacher neuron. The theorem we prove below is combination of Theorem. 3)
and Coorollry ffrm the main text.
Theorem 6 (Phase I: Alignment). Assume d = 2(log( ( with 8 (0, 1), for any G1 0, under
for
Assumption 1 with 10kς S e ol{?) and Assumptions such that < <2uk-2 [|ll|
10000myd in random
Gaussian initialization, and the stepsize satisfies 1 S
100k" Toll then by setting Ti = 100ykm for
t Tú, the following statements hold with probability at least 8:: 8:
si S [t)ll))) S $2 + 2kn 11 Vi [m],
with $1 Lova,
82 = :0v= (16)
and
sin? 0+- (4)(t) (t)
oे <
nk |001(]]
- -t sin"
0. (0)
at)
vi [m]. (17)
82
2
After Phase 1, we have:
0,0 ((i) S 4e1,
Vi [m].
(18)
and
h4 (T.) S 2h (Ti),
Vi E [m]
(19)
Proof. The proof is given by induction. We firstly prove and (19)
Eqs. (16) and (17) and then Eqs. (18)
At the initialization time 0. Eq. (16) and Eq. (17) directly hold according to Lemma Note
that the probability in this work only relates to the random initialization as given by Lemma For
description simplicity, we do include this probability during the derivation but just mention it in
our theorem.
first analyse the learning dynamics of 0... For any Vi [m] and 0 < t < Ti, according to the inductive hypothesis, we have:
Before proving Eqs. (16) and (17)
sin2 (0.. (t)
(4::00
0,. (0) 2 ci sin? 0g.
(0-(10)
4 2
< max
sin²
sin?
where the right part of the above inequality is given by the following fact with Assumption
sin 0r- (0-(
T
Si
2 = sin"
= + sin(C)) = 0(1) 2 = = o(1),
2
2
which means:
17

---

Published as a conference paper at ICLR 2025
04-(1) - Ci
(20)
Then assume Eqs. (16) and (17) hold for any 0 < t T1 to prove Eqs. (16) and (17) for + 1.
Proof of right part of Eq. (16)
According to the inductive hypothesis and 82 == 20 2
Lemma
we have:
16k+2
bl
of ||
|
||||]
|
[1:] S 82 + 2km || Ti
S
Vi E [m]
50m
50m
48m
3m
(21)
That means the teacher neuron's 's norm controls all of the student neurons' norm at E [0, Then by triangle inequality and Eq. (21) the gradient norm be upper bounded by
(1)l
my wny(1) (1)
+
1=1
(t)
m
sin (ty(t) (ey(t) Σ sin az [|
Σ
m
p)((t + Σ eutt))n(t)
2m
[t)ll
jm 1.j
||0|||
k ||a||
1
wll
|ill 3m
+ k
||l|||
mm
+ ko ||a||
3m
2
3m
<2
|0ll,
Vi E [m].
(22)
One can see that, the gradient norm is upper bounded by all of the teacher neuron's norm. Accordingly, based on the gradient iteration, by the above results, we have:
[t, (t + 1)ll ([w,(t) (t)]l (t)l + [n)]( s 2km |/// (t 1), Vi E [m]
(23)
which concludes the proof.
Proof of left part of. (16)
Here need to prove the lower bound, we have:
|| + 1)1l | ||w= 81,
Vi [m]
According to the gradient iteration:
||w + -lw, [t)] (t) V)V. -1w -2n (w, v.(t)) [t) Vi e [m].
we only need to prove E [m], (t), Vt(t) < 0. To be specific, we split (w, (t), V.(t)) into two parts:
18

---

Published as a conference paper at ICLR 2025
(t)(t), vi(())
wx((),
1 m
Σ
(ty(t)
1
Σ
12
2
2
j=1
1=1
&
1
wx(()
(f),
sin spir (f) ||00(()]1 -
Σ sin (t) ||0||
Σ
Pi (t)en (t) +
Dn (t(()
2T
Iw (t)l
J=1
1=1
W (t), (t))
(t)((t), vi)
2
==1
1
(w, (t)l
Σ
sin y (t) [|e2,
1
Σ
sin (t) |ol|
2T
2n
1
Pty(t) (w, (t), w,(())
(t) (x))
2T
2m
j=
Liii
1
= [[2x;(t)] (t)l
ww (t)l cos (Pt(tt)
||a cos (t)
|my (t)) sn (p.y(t) Σ
||aa| sin (t)
j= Lssi
1=1
& ||l|| cos (xt())
[w,(()] cos pir(())) +
Ø=1
k
1
(t)]l
||س|||
Σ
- cos (x()) - sin (t) + cos cos (t(())(((tt
2n
lm1
1 (l20; (4)ll
+ 2m
Σ
cos + sin (p((() cos pi)(())))))))))))
j=1
where the last equality holds by including the additional term related to pa = 0 for any i E [m].
One hand, for 4i, by Eq. (17) L is a monotonically increase function of (z(() on the interval [0, m].
Then by Eq. (20). we have (t( < (t) + + < - Gi, which implies that:
n cos (t) sin (t) + cos (t)0
S - x coS(TT ( - (1) - -inn - (.) + cos(T π - (1)( - (.)
cos((i) - sin({i)
53
4
c2. is always true on the where the last inequality holds by the fact that Ç; cos(Ci sin((i) < - interval [0, 7..
On the other hand, to estimate 7, recall [|w)((1|| < Iull o(1) in Eq. (21) and the fact [n S we have:
1
w
(t)l
Σ
D
(0,(t))
1
/w,(()]
Σ
m
Ne, (+)l [7
1 ww, [t)]
2tt
2
96
1=1
19

---

Published as a conference paper at ICLR 2025
Accordingly, combining the above derivation over I and I, we have:
[l,,(()) |elll (- ACy + 0(()) s0,
V.())) s 2n
due to oî 1000( o((), we conclude that [| + 1)ll 2 [[))(()] 2 8 and finish the proof for Eq. (16)
Proof of Eq. (17)
We analyze the learning dynamics of cos 0,. by splitting it into two parts (first-order term and the second-order term) as follows:
cos 0z. (L+1) cos 0:- (1)
(w:(t + 1), Vrr) (w;(t), vrd)
[w,(t + 1)1l || [[) |||||
[t)] (t + 1), vn.) [t: + 1)ll (t), v,,)
|1u0;(t + 1)ll [w: |||||
[t)]l (t) v)), v.,) ||,, (t++ 1)ll (t), vn
1w,, 1) [w, (+) ||l||
(1)x((t)l [ 1)ll) (t),
I|,, (t) (t), v,
1w + 1)1l (+)]l ||||||
[1)] || (f (t))(t), (t), v.,) (t)]l (t),, vn
[[.. (w.
(24)
I00 (1 + 1)ll (1) |||||
(t).
([V. (un; (4), vx.) - uw [()] (V+(1), vr.)
[e](]] + w. (e- 1)
[t + 1)ll (()l ||
[1 + 1)ll (1), ū. (t) U.,, Vz(1))
17
17 (t), vr.)
(t. (t)) (t)) || ww, - 11.
[w: ( + 1)ll
ww, + 1)]l + || (f)l
One can see that we need to estimate the respective two parts Ig and Ig. For term Iz, note that (1));()) (r.) (i:()) Ur,, (t)) 0, then we have:
20

---

Published as a conference paper at ICLR 2025
7 12 (1 + 1)ll ((w)(t), Ur,) (.) - Vi(())
[1,e((t + 1)ll ((100(()) Br,) (1) .e 2 j=1
1 m
1 k
7
(t)
Σ
11
2
1=1
+ +(1 (wo,(1), o.,)) (t)
1
Σ 0z (t) 1))
-ree 2n
Pis(t)ws(t) +
j=1,jıi
11 1)ll
17
{ (10,(1), 0r)) ():() - UyF, Σ w,(()
2
1
(t):(),, (r,) w..(1 - -0;;
11
j=l
1=1
(wo:(1), )r,) (0:()) - Drp
Σ
Pigl (t)(()))(1))
+
1
(10:(),, or,) (0:(1)) - -Or Σ
Ozg 025(1)oz) ))
2T
2nT
Ø=1,fşi
1=1
"
(( (cos (t) - Ür. Σ
m
(T (is((t)
cos 0z- (t)(0((())
T..
> (t (==-Bu(t))e ((=-B(())ee))
k
2n 12,,(t + 1)ll
jm
1m1
17 2 1|,, (t + 1)ll
cos 8 (t),, Σ ( po)(t))20,(t)
Ur
( (t)((t))
j=1
cos 8,. (t)
x - (a(t))vi Σ
+
Ür:: Σ
(T - Ba(())er
1=1
(=1
1
( [t)r (IT (in(t)) cos 0;- (t) cos (P:((t) - -E
(1wwy(t)) (IT - (iy(t)) cos 6 (t)
2x 1w, (t (t+ + 1)ll
j=1
(|//| (mr - (a)))) cos 0:- (t) cos Oz (1) + || sin 0.- - -(-(()((=(0-((1)) (-)
1-1.
n ||||||
sin? 0: (1) - (1))
Σ
Oz())) cos 0:. (1) cos 0n(()) 00
12 ei
Eq. (21)
2T lww (t + 1)ll
1=1,
17 ||||||
π
sin? 0;. (f) - -
π 12
2z ||w,(( + 1)ll
2
17 |||||||
17 2
sin 2 0:0 (t)
4 (t + 1)ll
30
(25)
which builds the connection between T2 and sin? 0,- (t). For term Ty:
I3 17 (t), v
(t). (t)) (I [[, (t)ll ||, + 1)I) - (V))
[t, + 1)ll
ww,(t + 1)]l + (wo
(()))
[using Eq. (23)
|/w:(( + 1)ll
1uo + +|
+ ||
[using Eq. (16)
(26)
||wy(t + 1)ll
2s1
172 || (t))
81 [1 + 1)ll
y ||/l|
Jusing Eq. (22)
31 [1: (+++) 1)ll
21

---

Published as a conference paper at ICLR 2025
Take Eg. (25) and Eq. (26)
into Eq. (24)
have:
cos (t+ cos (t) = 12 + 11
17 ||001
17
16 ||0||
sin (,.
c
(t ( 1)ll
30
$1
1) ||0|
133
sin 0₁ (1)
150 c
(t 1)ll
17 ||0|||
sin² 8,. (t) c
[t 1)ll
Accordingly, we transform the dynamics analysis on 0z- from cos to sin, which allows for estimat- ing Eq. (17) as below. Recall cos 2z = 1 2 sin x, the above inequality implies:
sin² (t-(t))
(t-(t+1)
sin?
2
2
cos 0.. (t + 1)
cos 02+ (t)
2
?) ||||||
sin 2 0g. (t) - 2î)
(27)
8 l|ww + 1)ll
17 ||e:1|
4sin² 0..
(.-(t))
2 cos²
o7
8 1/ + 1)ll
2
") ||
0:. (t)
2 4 wu + 1)ll
sin
- 27)
2
which implies:
sin? 00. (0..(t+1)
01-(t),
sin?
4 (1 + 1)ll
2
17 |||0|1
0r. (t) (z)
sin?
[using Eq. (16)
482 + 2nyh ||00||
(1 + 1))
2
6+1
17 /||w||
0:- (0)
sin"
- +
4(82 + 2np || 1))
2
/+2
17 ||
0, (0)
< exp
du
sin
- o
jusing 1 - x S e-
J==1
4(32 + 2mp/
|0||| u)
2
882 + 21ph ||00|| (t + 2)
0r (,)(0)
cxp
In
sin 2
ct
8k
$2 + 2np ||
2
nn
||
(t (t+1))
sin" 0:- (0)
(0)
<
1 +
2
82
2
Accordingly, we finish the proof of Eq. (17).
Proof of Eq. (18)
Let to I N, for any & [to, Ti], using Eq. (17) and definitions of 82, a, have:
22

---

Published as a conference paper at ICLR 2025
(t)
1 nk
-
((sin
(6-(0))
sin?
sin
$2
2
nk
||0||
+
$2
1 ||e tu
56.
82
vp | || Ti
100g vd
||/|| cf
10000mo
That means: sin?
(1)
< 2cq. So Vt E [T) and Vi E [m]] we have (t-(t) S 4e1-
Consequently, each student neuron has aligned to teacher neuron by the end phase 1.
Proof of Eq. (19) For any t [50, Ti], we study the dynamics of hę- (i.e,, the inner product between the projection of gradient and teacher neuron) admitting the following formulation:
h (t 1) h4- (t) = (t + 1), úr,) - -w, )y:)
(V), o..))
wy wt(() (t) Uvs Tr.
2
k:
17
w
sin (t) (122,(1()) (t)ll Σ
sin (t [|-l]) - Σ
(t(t)ey)()) + 0: (t)u,,, Br.
2m
|12,((1)
j=
||a||
hym (t)
2
/=1
cos f:: (t)(
Σ
sin (ty (t)l Σ sin Ou(t)
Σ
(t) + 0, (t) ||0||
2m
j=l
j=1jjti
(28)
To analyse this dynamies, need study the sin (l( at first. According to Assumption 2 have:
(0.())5aa0)
x +1) Vi [m], + Le [k] 2
2
So we have:
(t) < (x) < (t), vi [m], Ta # Le [k].
2
That is:
1 2 sin (1 = cos
-aa(t)
2 cos 0g- (1) 2 cos(461) 2 -8?, Vi [m], # LE [k].
2
23

---

Published as a conference paper at ICLR 2025
Then taking back to Eq. (28)
have:
he (t 11 1144 =
||||
cos 0s. (1)
Σ sin (t) [[el])
Σ
2x
j=1,jái
|||||
+
sin (t( || +
Σ
(t)(t)
2w (t(()]]] Jusing cos 0,. (t)
2m
5=1
of ||l||
|||||
+
k ||
+ (m 1) 48m
[using Eq. (21) and (Pi < T]
2
2x
0.5
||aa||
Vi E [m].
2x
Similarly, we can derive that:
k + π 1.5
has (1 + (1+
? |101
Vi [m].
2x
Then, we accumulate over the time:
49 1.5) 1.5)
49 π 0.5)
TfT || S h4s (Ti) h4:
n/2 |||||
Vi E [m]] (29)
100T
100T
The remaining thing left is to bound h4- (5)
Ti
Ti
k
S S2
2km
||0|||
I
"T
|||
Vi [m].
(30)
50
50
50
20"
Combine Eqs. (29)am and (30) we have:
49k + 49 5
73.5
49k + 49x + 5
-24.5
1/Th ||||| S hos (Ti) <
1/2i
|||||
Vi e [m]
100m
100x
Hence we finish the proof of Eq. (19)
G GLOBAL CONVERGENCE: PHASE (BEHAVIORS ON THE TANGENTIAL GROWTH)
In Phase 2, we are interested in the dynamics of h well as the tangential difference between the student neuron and its closest teacher neuron.
G.1 GLOBAL CONVERGENCE: PIASE 2 (TANGENTIAL GROWTII PROCESS)
In this section, we will restate and prove Theorem
Theorem 7 (Phase Tangential Growth, restate version of Theorem 4). Assume d (m (m/5)) ())
with 8 E (0, 1), for any > 0,e2 O. under Assumption with 10k < o = ={(?) olez ((k) /m),
<10t=2 wll
o(1), Assumptions 2
3 such that < 10000myd in our random Gaussian initial-
me o2
ovac?
2
ization, and the stepsize satisfies n
<
I00k" then by setting Ti == T00y and
Tr = Ti + 2pm 1 [k), the following statements hold with probability at least 6:
& In (1bte2)) then VT: S t < Tz. we define H((t) = ||l||| - Em Ir (t) for
24

---

Published as a conference paper at ICLR 2025
h4 (t) s 2hg- (t), Vi, E [m]] and TÜ Tj.
(32)
mp
t-T1
|| +8TE2 ||a||| 2 Hi(t) 2 ||a|| (1- (1-3mm
2
37m
(-T,
1
STEez
||0|||
2 24TE2
||l|,
E
[k]
9k
3
2k
(33)
2 |||0||
h4s h2-(t)) 2 vi [m],
(34)
m4.
and
01-(1) S vi E [m].
(35)
Proof. We use induction to prove this theorem.
First, for / Ti, according Eq. (19) and Eq. (18) have Eq. (32) and Eq. (35) hold directly.
For Eq. (33) by Eq. (21)
we have:
m I h
2 ||o||, AI E [k].
| 2 H(iTi)
=||0/|
Σ
h4s (t) 2
(36)
For Eq. (34) for the left part, by Eq. (21) we have:
2 |||||
2|1|| my
has (Ti) < 10 (Ti)l <
vi [m],
and for the right part, by Eq. (18) and Lemma have:
he (Ti) (])x:((T)) cos 0.. (Ti) (1 - -8ef) [[):(T))]ll
vi E [m].
Next step, we assume Eqs. (32) to (35) hold for T,, T, + 1...., , for any Ti < < Tg. and then
prove Eqs. (32) (35) for
Proof of Eq. (32)
By Egq. (28)
for any [m). decompose the tangential difference h,, (1 1) h4- (t) below:
25

---

Published as a conference paper at ICLR 2025
h4 (t + 1) 1)- h1: (t)
m
71
||8||
Σ h (t)
2
7
cos (.. (t)
Σ
sin (c)(t) [()r(()|| Σ sin Bu(t) [|01))
Σ
Pyy(th (t) 0. (t) (t)
2m
/=1,
5=1
m
"
(t)
Σ
, n h (t)
2
17
cos tz (t)
Σ
Σ Iyy= sin (cy(t) (t)ll sin (t) ||0|||
2π
1=1,lft
m
7) cos ass0;.00 (t)
Ury= sin (ts(()) 1|0, (t)l sin 0,- (t) ||0|||
2n
m
77
Piy (1)(()) (1) 0-- (1)
2x
j= L,
m
77 (t)
1 pr. hyr(
+
7 2n
cos (t)
(sin (sin@((t)(((t)))
2
1=1,17
11 cos 0i (t)
Σ
T - 1w (t)ll
sin (1;()) cos 0z sin eu(()) (in()((((())
2x
1=1,157;
1=1
m
71 2T
cos fg. (t)
I I sin (t)(t) [()r sin 6.- (t) ||0|||
m
77
Σ
(1) +0) 0z
2T
j=m Lipi
" Hr, (t)
+
17
Σ Ht(t)
2
2n
([ [cos0> (1) sin Ou(l) -1 (1(1)) ))(()
7
Iryyi, byr (1) +
17
Σ
2
2xt
1=1,177
77
cos Pr (t)
Σ
Ly |w (t)ll sin (1)()))))) cas (t) sin Bu((s) ))
2T
1=1.1f
1=1
17
cos Oz- (1)
Uryerr sin (() [()] sin (t) ||0l||
2n
m
m
1)
T TR (1) (t) + Σ T 0.- (t) |||||]
2TT
"
Hr (1)
"
Σ +((t)+Q000
2T
1=1,
(37)
where the (!) is defined as:
26

---

Published as a conference paper at ICLR 2025
Σ ([ommbr- Oa(l)) Hi(())
&
Q:(1) = 2
Σ
Jrn (1) +
2TT
1=1
1=1,477
77
cos Oz (1)
Σ
mr [|)(())]||
sin sis()))))) cas (oss0)(()) sin 0a(())
2π
1-1,7r
17 cos cos (1)
I m. sin (i)(() (0, sin
sin 0:- (1) (1)
2m
17
T Pis (t) (t)
Σ (, (t) 0.- (t) [|e||]
2T
To bound Qi we need to estimate Pay and 0₂ at first. By Eq. (35) and Assumption 2 we have that for T, = and T; # l:
π
π (t) (t) s (ty(t) s T
222 -
+ 0 + 8 (t) -
+ 262
2
2
2
2
And for a similar reason, we have:
π
S
0n(()
π
E2,
and
- s (1)(() S E2
2
2
which implies that for sufficient small eg:
sin (t)(()) cos (t) sin (t) S ssn (ty(t) 1/ cos (xx()) sin (z)
2 (v((())
+ 1 cos 0z1(1) cos 2
0a(1))
cos
(1 cos 202) + (1 cos² E2)
< 26 +
3e3
Then using this result well as Eqs. (34) and (35) to bound [Q:(1)), for Vi [m)) have:
27

---

Published as a conference paper at ICLR 2025
m
2 || sin (3)
[Q,(0) < !
Σ
1, ft
+
(k- 1) 0,- (t) ||e||
2
m, cos (t)
2
j=1
+
"
cos vos0,-(t) (t)
Σ
1 -
2 ||
362
2T
m, cos 0z-
1= 1=1,d
77 cos (os0,.(t) (t)
m
I
2 ||l|| sin (t)
+
17
sin 28, (t)
||0||
2m
my, cos 0z. (t)
2m
j=1
7
m
2 | sin 0z. (t)
7
m
2 |0l
1] 0.. (1) ||/
( + 2c2)
2
+
2T
mr Cos Oz (1)
2n
r2
m r
2x
j=1
j=1
50
Tr Tt
| c2 + x5)
+
"
kez |aa||
mry
2π
)=1
m
2 2||0|| + +62)
7)
I Tr
362
2n
mr,
1=1,47
j=l
7
2 2110| 222(1
+ Eg)
11 262
||a|||
T
2T
Mr.
2π 2
j=1
77
I k
π
+ 2kE)
|1
c2(1 + c2)
+ 2m j=1
I tn 26²
2 |
+
"
||
2T
2
mr;
m,,
2T
1=1
$1.11k622
|| + n/ez ||,0|| + 2xpke3 ||
+ 0.7.pe2
||||
0.211e2 || + 0.6ryke2 || + 0.71je2 |///||
+ 0.2r/e2 ||
<4n/ee2 ||00||
1 11
H (1)
1
Σ nn())
3
2
2T
1=1,1
(38)
where the last inequality use Eq. (33)
Then Vi.j [m/] and T; = Tj. we have:
& Σ
h4 (t 11++1)hhs-f + "
H (t) +
"
Hi(t) QQ(t)
2
2π
1=1,/7
k
22h (1) +
7 (1,
+
1
Σ
H.(1)+ H000)+;001)
2
2x
|=1,,
22hy
which finishes the proof of [q. (32)
Proof of Eq. (33)
Then derive the dynamics of H())), for any I [k), we have:
28

---

Published as a conference paper at ICLR 2025
Hi(t)
Σ
h4 (t ((+1) (t))
1=1
= -i((t) Σ 1m
(t,(t)) (t)
Σ +
2π
j=1.jæ1
-1
mmn
(t(()
mm
Σ
H2(t)
Q1())
2
2T
For ease of description, write the recursive iteration in matrix form
H(t+1)= I
" (1a + (x - 1)n) H(t) + Q()).
2x
by defining the following quantities
((t)
ER*
Diag(m) (m) == Diag (m) my...., mx) E R
Q(t) : 1. -.=1 E.
L. RR
In the next, we aim to derive the upper and lower bound of H + 1). Denote A == (8rkey wll S 85kegl T Rt, according to Eq. (38) and Assumption 3 x xtk-1
we have:
H(I (11+1 1) A x
Diag(m) (11 + (π - 1)1) (() + 41jke2 || Diag(m) - A.
2m
(m) [11T (11 + (π - 1)1) 11))
(t)) - A
1 m
(T - 1)1 H(L)
2x 3k
7mm(T -11 (t))
6xk
Here & means that all elements the previous vector are smaller than the following vector. Then for I E [k]. we have:
T) 1)
1+1-7;
847K2 || (T1 +
Ht <
67k
k 1
rmm( - 1)
1+1-2;
S ||00|| + k
|| +
6T
(+1-2)
m/m
||
+ STE2 |||||
9k
Similarly, have
H(++))+A +A
I
1] 3m
111' + (TT 1)1)
[H()) +
2x k
I 3mm
[11 T +
(π (f-1)))) H(I)
2
29

---

Published as a conference paper at ICLR 2025
Here > means that all elements of the previous vector greater than the following vector. The
eigenvalues of matrix 3ym
(11 (π -)) calculated to be one 3pm(k+*- and the
2nk
2
rest l are 3pm(a-1 1) Then according to Eq. (36) for E [k], we have:
2
2 ||aa|
31mm +
+ (k 1) 1))
6+1-T.
Hi(++
8 ||2/]
3
2
2
3rmm
1+1-1,
||a||
STEC
||0||
3
2k
Based on the above results, for E [At], we have:
rpm
1+1-7, T1
|0||| + STC2 |la|| 2 Hi(t)
2 2 3
|1a|| (1-
3mm
141-71
STC2 ||
(39)
9k
2k
Due 7ym < have (1 =) 2 exp( 11.5ar) with mym. Using this fact, for any & T2. the last inequality can be further lower bounded by:
2
3rmm
t-Ti
||||
88TE2
3
2k
2
2ppm
In ( (48Te))) )) -8rez ||a|||
|0|| exp
3
2nm
24
||0|||
Proof Eq..
To prove the left part, by Eq. (33) we have: Hi(t + 1) = -|10|| - -mm - have:
2 0. Then
Mrr hos (t 1), vi E [m]]
| (++
For the right part, we have:
h4 (1 + 1 1) (1) = Hr (1) +
Σ
H₂(1) + Q((1) 2
n/k
24TE2
|1a||
-mpeez ||l|| 2 0.
2T
2T
1=1,
So we have h;s (1+1) 2 hy (1) 2 h (Ti) 2 +
Proof of Fq. (35)
First, we prove that for Vi.j E [m], Ti $ t T2. we have
O(1).
my(ti
When = Ti, according to Eq. (31) we have:
h4- (Ti)
<
22,
Vi.j [m],
(Ti)
which implies:
I (Ti)ll
his (t) cos 0,, (Ti) @(1), Vi,j E [m].
(40)
|20,, (T:)
hy- (t) cos 0z. (Ti)
Then by defining t,
9k ln(2) 17
+ Ti, when Ti t < t., according
Eq. (33)
for any I E [k],
have:
30

---

Published as a conference paper at ICLR 2025
2
3rm
t-
Hi(t) 2
||a|
85TE2
||1||
3
2k
21pm
9k In(2)
|0|| exp
8TTC2
||a||
3
mmm
2
18
IV
31
||
STee ||l|
So for 12 e [k). we have
Hy (4)
((1).
(1)
Then for Vi,j E [m], according to Eq. (37) for Ti S to < t, we have - (fa)
(1).
(oo (16)
Then consider Eq. (31) have k. Ay* (1)
((1).
That means for Vi, E [m]) when Ti I t S ts. we have:
(t)(t)]
h (t) cos 0;- (t)
O().
(41)
(8)l
hy. (t) cos O (t)
When 1, St < T2, according to Eq. (33)
for vA E [X:]. we have:
t-
1/m
Hi(t) -
|| + STEE ||
9k:
17m 9k In(())
Sexp
||0|| + STEE ||0|||
[using (1 (1-x) x)) S exxp(-x), Ve 2 0
9k
1ym
||a||
||
Then we have:
1
I. /1 (t) = ||||| - H,({) 2 2
|aa||
8 ||a||
2
|
3
Then for Vi,j E [m]. we have:
m 1==
I.=n
m1 (1)
(t)
2m
||0>||
6m7
m -=n Ur,= hi- (t)
Gm
m,, hy- (t)
12m
h (t)
108
That means for Vi.j [m]] when to St $ T3. have:
(1)]l h (1) cos 0,. (1)
O(1).
(42)
|u (t)l hy- (t) cos 0 (t)
31

---

Published as a conference paper at ICLR 2025
So combine Eqs. (40)
(42) for Vi,j [m], when Ti St S T2, we have:
(1)l ha (1) 0, (1)
@(1)
(43)
w (t) (t) cos (t)
Then, analyze the change in angle, recall the dynamics of cos 0,- in Eq. (24) is given by:
cos 0os (t 1) cos 0,- (1) =: 12 13
For 1z. we have:
1
7
(100(1), (,.)) (0((1) - Dr.. V.(())
1ew,(t + 1)ll
m
"
Σ
[1wx(())]l (IT (oo cos cos f,r, (t))
2x 1las(t= + 1)ll
(1|al (Ir (t) cos 0,. (t) cos 0n((t) + ||]||| sin 0.- (t) 0,. (t))
1.
To bound 12, we need handle cos 0;- (t) cos sp (t) cos (s) (t) at first. For + Tj. without loss of generality, we assume that: ör, = 11,0, 0, ., E Rdd and ör = 10, 1, 0, ...... O]T E RRd. Let w [Wa,, W:2.-.... Wea]" T E IRd and wy [wy1, 1y2. ..... wjd] E Rdd, then have:
O0T
cos cos (f) cos (Pi((t) cos f.rt (t) = (w;, Ur.) (wi; w)g) (1. 07,))
=wa (1)
wa (a) - (1)(()
(t)
wa (12)) + Σ (n(()eu((t) wty((t)
-wa (1)
122 (12 Σ
wal wa(()w)((l) sin² 0,- (1)
+
(a(t)wa((t) sin 0: (t) | (t)
2 (w22(t)
(1(
sin? 0,- (t) (wy (t)
){
2 - (1
wa (1)2
( w3((1)2) | (z- (1)
-sin
(Cauchy-Schwarz inequality)
(wx2())u52(1) - sin 0:- (t) sin 0z- (t) sin? 0,- (1) (1))((1)
sin 0, (1) sin 0,- 2
For 7. Tj. without loss of generality, we assume that: ūr, =0r, = 01,0,0,.. ..... OJ Rd, Then,
let wn; (wa, W22.-.., Woa] Rdd and w, [Wis, 10y2,---, Wijd Rdd Then we have:
32

---

Published as a conference paper at ICLR 2025
cos Oz. (1) cos siy( (1) cos cos (1)
= (wi, Un.) (wi wz²) - (wz, or.)
(t)
wa(t)unz(() wy1 (t)
1=1
wi (t)w5 (t) + Σ walt ((ts(())())) up1 (t)
=w Σ (i((D)m(((1) - sin? 0.. (1)mz1(()
Σ wu(t)ao((t))
sin 0.r (t) cos 0jr, (t)
wi(())) $
w)((12) $ sin? 0.. [Cauchy-Schwarz inequality]
2 Σ
Σ
1=2
sin 0s- (t) sin 0z- (t) sin 0,- (t).
Then we have:
m
7
(1e00())) (or (t)( 0,. (t) cos (py() cos Ayt, (f)))
In
2x [w (t + 1)1l
Σ
(||/|l (or - (a)))) cos p.- (t) cos (t)) + ||l||| sin? 0:- -
0:- (t))
1=1,17
[w, (t) (2C + sin 0.. (t) sin 0z- (t)) + 1, [|2, sin 0,- (t)
ww:(t + 1)ll
+ (k 11) ||0||| Tς - ( - 8,- (t)) || sin? 0.- (t)
m
6kmg ||2/||
2 O'nssin Oz Σ
sin (t)
Eq. (43)
1w, (t + 1)ll
m Σ
12k7/C ||l|||
2 CC' 0z. (1)
sin 0z-
Eq. (34)
81
jml
(44)
In the next, we aim to bound I3. which requires the estimation of the gradient. Similar to Eq. (22)
have:
33

---

Published as a conference paper at ICLR 2025
VV(III
wy wny(1) (1)
+
2
1=1
(I)
Σ
m
sin pis(()
[10)(()]
Σ sin Oz (1) [1 (11l)))
Σ
())(1) + Σ moc0voo0)
2m
j=1,j
j=1j
171
9k
||e|
|0
9k
||ll
|| + mm
96 ||
wlll
km
||
m
2
m
m
<15k
||||
(45)
Combining with this result, we can derive the lower bound for 13:
Is = n (t), Un,) (t 1)ll
(t), V.())) (I) (f)ll [|,, (t 1)1))
1|,, + 1)1l + [t,(t)]l
[V.(()
(\
(t + 1)ll
$1
(46)
4n² (+)l
$f 900k x² ||01
Subsequently, need estimate the difference sin?
0,+11 2
sin? (+)(() for final
estimation for sin Oz.. Hence, similar to Eq. (27) combining Eq. (44) for Vi E [m]] we have:
0f. (t ( sin? Oi. (t)
sin²
1
cos Oz. (1
cos Pir (1)
2
1
(
12kr |01||
900x2r,² ||0| 2
<
O"nsin fz (t)
sin (t)
Jusing Eq. (44) and Eq. (46)
2
81
of
j=1
(t)
()-(1)
Gkag ||0|||
450k 17 |/011
<20"nsin
sin
2
81
Summing over all student neurons yields:
(g.(t (t+
(t)
sin
sin²
2
2
17
(1)
m
sin ()-(()
6kÇ1
|||||
25
||0 12
20 1) sin
2
81
$7
i=1
1=1
m
(g)(())
(1) 2
6kmςq
||
450 mn?
|00l
=20*n
sin
sin
+
+
2
$1
i=1
1=1
(t;((t)
6kmşn ||||||
450 mny |10l1
<20' 1777
sin?
+
Jusing AM-GM ineugality]
2
(47)
34

---

Published as a conference paper at ICLR 2025
Then we have:
81 (t+(++) 1)
sin
i=1
8, (t+ + 1)
3kg |00|
225 n |10||
sin
C*ts1
C*
1=1
m
0.- (t)
3kC ||l||
225k 1
||0133
5(1 + 20 mpm)
Σ
sin?
Eq. (47)
2
C181
C²
1=1
m
0,. (Ti)
3kC |
225kk21 |0/1 C*
<(1 + (mm
Σ
sin?
C*81
"meg 88
<(1 + nmm) -+1-Ts Ame, [by
Assumption
choosing C
1: ||
S exp
20* mm 21pm
In
48TE22 (1BTTe)
4mei
[using + x S [exp(x)]
4me?
(4
16
where the last inequality needs ef (48meg) (4*
61m
Finally we finish the proof for Eq. (35) i.c.
0: (1 + +1) E2
Vi E [m]
which finishes the proof.
G.2 GLOBAL CONVERGENCE: PHASE 2 (FINAL STATE)
Here prove the bounds the student neurons and the loss function the end of phase 2.
Lemma 3 (Final state of Phase 2, restate version of Corollary 2). Under the same conditions as Theorem at time T2, we have the following statements hold with probability least 8:
|0||| 3m,
||ll|| < m.
ww (T2)l
Vi [m],
and
((W( < k222
0.05 ||a||2
Proof. Firstly we derive the bound for the [w: (T>) By Eq. (33) in Theorem for any / [k]], we
have:
35

---

Published as a conference paper at ICLR 2025
T2-T1
7pm
(T2 -
|| + 8xT2 |||/]|
9k
Tpm
(T2 Ti) ||m|| + 8TE22 ||
S exp
9k
(48T)2)
is || 8 ||lill
s (49Te))
* ||0||
Σ ||0||
which implies
2
| S ||||| H7, (T2) = Σ
I h1 (T2) S 2m, h4s (72),
Vi 6
3
j=1
So we have the lower bound ||2)((2)]] 2 his (T2) 2
wil 3m-,"
For the upper bound, for any E [m].
we have HG, (T2) 2 0:
1 3 mrr (ww(TT2)
||a|| 2 (5- (T2) 2 2nh mr, his (T2) mr || cos 0;- (T²) 2
which implies l|w, (T2)l < v|ll| and the following estimation which is used for estimating the loss.
mr,
To be specific, for any E [k:]. we have:
h (T2) cos 0,. (T2)
Σ I (T2) Σ
S1 2 Σ I I. (T2) S (1+ ||a||
and
Σ -
(2, (T2)l
2
I h3s (T2) 2 (1 (1- (49TTcz) ||a||| 2 (1 (1-2200) |wa|
1=1
1=1
Combine the lower and upper bound, we have:
(1 - 0.05) || <
Σ T (1:
S (1++ +()) ||a|| ||o|]
(48)
1=1
Before we bound the loss, we need to analyze g(a, b) defined in Eq. (12) Li,, b) S 26a we have:
262 2x |all |b|| I g(a, b) ||a|| [bb||
( sin ((a (T ((a.b)) cos <(a,b)
1 ||a|l ||b||
2T
(49)
Besides, if 222 < = (a, b) 2eg, have:
1 4cz
1 462
|||| |||| S g(a, b) <
||||| |6||
(50)
2
2
According to Eq. (35) Theorem 7 then when Tj. have Pij 26g and when # Ty.
have -26 < Pij
<
262.
Then, according to Eqs. (48)
(50)
have:
36

---

Published as a conference paper at ICLR 2025
k
((W(
gw: (T2), (T2))
g( U₁)
gww; (T2);
2
i=
i= j=!
1
m
1 + 4e2
<
1
2
lo: (T2)ll 10, (T2) +
I rephi
||u,(T2)]] [[b)
2
2x
1=1
1=1
]=1
j=1
k
|0|
k-k
1)
|001
2
2
2
2T
Σ
Σ
T TT=1
2c2
(o;(T22) |102|
4e2
[w: |
2%
2T
1=1
1=1
m
m
m
1
7 Σ
I 1
[2 (T2)l I (T2)
+
I 7oT 2n
w (T2)l (
2
j=l
jml
k: b|
k:k 1) |01
1
1
Σ
1= 2
(T2)l
|,00||
I 2x
1|, (T2)
|0||
4T
k
E2
[ (T2) ||
1=1 1=1
2E2 Teyll
[T, ||02||
Eqs. (49) and (50)
π
A11 2 |0|
k(k 1)(1 + ) ||0||
ks ||
k:A 1) ||
4T
4x
k(1 - 2
|01|
k(k - 1)(1 0.05
||0||
2
2T
k(k 1)(1 +
- ||011 +
k(1 + e E2
|0| 2
2k(k
1)(1 + EZ)e2 1
Eq. (48)
$ 2 2,000 0.05
||0112
(51)
which concludes the proof.
H GLOBAL CONVERGENCE: PHASE (LOCAL CONVERGENCE)
In phase focus the local convergence of the network when the loss function has upper bound. First, we introduce some structural lemmas related to the loss function of neural network.
H.1 STRUCTURAL. LEMMAS
Lemma 4, We define that w = Lm
hy+ Dr. and 6max := max4e(m) 0s,, then we have:
m
0 L(W), w. - 202222(()) OKO2 max
&
||re| ||ol])
Owo
Proof. First, decomposes the residual function R(x)) into two terms:
37

---

Published as a conference paper at ICLR 2025
R(x) = Σ
O w σ:) Σ
21 x))
1=1
w x)0' (w x)
Σ
x) x)
[using ReLU property: Φ(z) xá xz)(
t=1
m
[w] a 2) (w) x²)
[w; - Ta) (w) œ:)
[using definition of ml]
1=1
i=1
k
(w. x))0 (' x)
I w;) ) x)) + > (n) x)) (w) x)
1=1
1 ( x)
o'(w, x) (((0T
Σ (rl w )
(2))
R2(
Then we can derive the lower bound for
m i==
8
L(W), w; w.
that:
Tw.
m
T.(W),
w0;; =0,"
Ow,
Enge
R(x))'(us x)x= (w: wx,
m R(x:))
ol (un ar)rr" m0?)
i=1
-2L(W) + EE
R(x)) Σ
m
(ww] x)) 06 (207 x) TooTuy no R(ær)
[using -(W)- - - 1. -1 1(x0)
('(w] x) wo ))]
-2L(W) + En
R(x))
pav) x)) Σ
[using definition of R(x)]
/b' (un x Too." ))
-2T(WW) + En
R(ar) Σ d'(x
)ar " Σ
[using definition of R(wx?)
-2L(W) + E.
R(x)) Σ
(x T wi²)
(((xxTw)) - p'w,,T xx)
1=1
:22(WW) + Ex
R1 Σ (x Tx,)
(0(xToo wi:) o' o' x))
1=1
L.
m
+ Eg
R2 (x
(x)
> x wx))
60(x (0(xTux)) (w())) - 66(w, x)
i=1
For term 1t, note that for Vi E [m], when w. x 2 O, we have d'ww, x)) = 1, which means
p'ww, x)) d'( x:) 2 0. Then we have (x: 2 0. Similar, we have Σm (x Too)) d' T ww))
p'ww, 2²) 2 0. So we have 1 2 0.
38

---

Published as a conference paper at ICLR 2025
For term Is. we have:
m
15 Ez
( x) (1) x) Σ (o T wii) ww;T)w))
wi) Ø (ww?? x))
k
Ean (r, x) (m) (x wi)) (axTwii)
('(x ( w[;) o (w) xx)
[=1 1=1
O 0g. ||,]])
where the last inequality is from the proof of Xu and Du (2023
Lemma 8).
Thus have:
L(W), w; -wi) -2L(W) + +
Ow;
22L(W) Σ> m
O 6%. |
2 2L(WW) - OkoP
Mrl [wll),
max
which finishes the proof.
Lemma 5 (Bounds of 0,, and |r]]). Given that then we have:
||l|| 3m..
wwlll S S
l||l| myy
and L(W) o(1|0 10))),
||/l s Ok | (W)),
EE [4]
(52)
00. 1 O(K+
Vi E [m].
(53)
Proof. The proof technique here heavily depends on (Zhou et al.
2021 1, so we simplify our proof
here. To be specific, using the same proof method as (Zou al. 202 Lemma C.6), we have:
m
we 1| 03. = =(LL
Similarly, following (Zou et al. 2021 Lemma 12), we have:
Eg Ry (zx) o (++ ||e| L (W))
Based on Zhou et al. (2021, Lemma 11), we can derive that:
Eg 22(x)) = 22 |ra]
VI E [k]]
Combine the previous results, for any 1 [Rs), the upper bound of ||m|| is:
39

---

Published as a conference paper at ICLR 2025
|ril OBB
4-
S 0(E R(x)) + (()))
0 Lb (W) k ||aa| 1.3 (W)
s s0 k |laa| (w))
[using ((W) - O ||]|
Accordingly, we finish the proof of
Eq. (52)
Based on this, using the same proof method as Zhou
al. 2021 Lemma 9), directly obtain
Eq. (53)
Lemma 6 (Bound of l|o, w ID) Given that for Vi E [m]. we have:
||lll 3m
I ||w:1
3llll and L(W) o( [|v|" ) then mn.
ke m 3mmLL33() [w wfll o
||w:ll
||0||
Proof. By Lemma we have 0,. o
kL (W)
(w)
and HH = 1(r, 0tح1 = ||r|l|
||w||
0 |ll| & (W)) ((()==])) Then we have:
Iw; will || S ||w; h1s h.-00.11 + || Dr, w
|
l|w; hus o..
+ h4s 1
2== Br by 14:
|| has |Hs
= || 0:- +
jusing definition of Hi)
Hi
|weel O #
|
1 (W))
kL (W)
<
[w: O
|00||
kL
W)
|| + o k L (W)
||mm||
||a||
|
AL (W)
||0|1 k2 )]
0
||w:|]]
[using L(W) 0
||01|
H.2 GRADIENT LOWER BOUND
In this subsection, we use the structural lemmas Appendix H.T to derive the local gradient lower bound.
Theorem 8. Given that |||||
< || S 3|ll| for Vi [m] and L(W) of 1 then we have:
Sme,
(L(W) dW
3 W
> Ω
122 ||01
Proof.
According Lemmas 4 and 5
have:
40

---

Published as a conference paper at ICLR 2025
0 L(W), ) w0 wi 2 2L(W) O,k max
rill [[ol))
Omn
KL (W)
2 2L(W) O
2- 2 ||ll| L (W)) ||a/]
|w||
si (W)
2 2L(W) o
||0|||
|| 1
2 L(W)
[using L(W) 0
2:12
Then according to Lemma 6, we have:
m
δ
L(W)
L(W).
wi w
Ow;
m
L(W)
|m w
Or.
((W) 00
1: (W)
|w
ow
||aa|
2 ( (W)
|1/||
(L(W))
oW
which concludes the proof.
H.3 LOCAL CONDITIONAL SMOOTHNESS OF LOsS
In this subsection, we deal with the non-smoothness of L. We will prove the smoothness of when the student neuron has upper and lower bounds
Lemma (Local Conditional Smoothness of L). Given that 5mm,, define the Hessian matrix of as A 8 L(W, 2W2 then we have || I O(m)).
|| < || |0
S 5| for any i E [m]]
mr,
Proof. According to Safran et al. (2021). we have that L is twice differentiable and the closed-form
expression of Hessian A LLL(W)
8W2 E mmddxmd can be write as:
A1,1
A1,
A
Am,1
A1.
where Ar Rddd Vi,j E [m]] we will discuss below.
For diagonal elements:
m
A I + Σ Aw, wy) Σ (w vi), Wi [m],
2 j=1,jæi
and by defining ne
8 cos L(w, v w. A can be rewritten as:
sin < v) ||0|| wuw + Dew TTo
A (w, v)
2T [w]
41

---

Published as a conference paper at ICLR 2025
We can bound that
||||
| e)ll
w
Then have:
A < 1/11 Σ
|| (wo, (w,, w;)1l
A (wi, vt)l
j=1,jæi
1=1
m
=0 + Ko
k
=mm), vi [m]
And non -diagoal elements satisfy that:
1 ( L(w;, [ + Aw., w, L._ + w.
00, ) Vi,j E [m]] and i # j.
Ar = 2π
So we have:
| || S 2m
( + 11, 1 Vi,j E [m], and i j.
Combining the above results, we have:
||| S
|A,i m(m 1) + m0(m) = O(m²).
H.4 GENERALIZATION ERROR BOUND
In this subsection, we prove the final convergence result, which is also the generalization error bound. aaad[Assumption
Theorem 9. Suppose the initial condition in Lemma I Hand Assumption 1[ 3and3hone holls. If we set 62 o(m 10mm-60-1000 and r o( 1 in Theorem 7 then VT E N, we have the following statements m hold with probability at least - 0:
1
L(W (T + +T)) <
(54)
3
L(W 2 k- |--||||113
1/("
and
||||
4 ||l0||
[w,(TT (T T2)
S
Vi E [m]
(55)
4m,
mn
Proof. We prove Eqs. (54) and (55) together inductively.
For = 0 Eq. (54) directly hold and by Lemma 3 we have Eq. (55) holds.
Then assume Eqs. (54) and (55) hold for 0, 1,..., .., for any t Ti to prove Eqs. (54) and (55)
for + 1.
Proof of Eq. (54)
42

---

Published as a conference paper at ICLR 2025
For Vi [m], similar to Eq. (45) we have Vv(tt)] O |e]]) Then for Ve [0. 11, we have:
|
(w;(t) (t)]l
[,)[t)] (.(+
n (|ll])
2
4m,
5m
and
4 ||/0|l
5|/0||
(t) mn (t)Il < (t)] + + ( s
+ 1/0 [ell -
my.
my,
Then, can use Lemma 7 for (()) V w (1) in the following proof.
For T2 1 L T + T2 1, according to the classic analysis of gradient descent in Nesterov al.
(2018), have:
1)) =
(VW)())
8L
(1 -1-11 w-)) OW? (W(I) (W)(()))
10
(LW(())-n/lV/ll) + [ (1-1)n) Vw O(m3)d (mm2d
Lemma
Then we have:
- ( / / (1- (t)l d
= =n/V (8)1 n (t)) O(m²)
2 5" wt)l1)
2 n1 (
Lemma 24). let C. S2 k (<=1|001=8)
According to Xu and Du (203
then we have:
1
LTW(T + +T)) S
3
L.- (W 2 h: 1-11|11 1|11 T/T
Proof of Eq. (55)
According to Lemma 3
we have L(W((2)) < 2.22 ||a|2
o( ||
Then for Vi [m). according to Xu and Du
2023
Lemma 24), have:
43

---

Published as a conference paper at ICLR 2025
T-1
(T.(T + Ta)ll 2 (ww;(T2)l
Σ
?
w + T2)1
1=0
||
||0||2
8C
3m+
kk3
|||||
80
h:
-
||aa|
3
3mr
3
|||||
||||||
3mr
|||||
4mhr
and
T-1 ww,(T + T2)ll s w0 (T2)
Σ
7 V w (t + T2)
||l||
80, 1/4
||8|| 2
mm
m 3,3
||0|||
880
k: -- ||
$
mm.
k
||ol||
k: |||l
o
mr.
4 ||l||
mr.
which finishes the proof.
44


Reviews:
[{"review_id": "xP6d86vZxB", "reviewer": "Reviewer_KUug", "summary": "The authors analyze the training dynamics of two-layer neural networks with ReLU activation in teacher-student settings, where both the teacher and student networks have multiple widths. Motivated by the analysis of (Xu and Du, 2023) for the teachers with single neurons, they provide a three-phase convergence framework, consisting of alignment, tangent growth, and local convergence,  to the training, and finally obtain the global convergence guarantee with $O(T^{-1/3})$ local convergence, where $T$ is the number of iteration.", "questions": "Besides the questions listed in the above, I am curious about the connection to [1], which also treats the three-stage convergence for regularized two-layer neural networks in the teacher-student settings.\n\n[1] Zhou, Mo, and Rong Ge. \"How Does Gradient Descent Learn Features--A Local Analysis for Regularized Two-Layer Neural Networks.\" arXiv preprint arXiv:2406.01766 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The teacher-student setting is one of the well-studied topics in deep learning theory literature, and treating teachers with multiple neurons is still lacking investigation. This paper tackles this critical problem and obtains certain results. The writing of this paper provides a detailed explanation of theoretical outcomes and their proofs, which makes the paper more accessible for readers to follow.", "weaknesses": "While this paper provides novel theoretical findings to the teacher-student settings literature, I have several concerns about them.\n\n- The first one is about the restriction of the teacher model. The authors impose several restrictions to the teacher model, such as orthogonality of each neuron and positivity of each coefficient of each neuron. Could the authors relax these assumptions? While the authors mention the orthogonality in the paper, is there any (possible) quantitative evaluation when the orthogonality does not hold? Moreover, I am curious about the accessibility to the case where both positive and negative teacher neurons exist.\n\n- The other one is the assumption of weak recovery, which the authors refer to in the conclusion. Although how the student neurons align to one of the teacher neurons is of interest, this assumption seems to impose this at initialization while the alignment phase still exists. Moreover, I could not find how $\\zeta$ in Assumption 1 can be small in the statements. Please correct me if there is anything I may have missed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhhiIAWhpu", "reviewer": "Reviewer_iXmg", "summary": "The paper presents a comprehensive analysis of training dynamics in a 2-layer  teacher-student model with ReLU activation and iid Gaussian inputs. The authors prove that, under reasonable assumptions, learning has three distinct phases:\n1. alignment - each student neuron aligns with one specific teacher neuron, and not too many students cluster around the same teacher neuron\n2. tangential growth - student neurons grow in norm \n3. final convergence - when all students neurons are sufficiently alinged with their respective teacher neurons, the loss converges at a rate of $T^{-3}$.", "questions": "- In Fig. 2-3, bottom rows, it seems that in **all cases** the long time behavior of the loss is significantly slower than $T^{-3}$. Is this not in contradiction to the analytical results?\n\n- The authors write in line 58 about sample complexity, though it seems none of the bounds depend on $n$.  Is sample complexity at all investigated in this work?\n\n- Is Assumption 3 (line 206) justified in a generic setting? It seems that if student neurons are initialized at random, the amount of student neurons that will be close to a given teacher neuron should be distributed binomially, and one should expect a small fraction of them to violate this assumption, no?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is overall well-written and the contribution is important, if true. I am not versed in the relevant literature and cannot attest for the validity of the proofs or derivations. I review the paper while accepting the claims of the authors in the main text at face value. The ACs should verify that the other reviewers can judge the content of the proofs.\n\nThe paper is very heavy on mathematical notation and all the \"juice\" is buried in the 30-page appendix. However, the authors provide intuitive informal explanations of the various theorems, which helps a lot in making the manuscript readable.", "weaknesses": "- As I wrote above, the notation is elaborate and difficult to follow, and all the actual scientific content of the paper is in the appendix. \n- The paper would benefit a lot from a paragraph or two, preferably accompanied by a diagram, that summarizes the main results, showing the 3 phases and the processes that occur in each phase, the bounds for the duration of each phase and so on. To save space, the current Fig. 1 can be safely omitted IMHO.\n- On the same note, it seems that some of the notation is introduced but never used (e.g. $r_j$ in line 152) and others is used only once", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJexu6JZIT", "reviewer": "Reviewer_FxhA", "summary": "The paper theoretically investigates a two-layer ReLU network in the teacher-student setup. The authors manage to derive a global convergence rate of $O(T^{-3})$ for a multi-neuron teacher and a multi-neuron student. The proof follows a three-phase structure, and the authors develop techniques to handle the interactions of multiple teacher and student neurons.", "questions": "1.\tIn phase 2, the authors claim that upper bound of the angle will increase, but this doesn’t necessarily mean that the angle will increase. Also, from the empirics, the angles appear to be monotonic. Will the angle actually increase as the authors state in line 337 that the angle is slightly larger than that of Phase 1?\n\n2.\tIn phase 3, is it possible to derive the convergence rates for the angle and the norm as well? If so, which factor dominates the overall convergence rate? Understanding this would provide deeper insight into the dynamics of this phase.\n\n3.\tIn the numerical experiments, how is the boundary between phase 1 and phase 2 determined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and clearly structured. The extension of previous results to a multiple teacher neurons case is a notable advancement towards understanding the learning dynamics of neural networks. The finding that GD balances the student neurons corresponding to the same teacher neuron also provides insights for the implicit minimum-norm bias in GD. The dynamical system analysis that handles the interactions of multiple teacher and student neurons is also an important theoretical contribution of the paper.", "weaknesses": "Major point:\n\n1.\tThe balancing results seems to reply heavily on the special initialization (a direct consequence from assumption 3 and lemma 1). The role of GD is mainly to preserve this balance throughout all three phases. While it’s interesting that GD can maintain the balance, the result feels somewhat limited due to the dependency on this initialization, making it seem more like a consequence of the setup than a profound discovery about GD itself.\n\n\nMinor point:\n\n1.\t$\\sigma$ is used for both the nonlinearity and the variance of initialization. It’s better to prevent notation overlap.\n\n2.\tThere appears to be a typo at line 189, where student neuron should likely refer to teacher neuron.\n\n3.\tIn theorem 3 (informal), $\\epsilon$ should be related to $\\zeta$ in assumption 1 but is not stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)