Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Authors: Hancheng Min
Zhihui Zhu
Rene Vidal
Venue: NeurIPS 2025 poster ()
Keywords: Neural Collapse
Gradient Flow
Implicit Bias
Two-layer ReLU Network
Primary area: optimization

Abstract: Among many mysteries behind the success of deep networks lies the exceptional discriminative power of their learned representations as manifested by the intriguing Neural Collapse (NC) phenomenon, where simple feature structures emerge at the last layer of a trained neural network. Prior works on the theoretical understandings of NC have focused on analyzing the optimization landscape of matrix-factorization-like problems by considering the last-layer features as unconstrained free optimization variables and showing that their global minima exhibit NC. In this paper, we show that gradient flow on a two-layer ReLU network for classifying orthogonally separable data provably exhibits NC, thereby advancing prior results in two ways: First, we relax the assumption of unconstrained features, showing the effect of data structure and nonlinear activations on NC characterizations. Second, we reveal the role of the implicit bias of the training dynamics in facilitating the emergence of NC.

Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
Hancheng Min"
Zhihui Zhu René Vidal!
"Inttitute of Natural Sciences & School of Mathematical Sciences, Shanghai Jiao Tong University Computer Science and Engineering, Ohio State University
Electrical and Systems Engineering Department of Radiology, University of Pennsylvania hanchmin@sjtu. edu. cn, zhu. ttu.34000osu.com vidalrgupenn. edu
Abstract
Among many mysteries behind the success of deep networks lies the exceptional discriminative power of their learned representations as manifested by the intrigu- ing Neural Collapse (NC) phenomenon, where simple feature structures emerge at the last layer of a trained neural network. Prior works on the theoretical un- derstandings of NC have focused on analyzing the optimization landscape of matrix catan--fattoizatio-liiiee problems by considering the last-layer features as uncon- strained free optimization variables and showing that their global minima exhibit NC. this paper, we show that gradient flow on two-layer ReLU network for classifying orthogonally separable data provably exhibits NC, thereby advancing prior results in two ways: First, we relax the assumption of unconstrained features, showing the effect of data structure and nonlinear activations on NC characteriza- tions. Second, we reveal the role of the implicit bias of the training dynamics facilitating the emergence of NC.
Introduction
Among many mysteries behind the success of deep learning lies the exceptional discriminative power of neural networks as manifested by
the intriguing Neural Collapse (NC) phenomenon [.] where simple feature structures emerge in the last layer of a trained network. The NC phenomenon is typically characterized by the following three properties (see the top plot in Figure I):
NC UFM
Intra-class variability collapse: The last-layer feature vectors of the data from the same class collapse into singleton;
2. Maximal separation of the class means: The class means, i.c., the mean feature vectors for each class, are maximally separated;
3. Self-duality: The classifier weights align with the class means.
Prior works the theoretical understandings of NC have focused on an- alyzing the optimization landscape of matrix factorization-lin problems by considering the last-layer features as unconstrained free optimization variables [ 9. showing their global minima exhibit NC. Extensions
NC in Two-layer ReLU
Last-layer features
Last-layer classifiers
of this so-called Unconstrained Feature Model (UFM) include adding nonlinearity and additional hidden layers 7 g 10 11, studying local
Figure 1: Visualization of the NC phenomenon.
convergence of gradient-based optimization algorithms around global
"Corrrepoonaacee Hancheng Min
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

minima 12 13. Until recently, Jacot et al. [14] show convergence towards NC in wide networks. Therefore, the question of how training neural network leads to NC has remained underexplored.
At the center of this problem lies the fact that practical neural networks are typically overparame- terized, i.e., their number of parameters is several orders of magnitude larger than the number of available training examples. As consequence, there are infinitely many parameter choices that can perfectly fit the data, and most critically, they do not necessarily correspond trained network that exhibits NC. However, NC is observed even for networks trained without explicit regularization [3] such as weight decay (often associated with the emergence of NC). This suggests a close relationship between the NC phenomenon and the implicit bias [15] of training algorithms.
Paper contributions. In this paper, investigate this relationship between NC and implicit bias by analyzing the dynamics of gradient flow (GF) on a two-layer ReL.U network for classification prob- lems, focusing on orthogonally separable data; i.e., any pair of input data with the same different labels is positively or negatively correlated, respectively. We make the following contributions:
1. In Section g| we present Theorem [ which shows that GF with small initialization provably converges to solutions that exhibit NC and provides precise NC characterizations for the trained network, as illustrated in the bottom plot of Figure [ Compared to prior works for the uncon- strained feature model, our results highlight the role of input data and ReLU nonlinearity in determining the NC characteristics. The former causes intra-class directional collapse instead of collapse to a singleton, i.e., the feature vectors of the data from the same class collapse into one-dimensional subspace. The latter leads to orthogonal class means, instead of maximally separated class means, and a projected self-duality where the classifier weights align with the projected class means (the projection is obtained by subtracting the global mean of all features).
In Section 4. through our proof of Theorem I for the case of binary classification, we explain how the implicit bias of GF facilitates the emergence of NC as training proceeds. Using results on the implicit bias of GF [1]. we show that in the early phase of training the neurons' directional alignment 26 with the training data makes inter-class features mutually orthogonal. Then, during the late phase of training, such inter-class separation subsequently promotes intra-class directional collapse due to the asymptotic max-margin bias (2 of GF.
3. In Section 4.2 in our proof sketch of Theorem for the case of multi-class classification, extend the aforementioned results on implicit bias for binary problems to multi-clas problems. We make technical contributions in addressing new challenges that arise in the dynamic analysis due to the multi-dimensional network output and the cross-entropy loss.
In summary, our work bridges the theoretical analysis of NC and implicit bias of GF by drawing explicit connections between the two. Moreover, we further advance the theoretical understandings for both by highlighting the role of input data and nonlinear activations in NC characterization and by addressing the challenges in analyzing the implicit bias of GF in multi-class problems.
Notations. We denote the Fuclidean norm of a vector by |||||| and its i-th entry by [u)]; denote
the inner product between vectors æ and y by (x, y) = x y and write x > 0 or x >> 0 if all the entries of æ are non-negative or positive, respectively. For an n X m matrix A, we let |||||] || denote the Frobenius norm of A. For a scalar-valued or matrix-valued function of time, F()), we let ai F() denote its time derivative. We define 1 to be the vector of all ones, whose dimension will be clear from the context. We let I, be the identity matrix of order n and sometimes drop the subscript if its order is clear from the context. We let [M] := {I,-.-, ,N} and let DD1 be the unit-sphere in RD.
2 Preliminaries
Orthogonally separable data. We consider a classification problem on a dataset {x, yi y of size n, where each data point x; E RD is associated with its label y; E Rdv, and the number of unique elements in {y: determines the number of classes K. Throughout this paper, we assume: Assumption 1 (Orthogonal separability). Any pair of data with the same (different) label(s) are positively (negatively) correlated, i.e., 30 < nt S 1 and 0 < <paS /K such that V1 S i, S n,
Me
y yj
|
m
||
Mad S
i/ + yg-
(1)
No dataset can satisfy orthogonal separability with pdd VK-I
2

---

Two-layer ReLU network. We are interested in solving this classification problem by training a width-h two-layer ReLU network f(: :)) RD - R parametrized by 0 := {W, V} e
RDxh RR xh with W [wi.... wo] and V = -Vi,,... .nl, and defined as
f(x; 0) = Vo (W = Σド= (as) x)),
(2)
where oT.)) maxq 0} is the dlement-wise ReLU activation function. We consider networks with width h 2 K; we call (wy, 0,) the j-th neuron pair in the network, w, its input neuron weight and vy
its output neuron weight. Moreover, we let Φ0(2) = [o x)), (( ..) . (((w,, x] Rh be the last-layer feature of æ. and V = 01, 12, -.. Un] E Rd denote the last-layer classifier. Note that we have considered bias-free ReLU networks; see the remark in the Appendix A for extending the results to networks with biases.
Gradient flow with small initialization. Given some L: Rdy x Rdy - Rg0 such that E(y 31)) (expressions shown in later sections) measures the discrepancy between the actual label y; and predicted label Dr. we let (()) = Σi=1 E(y,, fa 0)) be the loss function. We train the network via gradient flow (GF), which can be viewed as gradient descent with infinitesimal step size:
d 0 c -0o((0),
dt
where dg denotes the Clarke sub-differential
[2 operator. We study solutions (or trajectories)
(t), t 2 that satisfies (3) almost everywhere. We assume that the initialization (0) satisfies
Assumption 2 (e-small and balanced initialization). The initialization ( = { (w,(0) (0), (0)) satisfies the following: there exists initialization shape {woo, 1yo y=1 with w yo # 0, Vjj and an initialization scale > O such that Vj, wn (0) = =, (0) CUj., || |
Aside from the two assumptions, we will introduce additional assumptions different training scenarios when their respective settings become clear. For now, we shall remark on these two.
Remark 1. While the data assumption is strong, there are two main reasons for considering it: First, we investigate NC in shallow networks, whose single hidden layer has limited expressive power of collapsing features, thus one shall study more structured data, as also noted by Hong and Ling (23). Second, as it will become clearer in Section 4 the emergence of NC is closely related to the asymptotic convergence of the network weights, whose precise characterization is limited to cases with structurally simple data [ 20 24). Nonetheless, as we show in Section [S] simple real data satisfies orthogonal separability approximately, leading to NC characters that match our theorem.
Remark 2. Under the assumption of balanced initialization, we have [0)] I1, (0) I. Vj. and
this balance is maintained throughout the GF trajectories, i.e., [[,](()]] -l,,]]],, (0,() VL. Vj (25). This
assumption of balanced initialization has been common in prior works of this type [1S 20 24) for
the sake of tractable analysis. Our experiments in Section d not require balanced initialization.
3 Main result: Neural Collapse under GF on Two-layer ReLU Networks
Our main result shows that under small initialization, with some additional assumptions on the initialization shape, GF provably converges to neural collapse solutions on orthogonally separable data. Our results are presented for both binary classification and multi-class classification problems:
Case one: Binary classification:: We consider binary (K- = 2) data with scalar #1 labels, i.c. dy = 1 and y E {-1,+1) +1), Vi. Accordingly, the two-layer ReLU network f(x; 0) has scalar output ÿ. The loss function can be either the exponential loss f(y, g) = exp(-yg) or the logistic For this case, use plain font to suggest that label y and network output ÿ = f(x; 0) are scalars. Moreover, we define the index sets I+ == fi y: +1} and I_ : {i: 3/4 = 11} 1) for +1-class data respectively.
loss f(y, ú)= 2 log(1 cxp(-vt)
Case two: Multi-class classification: We consider multi-class (K >> 2) data with one-hot labels. i.c., dy = K. and y: E {ei, ek}. J. where e is the k-th column of the identity matrix Ik- Accordingly, the two-layer ReLU network has its output 9 = f(x; 0) E RK The loss function is the Cross-Entropy (CH) loss Eyy. Ū) EK_(v)) log Σド= exp((()))
exp(())=)v] Moreover, we define the
index sets I& == {i y: ex,, Vk E [K] for data from each class.
3

---

Main result. Our main theorem follows. Note that our theorem requires additional assumptions on the data and initialization shape that vary depending on the case, thus we feel it better introduce and explain them alongside technical discussions on the convergence analysis in later sections.
Theorem 1 (NC of GF on Two-layer ReLU Networks). Given orthogonally separable data (Assumption 7. e-small and balanced initialization (Assumption 2 for sufficiently small and some additional case-dependent assumptions on the data and initialization shape, the limit
== lim [|0(t]|F exists any solution O(t), 2 0 to G). Moreover, for the limit O = (W. we have: 3Φ2 E SSh-1,K E IC, where K. := {+.-) (case one) or K. == [K] (case two), such that
O(t)
1. (Intra-class directional collapse) The tustta of the training data satisfy that
(x(x;) (SW v.) 2R Vi E Ix. E,C, 8k =V 1E2 7ie
(4)
where "Ik = maX minGeIa (a,, u) k K. is the maximum margin achieved exclusively for class-k data and Uy is the corresponding max-margin direction;
(Orthogonal class means) The class means divections ΦR, k E K. satisfy that
ΦE 2 0, (Ōx, Φw) = O, vk, E
(5)
3. (Projected self-duality) The last-layer classifier satisfies that
V = 0_ 8 10__-ss__cca0 (case one) or V
K (T 1 11 [1101 " xKŌK] (case two).
(6)
Note that GF on positively homogeneous networks with classification losses drives the network weights to diverge to infinity (26) 27]. It suffices to study the properties of the asymptotic weight
e direction 0 since we have f(: 0) 0 fl: | due to the positive homogeneity of f w.r.t.
The following remarks compare the NC characterizations in Theorem with those prior works.
NC in two-layer ReLU. Theorem shows the following NC characters at the late stage of training:
(Intra-class directional collapse) Unlike previous works 2-8 which study unconstrained feature models and show that features collapse to class means with equal length, our work addresses more realistic and challenging setting involving input data. In this setting, the limited expressiveness of two-layer ReLU networks may prevent exact collapse to a singleton. Nevertheless, the result in (4) shows direction collapse in the sense that all data points in the k-th class Iş have their last- features و (z;) collapse into a one-dimensional subspace spanned by ΦR. though the features may have varying lengths. Consequently, the intra-class variability at the last layer is determined by the variability of projections U ax)) ). significant reduction compared to the variability of the original data {. Moreover, if the features are normalized to unit norm (e.g., by applying RMSnorm (8)), they collapse exactly to their corresponding class means, shedding light on the role of the normalization layer in the neural collapse phenomenon.
(Orthogonal class means) The result (5) suggests that the class-mean features are orthogonal to each other, forming a non-negative orthogonal frame when normalized. The orthogonal structure, rather than a simplex Equiangular Tight Frame (simplex ETF 3 arises because the features are always non-negative due ReI.U-but any orthogonal frame can be transformed into simplex ETF by removing its global mean. This finding aligns with results from the unconstrained features model using ReI.U as the activation [7].
(Projected self-duality) In the case of binary classification, the classifier V converges to 8 Φ.
s $-Q_, which yields maximum margin, as we will show in Section 4.1 For the case of multi- class classification, (6) implies that VV k-
[ 11 where $ = [s Ø1, ... (K Since Φ is diagonal matrix with positive diagonals, V forms a (scaled) simplex ETF, thereby achieving maximum margin. In particular, when the diagonal scales sk, k E K. are all equal, V becomes exact simplex ETF, and each classifier converges the corresponding projected class mean (projection is obtained by subtracting the global mean), up to a scaling achieving self-duality between features and classifiers weights
K (I R 111 Φ
K 'A K simplex ETF in R² a collection of points specified by columns of É V K- P(I -(1-1111))
where P g and P P 1.

---

Convergence of GF/GD to NC. Prior works on the convergence of gradient- gradient-based methods towards NC consider the mean squared loss 2 [2] Besides, additional conditions such as initialization close to global optimum 13 weight decay regularization 2
13 14], or large width [14] are needed. Compared with these works, we study the convergence under the cross-entropy loss without explicit regularization or of problems. Moreover, our results highlight the role of implicit bias of the training algorithm, which we shall discuss next.
showing that NC happens under broader class
4 Detailed Discussions: Connecting Neural Collapse with Implicit Bias of GF
4.1 Proof of Neural Collapse in Binary Classification
In this section, we provide a proof of Theorem for binary classification of orthogonally separable data. Recall that I+ and I_ denote the index sets for data with positive and negative labels, respectively. Let (t) := {j [h] (ign(u; = +1} and N- (t) := {j [h] sign(v, (t)
-1} denote the index set of neurons whose last-layer weights 2y (t) at time f has positive and negative
signs, respectively. Under Assumption 2 we have that sign( (1; (t)) sign(M, (0) I8 20 thus
(t) = N (t), = N (0), Vt, and we conveniently let N+ : : (0) and N :
Alignment phase of the GF. During the early phase of training, often referred to as alignment phase, several works 116188 20 24 29
30) have shown that the norm of the neuron weights, which
is initially of scale (e). remains small (of scale ((e2/2) for an extended period of time of length (log t). As result, one focuses on understanding the directional dynamics of the input neuron weights during this phase, which can be approximated as follows Vjj E [M]:
de Jam I sign(e;) (E E, O(e)) for some E,j E d,o(())
where w (I w,w" w/ /l P2) defines the projection onto the subspace orthogonal to wj. If one neglects the (e) term, the dynamics d w,
dt [| . E [h) are decoupled. The dynamic behavior of w, Tw
critically depends on the stationary points of which we shall address next.
The following discussions assume the (E)) error term is zero (only for the sake of the explanation here, the error terms are appropriately handled in the analyses). First of all, the directions || that render Sij- = 0, Vi E [n]] are trivial stationary points of (7). and they form the "dead region" for the neuron as all the activations to the data are zero (as its name suggest, neuron weights within dead region have zero gradient thus receive no update along GF). Next, the stationary points with some Ey * 0 are often called extremal vectors [16] 17 [2]], and the analyses in the work of Phuong and Lampert [7], Min et al. [20] have suggested that for binary orthogonally separable data the only extremal vectors are the class mean directions: #+ and w_.. More importantly, for neurons with j N x+ is an attractor, and æ_ is a repelle (the opposite for j E W_). Therefore, by following (7) the neurons weights with j E N either fall into the dead region, or converge in direction the average direction x of the positive class; while those with j E EN_ either fall into the dead region or converge to T_...
Transient analysis: inter-class separation via alignment dynamics of neurons. Based on the discussions above, the convergence analysis requires non-degenerate initialization shape { f such that 1) no input neuron weight is initialized to align with the repeller for (7). since moving away from the repeller can take long time that cannot be quantified; and 2) there must exist at least one neuron weight per class that is guaranteed to converge to the average direction of that class, avoiding the uninteresting case of all neuron weights entering the dead region.
Assumption 3 (Non-degenerate initialization). Let # T and x
I- where F4
EiEr. C; and x = EIEr_ at. The initialization shape (wo =1 satisfies that . + 0 and
max,EI+.jEN, Wo)) O, max;EI_.jEN_ (x,, Woo) 0; maxjeN, (m_, | < 1, maxjeN_ (0+,
(8)
wyo ||l 1.
(9)
'Roughly speaking, an attractor or a repeller is a stationary point that has the flow around its neighborhood pointing towards or against it, respectively.
5

---

(a) Illutraaiive Example
(b) Weight Initialization
(c) Alignment Phase
(d) Asymptotic Convergence
X Datn points Max mass-margin
Input weights Weight directions
directions
Figure 2: Convergence and implicit bias of GF in two-layer ReLU networks: (a) An example of orthogonally separable data (Assumption 1). gray region indicates the "dead region" for neuron weights: {w (w,,;) I 0, Vi E [n]]: (b) Weight initialization following Assumption 2 input neuron weights have small norm and random directions; (c) During alignment phase, inter-class separation is achieved through directional alignment between input neuron weights and data points, as described in (TOp: (d) Asymptotically, neuron weights diverge to infinity while their directions
align with the class-wise max-margin directions, as described in (2)
Given a non-degenerate initializationf] whenever the neuron weights converge to the vicinity of their respective attracting class averages, they stay close to the class averages for the rest of the training, leading to the following inter-class separation, due to the orthogonally separability of the data:
Claim (Inter-class separation via alignment, based the analyses from Phuong and Lampert [17] Min et al. (00). Given orthogonally separable data (Assumption[]) e-small, balanced and non non-degenerate
initialization (Assumptions 2 for a sufficiently small e, for any solution O(),, 2 to (3 77
and 0 + No S N+ and 0 + N SN- such that Vt Z T", we have
> 0, VIEI
I O, I > O,
W (1)a
W-(1) 24
Vi Vie and (t x4 S 0, Vi (10)
< 0, ViEI
where (t) = ر (t)(t))))
and (t) == (1) (NLUNN))) :- (w)))=
As a result, the inter-class separation (2:), )0(1(()))) - 0, ViE EI_ holds Vt 2 T".
We refer Figure 2 for an illustration of the weight-data alignment that achieves inter-class separation. As shown this claim, all neuron weights with index j outside VG, UN, will stay within the dead region after 7 and can be disregarded in the subsequent analysis. Therefore, without loss of generality, we assume until the end of Section 4.1 that N4 U = [h] and reorder the indices such that N4 = [[++ |] and AV= [h] []. To see that (10) indeed suggests inter-class separation characterized by last-layer features being mutually orthogonal, notice that Vt 2 T", we have
e (3:)) (Vigz.)
( [ (t)xx;) WI (f)=;] W (f)=:)
(METT+
[W (t) a [W (t)z;)
(0:c)(a;)
W (1)):
Asymptotic analysis: intra-class directional collapse and self-duality via max-margin bias. Now with the inter-class separation described in (IO). we are ready to study the asymptotic convergence of the weights through the lens of max-margin bias. In particular, notice that given inter-class separation (10), we rewrite the loss as (where we define V+ == [05(O]eN,, and V- == [0,(1) ):
C (0) Ei-, l (y:, ());0)) = -iEE, E(y., W.WTT2 2.) + Eier_ e(y,, V. z.).
(11)
This suggests that after T"* the GF on {W+, V-} is fully decoupled from that on {W-, V-J. allowing one study them separately. Moreover, each of the flows corresponds training two- layer linear network on positively correlated data with the same label, whose asymptotic convergence of the weight directions has been characterized in the work of Phuong and Lampert [17] mainly based on the analysis from Ji and Telgarsky [2I] on the max- max-margin bias of GF in linear networks.
To be precise, for the GF on losses of the form ΣiEZ, lly,, VW TW:.. Ji and Telgarsky 21 show that as time t + oc, both V and T diverge to infinity and their limiting directions exist. In
$ Min et al. (20) has shown that the non-degeneracy is satisfied with high probability when the input weight shapes are randomly initialized.

---

the case of ( this means 0 ( V+, W.. V-} V.) diverge to infinity, and lim;- [TaCe]
0(E)
= ( V+, W-, V-) Moreover, they show that the limiting directions satisfies the following alignment condition V+ W x u., the class-wise max- margin direction we have defined in Theorem
1 and balancedness condition V_V+ V+ = W W+ (similar for W-, V_, with 44 replaced by u.). It was first found in the work of Phuong and Lampert [17] that the only time these two conditions satisfied is when W has rank and the top left and right singular vectors align with V+ and u++ respectively. We show that this necessarily implies NC. The formal results are:
Claim (Directional collapse and self-duality via max-margin bias, based on the analyses from Phuong M(()
and Lampert [17). Ji and Telgarsky (2I). Given Assumptions & the limit f == lim, ToLL P
exists for any solution 0(1). 1 2 0 to B3. For the limiting direction O = { +1 V W V-}.
Egt E ESI-i-11 9 E 51W-1-1, such that
W+ u. gT. V+ S.9 W- = u- W-=s_ugg', V v.
(12)
where 84 and s_ defined in Theorem
7 As result, we have the intra-class directional collapse
(x.);))= =
(s. g+
= 2.+ 23))
g+
= (8+ u. mx) Tt, Vi E I
(13)
Φg(2.))
=
(s (s_ U- 2.)) 9-
(s_ us vii
u.
g-
= (s u. R.) क- Vi E I
(14)
and the self-duality between the last-layer classifier weight and the last-layer feuture
V
V+
0 0
V-]
s+
[ 0] s- s
[o
g s x+ s $
(15)
Note that the scaling factors 84, s_ are determined based on the results in the work of Lyu and Li 27 that the limiting 0 must satisfy another max-margin problem defined on the entire dataset.
Connecting NC with implicit bias of GF. In summary, we bridge the theoretical analysis of NC and the implicit bias of GF closer by showing how the latter facilitates the emergence of NC along GF. Notably, the inter-class separation is achieved by the directional alignment of neuron weights thanks to the small initialization scale. This resonates with a large amount of work 31 that identifies the small initialization as the active learning or feature learning regime, allowing simple weight and hidden feature structures to arise during the early phase of GF, which otherwise cannot be achieved if initialized in the so-called lazy regime 44] Then, we have shown how the asymptotic max- mar-mn bias promotes the intra-class directional collapse and self-duality after inter-class separation, where the key observation is that the max-margin bias often makes the weights asymptotically converge in direction to low-rank matrices, leading to -indimensional projections that significantly reduce the variability within the input data. We note that prior work [ studies the max- max-margin bias in UFM, while ours considers such bias in ReLU networks.
4.2 Proof Sketch of Neural Collapse in Multi-class Classification
As shown in the last section, the proof of the NC characterization for binary classification in Theorem T follows from existing results on the implicit bias of GF 17 20 21] However, to understand similar NC characterizations in the case of multi-class classification, one needs to extend the implicit bias analyses to multi-class problems, which prior work rarely does. In this section, we provide a proof sketch of Theorem for multi-class classification, emphasizing the additional challenges brings to the convergence analysis by considering a collti-ddmensionaal network output and the cross-entropy loss, and discussing our contributions in addressing these challenges.
Weight alignment in multi-class problems. Recall that in binary problems, the directional dynamics studied only for the input weights w..j E [h], because the output weights ny are sealars whose sign (i.e. the "direction" of the scalar) remains the same as its initialization. However, for multi-class problems, each 2y becomes K-dimensional vector, whose directional dynamics are non-trivial. Indeed, we show that (See Appendix C.2) during the early phase of GF, we have
EEin Euj
Eyg: T
my
x. + +(E))
(16)
at w
I
K
d
K-
IT
w, EE Eaj Wi, w,
Eyy + O(e))
(17)
de
7

---

where 1w Me and Eiy are defined similarly as in 77. and E be the k-th column of E and call it the pseudo-label of class k. as opposed to the one-hot label ex- From (16) (still, exclude the O(c) error terms for discussions), we see that although the dynamics 10, d
X1 We let ex
I,w, : de lle T E [h] are decoupled among neuron pairs, the directional dynamics of each neuron pair, now concerning both input and output weights, are described by Riemannian flow on SP x 5K that is highly nonlinear (due to the Ex²) and has non-trivial interactions between the input and output weights. This is the major challenge to the convergence analysis of GF under small initialization for multi-class problems. For our purpose, we discuss the alignment of the neuron weights through the lens of stationary points.
Aside from trivial stationary points that correspond to the dead regions for the input neuron weight, {æx, Jx A E [K]. where 3 Σlcz, "1/l1 Eier, æxll, are attractors of (16) To give a rough explanation, assume |eğ εk. i.e perfect alignment between output weights and the pseudo-label always holds, then one can write (16) as
my
d 3 de |mm
V K- nno (Erez, Eis (ēx, Di +V
K K---I Ino (ENEA EiEIg Eis (ēw, (ēw, Tx) xi
resulting on flow that pushes woyll class (other classes), and eventually towards 2g when sufficient alignment with the k-th class V has led to Eij = lIET.. Similarly, by assuming Tw,l Bg. we can write 17 as dt Te,
wyy towards (against) the directions of data points in the k-th
K - (EiET, (2,, ēx), pushing the output weight toward the pseudo-label éx.
Transient analysis: inter-class separation via alignment dynamics of neurons. Based on the discussion above, there is the region of attraction (ROA) for each attractor {æx, ēx} such that all the neuron weights initialized within the ROA are guaranteed to converge to (I é via (17) Moreover, the boundaries between two ROAs of different (class (pseago)-psecuoollabe)) pairs together form an invariant set that does not converge to any of the attractors {T },,}, where there exist saddle points of that no weight direction falls on the aforementioned boundaries and each ROA contains at least one neuron pair. However, finding an analytic expression for the boundaries is challenging problem by itself and far beyond the scope of this paper. Instead, our analysis identifies an invariant subset for each ROA, thus by initializing within those invariant subsets, we guarantee the directional convergence of the neuron weights to { ēx) &s E [K], which implies inter-class separation.
The exact generalization of Assumption to the multi-class case should assume
> 0} denote the index set for class-k data points that activates the input neuron weight w, let Ar = Everp (z:: |[
Formally, we let TX- = fi E I& : (Ii, ur
||
be the aggregate alignment with the k-th class of the input neuron weight w, and let B (ēs, v be the alignment with the k-th pesudo- label of the output neuron v. Then we define the following:
T
Assumption 4 (Semi-local initialization). The initialization shape (wz0. ) satisfies that 3 a partition {Nk, & [K]] of [M], such that vk E [K]. we have
I 1 12 Σ A 10 A0 2 2'
1050
B% 2
- VjEN. (18)
Given some additional assumption on the orthogonal separability of the data (see below), the condition (18) defines an invariant subset the ROA of the attractor (#x, :: any neuron weights initialized to satisfy (18) remains to do so during the alignment phase of GF, while getting attracted by (Øk, ēx }. leading to the desired inter-class separation:
Proposition (Inter-class separation in multi-class problems). Let K 2. Given orthogonally X < 2K 3, where Xmax == max4e(x] [| and separable data (Assumption 1 with X Baig
X Xmin == minig(n) ||ær,II, c-small, balanced and semi-local initialization (Assumptions 2 4 for a sufficiently small c, for any solution O(t), 2 to 3 37 such that Vt 2 T we have O, Vi ViE Is Vi E [n],k E [K]. where Wx(t) == [w, (t)lsENe As a result, the inter-class separation (Do()(Z)), )e(c((x)) =0, 0, Vi It, i' E Iw.k # k holds V*
Wx(t) 100 Vi Vi I
8

---

Asymptotic analysis:
intra-class directional collapse and projected self-duality via max-margin bias. Once the inter-class separation is achieved, we can agai decompose the loss function as
((0) = EiEZ, (CE W æ;), where Vi= = [yy]seNe thus it suffices to study the GF on {W&, V&J for training a two-layer linear network on exclusively on class-k data for each & E [K] with cross-entropy loss. Another major technical contribution of our analysis is to extend the max- margin results in the work of Phuong and Lampert [77), Ji and Telgarsky (21) to the multi-class
problems (albeit under special case that all data have the same label and are positively correlated),
leading to the following asymptotic characterization of GF:
Proposition (Variability collapse and self-duality in multi- problems), Let K > 2. Givvnn
Assumptions 7 & the limit == lim,,>oe [
0(t) exists for any solution ( t 2 0 to For
the limiting direction (W&, V K k=1 3g E S succh that
W suuag Vx sxērg
(19)
where Sks ls E [K] are defined in Theorem 7
As a result, the intra-class directional collapse (4 and
projected self-duality (6 hold.
Limitations of current analysis. Aside from the orthogonal separability of the data, for which have made remarks in Section
2 the convergence analysis for multi-class problems requires a stricter
X_
<K 3), as shown in Proposition | This assumption is required to show that the subsets of the parameter space defined in Assumption 4 are invariant under GF. We believe such an assumption is not needed in practice, but our limited understanding of the ROAs and their invariant subsets has led to this additional technical condition to ensure directional convergence. Future research on better characterizations of the ROAs and their invariant sets will naturally relax or even potentially remove this requirement. The additional assumption (Assumption 4 on the initialization shape for multi-class problems is another limitation of our analysis for the early phase of GF, requiring all neuron weights to have decent alignment with one of the (laass-avragge label) pair. Relaxing such an assumption necessitates a careful in-depth analysis of the neuron weight alignment dynamics shown in (7). for which we have discussed the underlying challenges, and we leave it as an important future research direction. Nonetheless, we would like point out that: First, our result asymptotic convergence of the weights is applicable whenever one can show that inter-class separation happens at sometime during the GF, and our transient analysis simply provides one condition under which the separation is guaranteed to happen; Moreover, the semi-local initialization can be satisfied if, instead of random initialization, one initializes all the neuron pair shapes {wo " by drawing uniformly from the (datu)-(pesudo-labbel) pairs {x; Yy.".. which is practically possible initialization scheme.
separability condition X
"iin "aaM?
5 Numerical Experiments
We conduct experiments primarily for the purpose of validating our theoretical results. We first train a two-layer ReLU network for classifying three MNIST [45] digits and visualize the neuron weights alignment at the end of the training, thereby showing the NC characterizationss in Theorem Next, based our remarks intra-class directional collapse, our Theorem [onorrmm[sugeee suggests that proper normalization layers such as RMSNorm (28) can potentially lead to a more significant level of NC, and we conduct some preliminary experiments with ResNet (46] to verify this conjecture.
MNIST digits classification. We train a two-layer ReLU network to classify three MNIST digits (0, 1, The experimental details are in Appendix B. Figure 3 visualizes the training results: First, we show that the dataset of MNIST digits, centered by the mean digit of the entire dataset, approximately satisfies the orthogonally separable assumption. Then, we visualized the neuron pairs, showing their respective alignment with the data and the pseudo-labels. Moreover, we visualize the top 3 principal components of the last-layer feature of the digits, together with the classifiers, whose structure matches the NC characterizations in Theorem I
Validating Theorem
Experiments on the role of normalization layers on NC. Next, we train a modified ResNet18 (by replacing the final linear classifier by a two-layer ReLU classifier) on MNIST and CIFAR 10 [47 datasets. In addition, we add normalization layer (Identity/None, LayerNorm 48 or RM- SNorm (8 before the ReL.U classifier and vary the methods for normalization. The experimental
details are in Appendix B.2 2
Figure reports (repeated for 5 runs; mean(line) and std(shade) are

---

(a) Digits Correlation
(b) Neuron Weights Alignment
(c) PCA of Iuput Data
(d) PCA of Last-layer Feature
Figh
Sanple lndex
Cantla
[B]
PC3
10 10
[[]
PC1
PCI
Sumple Index
Figure 3: Validating Theorem in classifying MNIST digits {0, 23. (a) Normalized correlation matrix of subsampled 500 MNIST digits; (b) (For the trained network) visualization of output neuron weights (as crosses, gray dashed line represents ég directions for references), the average input neuron weights (as grayscale image, surrounded by colored box), and the average of the digits (for comparison, next to the neuron weights); (c) PCA of raw digits data X, keeping the top 3 principal components (1000 points visualized) results in ~61 % relative approximation error for X; (d) (For the trained network) PCA of last-layer feature Φo! X) and classifiers (rows of V), keeping the top components (1000 points visualized) results in a 0.2% relative approximation error for Φ (X).
Intra-class Variability Collapse Class Means Separation
Self-duality
Accuracy
Accurncy 196)
$9.4
Layathore R34
0,15
59.2
home
$9.0
Epoch
Epoch
Epoch Self-duality
Epoch
Intra-class Variahility Collapse Class Means Separation
Test Accuracy
Aovmmn S (5)
1.0
1.0
1.0
Narasiatto
2
Z
LoyeNoor
0.0
20
0.0
20 Epoch
0.0
20
Epoch
Epoch
Epoch
Figure 4: Measuring NC in trained modified ResNet18 on MNIST (top) and CIFAR10 (bottom)
reported) the evolutions over training 50 epochs of the metries NC1, NC2 and NC3 that measure the NC characteristics in Theorem [oooremm[wwwweer (lower value implies more prominent NC; definitions in Appendix B.2 at the last-layer of the ReLU classifier. Notably, using the RMSNorm layer significantly im- proves the intra-class directional collapse, as we conjectured in the remark for Theorem suggesting potential practical value in using RMSNorm layers for promoting NC.
6 Conclusion
In this paper, we investigated the connection between NC and the implicit bias of GF through convergence analysis of GF on two-layer ReLU networks for orthogonally separable data and showed that the implicit bias of GF facilitates the emergence of NC along the GF trajectory. Future work includes relaxing the assumptions on the data and initialization, and extending the convergence analysis to understand the emergence of NC deeper networks; For example, similar early weight directional alignment and asymptotic max-margin bias have been studied in prior works 26, following the same high-level proof and utilizing these existing results on alignment and max-margin for deep networks might extend the current work to more practical setting.
Acknowledgments and Disclosure of Funding
This work is primarily done when H. Min was a postdoc at the University of Pennsylvania. The authors acknowledge the support of the NSF under grants 2031985 and IIS-2312840, the Simons Foundation under grant 814201, the ONR MURI Program under grant 503405-78051, and the University of Pennsylvania Startup Funds.
References
[1] Vardan Papyan, XY Han, and David I. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, 117640):2465-22463,, 2020.
10

---

[2] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory. Signal Processing, and Data Analysis, 20(2):11, 2022.
[3] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In ICLR, 2022.
[4] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. geometric analysis of neural collapse with unconstrained features. In NeurIPS, 2021.
[5] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. PNAS, 1 118(4))=2103009111 2021.
[6] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In ICML, 2022.
[7] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In ICMI., 2022.
[8] Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. NeurIPS, 2023.
[9] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu Dustin G Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. In ICML, 2024.
[10] Akshay Rangamani and Andrzej Banburski-Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. ICASSP, 2022.
[11] Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Under- standing deep representation learning via layerwise feature compression and discrimination. arXiv preprint arXiv:231 1.00960, 2023.
[12] Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research, 6:0024, 2023.
[13] Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
[14] Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse. In ICLR, 2025.
[15] Gal Vardi. On the implicit bias in deep-lcarning algorithms. Communications of the ACM, 66 (6):86-933 00066993.22 2023.
[16] Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXiv:1803.0883 08367, 2018
[17] Mary Phuong and Christoph H Lampert. The inductive bias of relu networks on orthogonally separable data. In ICLR, 2021.
[18] Etienne Boursier, Loucas Pullaud- -ivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. In NeurlPS, 2022.
[19] Nikita Tsoy and Nikola Konstantinov. Simplicity bias of two-layer networks beyond linearly separable data. In JCML, 2024.
[20] Hancheng Min, Enrique Mallada, and René Vidal. Early neuron alignment in two-layer relu networks with small initialization, In ICLR, 2024.
[21] Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In ICLR, 2019.
11

---

[22] Frank H Clarke. Optimization and nonsmooth analysis. SIAM, 1990.
[23] Wanli Hong and Shuyang Ling. Beyond unconstrained features: Neural collapse for shallow neural networks with general data. arXiv preprint arXiv:2409.01832, 2024.
[24] Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning neuron by a shallow reLU network: Dynamics and implicit bias for correlated inputs. In NeurIPS, 2023.
[25] Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced. In NeurlPS, 2018.
[26] Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning, In NeurlPS, 2020.
[27] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. ICLR, 2019.
[28] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NewrIPS, 2019.
[29] Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is two-edged sword. JMLR, 2025.
[30] Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two-homogeneous neural networks. TMLR, 2024.
[31] Andrew M Saxe, James L Meclelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In ICLR, 2014.
[32] Suriya Gunasekar, Blake Woodworth. Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. In NeurlPS, 2017.
[33] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In NeurIPS, 2019.
[34] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In COLT, 2020.
[35] Tao Luo, Zhi-Qu thi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at widtn limit. JMLR, 22(71):1-47, 2021.
[36] Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. In NeurIPS, 2021.
[37] Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv: /06. 15933, 2021.
[38] Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks. In International Conference on Machine Learning, pages 14422-1846.. PMLR, 2022.
[39] Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald, Jinqi Luo, Salma Tarmoun, Enrique Mallada, and Rene Vidal. Understanding the learning dynamics of lora: A gradient flow perspective on low-rank adaptation in matrix factorization. In AISTATS, 2025.
[40] Zhenyu Zhu, Fanghui Liu, and Volkan Cevher. How gradient descent balances features: A dynamical analysis for two-layer neural networks. In ICLR, 2025.
[41] Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B Simon, Michael R Deweese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A theory of feature learning in two-layer neural networks. arXiv preprint 2rX.v:2506.0664 2025.
12

---

[42] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization neural networks. In NeurIPS, 2018.
[43] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. In Neur/PS, 2019.
[44] Vignesh Kothapalli and Tom Tirer. Can kernel methods explain how the data affects neural collapse? TMLR, 2025.
[45] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1:211):222223211 2324, 1998.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE CVPR, 2016.
[47] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. Technical Report.
[48] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:1607.06450, 2016.
[49] Samuel Burer and Renato D. C. Monteiro. Local minima and convergence in low-rank semidef- inite programming. Math. Program., 103(3):427-444, July 2005.
[50] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. In ICLR, 2018.
[51] Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initialization the convergence and implicit bias of overparametrized linear networks. In ICML, 2021.
[52] Hancheng Min, René Vidal, and Enrique Mallada. On the convergence of gradient flow on multi-layer linear models. In JCML, 2023.
[53] Ziqing Xu, Hancheng Min, Salma Tarmoun, Enrique Mallada, and Rene Vidal. Linear con- vergence of gradient descent for finite width over-parametrized linear networks with general initialization. In AISTATS, 2023.
[54] Arthur Castello B de Oliveira, Milad Siami, and Eduardo D Sontag. Dynamics and perturbations of overparameterized linear neural networks. In IEEE CDC, 2023.
[55] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. NeurlPS, 2019.
[56] Liwei Jiang, Yudong Chen, and Lijun Ding. Algorithmic regularization in model-free over- parametrized asymmetric matrix factorization. SIAM Journal on Mathematics of Data Science, 5(3):723-744, 2023.
[57] Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon Shaolei Du, and Jason D. Lee. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing. In ICML, 2023.
[58] Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. In NewrlPS, 2024.
[59] Hancheng Min and René Vidal. Understanding incremental learning with closed-form solution to gradient flow on overparamerteriied matrix factorization. In IEEE CDC, 2025.
[60] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[61] Yuanhe Zhang, Fanghui Liu, and Yudong Chen. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. In ICML, 2025.
13

---

[62] Uijeong Jang, Jason D Lee, and Ernest K Ryu. Lora training in the ntk regime has no spurious local minima. ICML, 2024.
[63] Junsu Kim, Jaeyeon Kim, and Ernest K. Ryu. LoRA training provably converges to low-rank global minimum or it fails loudly (but it probably won't fail). In ICML, 2025.
[64] Salma Tarmoun, Guilherme França, Benjamin D Haeffele, and René Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. /CML, 2021.
[65] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In ICLR, 2019.
[66] Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparaaettrrizatoo:: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas Information Theory, 1(1):84-105, 105, 2020.
[67] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Neur!PS.
[68] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. NeurIPS, 2019.
[69] Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for under- standing neural networks. NeurIPS, 2019.
[70] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In COLT, 2019.
[71] Lenaic Chizat and Francis Bach. the global convergence of gradient descent for over- parameterized models using optimal transport. NeurIPS, 2018.
[72] Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn represen- tations with gradient descent. In COLT, 2022.
[73] Matus Telgarsky. Feature selection and low test error in shallow low-rotation relu networks, In ICLR, 2023.
[74] Spencer Frei, Niladri Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data, In COLT, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction clearly state our claims and underlying assump- tions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Remark 1 and 2 and paragraph Limitations of current analysis.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: Our assumptions are stated in the main paper, and we provide proofs of theorems in our technical appendices
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental details in Appendices B. and B.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We attach the code in the supplemental material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cipps.c/puullc//guu /ot.e//Suudeeooeomuummmiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental details in the Appendices B. and B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The experiments are for validating our theoretical findings and do not requires heavy computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::::///rrrriscecee http:///..c.cco//ccccccccleeeeea
Answer: [Yes]
Justification: the research follows the Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In this theory paper, there is no potential societal consequence that we feel must be specifically highlighted.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: original papers that produced the datasets cited
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs used for writing this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuree http://///////oofeeeeee/e/////// for what should or should not be described.
21

---

A Additional Remarks and Related Works
We make the following remark on ReLU networks with biases
Extension to ReLU nets with biases When considering two-layer ReLU networks with biases, since ((w,,a æ) b) ([w: b), [x) adding bias term effectively adds one homogenous coordinate to the entire dataset. Therefore. our results still hold if the augmented dataset satisfies the orthogonal separability condition. Notably, homogenous coordinate increases data correlation: ([m:; 1], [ 1) = ( 2z)) + 1, thus it mainly affects the negative correlation between data points with different labels. the case when min, | || > 1, the orthogonal separability (among augmented data {[ 1))=>> still holds with the bias term.
A.1
Additional related works
Matrix factorization in deep learning theory A major part of the research efforts in the theo- retical understandings of deep learning is through tractable mathematical problems, where similar phenomena can manifest to those observed in deep learning practice. Among these problems, matrix factorization [49 is an important one, which has been studied for understanding the convergence rate of gradient descent on neural networks 50 (54], the implicit bias of neural network training algo- 36 59), and, we already mentioned in the introduction, the NC phenomena 2 More recently, the prevalence of LoRA (60) in practice has motivated many works on its theoretical properties [3 63 through analyzing matrix factorization problems. While matrix factorization problems offer many valuable insights into various aspects deep learning, they generally neglect the role of training data in these problems. For example, the convergence analysis often assumes input data with isotropic covariance [50] 64], and as we have discussed, the analysis of NC often assumes the last-layer features as unconstrained optimization variable 24 Given that NC is characterizing the ability of neural networks to map input data to structured latent features, is also important to consider the role of input data. Indeed, our work highlights how the input data structure induces a directional collapse rather than a singleton collapse.
rithms 32 55
21 31
33
37
Gradient descent shallow neural networks Theoretical properties of the gradient descent algorithms on shallow networks have been studied in different learning regimes and from various perspectives. Earlier works 665 (66] concern the convergence of gradient descent in the so-called kernel regime, where under specific settings (large network width, random weight initialization with large variance, etc.) the linearization around network weight initialization holds valid throughout training [42] (67). However, limitations of such "lazy regime" training [43 are identified through some specific student-teacher learning settings [68] 69]. This motivates the study of convergence in small initialization settings, often called active or feature learning regime (34). From a dynamics perspective, in the infinite width limit with proper weight initialization scaling, the weight evolution during training can be characterized by some mean-fieed dynamics (70, [7]]: In the finite width with vanishing weight initialization, the early training phase can be characterized by the directional alignment between the weights and the input data [16 29, 19]. From a generalization perspective, learning in the feature learning regime can enjoy many advantages, such as sample efficiency [72 73
or benign overfitting (74). Our work studies the convergence of ReLU networks in the feature learning regime and contributes to this line of work in the following regards: First, prior works primarily concern the convergence of the weights, while our result discusses its implications the learned last-layer feature. Second, from a technical perspective, our result addresses several challenges emerging from considering a multi-class problem with the cross-entropy loss.
22

---

B Experimental Details
B.1 Experimental details on classifying MNIST digits
Preprocessing data. We first preprocess the training data, i.c. digits 1, by centering: f 2; Φ. where z Eie[n] z//n is the global mean image of the entire training data. Then we
have plotted the normalized correlation matrix K (I=i 2/ [|=||] >
(.reim) of the centered data, showing in Figure a that two data points of the same digit are likely have positive correlation and those different digits are likely to have negative correlations. This suggests that the orthogonality separability assumption in Assumption iis approximately satisfied.
Training. Given the centered data, for a two-layer ReLU network (2) of width-50, we initialize all entries of the network weights with i..dd Gaussians with variance 10 Then we run SGD of batch size 1000 with learning rate 0.1 for 50 epochs. For the trained network, we visualize the output neuron weights j E [h] and determine the N& by letting V (j E [h] k arg maxy (ēw', B2) }, then also visualize the average direction of the input neuron weights ΣSEN, w, E,,,,, will
for each group Ng. as
shown in Figure 3 bb)
B.2 Experimental details normalization layers
Modified ResNet. We take the ResNet18 and ResNet50 implementations (The first conv layer modified to accommodate MNIST and CIFAR10 10 input sizes) in Pytorch and replace the final linear classifier with two-layer ReLU network of width-1000, and also add normalization layer (Identity/None, LayerNorm, or RMSNorm) between the classifier and the feature extractor. The initialization follows the Pytorch default.
Training. For each choice of (model: ResNet18, ResNet50)-(Dataset MNIST, CIFAR 10), we repeat runs (with different random seeds) of SGD of batch size 128 and learning rate 0.1 (for ResNet 18) and 0.02 (for ResNer50) with momentum 0.95 for 50 epochs; and for every 20 epochs, we reduce the learning rate to 0.1 its current value. We plot the NC metrics and test accuracy against training epochs in Figure
Within-elass Variabilly Collanse
Means Separation
Self-duality
Test Accuracy
$9.6
0.5
Accuncc (1))
0.5
59
Normitaaton
2.
20
40
Epuch
Epoch
Egoch
Epoch
Within-class Variability Collagge 1.5
Class Menns Separation
1.5
Self-duality
Test Accuracy
1.0
1.8
1.0
Accorncy
% 20
Naraclatto
0.5
0.0g
0,9
600gg
Epoch
Epoch
Epoch
Epoch
Figure 5: Measuring NC in trained modified ResNet50 MNIST (top) and CIFAR10 (bottom)
NC metrics. The NC metrics follow those used in prior works except for the projected self-duality.
Σ Σ.Er, (1.))(m.)) (+))))/+=l//////// /|Z=| k==1 Given the class means Φ; Eiser, )o(œ.) and global mean Φ Σk= Φ2 Pa/K,NCO is defined to
be the ratio between intra-class variance and the inter-class variance
Σk= -|
NC2 is defined to be the proximity of the gram matrix of the class mean directions to the identity
matrix | VK
1 IlE,, where G = [
σx]
Φx]. NC3 is defined to be
the proximity of V t to an identity matrix
V |
JK-I
ElF..
In the main paper, have only provided the plot for ResNet18. We show the plot for ResNet50 in Figure
23

---

C Neural Alignment under Multi-class Orthogonally Separable Data
C.1 Basies neuron dynamics under multi-class problems
The differential inclusion 0 VVL(0) gives rise to the following characterization of the time derivatives neuron weights Vj E [h]:
So (y: Di. 05) Ci,
(20)
dt
==1
Σ ==1
(y - ÿi) (20e, wz)
(21)
dt
= 1,
(m); w) > 0
where Egp
[0, 1], 0, (z,, wj)
( w3 0 Sfftmax(f(x;; 0))
It will become clear soon that it is convenient to decompose the weight dynamics into those of the weight norm and of the weight direction, for which we use the balancedness that ||wr = || Vj:
(weight norm dynamics)
d
|,,l1 also |,,1 dt
Siz (3:
Dis v₁) (x wj)
Eij
Yi
Úi Vj
|
Zi:
||w
Wi
||0,11 (22)
i=1
(input neuron angular dynamics)
w
Eig (3)
Di.
dt
|
mi
1m1
n2
II
wy
Eúj
Y:
Úi.
10,
Vi
(23)
(output neuron angular dynamics)
Dj dt |
II
Σ
Sū - 41) (mx, w)²
||
i=1
II
Σ
Eij
Iss
wi
(y) 91)
(24)
|
where 11 = (I
2414 |
denote the project matrix onto the orthogonal complement of u.
note that the dynamic of each neuron pair (w,, U₁) is almost de- coupled from each other except for the interaction through ÿ., í E [n.. Interestingly, at the early phase of the GF, (we will show that) the norm of the weights remains close zero, resulting in 9; } 1, Vi E [n] thus fully decouples the neuron pair dynamics, the precise statement on such an approximation 9, ~ 1 1 is as follow:
By inspecting 22
Lemma 1. | A 1|l < 3 [I 0)] whenever ( 0)l < 1.
Proof. First of all, we have
[exp([=)) 1 =20 [1 exp(-1a)), <<0"
eep((2) 1 = max - 1, - exp(x))
We always have 1 - exp(-lla < |z]. Moreover, whenever |z| < 1, we have exp(1=|) - S 2/2/. Therefore, we conclude that
(exp(2) 1| - 2|2
M/ < 1
(25)
24

---

With 25
whenever (x;; 0)ll - 1, we have
max exp([f (a;; 0) 11 S 2 max ( 0)1al S fx,; 0)
(26)
Now we bound 1
1
using entrywise bound. Notice that
|
exp([f(an:0))))
R
Ex exp([((m;))))))
R+ (oxp((11))(
Klexpl (e)) (fx
K (exp ((xp((
[]exp[[f(<,, 1 |exp (2)))))))))
K(K Σx (
(iI
R ff 0)
(27)
[[K22K]
Finally, we have [/:: * 1|l
v K max:
|[xb
*
C.2 Analyzing neuron dynamics during alignment phase
In this section, we show the formal statements for the alignment dynamics we have introduced in 17). During the early phase of the GF training, the norms of the weights remain small (Lemma
2). leading to an approximate alignment dynamics in Lemma 3 which will be crucial for subsequent analysis.
Lemma 2. Given some balanced, e-small initialization (()) with S
VK 16X Vh' any solution O(t)
to the GF dynamics satisfies that Vt S 4n.Xmae
log 1 := T,
/hc
=|0,(0= Vh E [|f(;; 0()))1 S 26 Vh.
(28)
The alignment phase refers to the training phase until T In XnxXaax log Vhe With Lemma can
d w, approximate the angular dynamics at |[w and |o throughout the alignment phase as follow:
Lemma 3. Given some balanced, e-small initialization ((0) with 16X Vo' any solution 0(t) to the GF dynamics
satisfies that VI S 4nxmmx log The
T,
d wj
11
K
Exj
Eyg
U
<
16
en X Vh, Vj E [M]
dt
|
K
||0,111
VK K
i=1
0y dt ||,
11
K
Exj
Dis
W
Ey,
<
16
EnX Vh, Vj E [A]
K
|,,,ll
/ K
Proof of Lemma
From Section
we have
dt
d
wz 11
Eúj
y)
Di,
|
||rr
W;
|/00 ||2
(29)
Let T = [t: max; [f 0(1))
2e X
Xmax
TT.. then VI S T,j E [b]. we have
|2y112 2 = 2 Σ Eij ( (v. - U
2y |e
mi Øti
||,; ||
dt
w
i=1
n
<
y:
Dis T0
m
>l
Wis
Tw,
wy
||00,||
1=1
25

---

1/
y: a2 myl
1=1
1 2 Σ
(1 x + l| 8.l)) mmal |,,1
1=1
<
(V/x
K
K-1
8 (1: 0)1] X X 1|0,|
< 2
Xmuxx
16
max
||00112 2n (Mmax 16xX vh
||05112
(30)
VR
VK
i=1
Let == fnf [|mr(()]1 > be shown by contradiction:
vF and let j == arg miny Tj. Then Ty = min, Tj S T, which can
Suppose Ty. > T, then t 7 < Ty,, by the definition of Tys, we have max, ||un I c and by the definition of T and the continuity of ((t) w.r.t. t, 3i [n]] such that [f(. O(T)) = 2aX Vh, therefore,
vh
2c.X vh- f(x ;(xx:0((T))
Σ Eir 5 (w0, xx+)
JE[M]
Σ
|y |||||| [|| [|<00=1111111
jE(N]
1 Σ
X max
1,y1 S 1.X jE(h)
max [
(31)
jE(A]
which suggests that ma
12y1222
2x Vh
contradiction.
Now for t S Tj. I T, we have
d
Mnj- 112 2n Xmmxx + 16eX2 X6 Vh
||:05-- 112
(32)
dt
VK
By Grönwall's inequality, we have Vt S Tz.
[/w (1)1 exp (2m (Xmax
16eX2
tun Vh 1) 1/00-- (0)1 s exp (2n (Xmax + 16exx uan Vh
(1))))
1) c²
VK
vK
then by the continuity of 1|20- (t]l1 we have Suppose Ty- Inxmaxx log Vhe
vh = =1w, (Tx-)112 <
exp
2n
(maxx + 16eX2 mux Vi
Tz-
62
VK
< exp
2n (maxx + 16eX2 max VT
VK
Inxmaxx
log The 2
< exp
((3
8r 8 Vi
log The
1
2
VK
< exp
log
Vhe
where the last inequality due to c
vK
This leads contradiction. Therefore, one must
16.X via"
have T T22 Tz Tz-
log 11)
This finishes the proof.
Anxmax
Vhe
Proof of Lemma
3
We have shown in Section C.I that
dt
d w
11
Σ Eis
y:
|
2
(33)
T Therefore, V
w
17 w, EE Eig (y.
de w0
|o
26

---

Ei=ı Eiz
(3) 1 1)
+( +(
ÿt): |0;1 xs
= Ey.
II (V
K-1 D Eij
Ey:: T0;
xi
+ +I
Ei= Er²
( - ú:).
Te
2,
Finally, we have
d W dt
Il,
(V
K-1 K
Ei=i Eig
Eys: |[ x3)
II,,,
12
Ezj
(
||aا|||| us Bi);
(Lemia
(Lemmal
Ei=
||||
n. maxx I (æ;;; 0)
16 VR
X2
max Vh, (34)
VK
where we note that applying Lemma requires (a:; 0)1l S 1. which is guaranteed by Lemma and our choice S 16X vh" bound can be derived for d de |0,
We have shown the approximation enror bound for di [|w A similar
C.3 Neural alignment under multi-class orthogonally separable data
Sufficient statement for Proposition I is easy to check that the following proposition is sufficient for Proposition]] to hold.
Proposition 3 (Sufficient statement for Proposition D. Let > 2. Given orthogonally separable
data (Assumption 7p with Xaantaii < 2K - 3, where Xmax := max;e[n] |x;| and Xmin ==
X
mingg[n] || e-small, balanced and semi-local initialization (Assumptions) 2 for sufficiently
small e, for any solution ((1), 1,,1,22 to
33 and any j NR, k E [K]. define
Tj inf
0 IZ
Ixl:
20
O, Vk' + k}.
(35)
> O, VIEI.
then Vt2T3 T, we have _ 1
Iel
O, AK + k, thus wx(() x;
viEln].
O,
Vi Vi I
Therefore, we can study the dynamic behavior of each neuron pair individually, for convenience, let j E Ni. and we drop the index j.
For a neuron pair (w, v), we have defined the following:
w
04 _
(aligmment between input neuron and i-th data)
w
B, =
Eys: |0
(alignment between output neuron and i-th label)
I = )i Is : aa +0,
(number of active data points in k-tth class)
and
A = Σ 04 =
Σ
(alignment between input neuron and k-th class)
IEI
VEIz >0
we mosily use this notation for clarity
Bg =
ēk. |
(alignment between output neuron and k-th class)
Overview of the proof of Proposition show that (Recall that
I First, we utilize the alignment dynamics in Lemma to log
4nXuus
Vhe
Lemma 4. Given neuron (w,, Uz).j E Nr. during the aligument phase t S min (T T}, the following holds:
27

---

1. Ri remains close to its target pseudo-label: B₁ 2 2(K-1)'
||03]
2.
4
remains close its target class: Ak- + AK 2 As(0)) - 2 Ex Ar(0);
3. Neuron
does not deactivate data from target class, nor activate data from non-target class:
wyll (()
I
and 7
Ak # k.
The characterizations in suggest that the neuron weight directions {
w 2 remains close
1 Te to the attractor { } and as the weights move closer to the attractor, [Ix increases to [Tx] and I decreases to O. When the initialization scale is sufficiently small so that T is large, this Lemma will show that 7 is finite, and we will provide an upper bound.
Then the following lemma shows that the desired property for neuron (wy, 02) still holds after TjK Lemma 5. for any neuron (uj, wy).. E Nr. we have vt. > Tj
1.
v, remains close to its target pseudo-label: By 2 ||o
v2 2
2.
is exclusively activated by data from its target class: Ix Nk. |
0, Ak' / k.
|
The remaining parts of this section are dedicated to proving these two Lemmas. The next section will formally prove Proposition 3 thereby proving Proposition [
C.3.1 Proof of Lemma
Basic dynamics. The main proof concerns the time derivatives of the alignment classes
d As, de B.. With Lemma 3 have their approximations during the alignment phase:
de
d Aa di
EiET, 0
m Di. de [0ll
EiELEO
2g II (v (x En Ev (Eyr. |Ml] x. + O(c)
(36)
EiEtna (ar, Ihe (Er= (uBvarr) O())
V K
= V
K-1 1K1 Eietiaa Evsayza (10; Evarp) - a.Er)w))))))) Br + O(c)
(37)
K-1
EiEIna Da EisHsk EvEI ((T a.grar) Br O(c)
K EiSHeK EiEtea, EveI (1E aggay) BR + O(c) (38)
EisweK BR EEETroo Ev (12 aitrav)) + O(e)
EigNek Bx ((Eie Vi EveTp
-EisIaaa>o as) (Evetnna 20 (raw)) + +(E))
-V K- EisHek BR ((EiEtiaiso Fi, EvETpiay Every) AkAu) O()),
(39)
where uses the simple fact that EVeInaniZ
EE Oy Ak', 388 uses the
Eys Teel = (ēx, TTo if E Iw.
fact that 3
377 uses the fact that Evr 0 if oz 0, and
36 uses Lemma 3 with the O(e) term being
EiErnoiso 02: de |
IT (V
Er &v (Byu, Tिe xx.
K
28

---

whose norm can be upper bounded as follows:
Σ EieTomai
II,
(V K-
Σx -p=16 Er
Eyyr, lio| 2z¹) )
I|
K
|2>|||
w de |ll
-II (V
K-1 K
Ex= Er
(Byr, |oر xτ) $
16
en2 23
mux vh.
vK
and
d df
BB
Ckk de |To||
ēx, II,
(v
K-1 K
w 2i=1 (:
|wil| Ey.) O(c)
K-1
ēk I1!
(Eiswsk Eietyra a Ey.) ) +0(e)
k
K-1
e (é 11
(EisHsK Eiety ax ) + +0(e)
K
K-1
EiSKSK EiEIn (ēx 11, ēx) a O(c)
A EISKSK ((ēk, BBBw) E Eietyra as + O(e)
K
K-1 EigHek Aw (Yex, ex) B,BB) (EE)
K
K-1 (An(1- (A((1-B)) + Ex Av (-x--
Br -
+ O(e),
(40)
K
where multiple facts used to derive bounded by 16 VK max VT.
39 are also used here, and in (40). the ( has its norm upper
Axuillary Lemmas. The following lemmas will be needed.
Lemma 6. Given Be, 1,.... K defined for a single neuron pair (un, v), we have
-2(1 Bx) -- BB S 2(1 BB) R= Vk, K with k' + k.
Proof. With the following basic derivation
By = ( ||l]] ) @x |ل é + +xx) (ex:
v
- ēx :xx = ( evv: |oll êss
1
K-1
the desired result comes from the fact that
(ew,
v ||l||
S |aa|| ēx -2(1 Ba)-
Lemma 7. Given a dataset that satisfies Assumption 7
then the following is true:
Vk and some ai, Vi Ike we have
11 EEiET ax /ha Eier, a;Xmin
(41)
wk + W and some a,, by 2 0i Vi Ix, 1' E IIy, we have
EEiED, a,x,, Evet, byxr) S -all Eiez, axx EvEI, buzrll.
(42)
Proof. For the first inequality.
EiEE, a,x,ll = V EiEt, 1f + Enty a,dy 03)
2 EiEt, af 21,2 + Σits 14 po ||,,,,|||
2 VlA EieTo of Eiigi a,ax X = wiaa EiEt, a:Xmin-
For the second inequality,
(Eizz a, Ever, bybr) -iEEL abbr (2i, xv) S -ps EiEL.REI abbyl|2ss |aw||
S Spall Eier, a:w:lll Illl Evety bpar
29

---

Lemma 8. Given {Z,, i I with (z,, S0,vi, E I, + j. then Σ1EI 2ill S
Viez 1z
Proof. || 11EI zill
VEEEI || Enes (Z), 2z) V iEz 1
Lemma Given dataset that satisfies Assumption
7
30 < Ç < such that Ak E [K] and
Vw E (z:0< 12 < [kl], we have |Eie Li
Al 2 M2X
Proof. Notice that
10 AAe - =2= = IEie :0; Dz
E Eiezg Bi
to
20
LET
a.20 Fill
(Lemina z
Mi X
(43)
min
Licggop20 will
1 (1-(
Sicrparso Ri 1Z m0000 =0 mill
However, the nonnegative quantity
can be zero: Suppose
is zero, then = EieI= Ti which corresponds to either [ contradiction. We let its lowest value be Ç >>> 0. This finishes the proof.
or [
[xx]
The proof. Now we are ready to prove Lemma
Proof of Lemma
4
We define the following:
T inf
2 0: Bg < 2[K-1)
72 = inf
t 2 0: AA - 2 2
Av < AA(00) - -2EN Av (0)
T3 = inf
t = 0: I < | (0) or I
I Ii((0) for some k'
Then it suffices to show that min Fi, T2, }3} 2 min
j. T), for which we prove them by contra-
diction. Note: In the proof we will use sufficiently small.
to represent an inequality that holds when c is
Case 1: min min(ri 72, T3) = TI.
At TI, by the continuity of B, we must have (B(TT) = 1- 2(- Suppose TΙ S min(T); T}. then we have the following derivation
d
BR
de
K-1 K
Au BBB) EvER Av (-x-1- - BBaBBs) )) O(c)
(Lemma
K-1 (An(1 11 BB) + E Ex'sK Av (- - BB (2(1- BB) -Ba) xx=)))) +0(6)
1-1
(Ax(1 - -B²) -22 22 Av
1-B + BB - Bb))) + +((e)
V
K
K-1
2 Av. - (1) + + 2 Nish 2 Av (1 - (1_B3
1-B: - Ba -Ba)) + O(c)
K
2(K-1)
(1=11)
K-1 26d-2EnrR As)
R-T + +2
kk Avr (Z 2(A-TT) + O(e)
(m2m)
K-1 k
(At(0) 2 2N A (0)
x-1 +0(c)
ST
K-1 K
(A((0) 2 2 E AN Aw) (0)
1
16
en X2
Vh 2 0.
(44)
R-I
JK
max
30

---

The definition of Ti suggests that B5 must drop below 1- 1(K- right after t = which contradicts
that d 3x Be 2 0. Therefore min(Ti, T2, Ta > min(T], T can not be true under the case when
at
min 72, T3 T1-
Case 2: min 72, r3) 72.
Again, we derive a contradiction by supposing 72 I min Since min T2, 73} = 72, at 72
still have B& 2 -πーニJ 0, and by Lemma 6 we also have BB 2(1 Bx) K-I 0.
1
Starting from
39 restricted to 72, have for the target class,
dt
1=12
V
K-1 EisKsK BR ((Evezaanoo Li. ENEIwaa,205ER AnAR) + +((c
-VK
K-1
BR
(IEieIva - + (EieIea Fs. IVEIn Ev
20
20
K-1
By
(EieIeaauo Tis EVEIrra (xe0y)
A
+ O(E),
K
Kigk
50
50
K-1
BB
EiEI
20 xc 11| - + O(c),
(45)
K
and for non-target classes, we have
d
A
dt
ki
1=72
(LIieTara Vi 2v Erery) ) - Apr + O())
Eneph EiSNeK BB
vi
K
En Bi (EiETwan Eivepnaa Evary) ) Aw
K
Enech Ex Bx ((EiEzaroeso C3 CHEIenaai0Sraa Aw Aw + O(E)
K- K
Exeee Be EiiE Dis Ev E2
V
+
Enech EiETp Bú, Eveeh BB EV
K
K Ek (- -BRAAR Enepp BEr Aur + O(e)
(46)
K-1
[allg min B& E ER 12 R x2 Xaa Σ
Σx I OcE),
(47)
The last step to get (47) is upper bound the three terms in (46) separately, which we defer to the end of this proof. Combining (45 47), and recalling the upper bound on the nomn of the O(c) terms,
we have
d de
AA
Ex
1=T2
K-1 K
Br
23 A3
2 (Walls min
X2 BR
X2
2 X2 max
Ek'E [20 - ---32/7 332 Ken x3
max Vh
K-1 BB
2
ppapo min
(1 2(K-1)
2 x2
Σr I 32/ KeenKe2X aax Vh
K-1
max
20
31

---

X Vh (+) max 0. 0. vR
2V K-1 K 2(K-1) Mex 32
en
(48)
The definition of 72 suggests that A& -2= ki Ak must drop below (0)0 A& (0) right
aftert 72. which contradicts that d AAx - 2 ye Av
0. Therefore min 72: ra)
1= min min(T] T can not be true under the case when min(TI, T2, T3 T2.
Case 3: min(7 TI, T2: T3) T3- Finally, remains to exclude the case when min T2, T3} 73 and
73 I min(()]j (7,, T). At 73. either of the following must happen:
1. 3i E Ix such that a, and de a,
2. 3i E In for some k' o k such that a, 0 and d a; 0;
However, at t= T3, Vi Ix, we have
044 a.=0
Fig dr ||ll|
d w
(Lomm B
Eii l1 (V K
K-1 Ei-1 Eo EEyr, 2 |il||
xv)) O(t)
(a)=0)
K-1
(et, (Ep=1 SuBUZe O
Is, EVEIn Sert)) )
V
K-1 K
Bk Dis
EiEIe G(R0)) + +kugth By
+ +0(()
20
50
K- 1 BR Di. EieZ, E10..) + O())
(Lomm 7
(+) x Vh 2 0,
I[K-I] I (0)
16
(49)
VK
max
therefore it can not be that Di Is such that a, 0 and d as < 0. Next, Vi E Iw k + k, we have
d 043
K-1
(zi,
O(c)
K
By Dii Eiet, Giai) Ex By Ii, Eve Eoayy )) +0(()
V
K-
K
Bi Wis Eiezn,azoo + Enrph Ever By (vaw)) + O(e))
K
(Lemma
MaBBa|aailll Eiet Gorill || En EveTg B Oc()
K
(Tmmm
Pa X2 [I + || E Exyp Every Bw + O(e)
(emma
K-1 K
Pall By
X2 |I min
||l
E Bu ((())
K-1 K
Majua Inl + X2 2 V En BBw 11 + +0(e)
(TI5T))
Mafes (1- (1-ZAー1) 2(K-1] X2 [Tal +
2 -- X v /Emss [w + O(e)
(I=TT)
K-1 V (E.. -Mapps (1- 2(K
K [I
X2
min
Kk-
x-1
2 x2
max
VK
cnX3
max Vh (0)
-
(50)
32

---

therefore it can not be that 3i E In,k's * k such that a; = 0 and scenarios, min(TI, T2, T3} min The proof is complete once we add the derivations for 47
di > 0. By excluding both
j.k. ] can not be true under the case when (T. 72, Ta} = 73-
Complete the proot. Lastly, remains to prove (47) which comes from the following derivations: For the first term,
EN BR
CiETeRNDO Lis EveTynan SyEty)
Σ k-=k BR
(<EiTTe Ri EPeTyno 2x) + (EiEThag Li Ev ETyro Ev
20
Σ ko BR EiET Bi EveT 21
(Lemma
-Pd Ex B&l EiETi xill 20
eeemm
-Midlig X2 min
Ek'= Bx 1 T
(72STs)
-MdliX min Bk E 2
For the second term,
Ew # EiEInra Li, E Ex BR Σ Ev
(Ex Eietpraps0 Fi Ex Eivetama 20 BR Ex
Eueer DieTern a.ll 1 xa EvETpma 20 BB ) 1E
(Lemaa F
Ex |l IEiEpnaa 22 Exeep -Bxx) Ev Ez 20 Ey
(Lemmaa))
Σ get I
2x2 max (R=))² Exeee [T 2x2 max
(51)
2 X2
max Σn =k= [Iml]
where 51 uses that 2 S By S O, + k by Lemma 6 And for the last term,
ENsR -BLAANAR Exepee Be BEr AN
(1=12)
S AN (-2Bs Exeeh Av - Eneten Bar
EENfA AK Eneek A
Asr (-2Bs BBx)
Ensh Av Ennsa As (-2 + K-I 0.
3)
C.3.2 Proof of Lemma 5
We will use the following lemma:
Lemma 10. For any SK-1 such that (8,ē1) [0, 1]. then Vp such that p 0, [p] 0.
and (p, 11 = 1, we have
K-I Ets1 [0) (p. 8) s v1 82
(52)
Proof. First of all, since min
[0)a S (p,0) 0)
S max (o),, we know that
(p, (p.0) S max {
(0)
- Exs [o]a
min
[+)
max
[0]
Ā-1
Σ
[)
|0
Σxs (6)x
[0|2
K
33

---

Now given that (Ü, E1) = B, we can write ú = fē1 + v1 g8y' where y E =SK- and 1 ê1. Therefore.
E>>1
|0|2:K
= = Ev
Ei
(Eei /v le [ē112:KK + V1 32
(a 12:KK
K
k
k K-1
/1 $2
Eimgly 1e 11 - (y ]2
K-1
/ - 82
(1- マー yy 12:
Sv1 B 32111y 11,x112 S 11- Ba
Proof of Lemma
Without loss of generality, we prove this lemma for = 1. We define
Ti it Tj B&
2
T2 = inf(t > Tj | # [ or I + 0).
We need show that m = oc. We derive a contradiction by assuming it is finite. Case one: mi, [2] = 'T is finite
Assuming min [T, T3} = Ti is finite, our primary focus is the angular dynamics of |b
V, Vj E,
| 1I, Eiez, Tt |w (e1 9.)
"
and in particular those of its alignment with pseudo-label ě1.
d dt
(ē1, d vy at Twyll
ēi, mT Eiez,
Bia ww (el ú:)
Σier, ( (z) |
wi
(ēl || (el - 9i))
(53)
Ie, 8.)). For each i E II, we let 2 = [WWz =
We shall focus on the term Esen vyw then
exp(aa)
Σ exp(ase)
ER 1- exp( expla;g) Eis exp(E()
Ex= exp(2. exp(212) 2im exp(211)
exp(ai
exp(2ix)
exp( ((a)
1-1 (xp(2x))
Exs1 exp(Eie)
Desi (xp( exp(&.2) 14 D+>. (xp(
(a
exp(E,x)
Ekst exp(s(a)
Less exp(5xx)
T+2 121 exp(ā.))
exp($a)) x-n Σtsr exp(E 1+ exp(
ENst 1s1 exp($())
I+2 iss exp(Eic)
Eesi exp(3x)) Etsi exp
exp((ix) n=T Enst explex) 14 exp(E.1) (i.)
exp(212) Esg)
Ets1 exp(E(x)
V @ +
Σts1 exp(
K-I
(54)
I+ exp(2st)
exp(ăux)
+ 1
Σ Ees1 exp(2.x)
K-1
34

---

thus we have
Ei, MT (er :i))
exp(E12)
+ R-1
Eess exp(Ext) 14b1,1 exp(2it)
e1: (1 vz
K V
EnD1 exp(E,t)
11
K--ee ei +
expleix)
+ R-1
Ekss exp(asn)
0
Eep1 exp(
K (1- + ê1, I
exp(212) Eizs (xp()2x))
+
K-1
1
1+ exp(2it)
V
10 12
exp(Eix)
Exs1 exp(3m))
K-I
exp($nm) [mا
ΣnnI exp(ānn) )
Ensi explént)
V K- (1 (1- - 15.%
K
1 K-1 Ex>1
82
-E
142x exp(Fx))
|| &
-
(Lemma 10
Ensi exp(Fin)
(V
K (1 (B) 15,' /1 11 (1571)2
2
14) 142 exp(2sx)
K-1
Eess exp(
(35))2 )2 (V
K-1 v (B) )2 - B7:) )
K
(55)
1142,s1 exp(á.))
from which we see that at t = Ti, we have
di 5
8 -
E1==E
Ci.
wn |2yl ||
1 kz mxp(2) / (v
E oxp(E:x)
K-
K
V/
v2
2
> 0, (56)
contradicting the definition of Tj- Case two: min T3 is finite
Assuming min (T, T2} = Ty is finite, we shall focus on the time interval [T T2], when we have
d de w
=EiET (el U,) x. = Eier, (el
yi, Toyl
|00 2
(57)
From 54 we have Vt < TT
exp(Fut) |
EASI exp(Sna) )
2 ) e1 Di |a
Σni exp(3nn)
KK-BB Brg +
1 Σ k>1
Ek>11
kcss exp(E]
K-1
||
emmma
E exp(31e)
(V k- 3rh -
- -8
+ exp(3]
(T22))) Eey, exp( I+ exp(3)
K vス K-1
v2 0.
(58)
Therefore, by the Fundamental Theorem of Calculus, we have
(1%
To
(Tx wy (T) + +iEzT
e1
De
|0
Uj
||
(59)
20
which ensures that I wy (T,) [I and [T
(Ta) O, contradicting the definition of 72.
Therefore, the proof is finished by the fact that min (Ti, Tr} cannot finite.
C.4 Proof of Proposition] As we have discussed in Appendix C33 it suffices to prove Proposition 3
35

---

Proof of Proposition 3
We have shown that before min J) the properties of the weights in Lemma hold. We consider sufficiently small such that
16 VK
En X max
V S
K-1
1 2(K-1]
Mo
(60)
2
and
2X Le
log Vhc
(61)
AnXmaax
Then we show that min (T), we have, from 455).
T = Tjj by contradiction: Suppose that T S Tjkk then during [0,
J,k.
Ax
2
K-1 Bk K
(1 -1: + +(E),
ABt)
K-1 1
Ms 2(- min5
VK en X3
16
Vh.
max
R-1 1 K
MaX2ins
(62)
2[K-1]
Then by the Fundamental Theorem of Calculus, we have
(T) 2 A(0) + 2 K (1
2[K-1] P,X2 min
A(0) + Xaxx [Tx],
(63)
which is contradiction, knowing that As cannot exceed Xmax [Tal-
Therefore, we must have finite, Then the rest of the Proposition
min T x and T I T Anxmax log vhe
follows Lemma
36

---

D Asymptotic Convergence Analysis under Multi-class Orthogonally Separable Data
D.1 Basic results upon inter-class separation
With the loss decomposition upon inter-class separation, for which we have shown to persist after T
maxj T j,k.
K LCE
(
Σ
K, V W X₂)
EE (yk.s, V W ox..)
(64)
1=1
suffices to study the following GF on Σn tCE (yk,,, W Ik.i):
WR (Yn Yx) Vs
V = (Y, X W₁
where Y, = SoftmaxCoolVV X2)
(65)
The following basic results can be obtained from 21 27
|[alle; V&ll + oo; WR, Va exist;
V. V W W = 0:
3. WA V is a KKT point of
min W..V
W&
s.t. V.W, 2zlk VV x/) 2 1, 44 #
(66)
D.2 Proof of Proposition 2
Our proof of Proposition follows the same strategy as those in 17 21,, with the major difference being that are handling cross-entropy loss, in which we provide an extension of Lemma 2.11 (21), stated as Lemma 13 Lemma [3 is central to our proof.
Lemma 11. Let 7 = minigksK Yks where " = minjez, (u x;)), then y 2 Pa
Proof. For any 1 s ls S K, uoo Eier, 0, for some a; > 0, then immediately we have, Vi E It
(1,, = (Erez, avay, xi) 2 Mol EvEn, a/2r x| 2 Xm
(67)
Lemma 12. 7 minke(K] minjg]==1 maxiez, (E, > 0
Proof. This result is from Lemma 2.10 (2T) Note that the referenced Lemma requires additional assumption that the support vectors of æ; E In span the ambient space, but the authors of [21] have commented that this condition can be relaxed to the case that the span of support vectors is the span of æ; E Ie. which is true here given the positive correlations between æ,, E Ig-
Lemma 13. Given some + [0,,... 0k] E RDXK and some 1 k $ K. If holds that 3K' # k. (0x Oxx) -Tx))Tocc Toc.k > 0 and | (0, Ow)ll sufficiently large, then
ir ((een T x) O. where y (o
Proof. It suffices to prove the case when k = 1 (We discuss the others at the end of the proof). We start by the following derivations:
(le T 1x.0 x
= Ei= (lea
-
TT 350
x)..)
x
37

---

= Liz1 (el - U. θ "' m__ xz)
Lim1 (1 iu ) 11 ai + Exyi 10 10110 2;
En (CE exp(0% xi)) - exp(0] 2;) ) l1 2i + (- xi)) Uل az
kk=1 exp(O% xi))
Exyi exp(0) (0) - 0x) 0 I1 aa 22
Σ
Σ exp(0) x;)
ExEt exp( (0) - 0z) T (0) - 0k) T th 0z
Σ i=l
1 ++Ex+ E exp(-
0x)) x4))
exp(- (0) 0²) x; ( 0&) T
(68)
1+ I+Σ 1 expl - (01 0%) a.))
For the letth summand, let
1 = arg max (01 - 0x.) TI Zi
(69)
i
we have
exp(-(0) 0x) (0) - 02) T 2;
1=1
1+ Exep1 exp( -0) 0k) T x3))
exp(-(01- 0x) xc;; (-(0) - 0x) TII U.x 1077)
<
1+ Exyi exp( - 0z) xi;;)
exp( - (0) 02) TxxTT);( (0) 0x.) II xi
+ +E 0(1 02) II
2:20
1 Exyi (xxp(((0: 0s) xx;)
One can upper bound these two terms separately as follows:
exp(-(0) - 0s) क.. 2.;;; (- - 0x) Tx) IIl x.p))
1+
1n Ex exp(-
0x) 0x)T'x:(p)))) 2:;;)
< * exp(-(0) - 0n) T xx:;))( (- (0) - T Di;)
= K exp((((0 sap(-((:)) (0) 0n) T an Fi; exp( ((0p 02) I1 um L.; xx;)([]((( (- - 0x) 0x) T
u.c vi;)
S R (001- 0x) T 30 exp(- - -(0 -0x) T Txz ) (01 02) -1 T.4z)
S R exp (->(1= Q2) x exp (7 || (0) 0l)ll 7 [1 (0, 0x)]1,
(70)
and for the second term,
Σ 1:00 -0.) 2:20
exp( (0, - 0,) x;) (;) 0x) TII
10:0 8i
1+ Ex exp(- - (0, 0x) xx))
$ En(o, TT) 2:
- 0x) (0) - -0x) TI u.o x;
$ Ei(O,-o0) -.0 2; 20 exp( (0: - 0₂) TTa 2.)) exp( (-(0; 0x) TTT ax)(0: - 0x)) TTT 2;
0x
00
$ Ex(00-0N) -0x) 2; 20
(pp - 02) (ux) exp(-(01 0x) II x;) - -0x) "II__ x4
E 1.) 20 exp( (0) 0x) (xx)
< exp((- (00) 0%) xx))
(71)
Therefore, putting (68) 70 (1
together, we have
tr [e - T θ umx x)
< E. exp( (000 - 0x) Tux) ( - K exp (++ ITT
uso (0) - -x))ll) + (0, -0x)ll
= 0xx) (voo) ( - K exp (7 I - (0) - 0x))) (] 7 (0-00w)
[1g (0, -0 0w))l)
+ Σ k exp(->(00 0x) T Tūx))
38

---

exp( (0p(->(0)-00)) T [x)
C t exp (1 I (01 0x.) x- - (0) 0w.)
Exer expl -xp aax)
50,
when |11 (0) (01-0x)) is sufficiently large.
If k # consider the permutation matrix Pietk that swap the 1-st and k-th rows/columns of matrix, then
tr ((ex1T) - I uz x)
tr (exth - Yx) P O II
x)
tr (Prosent Prenk)) Piese II X
= =r SoltmaxCol(P θ X)) Piess θ TT t0o0 X)
Following the derivations for the case gives the desired result.
Proof of Proposition
Without loss of generality, we prove the case of k: The existence (W1, V1} is by (27 We first show that W1 Uigi which is equivalent to the statement that
[ Wille
[m Wille
0, and we prove by contradiction. Suppose
which necessarily implies
[[W ir
[W: Ir
[ W,W, l|e
that 3p > such that
= P. then for any c > 0, M.M > exists Te,M > 0 such that W.V.
[W, WT
IF
VI 2 Tr,M, we have |l
W.V.
IWV. Te S e and || V1 || 2 M. We will make clear the
Wi V,
choice of and M later.
Consider the time derivative of [T Ux
3
Willi = 21r (W d Wi)
dt
2tr (W, IT X(eit VV)
2tr ((e - Vi II x
We would like to use the result in Lemma 13 so should examine:
Inl W, W.
tr W, W.V. v.
t
Wi W, W. W, W W, || 0.
(72)
Therefore, > 0, k # such that [|TT W V (e: ex 22 6, otherwise, I W tsc
e+)]l = 0, Vk * 1, which can not happen. Since e1 ex k + 1 spans a k 1-dimensional subspace orthogonal to 1 VR. the projection I1 W, V onto this subspace is zero suggests II W, VT
rank-1 and all columns of V₁ are aligned with 1 which contradicts alignment result in Lemma VK' /2 cosine alignment with ē1).
4
(these columns must have at least
Then Vt. 2 Te,M, and for the l such that |TT W.VT(e, ex)1 6, we have
T Wi
V
(el - 22
|| Boc
| WIV'lE le (ei
W. W.
W. "
(e: ex)ll V2c)
W
W. W.V.
P
W. T
(ei - ex)ll /2e) M
(73)
W.V
Choose sufficiently small and sufficiently large M, we ensure that [I, WiV (e: ex)ll l suffi-
ciently large to apply Lemma 13 so that dt
W, 1 = 21r ((e V, x) s
39

---

[ Wille [ill
On the other hand, [Wll - contradicting our assumption that that W1 & uigi. T
0. This proves
By balancedness V W W 0, we know that V1 x øig for some i1 E Sk-1 remains to show that 81 ê1, which is proved by again contradiction.
Suppose 81 * éι, then 7K* such that [mi]a- [ūi]a Wk + k 1, and not all equalities be - exp( } /e exp(Ex) that appeared in (55), it converges to ex Vi E [n]. Based on this, for any 62, a. such that Vt > Ter.co we have - -exp( 21/2 exp(Eie)
obtained. As results, consider [0 exp((x2)
maxjEN, | iill e and maX4e[m) ||[0 - exp(2;2)
ex- S 6g. Therefore, for some j [h] and i Ter. we have, from 53
dt "
d (él de |eell/
é1, IT, Σier,
w,
(el - ÿ:)
3i Tw
Σier
w, ai Tiny
ēi 11 1 (El 1i:)
mii Tw.,T, wi
2ns1 explEin)
Σien,
V K-1 (1 (1-(5))) (1)
1 Exoi
exp(
exp(áre) l
R-I Ex>1
-Ess Σess exp(
Te k
Cu [w,ll Enst exp(
K K-I - (B8 13td (11)a>
Eiez,
261
1- 1+E exp((sn))
K-1
(E1, 62 sufficiently small)
Jnii mi Exst exp(Ea))
(V K-I (1-
Σiez,
(74)
1 Eisi (xp(Fa))
The right-hand side of (74) is positive and ( by the fact that weight WWll, [[] (log(t))) Therefore 4
grow at a rate
suggests the divergence of B contradiction.
Finally, we have shown W1 uig, and VI ē1g, and the same for other k. The choices of sk are determined by the fact that WR, V& must be a KKT point of (66).
40


Reviews:
[{"review_id": "vIwabiJEs3", "reviewer": "Reviewer_4Gmk", "summary": "This paper provides a theoretical analysis of the Neural Collapse (NC) phenomenon under gradient flow (GF) dynamics for two-layer ReLU neural networks trained on orthogonally separable data. Unlike prior work that studies NC using unconstrained feature models, this work investigates how NC emerges during training due to the implicit bias of gradient flow, even in the presence of nonlinearities and input structure.\n\nThe authors show an emergence of NC from the training dynamics. The paper proves that gradient flow with small initialisation on a two-layer ReLU network trained on orthogonally separable data converges to NC solutions. They also provide a more-refined characterisation of NC, especially intra-class directional collapse, where features of the same class converge to a 1D subspace, not a singleton; most likely due to limited representation power of a two-layer ReLU. Finally, they provide an explaination for the role of implicit bias of gradient flow, in terms of NC, in binary and multi-class classification, and have conducted experiments supporting their case.", "questions": "In your theoretical analysis, intra-class features converge to a one-dimensional subspace (directional collapse), rather than collapsing to a single point as in the classical Neural Collapse framework. However, given the universal approximation capabilities of two-layer ReLU networks, one might expect that full singleton collapse is still achievable in principle, even without an explicit normalisation. Do you view directional collapse as merely an emergent structure induced by gradient flow (with small initialisation), or as a reflection of a true global optimum under ReLU constraints? Additionally, your experiments suggest that applying RMSNorm can recover full singleton collapse—does this imply that normalisation layers are effectively necessary to realise the full Neural Collapse geometry in shallow ReLU networks?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper departs from the more \"classical\" analysis of Neural Collapse (NC) under the Unconstrained Feature Model (UFM) by characterising NC directly in terms of the input data distribution, without relying on free optimisation variables. While this direction has been explored in some recent works, the authors contribute to this growing line of analysis by incorporating the effect of ReLU activations on the emergence of NC. That said, similar considerations have already appeared in deep UFM settings, where the role of nonlinearity is also studied.\n- The paper rigorously demonstrates how gradient flow, even in the absence of explicit regularisation (e.g. weight decay), naturally drives the network towards Neural Collapse structures. By analysing the training dynamics in both the early and late phases, the authors show that inter-class orthogonality arises from early directional alignment of neurons (due to small initialisation), and that intra-class directional collapse and self-duality emerge from the asymptotic max-margin bias of gradient flow. This provides a compelling theoretical explanation for the empirical observation that NC often arises in trained networks without any explicit architectural or regularisation enforcement.\n- The paper generalises its results from the binary setting to the more complex multi-class case, where both input and output neuron weights evolve jointly under gradient flow with cross-entropy loss. This introduces non-trivial coupling between dimensions and breaks the relative simplicity of scalar output dynamics. To manage this, the authors identify structured attractors (pairs of class-average inputs and pseudo-labels) and introduce a “semi-local” initialisation condition that ensures convergence to these attractors. Their analysis captures how neuron pairs align with specific class directions and ultimately lead to inter-class feature orthogonality and intra-class directional collapse, mirroring the structure observed in NC. This is a significant step beyond prior analyses, which often restrict attention to binary settings or unconstrained models.\n\nWeaknesses:\n- While the analysis of two-layer ReLU networks is mathematically tractable and offers insights into how input data structure and implicit bias give rise to NC, it remains far removed from the dynamics of deep networks used in practice. In contrast, the Deep Unconstrained Feature Model (DUFM) arguably captures the emergent behaviour of deep networks more faithfully, particularly due to the implicit feature propagation and layer-wise alignment phenomena. As such, while this work contributes valuable theoretical insight, its explanatory power for real-world deep learning behaviour is limited. \n- The theoretical results rely on two key assumptions that, while enabling tractable analysis, limit general applicability. First, the orthogonal separability assumption on the data—that intra-class inputs are positively correlated and inter-class inputs are negatively correlated—is highly restrictive and rarely satisfied exactly in practice. Second, the initialisation conditions, particularly the semi-local initialisation required in the multi-class setting, demand that each neuron be meaningfully aligned with a specific class direction and pseudo-label from the outset, which goes beyond typical random initialisation schemes. While these assumptions narrow the scope of the results, the authors are explicit and upfront about these limitations, providing clear justification for their modelling choices and discussing how future work could relax these constraints to better reflect practical deep learning settings.\n\nOverall, this is a thoughtful and rigorous contribution to the theoretical understanding of Neural Collapse for shallow ReLU neural networks, and thus I recommend acceptance.", "was_revised": true, "final_justification": "This is an interesting work connecting neural collapse with implicit bias. I particularly liked the comparison that authors provided between the DUFM and their approach. I suggest that a brief discussion about this to be added in the final revision of the paper. I recommend acceptance.", "rebuttal": "We thank the reviewer for the constructive feedback. We respond to your questions/concerns below:\n\n**Comparison with DUFM** We agree with the reviewer that DUFM studies multi-layer networks, but we do not think the DUFM captures practical NC more faithfully due to its two main limitations: (1) it does not incorporate any input data, and (2) as a consequence, it exhibits exact collapse even at the shallowest layer, which is inconsistent with practical networks, where the shallowest-layer features typically show little to no collapse, and collapse becomes progressively more pronounced from shallow to deep layers. Furthermore, the analysis of DUFM lacks convergence guarantees beyond a local result. Our contribution is precisely to (1) incorporate input data into the analysis and (2) bridge the gap between landscape analysis (neural collapse under UFM/DUFM) and convergence analysis (implicit bias in shallow networks), establishing convergence to neural collapse. \n\nAs such, although DUFM studies a more practical model than ours, our analysis offers advantages over (D)UFM in several aspects (data dependence, convergence, etc.) Moreover, since the dynamic behaviors (alignment, and max-margin bias) also appear in GF on deep networks with positive homogenous activations(Kumar and Haupt, 2025; Ji and Telgarsky, 2020; Lyu and Li, 2019), we believe our work, although on shallow networks, opens up oppotunities to tackle the problem of convergence to NC in deep networks.\n\n**Can two-layer ReLU achieve singleton collapse** Based on our remark \"Intra-class directional collapse\" from line 144 to line 154, we believe directional collapse is the global optimal, and normalization along the collapsed direction is one way to achieve singleton collapse as in prior works. First of all, we note that shallow ReLU does not have a universal approximation capability in terms of mapping from input to hidden layer. The universal approximation generally refers to the ability to approximate any input-to-output function mapping $f^*: \\mathbb{R}^{d_x}\\rightarrow\\mathbb{R}^{d_y}$. Here, to achieve NC, the first layer mapping $\\sigma(W^\\top x)$ must approximate a function $F: \\mathbb{R}^{d_x}\\rightarrow\\mathbb{R}^{h}$ that maps training data to ETF features at hidden layer, but no universal approximation theorem exists in this case. Therefore, we believe the directional collapse is due to the limited representation power of $\\sigma(W^\\top x)$ for mapping orthogonally separable dataset to collapsed features. Then, our discussion on RMSNorm only suggests that there exists a shallow network with RMSNorm achieving full NC on orthogonally separable dataset, whether RMSNorm is necessary does not immediately follow from this statement. In fact, the understanding of the role of normalization on NC is mainly on an empirical level. Besides our experiments on ResNet with RMSNorm, (Pan and Cao, 2023) demonstrated that batch normalization has a critical influence on the emergence of NC, and their experiments show that models exhibit a stronger tendency toward NC when batch normalization is used. Theoretically proving such a relation warrants a separate paper in future research.\n\nWe hope our rebuttal addresses your questions. If you have additional questions/concerns, please feel free to post them during the discussion phase. \n\nReferences:\n\nKumar and Haupt, Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations, TMLR, 2025\n\nJi and Telgarsky, Directional convergence and alignment in deep learning, NeurIPS, 2020\n\nLyu and Li, Gradient descent maximizes the margin of homogeneous neural networks, ICLR, 2019\n\nPan and Cao. Towards understanding neural collapse: The effects of batch normalization and weight decay.\" arXiv 2023."}, {"review_id": "OdxG65Fyyo", "reviewer": "Reviewer_YaRJ", "summary": "This paper provides a rigorous theoretical analysis of the emergence of Neural Collapse (NC) under gradient flow in two-layer ReLU networks trained on orthogonally separable data. Unlike prior works that relied on surrogate optimization landscapes (e.g., unconstrained feature models) or required strong regularization (e.g., weight decay, infinite width), this paper directly analyzes the gradient flow dynamics of a finite-width shallow ReLU network with cross-entropy loss and small initialization. It proves that under orthogonally separable data and mild assumptions on the initialization, the network exhibits all three NC properties: intra-class directional collapse, orthogonal class means, and projected self-duality between classifier weights and class means. The analysis is carried out carefully for both binary and multi-class classification, and supported by proof sketches, detailed appendices, and qualitative experiments.", "questions": "1) Empirical results on CIFAR-10 underperform in the appendix: Test accuracy of ~80% with ResNet-50 is well below SoTA and not in the zero training loss regime, where NC is most clearly observed. This weakens the NC interpretation in this setting. \n2) Could you briefly explain (even informally) how the transition from approximate early-phase dynamics to exact max-margin behavior remains valid? Does any residual error affect final alignment?\n3) Does Assumption 4 hold with high probability under standard Gaussian initialization? If so, please clarify. If not, how robust are your results to neurons initialized near decision boundaries?\n4) You mention extension to deeper networks as future work. Do you expect the two-phase picture (alignment then collapse) to hold in three-layer ReLU networks as well?\n5) Please clarify lemma 3: my current understanding is the authors bound the deviation between the true directional dynamics and their approximation at each moment in time. If my understanding is correct, then it says nothing yet about how the trajectory evolves due to that mismatch. It remains unclear whether these small local errors can accumulate into large global deviations, which could affect convergence to the correct attractor.", "limitations": "YES", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength: \n1) Technically sound and rigorous: The main theorems for both binary and multi-class NC are carefully stated, well-motivated, and the proofs build logically on prior implicit bias and dynamical systems literature.\n2) Novel theoretical contribution: This is, to my knowledge, the first work proving that NC emerges without explicit regularization in a shallow ReLU network trained with gradient flow. This advances understanding of the implicit bias of optimization.\n3) Clear comparison to unconstrained feature models (UFM): The paper carefully contrasts the geometry of NC in ReLU networks with that in UFM (e.g., direction collapse vs. point collapse, orthogonal frame vs. simplex ETF).\n4) Unification of implicit bias and NC theory: The authors show how early-time alignment and late-phase max-margin bias together explain NC emergence, offering a conceptually clean two-phase view of the dynamics.\n\nWeakness: \n1) The main weakness lies in the use of decoupled neuron dynamics approximations under cross-entropy loss. The paper assumes that neurons evolve independently during the alignment phase (Equations (7), (16), and Lemma 3), despite the fact that the softmax function globally couples the outputs of all neurons. This approximation underlies the entire alignment-phase analysis but is not rigorously justified. In particular: 1) No Grönwall or Lyapunov-type bound is provided to control the accumulation of $O(\\epsilon)$ errors over the alignment time scale $T = O(\\log(1/\\epsilon))$ 2) The transition from approximate dynamics to max-margin behavior is assumed to be seamless, but no stability guarantee is given. 3) In the multi-class case, the approximation of neuron dynamics as independent is especially problematic due to softmax-induced coupling and feature competition across classes.\n\nThese points do not invalidate the proofs, but they significantly obscure where the approximation is safe and where it may break down. Clarifying these issues would help broaden the audience and ensure robustness of the claims.\n\n2) Assumption 4 (semi-local initialization) plays a critical role in ensuring neurons align to all class attractors, but the paper provides no quantitative analysis of the attraction basins.", "was_revised": true, "final_justification": "This work is interesting and the author's responses are thorough", "rebuttal": "We thank the reviewer for the constructive feedback. We respond to your questions/concerns below:\n\n**Why neuron dynamics decouple** During the alignment phase (any time t prior to $T=O(\\log(1/\\epsilon))$), the neuron dynamics decouple as (16)(17) even with the presence of softmax: The neuron coupling is always through $\\hat{y}_i$, and Lemma 1 suggests that during alignment phase, $\\hat{y}_i\\simeq \\mathbf{1}/K$, making the dependence on other neurons negligible up to some $\\epsilon$ error. Moreover, this approximation error is well controlled throughout the alignment phase. Specifically, at initialization, the norm of the weights satisfies $\\lVert w_j(0)\\rVert^2=O(\\epsilon^2)$, then one can show that the norm only grows linearly. Then a Gronwall argument shows that for any t before $T=O(\\log(1/\\epsilon))$, the norm can only grow up to $\\lVert w_j(t)\\rVert^2=O(\\epsilon)$, which gives rise to Lemma 2. The Gronwall argument is used in line 874 in its proof. Finally, $\\lVert w_j(t)\\rVert^2$ is used to control the error as shown in Lemma 3, and all the subsequent results (Lemmas 4 and 5) on the dynamical behaviors of neurons have taken this error into account. \n\n**What happens in the transition from decouple dynamics to max-margin behavior?** We note that our results adopt two different analyses for two phases. For the alignment phase prior to $T^\\*$, the approximation of decoupled neuron dynamics remains valid; for the dynamic phase after $T^\\*$, there is no neuron decoupling, nor does the convergence analysis rely on an approximate decoupled dynamics. The asymptotic stability of GF, i.e., the limit point $\\theta/\\lVert\\theta\\rVert$ exists, is shown through prior work (Lyu and Li, 2019) with the residual errors fully taken into account, and our results in Proposition show its connection to NC. We will clarify the different analyses for the two phases in the revision.\n\n**Quantitative analysis of the attraction basins** As we have discussed in Section 4.2, ROAs of multiple stationary points of the decoupled neuron dynamics pose unique challenges (to our best knowledge, our paper discussed them first) in understanding the directional convergence of neurons in the alignment phase in multi-class problems, and our current analysis can only provide invariant subsets of these ROAs, as defined by Assumption 4. Admittedly, the invariant subsets are relatively small when compared with the actual ROAs, because random initialization does not satisfy Assumption 4, yet empirically, the MNIST experiments show that the alignment with the attractors from random initialization. The quantitative analysis of these ROAs, as well as the relaxation of Assumption 4, has broader importance in understanding the implicit bias of GF on shallow networks in multi-class problems and is left to future research.\n\n**Accuracy gap between SOTA** We thank the reviewer for the observation. Similar gap between SOTA exists for ResNet18 in the empirical results in our main paper, for which we believe the main source of the gap is the lack of data augumentation: we rerun the ResNet-CIFAR10 experiments with data agumentation and report the NC and Acc. of the trained network after 50 epochs in the following table, from which we see the Acc. improvement from agumentation, but less prominent NC (it is harder to collapse data under augmentation).\n\n| Setting | NC1 | NC2 | NC3 | Loss | Train Acc. | Test Acc. |\n|---|---|---|---|---|---|---|\n|w.o. Data aug, w. RMSNorm | 0.2118 | 0.4848 | 0.2991 | 0.0000 | 99.99% | 81.95% | \n|w. Data aug, w. RMSNorm |0.3564 |0.8906 |0.3769 |0.1111 |96.12% | 89.30% | \n\nFor ResNet50-CIFAR10, there was an additional issue of training instability: we used a modified ResNet50, for which the SGD with an initial step size of 0.1 sometimes diverges. This led us to use an initial step size of 0.01. Given that empirically a larger step size leads to better generalization, the small step size in our experiments with ResNet50 further enlarges the gap between SOTA. We will refine the experiments to better align with practice.  \n\n**When does Assumption 4 hold** Assumption 4 does not hold under random initialization. We will add this to the remark on the limitations of Assumption 4. We believe the analysis can be extended to all cases when the weight initialization does not exactly land on the boundary between two regions of attraction, as mentioned in Section 4.2, but showing the inter-class separation time $T^*$ exists requires additional technical efforts, and thus is left to future research. \n\n**How to extend to deep nets** We believe a similar two-phase analysis is possible for three-layer networks, given certain initialization and data assumptions. Existing work (Kumar and Haupt, 2025) has shown that weight alignment also happens at the early phase of GF when training deep networks with positive homogeneous activations, and likewise, max-margin bias also appears in the asymptotic phase (Ji and Telgarsky, 2020; Lyu and Li, 2019). We believe our analysis suggests a recipe for connecting these results to NC in deep networks in future works.\n\n**Does Lemma 3 account for error** As we have discussed in previous point, the error accumulates but well controlled until $T=O(\\log(1/\\epsilon))$, after which we no longer need to control the error as the analysis in the second phase does not require decoupled neuron dynamics.\n\nWe hope our rebuttal addresses your questions. If you have additional questions/concerns, please feel free to post them during the discussion phase. \n\nReferences:\n\nKumar and Haupt, Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations, TMLR, 2025\n\nJi and Telgarsky, Directional convergence and alignment in deep learning, NeurIPS, 2020\n\nLyu and Li, Gradient descent maximizes the margin of homogeneous neural networks, ICLR, 2019"}, {"review_id": "k3h2jtKspk", "reviewer": "Reviewer_ti2o", "summary": "The paper describes the phenomenon of neural collapse (NC) in the classification setting, according to which the feature vectors associated to each data point converge to the same direction in the last layer of the network. Going beyond previous works on the topic, the paper rigorously proves that NC happens, under some conditions on the data, even in the setting of two-layer neural networks with ReLU activation function without explicit regularization.\nThe paper fully characterizes the NC solution in the binary and multi-class classification scenarios and explain how it emerges from the implicit bias of gradient descent. \nSome simple numerical experiments validate the analytical results.", "questions": "- Can you give some intuition on why it is required that the neurons initializations are exactly balanced? The output weights do not change sign also if $||\\boldsymbol{w_{j0}}||^2-|\\boldsymbol{v_{j0}}|^2<0$ when the output is a single neuron.\n- How much does the result depend on the neurons' balance property? What if the network has a non-homogeneous activation function, with no rescaling symmetry and, therefore, no balance under GF?\n- How does the inclusion of biases change the picture? In these cases, they are usually implemented by adding an extra dimension to the input vector that always takes value 1. Does this affect the assumptions of the theorems, like orthogonal separability?\n- Does the analysis provide estimates on the time scale $T^*$ required to have NC, and possibly its scaling depending on the network's hyperparameters?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strenghts**\n- The setting of validity of the results is limited, but overall the paper provides an interesting step in the direction of understanding the NC phenomenon. In particular, the paper is able to rigorously prove the emergence of neural collapse for the cross-entropy loss without the need of regularization, a result which was previously observed empirically but not treated analytically.\n- The paper gives novel technical contributions, like the multi-class analysis, which may be useful for other works in this direction.\n- The paper is clear about its contributions and how they connect to known results in the literature. Up to page 4, it is easy to follow and provides a very clear exposition to the NC phenomenon.\n\n**Weaknesses**\n- The scope of the theoretical analysis is quite limited, as it requires the strong assumption of orthogonal separability and is concerned with two-layer networks trained with gradient flow.  It should be noted, however, that these are common assumptions in the literature for this kind of analysis.\n- The main weakness of the paper in its current form is its dense and technical writing, which is very hard to follow. After page 4, the paper focuses on the proofs of the main theorems. Due to the space constraints, however, a lot of information, notation and details are presented quickly, without taking the required space needed for a reader to process them. For these reasons the proofs, which occupy almost half of the paper, are not particularly instructive and to me mostly feel like they should be in the supplementary material. Moreover, the key steps in the proof of neural collapse in the binary classification case i.e. inter-class separation (Eq. 10) and the claim of line 231 where the implicit bias is involved, are proven in other works. While this is not an issue per se, it means that reading the proof doesn't help a lot in understanding how the result follows from the implicit bias, and thus provides little added value. Overall, I think that section 4 should be much more streamlined, focusing on intuition and leaving the most technical aspects for the supplementary material. \n\n**Minor issues / suggestions**\n- The notion of \"implicit bias\" may not be immediately familiar to every reader, and so it should be explained as early as possible. At the moment is taken for granted and so it is practically undefined. I would add a few sentences in the introduction after line 41 to give some intuition.\n- It would be useful to give very specific pointers (equation number) for the parts of the proofs which come from other works, so that it is easier to look them up. For instance in Eq. 7 and in the Claim of line 212.", "was_revised": true, "final_justification": "My main concern about the paper in its current form is the lack of clarity in the exposition of the proofs of the main results. The authors addressed this point in the rebuttal by outlining a plan on how to improve this aspect. If this is implemented as promised in the revised version, I think the paper would greatly benefit in terms of clarity and accessibility, thus deserving to be accepted.", "rebuttal": "We thank the reviewer for the constructive feedback. We respond to your questions/concerns below:\n\n**Section 4 is too technical** We thank the reviewer for the suggestions. We will, in Section 4.1, add high-level explanations to the proofs with illustrative figures, and also improve the appendix to better explain the technical aspects of the proofs. \n\n**Requirement on weight balancedness** Indeed, the analysis can be extended to the case when $|v_{j0}|>\\lVert w_{j0}\\rVert$. We will check how much editing is required in the proof to determine whether to modify the main statement or to add a remark.\n\n**Requirement on homogenous activations** We believe it is possible to extend the current analysis to networks with approximately homogenous activations. Specifically, we studied the dynamical behaviors of GF in different phases: alignment in the transient phase and max-margin bias in the asymptotic phase. For the alignment, it is mainly derived from the linearization of the network around the origin; thus, similar analysis can be done with non-homogeneous activations, albeit losing weight balancedness, potentially making the alignment dynamics harder to work with. For the max-margin, it requires positively homogeneous activations. Nonetheless, we note that recently there has been work on extension to approximately homogeneous activations (Cai et al., 2025). We are happy to add the discussion here to the appendix if needed.\n\n**Adding bias** We will add a discussion on the network bias term in the appendix. Specifically, since $\\sigma(\\langle w, x\\rangle +b)=\\sigma(\\langle [w;b],[x;1]\\rangle)$, adding a bias term effectively adds one homogenous coordinate to the entire dataset. Notably, homogenous coordinate increases data correlation: $\\langle [x_i;1],[x_j;1]\\rangle=\\langle x_i,x_j\\rangle +1$, thus it mainly affects the negative correlation between data points with different labels. In the case when $\\min_i \\|x_i\\|\\gg 1$, the orthogonal separability (among $[x_i;1]$) still holds with the bias term.\n\n**Time scale of $T^\\*$** We first note that $T^\\*$ is the time to achieve inter-class separation, but not necessarily NC. For binary problems, its scale is characterized in (Min et al., 2024), which mainly depends on the data separability $\\mu_s,\\mu_d$ and dataset size. For multi-class problems, our current Proposition 1 did not provide a scale for $T^*$, but we believe it should have a similar scale as the binary case. We will work on adding the scale to the revision. \n\nWe hope our rebuttal addresses your questions. If you have additional questions/concerns, please feel free to post them during the discussion phase. \n\nReferences:\n\nY. Cai, et al., Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks, ICML 2025\n\nH. Min, et al., Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization, ICLR 2024"}, {"review_id": "t5NSZCHcnq", "reviewer": "Reviewer_kUvC", "summary": "In this article, the authors analyze Neural Collapse (NC) for two-layer ReLU neural networks, optimized using gradient flow, in the context of a classification problem. Their analysis is conducted under certain assumptions on the weight initialization and the data.\n\nAssumption 1, referred to as orthogonal separability, pertains solely to the data and represents a clear limitation of the results presented in the article.\n\nAssumption 2 concerns only the initialization and appears to be relatively weak.\n\nAssumption 3 applies to binary classification and also seems weak when the data is spread out. However, due to the constraints imposed by Assumption 1, it becomes more restrictive. In this case, it requires a good initialization.\n\nAssumption 4 applies to multi-class problems, involves both the data and the initialization, and appears to be much stronger. If understood correctly, it requires that, at initialization, the activations are already essentially separated. Ideally, at initialization, each class $k$ should correspond to a group of neurons in the hidden layer that are activated only by examples from class $k$.\n\nUnder these assumptions, the authors show that gradient flow converges, and its limit satisfies Neural Collapse (NC).\n\nThe authors also present experimental results demonstrating that NC occurs:\n- on MNIST, for two-layer neural networks as well as ResNet-18 and ResNet-50, and\n- on CIFAR-10, for ResNet-18 and ResNet-50.", "questions": "- Footnote 1: I do not understand this remark. A simple dataset where $(x_i, y_i) \\in \\\\{(1, 1), (-1, -1)\\\\}$ for all $i \\in [n]$ satisfies equation (1) with $\\mu_s = \\mu_d = 1$.\nDoes your footnote imply that such a dataset cannot exist? If so, could you clarify why? Is there a reason to exclude such a dataset from your analysis?\n\n- Line 84: I believe that $V \\in \\mathbb{R}^{d_y \\times h}$. Could you confirm if this is correct?\n\n- Theorem 1: Theorem 1 would benefit from including the \"additional case-dependent assumptions on the data and initialization shape.\" Providing them explicitly would make the statement more formal and self-contained.\n\n- Equation (6): Please define the identity matrix $I$ in the notation section for clarity.\n\n- Footnote 2: Please define the acronym ETF. In the formula for $\\tilde{E}$, I believe it should be $K$, not $k$.\n\n- Equations below Line 221: There appears to be a typo. In the second equality, it should read: $\\forall i \\in I_-$ instead of $\\forall i \\in I_+$.\n\n- Claims in Section 4.1: Could you provide a reference to a specific proposition in the appendix where each claim is stated and proved?\n\n- Line 233: I may have missed it, but could you clarify what $u_+$ and $u_-$ refer to? Please define them or provide a pointer to their definitions. Similarly, could you provide references for $u_k$ and $\\tilde{e}_k$ in Proposition 2?\n\n- Line 329: Typo: \"pesudo\" should be corrected to \"pseudo\".", "limitations": "As already mentioned, the assumptions significantly limit the scope and practical relevance of the article. To better assess the extent of this limitation, it would be valuable to conduct an experiment testing whether, in ResNet-18 and ResNet-50, the data once mapped in the penultimate layer are orthogonally separated. For instance, the authors could report the empirical values of $\\mu_s$ and $\\mu_d$. The limitations could be better explained (see suggestions above).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The article studies an intriguing phenomenon known to occur in deep learning and believed to contribute to the success of neural networks in classification tasks. It is, therefore, an important topic of investigation. The conclusions of the theorems clearly explain Neural Collapse (NC). While I have not verified all the details, the parts I did check were correct.\n\nThe main drawback is that the hypotheses of the theorems are very strong and quite far from the settings in which NC has been observed empirically. Although this limitation is common in many theoretical works on deep learning, it is worth emphasizing that the article does not provide a clear path for extending its theoretical results to settings that would explain NC as it appears in practice.\n\nAnother drawback is related to readability: the assumptions would benefit from being illustrated with diagrams, examples, and explanatory comments.\n\nIn Proposition 1, the assumption $\\frac{X_{max}^2}{X_{min}^2} < (2 K -3) \\mu_d\\mu_s^2$\nappears to be strong. In particular, it becomes vacuous if $(2K - 3) \\mu_d \\mu_s^2 < 1$. This represents a significant limitation on the data, given that $\\mu_s < 1$ and $\\mu_d \\leq \\frac{1}{\\sqrt{K - 1}}$. For instance, when $K = 3$ and $\\mu_d = \\frac{1}{\\sqrt{K - 1}}$, one must have $\\mu_s \\geq \\sqrt{\\frac{\\sqrt{2}}{3}} \\simeq 0.69$. This implies that data from the same class must be strongly correlated. Such limitations should be better explained in the discussion of the assumptions.\n\nThe experiments confirm that NC occurs on standard datasets. However, they do not seem directly related to the theoretical statements. These could be enhanced—or even replaced—by experiments designed to test whether, in ResNet-18 and ResNet-50, the data in the penultimate layer are orthogonally separated. For example, one could report the values of the constants $\\mu_s$ and $\\mu_d$.", "was_revised": true, "final_justification": "The assumptions made in this article considerably limit its scope. For this reason, my recommendation is a borderline accept.", "rebuttal": "We thank the reviewer for the constructive feedback. We respond to your questions/concerns below:\n\n**Extension to deep nets** We have acknowledged the limitations of our results in the paper through multiple remarks. However, as the reviewer pointed out, the discussion on extensions to deep nets was not sufficiently detailed.. We will revise our conclusion section to elaborate more. Specifically, it is our belief that the key technical ideas (alignment dynamics in transient phase, and max-margin bias in asymptotic phase), whose connection to NC was not well explored in prior work, will be useful to study the convergence to NC in deep nets as both ideas appear in deep nets with positively homogenous activations (Kumar and Haupt, 2025; Ji and Telgarsky, 2020; Lyu and Li, 2019). Therefore, following the same high-level proof and utilizing these existing results on alignment and max-margin for deep nets might be the way to extend the current work to a more practical setting. \n\n**Assumption in Proposition 1** We will add a remark to explain the limitation of this assumption on the data correlations. Specifically, this assumption is required to show that the subsets of the parameter space defined in Assumption 4 are invariant under GF. We believe such an assumption is not needed in practice, but our limited understanding of the ROAs and their invariant subsets, as we have discussed from line 279 to line 289, has led to this additional technical condition to ensure directional convergence. Future research on better characterizations of the ROAs and their invariant sets will naturally relax or even potentially remove this requirement. \n\n**Features in ResNet** We clarify that the main purpose of our ResNet experiments is mostly to examine the role of the normalization layer in determining the NC level in deep networks. It is unlikely the features in ResNet would satisfy our separability assumption (in fact, our MNIST experiment already shows that it is only satisfied approximately for MNIST). Nonetheless, the fact that our results align with some of the empirical observations (RMSNorm improves NC) shows that results with the separability assumption are not uninformative for practice, and relaxing this assumption should lead to more practically relevant results.\n\n**Clarifying Footnote 1** Footnote 1 formally says that there exists no dataset satisfying equation 1 with some $0<\\mu_s\\leq 1$ and $\\mu_d>\\frac{1}{\\sqrt{K-1}}$, we will add its proof to the appendix. Note that the simple dataset the reviewer mentioned does not invalidate this statement, as this dataset satisfies equation 1 with $\\mu_d=1\\leq \\frac{1}{\\sqrt{K-1}}$ (note that $K=2$ here) \n\n**Claims in Section 4.1** The Claims in Section 4.1 are directly implied by existing results, which we have referenced at the beginning of the claims. We understand that this makes the paper not self-contained (which is also pointed out by Reviewer ti2o). We will, in Section 4.1, add high-level explanations to the proofs with illustrative figures, and in the appendix, explain how the claims are derived from previous results. \n\n**Definitions of u vectors** $u_+$ and $u_-$ vectors are max-margin vectors for positive-class and negative-class data, respectively, defined in Theorem 1, right after equation (4). In Theorem 1, we have stated two cases, and the \"case one\" is when $\\mathcal{K}=\\lbrace +,-\\rbrace$, and $u_k, \\mathcal{K}=\\lbrace +,-\\rbrace$ defined for this case gives rise to $u_+, u_-$\n\nLastly, we will fix the typos the reviewer mentioned (in particular, V is in $\\mathbb{R}^{d_y\\times h}$, as the reviewer pointed out), and add the definitions for identity matrix and acronym ETF (Equiangular Tight Frame).\n\nWe hope our rebuttal addresses your questions. If you have additional questions/concerns, please feel free to post them during the discussion phase. \n\nReferences:\n\nKumar and Haupt, Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations, TMLR, 2025\n\nJi and Telgarsky, Directional convergence and alignment in deep learning, NeurIPS, 2020\n\nLyu and Li, Gradient descent maximizes the margin of homogeneous neural networks, ICLR, 2019"}]

Author rebuttal:

Meta-review: This paper proves the emergence of Neural Collapse in shallow networks trained with gradient flow, under the assumption of orthogonally separable data. This improves upon previous results that assumed unconstrained last-hidden layer features, or required specific architectures. This paper brings tools from the convergence analysis of shallow networks to the Neural Collapse anlysis, which is relatively new, and might have other applications.

In agreement with the opinion of the reviewers, we are happy to accept this paper as a poster at NeurIPS.
Decision: Accept (poster)