Neural Collapse: A Review on Modelling Principles and Generalization
Authors: Vignesh Kothapalli
Venue: Accepted by TMLR ()

Abstract: Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural collapse essentially represents a state at which the within-class variability of final hidden layer outputs is infinitesimally small and their class means form a simplex equiangular tight frame. This simplifies the last layer behaviour to that of a nearest-class center decision rule. Despite the simplicity of this state, the dynamics and implications of reaching it are yet to be fully understood. In this work, we review the principles which aid in modelling neural collapse, followed by the implications of this state on generalization and transfer learning capabilities of neural networks. Finally, we conclude by discussing potential avenues and directions for future research.

Published in Transactions on Machine Learning Research (04/2023)
Neural Collapse: A Review on Modelling Principles and Generalization
Vignesh Kothapalli
1k2115@myu.ed
Courant Institute of Mathematical Sciences New York University
Reviewed on OpenReview: https: openreview. net/ forun? id= QTXocpAPPp
Abstract
Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural collapse essentially represents state at which the within-class variability of final hidden layer outputs is infinitesimally small and their class means form a simplex equiangular tight frame. This simplifies the last layer behaviour to that of nearest-class center decision rule. Despite the simplicity of this state, the dynamics and implications of reaching it are yet to be fully understood. In this work, we review the principles which aid in modelling neural collapse, followed by the implications of this state on generalization and transfer learning capabilities of neural networks. Finally, we conclude by discussing potential avenues and directions for future rescarch.
1 Introduction
With unprecedented growth in the size of neural networks billions and trillions of parameters, their capabilities seem to be limitless in the modern era (Liu et al. 2019 Brown al.) 2020 Thoppilan et al. 2022 Chowdhery et al., 2022 Yu et al. 2022 Zhai et al. 2022). Yet, their generalization capabilition continue to evade our underslanding of deep learning techniques based on model complexily (Hu et al. 2021). One can aim to reason about these overparameterized networks by tracking and analysing the feature learning process over time, or by understanding simplified thcoretical models. Although the thcoretical foundations Gooddellow et al. 2016 He & Tao 2020) are being steadily improved, the role of novel empirical analysis is paramount. importance to speed up the process. Tn our work, we take a principled approach review and analyse one such intriguing empirical phenomenon called "Neural Collapse" (Papyan et al. 2220).. NC essentially defines four inter-related characteristics of the final and penultimate layers in deep classifier neural networks when trained beyond zero classification error (see figure T
NC1: Collapse of variability: For data samples belonging to the same class, their final hidden layer (i.e the penultimate layer) features concentrate around their class mean. Thus, the variability intra-class features during training lost as they collapse to point.
NC2: Preference towards a simpler equiangulor tight frame: The class means of the penultimate layer features tend to form simplex equiangular tight frame (simplex ETF). A simplex ETF is symmetric structure whose vertices lie on hyper-sphere, are linearly separable and are placed at the maximum possible distance from cach other.
NC3: Self-dual alignment: The columns of the last layer linear classifier matrix also form a simplex ETF in their dual vector space and converge to the simplex ETF (up to rescaling) of the penultimate layer features.
NC4: Choose the nearest class mean: When a test point is to be classified, the last-layer classifier essentially acts nearest. (train)-class mean decision rule w.rs penultimate layer features.

---

Published in Transactions on Machine Learning Research (04/2023)
Figure 1: Evolution of penultimate layer outputs of a VGG13 neural network when trained on the CIFAR10 dataset with randomly selected classes. The green balls represent the coordinates of a simplex ETF, rod ball-and-sticks represent the final layer classifier, blue ball-and-sticks represent the class means and the small blue balls represent the last hidden layer (penultimate layer) activations. Image credit: Papyan et al. (2020)
Intuitively, these properties portray the tendency of the network to maximally separate class features while minimizing the separation within them. The benefits of such properties extend not only to generalization
but to transfer learning and adversarial robustnoss as well (Liu et al. 2016 Wen et al. 2016: Sokolić et al.
2017
Liu et al.)
2017 Cisse et al. 2017: Snell et al.] 2017 Elsayed et al.) 2018 Waan et al.) 2018 Soudry
et al.
2018
Jiang et al. 2018 Deng et al. 2019 Sun et al. 2020). On a related note, prior work by Giryes
showed that even in a shallow neural network with Gaussian random weights, the angle between intra-class features shrinks faster than the inter-class fcatures. The variability collapse (NC1) property an extreme version of this shrinking process when the network is deep enough. From theoretical standpoint, by leveraging scattering transform based convolution operators fundamental results of group invariant scattering by Mallat (0012); Bruna & Mallati 2011 Sifre & Mallat 2013 show the tendency of the network to reduce the scattering variance as the number of layers increases.
et al.
2016
alternative to trainable filters, the
Furthermore, constraining the network weights to tight-frames has been shown to improve the adversarial robustness and training efficiency of the networks. For instance, Cisse et al. (2017) enforce the hidden layer weights of wide residual networks (Wide ResNet) (Zagoruyko Komodakis 2016) to be parseval tight frames (Kovačević et al. 2008) and show improved robustness to advessarial examples on CIFAR1O and SVHN data sets. Unlike the parseval networks which enforce structural constraints on all the hidden layers, the line of work by Peeriic et al. (2019); Pernici et al.; 2021 fixes the final layer classifier regular polytope i.e, either a simplex, cube or an orthoplex, (Coxeter 1973 and observe similar performance to learnable baselines on image classification tasks. Additionally, there seems to be an interesting, yet unexplored connection between the low-rank nature of these symmetric structures (such as a simplex ETF) and the 'rank-collapse" phase of training. Martin & Mahoney (2021) descríbe the 'rank-collapse" phase as a state of over-regularivation during training, where the empirical spectral density of the weight matrices is dominated by a few large eigenvalues. On similar note, the spectrum of the hessian of training loss was also shown to exhibit outlier eigenvalues which inherently represent. the class information in image classification settings (Sagun et. al.
2016 2017
Wu et al. 2017: Papyan 2018 Ghorbani et al. 2019).
The motivation behind reaching this collapsed state during TPT scems comter-intuitive as one would prefer to avoid over-fitting on the training data. However, recent observations based on "double-descent" by Belkin 2019 2020) and the benign effects of interpolating on the training data with over-parameterizee networks MMa et al. 2018 Belkin et al., 2018: Allen-Zhu Zhu et al.) 2019 Feldman] 2020 Papyan al.] 2020: Bartlett et al. 2020 Zhang et al. 2021a) provide sufficient justification for further experimentation and analysis in this regime. To this end, prior work by Cohen et al. (2018) showed that the behaviour of k- Nearest Neighbor(k-NN) and deep classifier neural networks tend to be similar upon memorization. In fact, their experiments on the KL divergence between k-NN and Wide ResNet. classifier outputs provide evidence for the emergence of NC4 property.
ct al.
The differentiaaing factor of NC from prior efforts lies in the fact that canonical deep classifier networks nat- wrally exhibit all four properties without explicit structural constraints during training. Questions pertaining
2

---

Published in Transactions on Machine Learning Research (04/2023)
to theoretical explanations, modelling techniques and implications of NC naturally arise in this situation and we aim to address them in this work. The rest of the paper is organized as follows: section details the preliminaries and setup that we use throughout the paper, section 3 introduces the modelling principles and reviews community efforts in a bottom-up fashion, section presents the implications of NC on generalization and transfer learning, followed by lakeaways and potential research directions section
1.1 Contributions
We review and analyse NC modelling techniques based on the principles of unconstrained features and local elasticity, which is currently missing in the literature.
We review and analyse the implications of NC on the generalization and transfer learning capa- bilities of deep classifier neural networks. Through these discussions, we hope to clarify certain misconceptions regarding NC on test data and provide directions for future efforts.
2 Preliminaries
In this paper, the term "network" refers to neural network. Architectural details such depth, presence of convolution layers, residual connections etc are omitted for brevity and will be mentioned as per the context. Since NC analysis using recurrent neural networks or its variants is absent in the literature, we omit such assumptions in this paper. We primarily focus on classification settings and employ a common notation scheme for all modelling techniques.
2.1 Data
Let's consider a data set X E RaxN for which is assumed to be the underlying probability distribution. is associated with labels [K] = =1,........ K), where K e N. The label function & : Rdd - (ei,..., ek) E R is a P measurable ground-truth provider that maps an input its respective one-hot vector. Formally, by representing the tt data point of the clttsset X as H Rd Vi [N] and with slight abuse of notation, the 8th data point of the 1tth class as a. E Rd, the function call ((x)) outputs the ground truth vector ex- For notational convenience, we overload {(x{)) Eaz. We define set C (fex)) as the set of data points belonging to class l E [K], with Pc, their class conditional distribution. When indexing on the data is not necessary, simply use RR to represent random data point. We represent the size of each class as ni, & [K] where > R& = N. For the majority of this paper, we assume balanced class sotting and consider n N/K for convenience. Additionally, ||-I|= denotes the frobenius norm, ||(r, s))l = ||r|| + ||s|]² ()) denotes the inner product, tr(-) denotes the trace of matrix and denotes the pscudo-inverse.
2.2 Setup
A classifier network hr Rd R belonging to function class H. can be formulated as a composition of L, layers followed by linear function. Formally, hr az, o ST.-1 = 0r, 91.-1 9r-22 9 where 9. m + 1T Vi [L are parametric layers of the network, ft-1 9L-1 9L-2 91 is the function obtained by composing L - layers and az R - Rm is the final layer lincar function. For better readability, the number of layers L is implicitly assumed and the sub-scripts are dropped. This gives us := h4.0 := o := fL-1 and simplifies mp = d, mn = K. We also consider m := mn-1 for the majority of the analysis that follows. As we will be dealing with data and label matrices, we formulate the matrix representation of a network as:
H = AF AF+b +
(I)
Where H E KNN is the network output matrix, F E R the penultimate layer feature matrix, A E Kxm is the final layer weight matrix and b EK is the final layer bias vector. We represent the outputs of Rd 4 k over x, E R E [N] as columns of H E R outputs of f Rd - R over m; Rdd, E [N] columns of F E R and treat the one-hot label vectors HEIN) the columns
3

---

Published in Transactions on Machine Learning Research (04/2023)
of label matrix Y. For simplicity, consider the ordered versions of A = [a1, ag,.aa,, aax]' R k'mm with rows E "R vk E [ F= [f..1... fi.m:-.. _K.n] RRxN with penultimate layer features for af given by the column fK.i E R" H [h1,1,".. hin. k..] E RKxN with network output for x given by the column hg E R and consider label matrix as a Kronecker product matrix Y = Ix & 1, E RR To measure the performance of nelwork h, we use a generic loss function & RK x R + [0, oo) and define population risk functional R: H -> [0, 00) as:
R(h)= = / (()())
(2)
The population risk can be approximated by the empirical risk on data X, leading to the ERM problem:
mig R.(h) N Σ ((h(a)))
E
(Afrr + b,
(3)
hEH
Kn k=1
Most the community efforts that we review in the following sections focus on the theoretical aspocts of modelling neural collapse. From an empirical viewpoint, we observed that most of them employ the SGD optimizer for the ERM problem to validate the theory. The experimental details will be provided per context but the reader can assume SGD with momentum as the default choice of optimizer unless specified otherwise, An extended set of notations for following the theory available in Appendix A.1
2.3 Neural collapse
When a sufficiently expressive network h is trained on X to minimize R(h), a zero training error point is reached when the classification error reaches 0. The network enters TPT when trained beyond this point and exhibits intriguing structural properties as follows:
Collupse of Variubilily: For all classes [K] and data points E [n] within a class, the penultimate layer features fa.s collapse their class means μy = 1 Ei= fk,t We consider the within class covariance E mmxm global mean μc K Σ P E R" and between class covariance ΣB = t ΣK ((us - (G) (G)T) E Rmxm to measure the variability collapse.
NC1:
Σw Kn D ((fr, (fR, T)
Empirical metric:
NC1 == - tr
(4)
Preference towards simplez ETF; The re-centered class means WG- Vk [K] are equidistant from each other: || - Pallg l| - pGllz for every k, /' E [K] and by concatenating и4-Pa ER, Vk E [K]
NC2:
||a=-
to form a matrix M E I Kxm, M now represents simplex ETF such that:
K K-1 IK
1
MM =
K - 1x 1x1k
(5)
(1 PG. px maa
cpsk PG, pi mc)=
Vk R [K]
(6)
|m4 Hall2 || HGll2
K
Empirical metric:
MMT
R 1x1k)|
1
NC2
IK
(7)
[M
VK-
NC3: Self-dual alignment: The last-layer classifier A is in alignment with the simplex ETF of M (up to rescaling) as:
A
M
A P
M FF
4

---

Published in Transactions on Machine Learning Research (04/2023)
Empirical metric:
AM
1
NC3
IK K
x
(8)
|MM
1|
VK
NC4:
Choose the nearest class mean: for any new test point Litest, the classification result is determined by: arg [i]] (Itest) px During training, one can track this property on X as sanity check. Empirical metric:
K
NC4: = Kn
[arg max SR,) + be) # arg min [[fk - Hell2) cE[K]
(9)
cE[K]
Where I: (True, False] {0, 1) is the indicator function and bg R is the ct element of the bias vector. The metric essentially represents the fraction of misclassified data points using the nearest class center (NCC) rule on the penultimate layer features. Based on this setup, consider a network to be collapsed if the empirical metries NC1-4 - O. Without loss of generality, when we consider a network to exhibit NC1, it means that empirical measure NC1 + 0. The same holds for NC2-4. In the following sections, we review recent efforts understanding the desirability, dynamics of occurrence and implications of these properties.
3 A Principled Modelling Approach
In this section, we focus on the principles of Unconstruined Features" and "Local Elasticity" to model NC. The " "Unconstrained Features Model (UFM)" analyzes the ideal values of F,A, b for perfect classification and the training dynamics that lead to them. This line of analysis assumes that and disconnected from the previous layers, including input. To the contrary, the "Locol Elasticity (LE) based model aims capture the gradual separation of class features using stochastic diffcrential equations (SDE) and similarity kernels. Additionally, this approach imitates the training dynamics of the network in data-dependent fashion. Although the principle of unconstrained features has been the relatively popular approach, the earlier efforts by Wojtowytsch et al. (2020b; Lu & Steinerberger (2020); Ergen & Pilanci (2021)
freely optimizable
did not explicitly use the UFM terminology. On the other hand, Mixon et al. (200 and Fang et al. (2021)
formalized and termed this approach as UFM and "(N-)Tayer-Peet Peeled Model (I.PM)* respectively" Thus,
avoid confusion due to terminology in the presentation of idcas, we stick
UFM for the rest the paper.
3.1 Networks with "Unconstrained Features"
The unconstrained features model builds on the expressivity assumption of function class H and attempts to explain NC w.r.t ideal geometries and training dynamics. We consider a network to be expressive enough for X it achieves perfect classification on X. Under this assumption, the penultimate layer is disconnected from the previous layers and treated as free optimization variables during training (see figure 2 Since the last two layers of canonical classifier networks are fully seunected/dense UFM analysis as well. To this end, we study the properties of (F, b) under various settings pertaining to: loss functions, regularization, and normalization to derive insights on their collapse propertics.
assume this to be the case for
3.1.1 Role of cross-entropy loss without regularization
A note on desirability: Prior to analysing the NC properties, we start by briefly reviewing the work of Wojtowytsch et al.) (2020) which analyses the idcal outputs of h that lead to minimal risk. By collapsing the network outputs to single point based their class, and repeating it for all classes k [K], we gel:
2k = 1 / h((a' (Pa''')(da))
(10)
Here indicates the layer features (from the end of the network) which can freely optimized. Additionally, that the UFM and extended version by Tirer Bruna (2022) now be considered the 1-I.PM and 2.I variants.
5

---

Published in Transactions on Machine Learning Research (04/2023)
Peeled layers
Peeled layers
VER
V
L
L-1
L-1
(a)
CNN UFM
(b) MLP UFM
Figure 2: From left to right, illustrate UFMs corresponding to expressive Convolutional Neural Network (CNN) and MLP respectively. The shaded regions in both plots pertain to the first. L-2 layers which are 'peeled away' from the last two layers. Here θ = (F, A, b) indicates the trainable parameters and VeR indicates the gradients of the empirical risk w.r.t @ that are backpropagated. Under the expressivity assumption, observe that the nature of the first L-2 2 layers is impertinent and allows the UFM to encompass variety of network architectures for analysis.
Without loss of generality, if we consider a network h E H to exist such that h(æ)) =k. Vo E Cke then Wojtowytsch et al. (2020) showed that R(h) R(h) when & loe:
exp((h(z), Ex))
(cr(((a)),
(11)
1== exp (()((h(()),e))))
Thus, indicating the desirability of variance collapse for the network outputs (refer Appendix A.2 for further details on this result). Note that for sufficiently expressive function classes which are scale-invariant, if (h(a)) Ex) > max (h())),)) (i.e in TPT), then limx R(hh) = 0. Out of these infinitely many c.EEE)
possible solutions, one can assume norm bounded H and focus on minimizer results whose structure can be analysed. Based on this assumption, let's consider H to be expressive class of functions from the input space to the Euclidean ball of radius R and center at the origin: Bx(0)= E R² The empirical risk now be minimized over the class means when variance collapse of network outputs occurs. Observe that when:
(-log log exp((h(at)) ex)) exp((((at).ea)))
h (x
arg min (xt))BBx())
Σ exp((1(a)t))))))) e3,))
(12)
The Lagrange multiplier equations for this minimization problem lead to:
K K
R
K
h" (x(x))
Reg
ej
VK(K
Thus, forming simplex ETF of the collapsed network outputs. Note that the bias b led to re-centering of Bn and didn't t affect the simplex ETF formation (Wojtowytsch et al.) 2020). In concurrent line of work, Lu & Steinerberger (2022) showed related results for (cg, where the collapsed outputs of the network h satisfy the angle property given in equation t and indicate the desirability of forming a simplex ETF.
The transformation from the penultimate layer the final layer is affine w.r.t A, as per equation | Now, by noting that (A +b)-1(z)) is an m K dimensional affine subspace of RT is desirable for collapse occur in the penultimate layer features when is Porormm constrained. This is due to the strictly convex

---

Published in Transactions on Machine Learning Research (04/2023)
nature of [--norm (1 p < 00) which leads to unique mapping from the collapsed final layer outputs to collapsed penultimate layer features. See section 4 in Wojtowytsch et al. (2020) for detailed proofs and additional analysis of this result.
Gradient Flow: With the desirability of collapse being established w.r.t population risk, a question that naturally arises whether the dynamics of the optimizers in ERM settings lend towards such states, especially without the norm-coww conn-constraints which leads to non- unique minimizers? The work by Ji et al. (2021) addresses this question by analysing the gradient flow@of of the empirical risk with cross-entropy loss under the zero bias assumption. We present the key results from their analysis of the gradient flow and its relations with max-margin separation problem below. Firstly, observe that the empirical risk R with & = fce leads to ERM with Reg as:
K
explaz fk.i) j= E exp(a) smri)
min
RC F, A) F.A
=
log
(13)
N k=1
1=1
This leads to the gradient flow formulation as follows:
dF(tt
ORCE [((t).
dl
OF
(14)
(A ORCE [(t), A(t)
dt
0A
Where F(t), A(t) are indexed by time of the gradient flow. Formally, by defining the margin of data point ot and the associated penultimate layer feature fE.S (k, A) == a fk. max a fk,s, then reaching TPT indicates that (F, A) 2 O, vk E [ E [n], ie, the features can be perfectly separated. In such setting, Ji et al. ( proved that, if (F((), A(t) evolve as per the gradient flow of equation 14 then F())
any limit point of the form: ((F(1, Á()) ( (A())
A(t)
)} is along the direction of
FFFE VIAOR FOD
an (c, appproxxiee Karush- Kuhn-Tucker (KKT) Gordon & Tibshirani 2012 point with 6, - 0, of the
following minimum-norn separation problem:
1
min | A.F 2
|
2
(15)
s.t a fr. fRA - aa fk. 2 k $ [K] [n]
Now, by considering Qmin (F, AF ==
min ke(K],jeml
Gh (F A) as the margin of data set X, which is bounded by
LI 2(K-1)/n Ji et al. 2021 show that the maximum separation is attained when |||||
Gmin (F, AF <
||F||p and AF satisfy the NC properties. Finally, to show that the approximate KKT points are indeed
the desired global minima, it sufficient to show that the separation problem of equation 15 satisfics the
Mangasarian--Frooviiz Constraint Qualification (MFCQ) (Mangasarian & Fromovilz) 1967); refer Dutta
et al. (2013) and theorem 3.1, 3.2 in Ji et al. (2021) for detailed proofs. Although this line of analysis focuses NC properties of the max-margin solutions, similar proof sketches were employed by Nacson et al. (0019
Lyu & Li (2019) to study the implicit bias of gradient descent in homogeneous neural networks.
Loss Landscape: The non-convex nature of the risk in equation 13 can result in KKT points which are not global minimizers. (F, A) don't satisfy NC properties or || + [[Fl) then a direction exists in the tangent space of (F, A) that leads to lower RAE values. Formally, by defining the tangent space of (F, A) as (AF E R AA R Kxmm ir(FTT (.FFFAFFF +F)++r(AA AA) 0}, then Ro (F A+ŌAA) < &R A), for constant 8maz 0 and VO < < 6man- This result by Ji et al (021 b
proved using second-order 13 (without the 1/N scaling factor) and analysing the cigenvector
analysis of the empirical risk in equation corresponding to a negative eigenvalue of the resulting Riemannian hessian mairix.
Gradieen flow can be treated gradient descent with infinitesimal step sizes Chizat Rach 2018
Du
2018

---

Published in Transactions on Machine Learning Research (04/2023)
These theoretical observations pertaining to gradient flow were empirically validated by Ji et al. (2021 using ResNet 18 and VGG13 networks to classify CIFAR1O, MNIST, KMNIST and FashionMNIST data sets. SGD with momentum 0.3, and learning rate 0.01 was employed as the optimizer. Thus, the implicit regularization due to cross-cntropy seems to be sufficient for converging to NC solutions.
3.1.2 Role of (mean) squared error without regularization
Following the cross-entropy case, let's consider the squared error & = tSE and the ERM formulated as:
min F.A.b
+b 11 YI
(16)
Where IN RN is the all ones vector. Unlike cross-entropy, it convenient to deal with matrices instead of individual feature vectors for analysing the squared setting. We consider the squared error in variety of NLP and vision-based classification tasks (Janocha Hui & Belkin 2020 Demirkaya et al. 2020
analysis due to its empirical effectiveness
& Czarnecki 2017
Gradient Flow: In this section, we review the NC properties of squared error minimizers based on the gradient flow analysis presented in a recent effort by Mixon et al. (2020). With near 0 initialization of (F, b,, Mixon el al. (2020) observed the following 'Strong Neural Collapse (SNC)' properties of the minimizers follows:
1 K 1x11)
SNC1: AAT Vn(lk
SNC2: F (A)
(A
(17)
Vn
1
SNC3 6 1K
K
Where n = N/K (as per setup). These properties are called 'strong' as the standard NC properties can derived from them. For instance, observe from SNC2 that μa v ag satisfics NC1. Similarly, from SNC1 we can deduce that:
AA = Vn(Ix [ = Vnl1 =
= AA I
This result leads to:
K
1
1
1
pa mall2
a
aj
ak
A
1k
aj
j=1
V
nn
Vnk
2
VT
Where
ak 2 is the thhh diagonal element of AAT and is equal to J. (1 k).
Thus, when A is
vm
n
normalized, we can from SNC1 that:
(AA) ke VM(IK 1 [1 -K-IIK K K-1
K K
a aj
1 1x1
|a
|all
vn(1
Vn(1- +)
K
kt
Where (AAT) Jat indicates the (k;,) 1)th element of AAT resulting in the formation of simplex ETF by A. Now, to understand how these properties were obtained, note that the gradient flow equation with e (F. A) and the respective derivatives is given by:
O'(1) (RsE(O(t)
((()) = AT (AF + tb1 - -Y)
(18)
(AF + -bi FYFFT
(oRT)((((t) (AF + Y)ln
8

---

Published in Transactions on Machine Learning Research (04/2023)
The optimal value of b can be partially decoupled from F, A when they are assumed to be small. Thus, the resultant ODE of the parameters:
(t) - A(t)
A(t))((()))))))))) (b(t)IN -Y; (1()) (6(1)) F)F(!), b(()) - (b(1)I Y)IN
with initial conditions F(0)= Fo; (() = Ab,b(0) 0 has solution satisfying:
(F(1), ev T Fo; An) Sexvi ||, (Fo, Ao)l b(t) (-
(1-e
1K 1K, 22 0 (19)
Where A)ll 2 + | as per setup and II, is the orthogonal projection onto the subspace:
1 J/ (A 1m) 1 =0
= (F.A):F=
vn
For a detailed proof, see theorem 2 in Mixon et al. 20220). This result implies that, during the initial stages of the gradient flow, large component of Fo, An resides in subspace T while b tends towards the span (1x) Thus, the initial trajectory of O lies along the subspace S given by:
s= (F. F=
1 (A 1,) 1 = =0,b E span [xk
(20)
To this end, when lies along S, the risk modifies into:
REE S,,FF,AB. A,b)
1 +
AF
b
Y
- - A -
1
(A01.)) - Y 2
2
Vn
1
1
A(
AA 2 1,) 1n) (Ix - K
1 + 1x 01
2
Vni
K
AAT - - K
1
+ (b
2
vn
K
2
N
2
AAT Vn(Ik
K
1
+ 2
K
K
F
Thus, showing that (F, A. b) satisfying SNC properties are indeed the minimizers of Rsg. The last equality is valid since the two terms orthogonal when θ lics along S. The empirical sctup to verify this behaviour
is simple. For some choice of N,N,m, one can randomly initialize Au. Fo and set b 0, Y Ix @ I This setup sufficient for performing gradient descent w.r.t R&E and tracking the NC properties of weights and features across steps/iterations. Such an empirical analysis by Mixon et al. (2020) showed that SNC properties are highly sensitive to initialization of (Fo, Ao) i.e, by defining the SNC errors as:
8SNCi = LAA vn(lk- 11
OSNC2 = . - (A (1481.)7// 1T) p = ||b 1 1
J
Figure X illustrates SNC errors that are orders of magnitude higher as initialization moves away from Furthermore, Mixon et al. (2020) also confirm that, as Foll - ||Aoll tend towards 0, the entire trajectory of gradient descent lends to stay along S. The theoretical analysis of the full trajectory behaviour, especially pertaining to the implicit bias of gradient descent towards NC solutions is still lacking and open for research. We show in Appendur[A.. that the mean squared error can be theoretically analysed in the same fashion, and show resemblance to the above results.
Loss Landscape: As the UFM can be considered as a linear neural network with optimizable inputs, the existing work by Baldi & Hornik (1989); Saxe et al. (2013): Kawaguchi (2016); Froeman Bruna (2016) is relevant to our study. Based on the key results of these efforts, it can be shown under mild assumptions that the landscape of squared error loss for linear networks contains critical points which are either global minima
9

---

Published in Transactions on Machine Learning Research (04/2023)
12"4
1=10
1010
10
10"1
10
12"1
1011
39
39
12-4
10
10115
g 10
2008
3900
0000
6000
1668
(a) SuCI steps
(b) SNC2 steps
(c)
Sscs vs steps
Figure 3: (Left-right) Visualization of òsNC1, 6SNC2, oSNCS with initialization | Follp c, bu
0, K = 3, N 9, m = 15 vs gradient descent steps on Rsg. Image credit: Mixon et al. (2020)
or strict saddles with negative curvature. Although the landscape benign, note that we always need some randomness to escape these saddle points. Due this reason, the vanilla gradient descent which doesn't 't include randomness in its updates may get stuck in one, which believe is a possible reason for large SNC errors in figure Similar observations were made by Ji et al. (2021) for RoE and gradient ceeccntt.
3.1.3 Role of cross-entropy loss with regularization
Our analysis till now has been limited to simplified risk formulations. In this section, we move closer practical settings by incorporating weight and feature regularizations.
Lower bound: As the empirical risk is dependent on F, P,A,,, the regularized version of ERM with RCE can be formulated as follows:
(cE(Afr; + bb) + AA
Ab
min RC (F A,b) F.A.b
|||
IF
|
(21)
N
2
Where λA, do > 0 are penalty teriss. To this end, Zhu et al. 2021 lower bound Ro (F, A,A,) by:
ROE (F,
1
(CE(Afe + bx,
Aa |
+
AF
F 2
Ab
|
N
(22)
|||
AA ndF
+ + A
(1+ ci)(K
Where 61 > C2 c1+1 (og((1cc:)(t 1)) Ites log( Lta) ) and equality holds true when F, A satisfy NC properties and ||||| finite. Zhu al. (2021) arrive at this result by expanding the cross-entropy formu-
lation, identifying this lower bound and showing that it can be achieved when F, A satisfy NC properties.
Loss landscape: To analyse the loss landscape, we summarize the approach of Zhu et al. 2021 which leverages the presence of regularizing terms to form a connection between the non-convex problem in equa- tion 21 and a convex program as follows:
n min RoE (Z, = ΣΣ 2.b
lcr Zz, +b, + VAAdE Z11, + d |bb
(23)
Where Z = AF E RKX and I1-11. denotes the nuclear norm. The validity of this connection can be understood from the following result by Zhu et al. [002]] (with similar results in Srebro (004)); Haeffele &
Vidal 2015 given as:
min AA |
AF F 2
VAA [|A|2-
AF | = VAAdE Z +
+
VAAdE
min
(24)
AF=2
2
AF=2 2/Ar
XA
10

---

Published in Transactions on Machine Learning Research (04/2023)
By exploiting this connection with the convex program, Zhu et al. (2021) find the global minimizers of ReEr equation 23 and show that they provide lower bound for Rogr (F, A, b). Formally, if (Z., b.) is the global minimizer of RCEr Z, b), then Reg (Zx, b.) S (F, b), and this optimal state can be transferred to the non-convex RCE- (F, A, For the sake of conciseness, we directly state the result of lemma C.4 in Zhu et al. (2021) that, any critical point (F,A, b) of equation 21 satisfying:
AF N
1
ΣΣ k=
n cE (AEEAf +b,ex) (,)(A)(stb)ess))
SvinAr
(25)
is a global minimizer of ReEr with Z AF. To this end, Zhu al. 2021 classify the critical points of RCEE(F. A, b) into two disjoint subsets as follows:
C - F, F,A,b: VARCE (,b, - VERCE
A,b) - VORCE A, 0
C: G=-on F, b,
VZ=AF
N
le (cf(A((i+b,e)))
VAAdF
(26)
K
C2 = -2 F, b;b:
VZ=AF
ΣΣ
VAAdF
k=
Note that points in Cj already satisfy the global minima conditions based on equation 25 assumption of m > K is needed to create a negative curvature direction for the hessian of RoEr as follows:
For C2 stronger
(-(((w)
A AF
1/4
AAA AF
1/4 uww
A A=(
we
(27)
where u E R.K. E N are the left and right singular vectors corresponding to the largest singular value of V2 AZ=AF(( N E Ei= (AE(Afk + b, ex)) and w E R" a non-zero vector such that Aw = 0. Observe
that m > is necessary to obtain a in the null-space of (sce theorem 3.2 in Zhu al. (2021)) ). Thus, stochastic optimizers can escape these strict saddle points along A and reach global minima that satisfy NC.
3.1.4 Role of (mean) squared error with regularization
Lower bound: similar to cross-cntropy, we define the ERM for MSE with regularization follows:
min RMSSE,FFAA, 2N |F +IJ YI + XA A F.A.b
XF 2
A
F
|
(28)
2
Where Xa, Ar, do 0 are penalty terms. In series of recent. efforts, Han et. al. (021 analysed RMSE (F,A, A, when dF = (with bias b concatenated A) and Tirer & Bruna 2022 analysed the
"bias-free' (b 0), 'unregularized- bias' (A, = 0) cases. For simplicity, we consider the b 0 case and present
the lower bound given by Tirer Bruna 2022 based on Jensen's 's inequality and strict convexity of ||-1 as:
RMSE (F
1 AF
Y
F +
AA
+
XF
|
2N
2
F
2
K XF
1
Af i
1
XA
||||
+
Σ n1 k=
2Kn
2
n
(29)
2Kn
1
K
A
Σ
fa.i
+
XA
2
| 7
+
A
2
K
k
isl
k=
A=
11

---

Published in Transactions on Machine Learning Research (04/2023)
Where the final equality holds when NC1 is satisfied, i.c fk.1 = fk.n: leading to F == F & 1 Here F E mRm is matrix with class feature means fx = nn Ei= Ski Vk E [K] as columns. Under the assumptiin of balanced classes, Tirer & Bruna (022)) consider VERMSER (F, A) V VARMEEE A and obtain a closed form representation of A = F (FF' + +Kdalm)- This formulation simplifies RMSE to solely depend on F, with the minimizer characterized by flat spectrum: F F Ix- The tight-frame obtained in this analysis is not simplex ETF but an orthogonal frame. However, by centering the columns of F around their mean fo == R Ek=n fkr we obtain the matrix: F felk which is indeed simplex ETF. Note that k fG are essentially Mk: MG (as per setup). Additionally, Tirer & Bruna (2022) showed ag PG. Thus, the ideal bias turns out to be the global mean subtructor that was necessary for F to form a simplex ETF. A similar observation was made by Han et al. (2021) in their bias concatenated setup.
that when b + 0, > 0, the closed form of bias for each class is bjg
Loss landscape: Although the results by Saxe et al. (013 Kawaguchi (2016) Freeman & Bruna (2016)
have been influential, they don't 't deal with regularized setlings for squared error. To this end, it is essential to characterize the deviations of critical points from global minima or strict saddles when regularization is
introduced. By considering AF = A = 0 and small values of AA - 0+, Taghvaci et al.] (2017) modols this regularized ERM problem for linear networks as an optimal control problem (Farotimi et al. 1991) and show that not all local minimizers are global minimizers in the presence of regularization (see Mehta et al. (202T)
for an empirical analysis). However, a second-order analysis of this regularized landscape is yet to be fully studied, espocially the nature of critical points and their NC properties.
3.1.5 Does normalization facilitate collapse?
Gradient flow perspective: In the gradient flow analysis of the squared error without regularization, recall that it was necessary for (F, A) to lie along the sub-space S (in equation 200 to exhibit NC. In the regularized squared error setting with = Ap = Han et al.] (2021 draw similar yet rigorous conclusions by decomposing the mean squared error into terms that depend on the lcast-squares optimal value of A (which is a function of F), and the ones that capture the deviation of A from this optimal value. The terms which come under the former category are of particular interest. Formally, when A is set to the least squares optimal value ALS, let F- F
Pclyy F-Pcll E R M [M] (M1-M,,- ,KK- -MG] E Rm and 2i FFF E @m
then Han et al. 2021) show that the parameter space of {(ALs, F): ALS 1 M Σ=1) holds the following F
K
property for any symmetric full rank matrix D E mmmxm
(DM) (DF)) 'DF DF= M M E
1
(30)
The proof is straightforward expansion of the transpose and inverse terms. This result implics that ALs F is invariant to the transformation F + DF. Han et al. (2021) exploit the freedom to choose D and set D= > resulting in 'renormalized features DF (similar to 'whitened' features in statistical terms). By incorporating this continual renormalization (N = DF) into the gradient flow, they obtain:
dt N = IINNI (WNRMSE-LS(NN)
(31)
Where RMSE is the empirical risk pertaining to the parameter space of (ALs, F) and Hzxz is projection operator onto the tangent space of the manifold I, of all identity-covariance features (refer Absil et al. (2009) for additional information on matrix manifolds and optimization). Informally, one can think of this operator as applying 'renormaizatiion at every step of the flow. Han et al. (2021) ) show in this setting that as - co, the non-zero singular values of Ew / M tend to infinity while approaching equality. In simpler terms, the signal dominates the 'noise" (where 'noise" pertains to the deviation terms of A from A,s during the gradient flow) and the limiting matrix of (Σ M)T E K is a simplex ETF. Thus, demonstrating the role of such renormalization steps in exhibiting NC. In addition to these intriguing theoretical properties, the benefits of such 'whitening' techniques have been widely studied and are typical in
12

---

Published in Transactions on Machine Learning Research (04/2023)
modern-day deep learning settings (LeCun et al.
2012
Wiesler et al. 2014
loffe & Szegedy 2015 Salimans
& Kingmal 2016: Cyyanov et. al. 2016; Ba et al.
2016 Wu & Hel 2018 To further demonstrate the effects
of normalization, we complement the UFM analysis with the results of [rgen & Pilanci (2021) for ReLU
networks with batch-normalizztionn and some minor additional calculations of our own in Appendix A.4
3.1.6 Discussion
The simplicity of UFM allowed us to leverage the rich literature on matrix factorization, and optimization theory and identify the ideal configurations for F, A. In this section, we discuss additional properties and limitations of this modolling technique.
Data independence: dala-indepenndnt models such as the UFM, we are mainly concerned with F, A, b and Y. As a consequence, under the balanced class assumption, networks can exhibit NC after sufficiently long training on completely random (X, Y). This interpolating behaviour leading to NC properties was observed by Zhu et al. 2021 in canonical networks such as ResNet18 and MLP when trained on a randomly labelled CIFARTO dataset (see figure 4 Thus, if a network has sufficient capacity to memorize the training data and reach TPT, we can expect its penultimate layer features and final layer weights to satisfy NC prop- erties. To the contrary, experiments by Papyan et al. (2020) (see figure 5 show varying magnitude/ /exteetteeeeeteeeeeeeeee of variance collapse depending on the complexity of data. For smaller data sets such as CIFARIO, a ResNet18 network attains a NC11 value of = 10-2, while for ImageNet, a ResNct152 network attains a NC1 value of & 1. Thus, even after memorizing the training data, empirical results show deviation from the ideal UFM behaviour for larger, complex data sets. To understand the behaviour in figure one needs to incorporate notion of data complexity into the UFM, which is not straightforward as it goes against the premise on which the UFM is based. Instead, one can attempt to analyse the role of a large number of classes K on the NC properties while enjoying the simplification ions of UFM.
Ncc (log scale)
to
width 16
width =
0.8
1.0
withh 16
witth -8
1.0
withh 15
witth =8
80
90
widhh
with 16
0.6
-12th-32
0.0
witth 32
0 50
60
with 64
with 32
MLP
withh-12
0.0
witth
withh 128
width 256
0.4
wisth 256
with 256
Training
with
width - 512
withh - 512
0.4
witth - 512
witth 512
width 1024
02
wisth - 1024
0.2
with - 1024
width 1024
width 2048
wisth - 2048
witth 2048
with 2045
100
200
300
400
500
0.0
100
200
300
400
500
0.0g
100
200
300
400
500
%
100
200
300
400
500
Epoch
Epoch
Epoch
Epoch
1.0
1.2
40
width
with =4
widhh 4
Training Error Rate
width
0.8
1.0
30
Ne; (log ResNet18
wilth
0.6
0.8
wisth
with
0.6
width
0.4
width 16
0.2
0.2
width 32
width 64
200
400
600
800
0.0
400
500
800
0.0g
800
%
400
600
000
Epoch
Epoch
Epoch
Epoch
(a)
NC1
(b)
NC2
(c) NC3
(d) Training error
Figure 4: NC metrics of MLP and ResNet18 on a randomly labelled CIFAR10 dataset using cross-entropy loss. The width of the network is maintained across layers and varied across experiments. The first row corresponds to a 4-layer ML.,, optimized using SGD with a learning rate 0.01 and weight decay 10-1 The second row corresponds to ResNet18, optimized using SGD with momentum 0.9, weight decay x 10-4 initial learning rate 0.05, decreased by factor of 10 every 40 epochs. Image credit: Zhu et al. 2202))
Implicit label dependence: Unlike cross-entropy and (mean) squared error losses that explicitly require Y, contrastive losses such as Noise Contrastive Estimation based InfoNCE (Oord et al. 2018), Seasen-Shannon Divergence (JSD) [Lin] 1991) etc are independent of it. In the absence of labels, such losses aim to maximize the feature similarity of closely related training samples (for instance, of samples which inherently belong to the same class) while maximizing the dissimilarity with unrelated ones. A recent surge in unsupervised
13

---

Published in Transactions on Machine Learning Research (04/2023)
MNIST
PasiieaMNIST
SVIE
CIPAR10
CTF
1.0x :d4
VOC 0
Emuor
1.h-14
1.8r-02
RooNee
1.8x-01
1.8x
Dotoseee
14x-411
1.0c 00
100
2011
2001
180
300
200 100 2311 2001
1000
300 100
300
100
233
201
1001
200
350
Figure 5: Plots of AC1 (variability collapse) for combinations of data sets and canonical networks. VGG11, R8 and DenseNet40 FashionMNIST,, VGG13, ResNet18, and DenseNet40 for CIFARIO, VGG13, ResNet55, and DenseNet250 for CIFARI00, VGG13, ResNet50, and DenseNet 250 for STL10, VGG19, ResNet152, and DenseNel201 for ImageNet. The networks were trained using SGD with momentum 0.9 and weight decay of 10 for ImageNet and x 10-4 for other data sets. The learning rates were chosen by sweeping over logarithmically spaced values between 10-4 and 0.25 and the value resulting in the best lest error was chosen. During the parameter search, the learning rates were reduced by factor of 10 after 100, 200 epochs for ImageNet (total=300 epochs) and 175, 265 epochs for the rest (total-350 epochs). Image credit: Papyan al. 2020
chosen for MNIST and SVHN, VGG11, ResNet18 and DenseNet250 for
representation learning can be attributed the effectiveness of such objectives (Saunshi al. 2019 Chen
et al. 2020 Baevski et al. 2020; Jaiswal al.) 2020 Jing & Tian 2020). However, with unknown number of inherent classes, when F has rank m < K, it is impossible for F to form K-simplex. Nevertheless, the supervised contrastive learning (Khosla et al.) 2020) settings provide interesting insights on NC where the label information is implicitly used in the objective:
1 N
n log
exp(SE.fAa/t Ek'= Σ Er= exp(FT.
min RCL
(32)
Where T > known as the 'temperature" parameter, which controls the hardness negative samples
(Wang& Liu 2021). Owing to its similarity with RoE, Fang al. (2021) obtained lower bound for ReL follows:
exp(FT,.fnssT)
o2K0F (cr + + - 1)r
log
+ logn
(33)
K kk=1 vi
Where 6 exp(vRA!lp 02 (K [)exp(-VAAAlF)).. (or) log log( (1t+))) + (cs))g) log( (kx+e))(-1)) 02
Ek= 1lax12 < SA, Kn Σk En || Ωr. Similar to our previous observations, equality at- tained when variance collapse occurs and the columns formed by class means in optimal F resembles a simplex ETF (Fang et al. 2021 As takeaway, observe that even without explicit label matrix Y, a loss function which promotes variability collapse and marimum separation leads to newral collapse based solutions.
Class imbalance: Assuming an equal number of training examples for all the classes has been critical for analysis till now. Having n, = ng = #g = N/K gives symmetric structure Y the form of Y = Ix & 1 which results in relatively casier derivation of variance collapse and simplex ETF, When the classes imbalanced, the analysis is not straightforwwrd. By considering & to be any convex loss function,
14

---

Published in Transactions on Machine Learning Research (04/2023)
the ERM objective with norm constraints can be given by:
1 Σ ((Afa, ex) 8.8 PAN k=
1 K
K
1 n
n
min
|ax12
fk.ill2 S S2
(34)
K
Without loss of generality, one consider the cross-entropy loss and analyze the ERM by performing convex relaxation into a semi-definite program. Fang et al. (2021) analyse such convex program by considering subset of CIFARIO dataset with Kmaj majority classes such that ιu = R22 ....... R.Kaa = Rmaj and Kmin minority classes such that RK = = [KK = Temin- By defining a class imbalance ratio of ri Rmaa// Remas/Tman Mmann > 1, the authors empirically observed that when rs 2 to. for some threshold to > 1, the average angle between the minority class classifiers becomes zero, i.c, the rows of A pertaining to these minority classes collapse to a single vector. The authors term this phenomenon 'Minority Collapse'. Additionally, they observed that the threshold to tends to get smaller (larger) with smaller (larger) Ra,9r Kmin- Intuitively, when the constraints aA f2f tighter, observe from equation 34 that majority classes (Kmaj) dominate the objective and there is a little 'budget' in the gradient updates for data in Kmin minority classes. In a formal sense, let's consider the gradient of cross-entropy loss w.r.t ags k E [K]:
OCCE = Σ fu. 1=1
exp(a fk.))
K
ng
exp(a fk.
1 expla fk.s) -1 1
+
fw J Σ =1==1 expla fw
(35)
dak
K
K
f=1
"pull"
"push" "
The "pull" term represents the tendency of ag to move towards features of the same class while the "push" term represents the tendency to move away from them (see Yang et al. (2022) for analysis based on this formulation). Tn the case of minority classes, the "push" term dominates the gradient, potentially leading to minority collapse. The manifestation of minority collapse was even observed in ResNet18 networks CIFAR10, MNSH data sets for sufficiently large Tib. In practical settings, one way of avoiding this state is to oversample data from the minority classes or under-sample from the majority class to decrease
Fib (Drummond et al.] 2003 Zhou & Liu 2005 He & Garcia 2009 Huang et al. 2016 Buda et al., 2018
Johnson & Khoshgoftaar] 20199] Cui et al. 2019 Fang et al., 2021). Alternatively, when we are aware of
the imbalance, fixing the last layer classifier to the desired simplex ETF seems like a clever hack to prevent
minority collapse. YYang et al. (0022 confirm this intuition and achieve improved performance even in
fine-grained image classification tasks.
Extensibility: Extending the UFM with multiple non-linear layers quickly turns tractable model into an involved one. Thus, a good starting point in this direction is to add a single linear layer and analyse the model properties. To this end, consider the following ERM based on MSE with regularization and an extra linear layer as proposed by Tirer & Bruna 2022):
min RMSE-cat ent F.Ar.A,
2N |44 YI 1/22 +
AAa 2
|| 1
AA |Ail
+
AF |F 2
(36)
Where F R=xN are the unconstrained features, A E Rmxm Ao IR Kxm are the linear layer weights
and XAa AA, > 0 are penalty terms. Tirer & Bruna 2022 lower bound this risk by following the same
sketch as equation 29
XA2
XA1
AF F
A2A
F-
Y +
||
A211
+
||
1|
+
2N
2
2
2
1
n
Xd
Xa
XF
|A2
11 +
||A211
+
|||1| 1
Σ
||
2Kn
2
2
2 k=
(37)
k=1
i=1
1=1
K
12
K
AAa
XA
XF
n
A₂ A
Σ
fk.i Ck
+
|| 7
+
|
11
+
Σ
n
2Kn
2
2
2
1m
i=1
k=
2=1
15

---

Published in Transactions on Machine Learning Research (04/2023)
Where the final equality holds when within-class variability in F is O, i., F= E R (similar to the RMSEr case). The authors connect this three- minimization problem with two-factor objectives based the result in equation 24 and split the risk into two sub-problems as follows:
min 1
AAZ
RMSE-cat1
A2Z
A.
IK
+
| VndA, AF
Z ZA.F
Af 2K
2
(38)
RMSE-ca12 = min
ZA,A, F FlLA,-IA. Ixl +
ndF
P +
VAA,A, ZA.A,
Zazar FF 2K
2
Where sub-problems RMSE- eztt RMSE-eare e212 have close resemblance to the one layer UFM risk formula- tion. If (F*, A5) is the global minimizer of MSE--eat such that F"= F I, then:
(AAAj)F&&l & F Fo (A;A) Ix
(39)
Thus, F collapses to an orthogonal frame (to a simplex ETF if we recenter around the column mcans) and is aligned with (A) T. Similar results can be shown for Až and F (see theorem 4.1 in Tirer & Bruna (2022) for proofs). Now, by converting the linear layer F into a FA),), where represents a non non- linerr activation (such ReLU), we are presented with a non-linear UFM model. Factorization of A2a F) into 2 factors is possible in the case of RMSE--zt1 defined above but not for RMSE--Z12- Tirer & Bruna (2022)
analyse the former by considering ReLU activations as a non-negativity constraint F, and proceed with the 2-factor problem. Refer to theorem 4.2 and Appendix.E in Tirer & Bruna (2022 for further details.
3.2 Models of "Locally Elastic" Networks
Theoretically modelling the training dynamics in neural networks has been long-standing challenge and is mostly tackled in the shallow settings (Jacot et al.,
2018
Arora et al.) 2019 Goldt et al., 2019
Yang.
2019 Hu et al. (2200) or in linear networks (Saxe et al., 2013
Kawaguchi 2016 Ji & Telgarsky 2018 Arora
ct al. 2018 Lampinen & Ganguli, 2018). Nevertheless, with an understanding of the 'desired' structures
that sufficiently expressive network should achieve, we transition to modelling technique which attempts imitate the feature separation dynamics in canonical deep classifier neural networks and demonstrates neural collapse as by-product.
3.2.1 Primer
Introduced the work of He & Su (209)), local elasticity is a phenomenon which describes classifiers whose prediction of a training sample af, i [n], k E [K], is insignificantly affected by SGD updates pertaining to gradients of dissimilar samples ax i E [m], + k E [K]. Intuilively, local elasticity represents the influence that training samples have on each other during training (see figure 6 Analysis along those lines can be traced back to the seminal work inffuence functions and curves by
Local Elasticity (LE):
Hampel [1974] followed by efforts in developing robust statistical models (Reid & Crépeau 1985 Weisberg,
2005 Huber 2011 Kalbfleisch & Prentice 2011 Koh & Liang, 2017). While influence functions have been
widely adopted in the machine learning community for studying interpretable learning techniques AAdadi &
Berrada, 2018 Molnar 2020). we restrict our focus to locally clastic networks and their training dynamics.
He & Su (2099) present preliminary analysis of this idea image classification settings and geometric interpretation using the Neural Tangent Kernel (NTK).
Stochastic differential equations are statistical variants of classical differential equations and can be traced back to the works of Itō [951]: Van Kampen (1976);
Stochastic Differential Equations (SDE):
Klocden & Platen (1992). Their presence can be found in wide range of important settings such as filtering
(Welch et al.) 1995), boundary value problems (including the influential 'Dirichlet problem"), the Fokker- Meistein-Uhllnbeckk processes [koda, & Watanabe 2014) and many more. From a deep learning perspective, since we are modelling the impact of SGD updates on the gradual separability of features, the idea is to represent these changes as a Zhang et al. 2021b) and study its implications.
Plank equation (Risken 1996), the Black-Scholes model (Karoui et al. 1998),
stochastic differential equation (L. et al. 2021
16

---

Published in Transactions on Machine Learning Research (04/2023)
s=1
$=2
s=3
small drift
large drift
f
f2
f3
Figure Mustration of the local elasticity phenomenon in neural networks with a loy example. At step I, the image of football passed to the network, resulting in its penultimate layer representation ft. In the next step 2, the image of tree is passed, which is dissimilar to the football and results in small feature drift in the learnt representation of football. Finally, when an image of a basketball is passed to the network at step = 3, the drift from fa to f will be larger than f to f2 due to visual similarity.
3.2.2 Feature separation via Locally elastic stochastic differential equations (LE-SDE)
To provide intuitive understanding of this model, we take a bottom-up approach to present the ideas. Recall that the penultimate layer features for the 4th data point of class & are given by fk.i R=. We extend the notation to denote this feature at iteration/step s of training as fū.;- Without loss of generality, by randomly sampling an i' Uni [(n))( Uni]((K) at iteration 8, our goal to model the impact of training a network with xy on fk.S. A reasonable formulation can be given by:
Intuition:
fk. sk.7
J. 0" (x.)
(40)
This indicates that the 'drift' features (JÉ, SK5 is proportional f scaled by some impact term E*, plus data dependent noise d=
(at
LE-SDE Model: The work of ZZhng et al. (022b formally captures this intuitive idea in their LE-SDE model. Since back-propagation iteratively updates the features, Zhang et al. (2021b) capture the impact of learning rate as well as transformations of f on fūr Thus, the formal and refined version of 'drift' is presented by the authors as:
fū, fE. n)E (T") JKK fir + -no=-
(41)
Where 17 is the step size, o² (x+) is the Gaussian noise associated with the data point at (independent of it's feature fo-1). E E k is the local elasticity impact matrix at iteration and (E") E IR is the (k, W'th entry of E which represents the LE impact that ay has on ot. Similarly (T*) E 1mxx
3Observe that this formulation allows track the separability of features in the pre phases well.
17

---

Published in Transactions on Machine Learning Research (04/2023)
is the transformation matrix on features at iteration s. Next, by considering fx to be random sample from class k (can be informally thought of as a representative sample as well), the authors represent F [fll .... E kK to be a concatenation of per-class representative features and F - -Miil [u E Rkm the concatenation of per-class feature means at iteration 8. This assumption and set follows from NC1 where f eventually collapses to g.g Thus, single representative data point for each class is amenable for analysis. Now, the continuous version of the SDE in equation 41 with t = st). 17 - 0 can be represented as:
= dt + (23)
(42)
This is the LE-SDE formulation where W represents standard Wiener process E RKmm E' E R R Km is the covariance matrix of representative features and B E R KmmxKm is a K x K block matrix with m x m sized blocks, which models the combined effect of LE impact E' and transformations T". The (k: kith block of B' is (E') Ja (T" (T)) /K, Vk, k' E [K]. To account for randomness involved in selecting the representative samples F", the class mcans F' satisfy:
d dt (F') B'F'
(43)
This is obtained by laking expectation on both sides of equation 42 and noting that the Wiener process can be characterized a martingale with Wo = O. This implies B[W 0, resulting in equation 43 Zhang et al. (021b)) call it the LE-ODE. With this setup in place, the authors assumne the diagonal entries of E* to be a and the off-diagonal entries to be 8.. Here at 3r E R pertain to "intra-class" and "inter-class" LE impacts respectively. Now, as + oc, when v = min - Bt, 0r + (K - 1) > 0, and T is a positive semi-definite matrix with positive diagonal entrics, then Theorem 3.1 in Zhang et al. (2021b) states that:
The features are separable with a probability p- when v (1/t)
The features are asymptotically pairwise separable with a probability p + when 1/2 = o(1/t) and n + oo at an arbitrarily slow rate.
Here, pairwise separation at any time t implies, for 1 1 k K S K, there exists a direction o such that:
min(v fka,) > y max fkr
(44)
Similarly, asymptotically pairwise separable implies:
P(min(v fi.d) 3
max( fkrz))
(45)
3.2.3 Neural collapse as a by-product
A powerful yet simplified aspect of this model is the freedom to choose T. This flexibility was exploited by the authors by setting it as the outer product of residuals d,d], where residual roughly aligns along d, =, 1m Vj e [K]. Intuitively, these residuals roughly indicate the direction in which fy need to be pushed for perfect classification. Thus, by setting:
d,d] 2
1 _ 1m E R", je [K]
(T),
E
R
m
where d, =, K
(46)
l
the transformation T always aligns the changes in fy along dy for all E [K]. Formally, by considering the "intra-class" and "inter-class" LE impacts on Bg to be constants a and B & (E & Ig) @ T, where o represents the Hadamard product, (E & IR) O T has eigen values {a 3,a a + 0 -) with multiplicities (KK 1),K K 1}} respectively. Additionally, by assuming m K for satisfying the psd property of the transformation matrix and using these quantities to solve the LE-ODE in equation 43 the authors get:
18

---

Published in Transactions on Machine Learning Research (04/2023)
F c T TI de
T2tty ex ex x.,
(47)
Where d and u, E ER E K - 1] are the eigen vectors (E 2 Ix) @ T with eigen values a - 3, + k-t respectively, co E RK 72, 721 E IR, E [ - 1] are constants and A, = So adt, B, = So 3dr. With the assumption of local elasticity (v: > 0), B, 0, as + 00, F' will eventually align towards v. Since d is concatenation of residuals d,.j [K] from equation 46 the matrix formed by dj 's as columns forms a simplex ETF (refer A.2. in Zhang et al. (2021b) for details of the proof). Thus, after certain point in time 1 2 of evaluating the LE-ODE, the class means ue vk [K] which were evolving through the concatenated matrix F', lend to a simplex ETF as I + co.
3.2.4 Discussion
The LE-SDE/ODE approach implicitly captures the impact of data complexity the feature separation dynamics and provides unique way of modelling feature evolution. Note that the purpose of this modelling technique is not to approximate the actual non-linear dynamics of deep classifier neural networks, but to mimic the dynamics the LE phenomenon during training. The key takeaway here that by choosing the LE transformations to be biased towards an orthogonal structure of labels ej. E [K], neural collapse is manifested as a by-product. For additional results and experiments pertaining to the study of LE, please refer to He & Su 2019); Zhang et al. (2021b). As we have already observed the effects of realignment in analysis of normalization and UFM, the LE-SDE/ODE model reinforces the role of continuous realignment towards a maximally separable configuration for facilitating NC. As follow-up of this observation let's consider the Taylor approximation of feature drifts as presented in Zhang et al. (021b) to better understand the realignment bchaviour. As a first step, assuming that a network is being trained using the cross-entropy loss, and considering the "logits-as-feal ure'" model Zhang el al. (2021b), observe that the drift features can be approximately given by:
fh. fk. Gi Jofttman (f
(48)
0ft 0f 30 20
E RR is the time dependent gram matrix for class k, trainable parameters 0 and softmax function asaftmaz- Recall that we assumed m for the transformation matrix to be psd. With this approximation, the feature drift D(F', 1) can be defined for all representative features F when = 87), 1 + as:
Where G
D(F",t) G²
Ge -'((ee--Ooof [(e1--ooftmaz
Dzoftmaz (fttmaaz(( (f .... .... (es soottman ( (kk))
(49)
Where G¹ E R is the gram matrix of all classes. As the actual dynamics of separation are non-linear, the LE-SDE attempts to model it by linearizing the drift around the mean value of F² (see Särkkä & Solin 2019h for similar lincarization approach) to obtain:
D(F 1) D(F .t)
+ V DF t) F F'
2 Gd V D(F F' + [(el Fo)) .e. -FFR -V v D(F', 1)F"
(50)
"residue"
is the Jacobian of the drift of class means w.r.t Fr. The "residuc" term derived in the original proof by Zhhnng et al. (2021b) an approximation of the one mentioned above and is shown tend to around convergence. The negligible residue implies close resemblance of the class mean drifts by the
Where v.. "F,D(F', ,t) ))
19

---

Published in Transactions on Machine Learning Research (04/2023)
representative feature drifts for all classes. Thus demonstrating the emergence of NC properties under the effective training assumplion. We found that the experimental setup of Zhang et al. (2021 focuses mainly on LE behaviour, and analysis pertaining to AC1-4 metrics was lacking, For closure, we mention that, despite the non-negligible residues and lincarization effects of the LE-SDE/OD /ODE model, this approach provided a faithful approximation of the feature separation dynamics on CIFARIO and a synthetic GeoMNIST dataset. Furthermore, if one wishes to track NC1-4 metrics, a batch size of 1 is needed to explicitly track the drift of each and every feature, making the model susceptible noise and extremely slow for large scale data sets such ImageNet. Furthermore, the complexity of data sets (w...t mumber of classes) might have different implications on the drift. behaviour, leading to unexpected deviations from exhibiting NC as by-product.
3.3 Retrospect
After detailed analysis of UFM and LE-SDE modelling techniques, we summarize our observations in this section. In the work of Lu & Steinerberger (200)]; Wojtowytsch et al. (2020), the authors impose norm constraints on the weights, and features and observe that the simplex ETF configuration leads to a global minimizer solution for the ERM with cross-entropy loss. Additionally, [Wotowytsch et al. (2020) analyse two-layer neural networks in the mean-field setting and for non-convex input classes in their paper to conchude that NC might not be desirable in such cases. Thus, suggesting an inherent difference in the expressiveness of shallow networks with infinite-width and deep neural networks. In the concurrent work of Mixon et al. 2220)), the authors prosented gradient flow analysis of F, A by employing a lincarized ODE. By identilying an invariant subspace S along which strong neural collapse holds, the authors show that solutions along& S are indeed the minimizers of the ERM with squared error loss. However, the linearization by partially decoupling F, A is valid in regimes where their norms are small, which is typically not desirable in practical settings with deep networks suffering from vanishing gradients.
In the following work by Zhu al.) (2021), the authors take a different approach and analyse the loss- landscape with cross-entropy and regularization. Unlike the work of [i et al. (2021), which presented an analysis of the cross-yntropy loss and concluded that the implicit regularization of gradient flow is sufficient for attaining NC solutions, Zhu et al. (2021) emphasizes the benign nature of the regularized landscape which allows stochastic optimizers to reach global minima and exhibit NC properties. Additionally, by fixing the last layer classifier of ResNet50 and DenseNet169 as a simplex ETF for classification on ImageNet, (Zhu et al.] (2021) report Rv 10% reduction in the number of trainable parameters with a negligible drop in performance. For the regularized (mean) squared error setting, the works of Tirer & Bruna (2022) and Han et al. (2021) present a multifaceted approach to analysing NC. [Han et al. (2021) obtain a closed-form solution for A in terms of F and show that the gradient flow trajectory of normalized features along this 'central path' is of particular importance to the learning dynamics. Their experiments employ canonical deep classifier networks and datasets to validate the theoretical results and in a sense complement the trajectory analysis of Mixon let al. (2000) in a more rigorous fashion. The work of Tirer & Bruna (2022) analyses a similar ERM set seiting with varying choices pertaining to bias and reglaarzuiii.oioon. Furthermore, Tirer & Bruna 2022) and Han let al. (2021) draw similar conclusions that the idcal bias value pertains to that of global mcan subtractor for the penultimate layer features. Additionally, Tirer & Bruna (2022) presented an extended version of the UFM by adding an extra linear layer. In the absence of non-linearity, they show that NC solutions are indeed desirable by splitting the 3-factor optimization problem into multiple 2-factor variants. Although they partially address the non-linear case, rigorous analysis is still open for research. To this end, the work of Ergen& & Pilanci (2021) analyses this extended variant of the UFM with respect to practical settings such as ReLU activations and batch--normalizationf Their approach is based on a convex-dual formulation of the ERM for any convex loss function and highlights the importance of batch-normallzzation in obtaining bounded and NC-satisfying optimal solutions.
Unlike the above efforts which rely a balanced class sctting, Fang et al. 2021 explored the implications of class imbalance on the geometry of minimizers. Their approach relies on the work of /Strmm & Zhang (2003) and relaxes a quadratically constrained quadratic program as a semidefinite program. Their work highlights the 'minority collapse' phenomenon and studies the effects of class imbalance ratio on the features of minority classes. To this end, the empirical results of Yang et al. 2022 complement the work Zhu et al.
We present simple application of their results the Appendix A.4
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table Unified comparison of modelling techniques (MT) (UFM, LE or generic analysis) based on constraints or regularization (Rog), loss function l, class distribution constraints (balanced (B), imbalanced (IB)) for theoretical modelling, the main approach for analysis, the NC properties (NC1-4) that are derived and finally whether the authors provide the relevant empirical analysis (EA).
MT
Reg
71
Approach
NC
EA
Wojtowytsch
2020
UFM
CE
B
Global Minimizer
1,
& Steinerbergor berger
(2020)
UFM
CE
B
Global Minimizer
1,
Gradient Flow
Ji et al.
(2021
UFM
CE
B
KKT
1,
+ Loss Landscape
Global Minimizer
Zhu et al.
(2021
UFM
V
CE
B
1,
+ Loss Landscape
Convex Relaxation
1,
Fang et al.
20021
UFM V
CE,CL
B, IB
+ Global Minimizer
+ MC
Gradient Flow
Mixon ct al.
(0020)
UFM
SE
B
at Initialization
2,
+ Linearization
Gradient Flow
Han al.
(2021)
UFM V
MSE
B
1,
+ Central Path
Global Minimizer
Tirer Bruna
(0022)
UFM
V
MSE
B
1,
+ Extended UFM
Poggio & Liao
20199
V
SE
B, IB
Gradient Flow
1,
Convex Dual
Ergen & Pilanci 2021)
V
SE
B, IB
1, 2,
Whitening
Dynamics SDE
Zhang et al. (2021b)
LE
CE
B, IB
1, 2
+ Linearization
2021 and highlight the practical relevance of fixing the final layer weights as simplex ETF to address the minority collapse issue.
The 'local clasticity' based approach is relatively less explored than the UFM and primarily analyzed in the work of /Zhang et al. (2021b). In the case of UFM, the expressivity of the network was exploited by considering the penultimate layer features to be freely optimizable. Whereas in the LE-SDE setup, the local clasticity of the network was expected to result in sufficiently expressive features. Based this assumption, Zhang et al. (2021b) showed that similar drift patterns of representative features and class means as f lead to NC-based solutions. However, the authors did not present any experimental results to validate these theoretical results. On a related and non-rigorous note, the LE-SDE formulation can be considered as a flexible version of the gradient flow of features /logits. Especially, the lincarization of the drift in LE-SDE to model the non-linear dynamics allowed Zhang et al. (2021b) to faithfully approximate the training dynamics around convergence (see Appendix.B B.2 Zhang et al. (2021b) for additional discussion on the initialization regime). Such an approach avoids the negligible weight norm assumptions that Mixon al. (020) had to employ for linearization of the gradient flow ODE.
Overall, by employing the UFM approach, one can analyse the optimal closod-form solutions and gradient flow dynamics of F, A while also leveraging convex relaxation approaches to study the otherwise non-convex ERM problem. On the other hand, the LE-SDE approach is limited to gradient flow-like analysis and heavily relies on the linearization of the SDE to derive insights on NC. Although both approaches fail to provide an accurate picture of the pre-TPT phases of training the drift formulation and transformation matrix choices in the LE-SDE approach present good starting point to analyse the initial phases of training. Similarly,
Note that the difficulty primarily lies in accurately modelling the non-linear dynamics of the gradient flow. Thus, majority of the efforts aim faithfully approximate it through linearization.
21

---

Published in Transactions on Machine Learning Research (04/2023)
in the case of UFM, we believe that incorporating non-linearity and additional layers can be first step in deriving new insights for inner layers. Especially, as discussed above, the difficulty in solving this problem lies in factorizing the matrices after an activation function has been applied. A detailed comparison of various implementations of these models is presented in Table
4 Implications on Generalization and Transfer Learning
In this section, primarily focus the connections between NC and the generalization capabilities of wer-parameterized networks. The modelling techniques discussed in the previous section are limited to the training regime and explain the desirability /dynamics of attaining NC-based minimizers. To this end, we shed light on the empirical results by Zhu et al. (2021), which indicate the discrepancy in test performance of networks exhibiting NC during the training phase (see figure 7 This observation highlights that NC during training (let's 's call it train-collapse for convenience) doesn't necessarily guarantee good generalization. Also, one might wonder train-collapse is just a result of the optimization process and doesn't necessarily describe the effectiveness of the learnt features. So, what is an effective approach to capture the impact of train collapse test data? If train collapse doesn't guarantee generalization, how does it affect transfer learning? We address the questions by analysing recently proposed generalization and transfer learning bounds based on NC and discuss their validity.
100
100
100
100
Training accuracy
Testing accuracy
Training accuracy
Testing accuracy
90
98
80
70
SGD
SGD
SGD
60
SGD
50
Adam
9/
50
100
150
200
94
50
100
150
200
40g
50
100
200
40g
100
150
200
Epoch
Epoch
Epoch
Epoch
(a)
MNIST Train
(b) MNIST Test
(c) CIFAR10 Train
(d)
CIFAR10 Test
Figure 7: Train vs test performance of ResNet18 on MNIST (first two plots) and CIFAR10 (last two plots)
with SGD, Adam and LBFGS optimizers. SGD with momentum 0.9 and Adam with B1 = 0.9, 82 0.999 were initialized with learning rate 0.05, 0.001 respectively and scheduled to decrease by factor of 10 every 40 epochs. LBFGS was initialized with memory size 10, learning rate 0.1 and employed Wolfe line-search strategy for following iterations. Weight decay is commonly set to 5 x 10 Image credit: Zhu et al. (0021)
4.1 How to evaluate a test collapse?
NC properties during training are measured when perfect classification has already been achieved by the network. Since we generally cannot guarantee perfect classification on test data, we consider relaxation of perfect classification during testing as proposed by Hui et al. (2022) and define:
Weak test-collapse: This variant of collapse mandates that test samples should collapse to either one of the K class means: M1, , and not necessarily to the mean of the class that it actually belongs to.
Strong test-collapse This variant mandates that test samples should collapse to the 'correct' class mean.
Intuitively, the "weak" and "strong" notions test. collapse seem reasonable but they bring forward additional challenges. Strong test-collapse requires a Bayes-optimal classifier to exist based on the features limited number of samples. This is infcasible and too rigid of a requirement. On the other hand, weak test-collapsc can be satisfied by fixing the penultimate layer of network an orthogonal frame representing the one-hot hot logits. However, this setup doesn't 't guarantee good performance as the network can misclassify all the test points to the wrong class means and still attain weak test-collapse.
6 Although most of the approaches derive NC1-3 properties in their results, sufficient. Papyan 2020 showed that NC1-2 imply NC3-4. Additionally, the experiments column ticked only if the paper presents NC-related expermentt.
22

---

Published in Transactions on Machine Learning Research (04/2023)
In the empirical studies of Hui et al. (2022) when ResNet18 was trained and tested on CIFAR1O, the poggittud//extent of weak/stron test collapse was observed to be lesser than the train collapse. Surprisingly, we observed that the value of weak/stron test collapse of ResNet18 on CIFARIO in Hui et al. (2022) is on- por with the train-collapse of ResNet152 on ImageNet. Thus, the notion of NC occurring on test data is entirely data-dependent and is not convincing enough for understanding generalization.
To the contrary, recent work by Galanti et al. (2021) presents generalization bound based on the variance collapse property (NC1) and states that train collapse favours generalization. Formally, by defining a " "Class
Distance Normalized Variance (CDNV)" metric over class conditional distributions:
(y(P,, (Pr, + (Pc,).
(51)
2[(u)(Fo,) uy ( 112
Where Hy (Pc.)) EzwPe, [f(z] and (ar,(Pc..) = EewRe [llf(x) - us (PPc)) the generalization bound by Galanti et al. (2021 states that:
P(V) (V,(P,, B) +)( 2 1 -
(52)
[2). x (P() (Pc;)))))) (Dc,,. is the mcan of features I (x))), Vj E [m.] pertaining to class sampled from the empirical distribution Do, and Do. ~ Pć denotes empirical distribution on class with n, data points. The bound in equation 52 essentially relates the population CDNV with empirical CDNV and the generalization gap (details omitted here for brevity). From the UFM analysis presented in the previous sections, we saw that (D,,)Dc,) - Ju (D, Il is maximized when the class means (Dc)),,), i E [K] attain the ideal simplex ETF configuration. Thus, by bounding A, B and considering sufficiently large values of ni, Ryj the generalization gap can be reduced and train-collapse on classes i, j Le, (D,,, Dc,) + 0 would indicate (PIP,,,Po,))
Where B x - My (P()
0 with high probability.
From a theoretical standpoint, the assumption of large n4, is a natural way to approximate a true data distribution and seldom applies to practical settings (since a large amount of labelled data is usually hard to obtain). Thus, even though the experimental values of strong /weak test-collapse in Hui et al. (2022) showed higher values than train- coalnccolapsse, their analysis conveys the same observation that, with an increase in the size of train data, the weak/strong Lest collapse can potentially tend to lower values (although the setting itself is not good indicator of typical practical scenarios). Furthermore, from the experimental results Mini-ImageNee by Galaati et al. (2021) in figure [] we observed that for the same nunber of classes, the test CDNV is approximately an order of magnitude larger than train CDNV and holds resemblance with the trend of plots presented by Hui et al.] (2022). This observation is of paramount importance as the absence of thresholds for CDNV or NC1-4, which indicate whether collopse has occurred or not, can lead to misleading/ bankeeadding/onluudiihthe /cntraddicingg results in the communiity. We believe that beller metrics or thresholdss can help in drawing objective conclusions on the occurrence of collapse (train/test and avoid subjective interpretations in future efforts. Developing such objective metrics can prove to be tricky, especially when considering the varying number of classes in data sets, network archilectures, optimizers etc and we open this question to the community for further thought.
4.2 A 'depth' based generalization bound
Till now, we obscrved that networks exhibit NC when they interpolate on train data and such memorization doesn't 't necessarily guarantee good test performance (for instance, based on the choice of optimizers). The CDNV-based generalization bound fails to explain such discrepancies. In this section, we analyse a gener- alization bound basod on the seemingly obvious NC4 property and shod light on the occurrence of NC in random label settings. We begin by presenting simplified definitions from Galanti (0022) as follows:
Thinkk of it as empirical approximation of (Pc()))) based on Dc
23

---

Published in Transactions on Machine Learning Research (04/2023)
2a
21
21
CDNV
CDNV
20
2
20
20
2
50
2-1
50
2-2
60
lteration
lteration
(a) Train data
(b) Test data
Figure 8: CDNV (lain(left) and (rih(tight) data of a Wide ResNet-28-4 (i.e depth factor of 28, width factor of 4) trained on Mini- Image-t using SGD with momentum 0.9, learning rate 2-4, The legend indicates randomly selected classes for training/testing. Image credit: Galanti et al. (2021)
c-effective depth: For a network composed L layers, let h((x)) = arg min lff(z)) un (Xc))l [L].
kE[R]
The eeefecttive depth xxx (h) of h on the minimum depth E [Z]] for which errx (hx) e. If such an doesn't exist then px (h) =
Here ertx (h+) + Σ (x;)) * arg [ E(x;)] is the nearest class center (NCC) misclassification error, Mr (Xc,) indicates the mean of features fi(.) for samples of class k and finally recall from the preliminaries that fi = 91 o the e-effective depth p5x (h) represents the minimum depth at which the fentures can be classified by the NCC decision rule and achieve at most classification error.
Rd " the composition of / layers of the network. Essentially,
c-Minimal NCC depth: Let G represent function class of layers in network, the e-Minimal NCC depth Pinin (9, X) on data set X is the minimum number of layers (belonging to G) that can be composed to result in an output function f such that errx (h) < where h(x) == arg minke(k] || [(x) Px((
Now, consider the following setup: X2 P are two balanced data of size N. With a slight abuse of notation, we represent hi,, Rd RKK as network with weight initialization and trained on dataset X1. When this network is evaluated on X2, assume that the misclassified samples uniformly distributed over the samples in X₂ with probability 1 o be thought of as representing scenarios with non hin-aiirarl classes. As second assumption, X2 contain
ol.. In practical settings, this assumption
labels and both were to be used for training, we consider that with probability 1 NN P E (0, 1/2) a E 10, the e-minimal NCC depth to fit (2 p)N correct labels and p.V random labels is upper bounded by the expected e-minimal NCC depth to fit (2 - N correct labels and g random labels for any q 2 (1 + a)p. Under these assumptions, the generalization bound proposed by Galanti (0022) be formulated follows:
Ex, En.erre (xxx Px,, X,, Exlex, (1x,) 2 Pinin (G, (, X²) +11 (1+a)pp 5,y
22 N
533)
Where chr = EpI[arg ma h(xx) o arg kc {(x) and X. is obtained by randomly relabelling PN samples from X₂. The noisy labels are now represented as Y, Essentially, the bound indicates that, by randomly selecting p E (0, 1/2), if the expected e-effective depth of hix, for all smaller than the e-minimal NCC depth Pmmin (9, X1 U X2), then the network bound to do well on the test data. For comprehensive proofs and tightness comparison of the bound, refer Galanti 2022)
24

---

Published in Transactions on Machine Learning Research (04/2023)
Additionally, based on the empirical results of Galanti (2022) presented in figure 9 one observe clear indication of CDNV collapse as NCC train accuracy reaches after certain depth. Cohen et al. (2018) performed a similar study using a Wide ResNet on MNIST, CIFARIO, CIFARI00 & their random counterparts. It was observed that the layers gradually learn the k-NN based features when the dataset is clean and demand sufficient depth for randomly labelled data. The experiments by Galanti (2022) corroborate this observation and align such behaviour with the learning gap induced by &x in equation 53 presented above. Furthermore, observe from figure g that NCC test accuracy and CDNV test of the final layer of the MLP saturates at = 0.6, = 2 respectively in both the cases. Interestingly, these values during training, i.e when NCC train is z 0.6 in the pre-TPT stages, the CDNV train is R 2-1. On the other hand, CNN whose final layer was able to achieve 2 0.85 NCC test accuracy showed a CDNV Test of x 2-2 (see figure.5 in Appendix.A of Galanti (2022)). Such a paltern underscores the definition of strong test-collapse and a Bayes-optimal classifier that Hui et al. (2022) demand is the one we would idcally require to achieve that state. As a takcaway, note that for a network which is collapsed on train data, attaining collapse on test data depends on various factors such data distribution, the implicit bias of network and the optimizers used. Since attaining strong test collapse is usually an ambitious setting, one can nevertheless employ multifaceted approaches to track the extent of test collapse and gain better understanding of the learning dynamics. For instance, Ben-Shaul & Dekel (2022) extend the cross-entropy loss by enforcing NCC behaviour across intermediate layers and show improved performance across vision and NLP sequence classification tasks.
can observe similar pair of
12 layers
tpoch
730
4.3
-40
153
300
18 layers
0.78
209
E20ch
400
Epocch
+00
5.0
130
330
Epoch
123
435
105
300
300
500
(a)
CDNV Train
(b) NCC Train
(c)
CDNV Test
(d)
NCC Test
Figure 9: Layer-wise CDNV and NCC plots of an MLP trained on CIFAR10. Each hidden layer has a width of 300 and undergoes batch-normalliation followed by ReLU activation. Cross-entropy loss is minimized using SGD with momentum 0.9, weight decay x 10-4, and initial learning rate 0.1, which is reduced by a factor of 10 at epochs 60, 120, 160. Image credit: Galanti (2022)
4.3 Does collapse favour transferable representations?
Good classification performance achieved when the networks are powerful enough and sufficient data is available for ERM (in statistical sense). In settings where quality labelled data is scarce, transfer learning is
widely adopted technique for classification (Caruana 1994 Thrun 1998; Pan & Yang. 2010 Bengio 2012;
Weiss et al. 2016 In typical transfer learning settings, large network is trained on plethora of source
tasks, followed by fine-tuning on downstream target tasks.
The literature on the empirical effectiveness of
this approach is quite rich (Long et. al.] 2015 Zamir et al.]
2018 Houlsby el al.) 2019 Raghu et al.) 2019 2020) and various attempts have been made to
Raffel et al.) 2020 Kolesnikov et al. 2020 Brown et al.]
theorctically understand this capability (Ben-Davie al.
2006 Blitzer et al. 2007 Mansour al.] 2008
Zhang et al. 2013 Tripuraneni et al.) 2020). natural question that arises in this context is the role of
collapse in learning transferable features.
25

---

Published in Transactions on Machine Learning Research (04/2023)
addresses this question by extending the theoretical analysis of collapse from unseen data (as discussed in the previous section) to unseen classes. Specifically, their anal- ysis focuses on multi-task settings where the source and target class conditionals are sampled iidd from a distribution over class conditionals. For a better understanding of the framework, consider a transfer learning setup with a I-class classification problem as the downstream lask and a 8-class classification prob- lem as the source/ source/auxiliary /axiliary task. Formally, we extend our notation and assume the data for downstream and source tasks is sampled from P, respectively. Next, similar to the setup for CDNV-based general- ization bound, let Dc, ~ PC, denote a target data set for class E [t] where n data points have been sampled from Fc,, Along these lines, the target data set is comprisod of D Dc, U... Dc,, and the source data set is comprised of D = Dc. U .... U Dc,- Finally, the class conditionals Pc,, Vi E [4] and Fc,, Vj E [s] are assumed to be sampled i.i.d from a distribution Q over class conditional distribu- tions U In the work of Galanti et al. (2021), , the authors randomly select two classes k,k'= E [t] from the target task and bound the expected CDNV between them, Re (TP Pc,.)] using the aver- age CDNV of the source classes, 5(8-1) Ei=1 (Pe,(V((Pc,FFo))) and terms which inversely depend on infy infre, Poy [Py(()))) - (P((Pv))ls Now, by defining the expected transfer error as follows:
The work by Galanti et al. (2021 2022
2
= : (xa))
[ f))(z) + y]]]
(54)
the authors bound the transfer error by Bec, #Pc, (P,,,,,, Fc..)] )] up to scaling (see Proposition.2 2 and Ap- pendix.D D in Galanti et al. (2021)). Here the expectation is taken over randomly selected target tasks from IP and the limited available data D. and (ao f)D indicates the network (as per preliminarios) trained on D. The issue with this bound pops up when (Pc,) - (y (P,(Pc tends to be very small. Thus, settings where the transfer error is small, there is a possibility that the upper bound is very large and not indicative of the network's performance. Such scenarios occur when the support U for Q is infinitely large and anomalous pair of larget classes can lurn this bound vacuous. To address this issue, Galanti et al. (0222) consider a specific case of ReLU networks with depth r and bound the transfer error with the averaged CDNV over source classes (instead of target classes presented above) plus additional terms that depend on s, r and a spectral complexity term which bounds the Lipschitz constant of f (Golowich et al. 2018). Their theoretical results indicate that, with a sufficiently large number of source classes 8 and data samples per source class, the prevalence of neural collapse on source data leads to small transfer errors, even with limitod data samples in target data sets.
0.19
0.55
0 C Inxian
C
41
C
12
(a)
Train CDNV
(b) Test CDNV
(c) Target CDNV
(d) Target accuracy
Figure 10: (a)-(b) Source train and test CDNV, (c)-(d) Target CDNV and accuracy for 5-shot classification on Mini-ImageeNet using Wide ResNet-28-4. A varying number of classes (as per legend) are chosen for the source task and 5 classes are randomly selected for target tasks. SGD with momentum 0.9, learning rate 2 was employed to minimize the cross-entropy loss during training. Image credit: Galanti et al. 2021
Empirical results on Mini- ImageNNe pertaining to 100 5-shot classification experiments by Galanti et al.
2021 are shown in figure 10 In this experimental setup, a varying number of classes are randomly chosen
8On lateral note, the setup is also amenable covariate shift (himodaira 2000) analysis and has been studied by Gretton et al. 20066: Huang et al. (2006): Zhang al. 2013
26

---

Published in Transactions on Machine Learning Research (04/2023)
from the data set for pre-training a Wide ResNet-28-4 network. Next, a ridge-regression classifier is used as the final layer (with all the previous layers kept fixed) and trained on 5 randomly selected target classes. Finally, the network is tested on 100 random test samples from each of the 5 target classes for reporting the metrics. Observe that a clear pattern emerges in this plot, showcasing the benefits of pre-training on a large number of source classes as per the theoretical analysis.
Although the results look promising, recall that the bound and empirical analysis is restricted to setting where the class conditional distributions are sampled from common Q. Thus, camnot guarantee similar results for settings where the source and target distributions come from different Q. The empirical results presented in Kornblith et al. (2021) are of significance in this context. The authors show that networks pre-trained on ImageNet tend to perform poorly on different downstream data sets (such as CIFARIO, CIFAR100, Flowers etc) when exhibiting a higher extent of collapse during pre-training. They especially show that softmax cross-cntropy loss leads to relatively smaller margins between classes during pre-training and in turn results in better downstream performance when compared to losses such as squared error and cosine softmax. The impact of discribution changes (also known as model shift (Wang & Schneider, 2014)) is clearly evident from the experiments by Kornblith et al (2021) and the bounds presented above fail to analyze this generalized sctting.
5 Takeaways and Future Research
In retrospect, the study of neural collapse is essentially a study of desirable geometries and feature evolu- tion dynamics in deep neural networks. Although each of the four NC properties has been studied in the literature, their unification under a common notion of 'collapse' has certainly piqued the interest of the community. Firstly, an extensive theoretical analysis of the global minimizers for cross-entropy loss from NC perspective by WWojooytsch et al.] (2020); Lu & Steinerbergen (2020) highlighted the difference in the expressive power of shallow and deep neural networks. In the following work by Han et al. (2021), the authors decomposed the MSE loss based on the gradient flow trajectory of features aligned with NC properties and demonstrated this behaviour for canonical deep classifier networks. Thus, shedding light on their training dynamics. Additionally, Zhu et al. (2021) leverage the simplex ETF property (NC2) and show Rs 20% reduction in the memory footprint training ResNet18 on MNIST and ss 52% reduction in the number of trainable parameters of ShuffleNet for ImageNet with 1000 classes. On a related note, Fang et al. 20011; ;; Yang et al. (2022) propose solutions to address class imbalance training by analysing the collapse properties of represeedd classes. Furthermore, the study of NC has led to series of efforts which atlempt to explain the effectiveness of learnt representations for generalization and transfer learning. In summary, the studies pertaining to neural collapse have presented a unique approach to questioning and understanding the heuristic design choices and training dynamics involved in deep network training. In the following, we discuss open questions and future efforts that can have a broader impact:
Modelling techniques: The "unconstrained features" and "local elasticity" based models have provided good theorctical ground for analysis. However, both are simplified presentations of the actual networks. Since it is extremely challenging to model every aspect of training deep neural network in theoretically tangible fashion, one can either extend these models incrementally (such as adding more layers to the UFM analysis) or approach this problem in a radically different way. The complexity lies in modelling the role of depth, non-lincarities, normalization, loss functions, optimizers etc, all in a single model. During our review, we analysed each of these aspects individually but the challenge of unification still an open problem and is of significance.
Desirable geometries: A m-simplex is one of the three regular polytopes that can exist in an arbitrarily high dimension m. The other two are m-cube and m-orthoplex (a cross- pross--olyyope) (Coxeter) 1973). Pernici
2019: 2021) empirically show that by fixing the last layer of a VGG19 network as a m-cube with loga K or as a m--rthoplex with m = K/2], the performance on CIFARIO, MNIST, EMNIST and FashionMNIST is comparable to that of a learnable baseline. However, if we desire maximum intra-class soparation from our network, a m-simplex fits our nceds. Interestingly, when m increases, the angle between the weight vectors that form a m-simplex tends towards x x/2, which is similar to the case of m-orthoplex. The question that arises in this context is whether deep classifier wotw attains the m-orthoplex configuration
ct
al.
m
27

---

Published in Transactions on Machine Learning Research (04/2023)
if the penultimate later features dimension is fixed to [K/2]. Also, how does this structure affect class- imbalance training? Similar questions arise for m-cube well.
Learning objectives: In addition to supervised classification settings using cross-entropy or squared error losses, we analysed that contrastive losses in supervised settings also favour collapse. This implies that either explicit or implicit label information necessary for attaining the collapse state. Tt would be interesting explore a similar phenomenon in unsupervised clustering tasks where labels are entirely absent (Xie et al,, 2016 Min et al.) 2018). Additionally, based on the results in Kornblith et al. (2021). network exhibiting relatively high extent of collapse
ImageNett was shown to transfer relatively badly to downstream classification tasks. To this end, it would be interesting to explore schemes such as "marimal coding rate reduction (MCR²)" (Yu et al., 2020 Chan et al. 2020 Wu et al. 2021 Chan et al., 2022), which aims to preserve the intrinsic structure of within-class features along a subspace, while also increasing the distance between these subspaces. A rigorous analysis of such objectives from an NC perspective can shed light on the scemingly clusive implications of collapse.
Generalization: From review of efforts which analyse test-collapse, it was highly subjective whether certain value of .C1 CDNV can be deemed as exhibiting collapse or not. It is necessary for the community to standardize such observations in the early stages of this rescarch direction and promote objective results. Furthermore, we observed that empirical results by Zhu et al. 20021 | showcased the disparity in generalization performance of networks which attained train collapse using different optimizers. This is at odds with the CDNV-based generalization bounds that Galanti et al. [2021 propose, which heavily rely on train collapse. Further empirical and theoretical analysis is required to model such observations and improve the bounds.
The special case of large-language models (LLM)) Based on the transfer learning bounds and ex- periments by Kornblith et al. (2021); Galanti et al. 2021 2022), il would be interesting to track collapse metrics on the learned representations of large language models. Specifically, consider a case where an LLM is pre-trained in an unsupervised fashion on billions of text sequences and fine-tuned for a variety of clas- sification tasks. In this setting, how would the collapse metrics differ when fine-tuning on tasks such as sentiment classification, docurnett classification etc? Is it possible propose transfer learning bounds when pre-training is unsupervised? Although such questions are primarily relevant at the scale of LLMs, novel methods to analyse such scttings at a smaller scale can be of broad interest.
From a geometric deep learning perspective (Bronstein et al. 2017/. neural networks have been quite effective in learning on non-euclidean data such as graphs and manifolds. Since the architecture of such networks is highly dependent on the topological structure of data, novel modelling techniques are required to empirically and icalretically analyse NC in such settings.
Data domains:
6 Conclusion
In this work, presented a principled review of modelling techniques that analyse NC and discussed its implications on the generalization and transfer learning capabilities of deep classifier neural networks. We presented a comprehensive review of the unconstrained features and local bcit- models by analysing their assumptions, settings and limitations under a common lens. Next, we discussed the possibility of neural collapse on test data, followed by an analysis of recently proposed generalization and transfer learning bounds. We hope our review, discussions and open questions would be of broad interest to the community and will lead to intriguing research outcomes.
Acknowledgments
The author would like to thank Joan Bruna, Tom Tirer and Federico Pernici for their informative discussions and the anonymous reviewers for their valuable feedback and time.
References
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matria Manifolds. Princeton University Press, 2009.
28

---

Published in Transactions on Machine Learning Research (04/2023)
Amina Adadi and Mohammed Berrada. Pecking inside the black-box: survey on explainable artificial intelligence (xai). IEEE access, 6:52138-588 5216002260 2018.
Farid Alizadeh. Interior point methods in semidefinite programming with applications to combinatorial optimization. SIAM journal Optimization, 5(1):13-514 51, 1995.
Zeyuan Allen- Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. arXiv preprint arXiv:1810. 02281, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322 332. PMLR, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ar preprint arXiv:1607.004450 2016.
Alexci Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in Newral Information Processing Systems, 14:12449-12460 2020.
Pierre Baldi and Kurt Hornik. Neural norksrks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58 58, 1989.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117((8))3006333000 2020.
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in nearal information processing systems, 31, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 116((2))15849 15854, 2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 144 2020.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Percira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. arXiv preprint 20..2994.. 2022.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and tronsfer learning, pp. 17-36. JMLR Workshop and Conference Proccedings, 2012.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jennifer Wortman. Learning bounds for domain adaptation. Advances in neural information processing systems, 20, 2007.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872 1886, 2013.
29

---

Published in Transactions on Machine Learning Research (04/2023)
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-2583 2018.
Rich Caruana. Learning many related tasks at the same time with backpropagation. Advonces in neural information processing systems, 7, 1994.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Deep networks from the principle of rate reduction. ar preprint orxiv:2010. 14 4775,, 2020.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunct: A white-box deep network from the principle of maximizing rate reduction. Journal of Machine Learning Research, 23 (114):1-103, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. aixiv preprinl arXiv:2204.025: 2022.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pp. 854- 863. PMLR, 2017.
Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Dan or k-nn: That is the generalize vs. memorize question. arXiv preprint orXiv:1805 008206 2018.
Harold Scott Macdonald Coxeter. Regular polytopes. Courier Corporation, 1973.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class- balanced loss based on effec- tive number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass clas- sification. In 2020 54th annual conference on information sciences and systems (ciss), pp. 1 5. IEEE, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CV CVF conference on computer vision and pattern recognition, pp. 4690 4699, 2019.
Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost scnsitivity: why inder-samping beats over-sampling In Workshop on learning from imbalanced datasets II, volume 11, pp. 1-8, 2003.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. anXiv preprint arXiv: 1r10::1810.02054, 2018
Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. Approximate kkt points and a proximity mcasure for termination. Journal of Global Optimization, 56(4):1463-144 1499, 2013.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in neural information processing systems, 31, 2018.
Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. International Conference on Machine Learning, pp. 3004-3014. PMTRR, 2021.
30

---

Published in Transactions on Machine Learning Research (04/2023)
Tolga Ergen, Arda Sahiner, Batu Ozturkler, John Pauly, Morteza Mardani, and Mert Pilanci. Demystifying batch normalization in relu networks: Equivalent convex optimization models and implicit regularization. arXiv preprint 00f:::1103.0149,, 2021.
Cong Fang, Hangfeng He, Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021.
Oluseyi Farotimi, Amir Dembo, and Thomas Kailath. A general weight matrix formulation using optimal control. IEEE Transactions on newrol networks, 2(3):378 394, 1991.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proccedings of the 52nd Annuaal ACM SIGACT Symposium on Theory of Computing, pp. 954 959, 2020.
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprinl ar 1511::611.01./// 2016.
Tomer Galanti. A note on the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.0900 09022, 2022.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint Xix:22112. 15121, 2021.
Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-truining: Perspectives, Pitfalls, and Paths Forward ICML, 2022, 2022.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp. 2232 2241. PMLR, 2019.
Raja Giryes, Guillermo Sapiro, and Alex M Bronstein. Deep neural networks with random gaussian weights: A universal classification strategy? IEFF Transactions on Signal Processing, (4(13):344433455 2016.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural net- works. In Conference On Learning Theory, pp. 297 299. PMLR, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Geoff Gordon and Ryan Tibshirani. Karush-kuunn-tuker conditions. Optimization, 10(725/36):725, 2012.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-probeen Advances in newral information processing systems, 19, 2006.
Benjamin Haeffele and René Vidal. Global optimality tensor factorization, deep learning, and beyond. arXiv preprint Xio: 07540, 2015.
Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(36))3383 393, 1974.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprinl orXiv:2106.02073, 2021.
Fengxiang He and Dacheng Tao. Recent advances in deep learning theory. arXie preprint arXiv:2012.1099. 2020.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and dala engineering, 21 19):1233 -284, 2009.
31

---

Published in Transactions on Machine Learning Research (04/2023)
Hangfeng He and Weijie J Su. The local elasticity of neural networks. arXiv preprint arXtu:1910.0692 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In International Conference Machine Learning, pp. 2790 2799. PMLR, 2019.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the carly-time learning dynamics of neural networks. Advances in Neural Information Processing Systems, 33:17116 17128, 2020.
Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian. Model complexity of deep learning: A survey. Knowledge and Information Systems, 63:2585 2219,, 2021.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imba imbalanced classification. In Proceedings of the IEEE conference on computer vision and pottern recogration, pp. 5375 5384, 2016.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
Peter J Huber. Robust statisties. In International encyclopedia of statistical science, pp. 1248 1251. Springer, 2011.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification lasks. arXiv preprinl 20K::20006.0322, 2020.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding general- ization in deep learning. arXiv preprint orXiv:2202.0833 08384, 2022.
Nobuyuki Ikeda and Shinzo Watanabe. Stochastic differential equations and diffusion processes. Elsevier, 2014.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Kiyosi Itô. On stochastic differential equations. Number 4, American Mathematical Soc., 1951.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in newral information processing systems, 31, 2018.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Katarzyna Janocha and Wojeiech Marian Czarnecki. On loss functions for deep neural networks in classifi- cation. arXiv preprint arxiv 100:1770.0000 05659, 2017.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained layer-poeled per- spective on neural collapse. ar preprint ar Xiv:2110 fau:2110.029900 027900 2021.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep lincar networks. arXio preprint arXiv:1810. 02032, 2018
Yiding Jiang, Dilip Krishnan, Hosscin Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. ar preprinl 1xxx:1810.000 2018.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pullern analysis and machine intelligence, 43(11):4037-4055 2020.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
32

---

Published in Transactions on Machine Learning Research (04/2023)
John D Kalbfleisch and Ross L Prentice. The statistical analysis of faslure time data. John Wiley & Sons, 2011.
Nicole El Karoui, Monique Jeanblanc-Piequéè, and Steven E. Shreve. Robustness of the black and scholes formula. Mathematical Finance, 8(2):93-126, 1998. doi: http:///oo..oogg11111111111111 https://onllnellbraayy wwwlly..com/o////////aaaaa 1111/ 996.999950004404
URL
Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661 18673, 2020.
Peter E Kloeden and Eckhard Platen. Stochastic differential equations. In Numerical Solution of Stochastic Differential Equations, pp. 103-160. Springer, 1992.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885 1894 PMLR, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigoerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pp. 491 507. Springer, 2020.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead less transferable features? Advances in Neural Information Processing Systems, 34, 2021.
Jelena Kovačević, Amina Chebira, et al. An introduction to frames. Foundations and Trends0 in Signal Processing, 2(1):1 94, 2008.
Andrew K Lampinen and Surya Ganguli. An analytic theory of gencralization dynamics and transfer learning in deep linear networks. arXiv preprint arXio:1809. 10374, 2018.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Newral networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). Advances Neural Information Processing Systems, 34, 2021.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 3711):145-151, 1991.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. Xriie preprinl urXiv:1612 02295, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint XrX:::1907. 1169, 2019.
Jonathan Long, Fvan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 3431 3440, 2015.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv:2012.08440 2020.
arxio preprint
33

---

Published in Transactions on Machine Learning Research (04/2023)
Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-cntropy loss. Applied and Computational
Harmonic Analysis, 2022. ISSN 1063-5203. doi: httpss//abb..og///1110///aaaa 12.011. 1..00 URL. https:
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprint wwXi::18990000 2018.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprinl ar 1906. 05890, 2019.
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness sgd in modern wer-parametrrzed learning. In International Conference on Machine Learning, pp. 3325-3334. PMLR, 2018.
Stéphane Mallat. Group invariant scattering. Communications Pure and Applied Mathematics, 65(10): 1331-1395 2012.
Olvi I. Mangasarian and Stan Fromovitz. The fritz john necessary optimality conditions in the presence of equality and incquality constraints. Journal of Mathematical Analysis and applications, 17(1):37-47, 1967.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. Advances in neural informalion processing systems, 21, 2008.
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. J. Mach. Learn. Res., 22(165):1-73, 2021.
Dhagash Mehta, Tianran Chen, Tingting Tang, and Jonathan Hauenstein. The loss surface of deep lincar networks viewed through the algebraic geometry lens. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5664 5680, 2021.
Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514, 2018.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXim:2011.11611 2020.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Mor Shpigel Nacson, Suriya Gunasckar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pp. 0683-4692. PMLR, 2019.
Arkadi S Nemirovski and Michael Todd. Interior-poin methods for optimization. Acta Numerica, 17: 191-234, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
03748, 2018.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1445-1359, 2010.
Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXie preprint arXio:1811.00000 2018.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40) 24663, 2020.
Federico Pernici, Matteo Bruni, Claudio Baccchi, and Alberto Del Bimbo. Maximally compact and separated features with regular polytope networks.
34

---

Published in Transactions on Machine Learning Research (04/2023)
Federico Pernici, Matteo Bruni, Claudio Baccchi, and Alberto Del Bimbo. Fix your features: Stationary and maximally discriminative embeddings using regular polytope (fixed classifier) networks. arXiv preprinl arXiv:1902.10441 2019.
Federico Pernici, Matteo Brumi, Clandio Baecchi, and Alberto Del Bimbo. Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Tomaso Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square loss, Tochnical report, CBMM Memo No, 2019.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. Xix preprint arXiv:2101.00072, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140): 67, 2020.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
Nancy Reid and Helene Crépeau. Influence functions for proportional hazards regression. Biomelrika, 72 11:11-9 1985.
Hannes Risken. Pokker-Planck Equation, pp. 63-95. Springer Berlin Heidelberg, Berlin, Heidelberg, 1996. ISBN 978-3-612-61544-3, doi: 10.1007 00.1007/978-33662-654444 4. URL https: ott.ao.rgo11.00.. 1007/
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hossian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07/77 2016.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Botton. Empirical analysis of the hessian of wwer-parametrized neural networks. Xr preprint ar S:3::17060044 2017.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Nikunj Saunshi, Orestis Plevrakis, Sanjecv Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoret- ical analysis of cont unsupervised representation learning. In International Conference on Machine Learning, pp. 5628 5637. PMLR, 2019.
Andrew M Saxe, James I, McClelland, and Surya Ganguli. Fxact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the ligelikeldhood function. Journal of statistical planning and inference, 90()):227-244 2000.
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognation, pp. 1233 1240, 2013.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Advances in neural information processing systems, 30, 2017.
Jure Sokolié, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16):4265 0001442666-40,,, 2017.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822 2878, 2018.
35

---

Published in Transactions on Machine Learning Research (04/2023)
Nathan Srebro. Learning with matrix factorizations. 2004.
Jos Sturm and Shuzhong Zhang. On cones nonnegative quadratic functions. Mathematics of Operations research, 28(2):246-20) 2677 2003.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei, Cir- cle loss: A unified perspective pair similarity optimization. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 6398 6407, 2020.
Amirhossein Taghvaci, Jin W Kim, and Prashant Mehta. How regularization affects the critical points in linear networks. Advances in neural information processing systems, 30, 2017.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprinl ar trXiv:2201.0822 08239, 2022.
Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pp. 181 209. Springer, 1998.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. arXiv preprinl Xre:22200..00 08087, 2022.
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity, Advances in Neural Information Processing Systems, 33:7852 7862, 2020.
Dmitry Ulanov, Andrea Vedaldi, and Victor Tempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint orXi::107700000 2016.
Nicolaas G Van Kampen. Stochastic differential equations. Physics reports, 24(3):171-22) 1976.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2265-5274. 2018.
Xuezhi Wang and Jelf Schneider. Flexible transfer learning under support and model shift. Advances in Newral Information Processing Systems, 27, 2014.
Mingwei Wei, James Stokes, and David J Schwab. Mean-fiel field analysis of batch normalization. arviv preprint arXiv:1903.020 2019.
Sanford Weisberg. Applied linear regression, volume 528. John Wiley & Sons, 2005.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
Greg Welch, Gary Bishop, et al. An introduction to the kalman filter. 1995.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Simon Wiesler, Alexander Richard, Ralf Schlüter, and Hermann Ney. Mean normaioomal stochastic gradient for large-scale deep learning. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 180-184. IEEE, 2014.
Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXio:2012.054200 2020.
Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. Náx preprint arxiv: 176.. 10239, 2017.
36

---

Published in Transactions on Machine Learning Research (04/2023)
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
Ziyang Wu, Christina Back, Chong You, and Yi Ma. Incremental learning via rate reduction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 1125-1133, 2021.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. Tn International conference on machine learning, pp. 478 487. PMLR, 2016.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902. 0ete:1902.477000 2019.
Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203. 09081, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-lext foundation models. ar preprint arXin:2205.01913 2022.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discrimi- native representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422 9434, 2020.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. arXiv preprint arXic:1605.0714 2016.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018.
John Zarka, Florentin Guth, and Stéphanc Mallat. Separation and concentration in deep networks. arXiv preprint Xiv: 10424, 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021a.
Jiayao Zhang, Hua Wang, and Weijie Su. Imitating deep learning dynamics via locally clastic stochastic differential equations. Advances in Neural Information Processing Systems, 34, 2021b.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference on machine learning, pp. 819 827. PMLR, 2013.
Zui-Hua Zhou and Xu- Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on knowledge and data engineering, 18(1):63 77, 2005.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geomet- ric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34, 2021.
37

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Extended notations
Table 2: An extended set of notations for lookup.
Notation
Description
X E RAXN
Input data to the network Imbalanced and balanced class sizes one-hot vector w.r.t index vector of all zeros of suitable dimension Vector of all ones, Identity matrix 8th data point (ith column of X) 7th data point of b class Set of all data points belonging to class l
Tkk n EN
ei
1k R
1K E
R
E Rd
2 Rd
Ck
E: Rd {el,
ek}
A ground truth labelling function Data and class conditional data distributions
P. P.
A function class of networks
h1 Rd RK overload:
A network composed of layers
01 R. R overload:
Final layer linear function The feature function for layer I Composition of L 1 feature functions
91 : m- m
f--1 Rd Rmm. overload: f
Hr E R RK*N overload: H
The network output matrix
Ar E I Kxm overload: A
The final layer classifier matrix Ponultimate layer feature matrix
FL-1 E R overload:
by. E RRK overload: Y KxN
Bias vector for layer
Label matrix
8: RK RK - [0, 00)
Generic loss function
R: 71 + 00)
Population risk function
R: o0)
Empirical risk function Mean of penultimate layer features of class
14 E Rm
PG E Rm
Mean of penultimate layer features class means
Ew Rmxm
Within class covariance matrix for F Between class covariance matrix for F Indicator function
ΣB E Rmxm
I: (ruue, False] + 10
(k.i A) E
Margin of a data point xt Batch-normalizatioon scaling, shifting constants for Fj.
Yi Uj E R
on E R
Intra-class, inter-class LE impact at time Local elasticity condition for feature separability Penultimate layer features for of at iteration s
14 IR
fū, Rm
E' E R T E RKmKKm
LE impact matrix at time t
LF-SDE transformation matrix at time t LE-SDE block transformation matrix at time
B'ERE PERKKMKK W' ERK
Wiener process al time L Population CDNV for classes i,
(PC,,, ER
(D Dc,)) RR.
Empirical CDNV for classes i, e-ellective depth of h w.r.t X
xx R
erT' ETT), E
Empirical, population NCC misclassification error Transfer learning crror Transfer crror Frobenius norm, Nuclear norm Inner product, Hadamard product Trace, Pseudo-inverse and Transpose of a matrix
Ce(f) ER
-1l1e,1-11e (), tr(.)t.T
38

---

Published in Transactions on Machine Learning Research (04/2023)
A.2 A note lower bounds for population risk with collapsed outputs and cross-entropy loss
Withoul loss of generality, for subspace B E R2 where tce is striclly-conver, cun show that R(h) S R(h)
for F-almost every a, where h E H such that h(a) = Van E Ck (Where zn is given by Eg 10 A brief sketch of the proof by Wojtowytsch et al. (2020) is as follows:
Let 4 R + I2tt be the softmax function, given by:
exp((h,e))) e1))
exp exp( ex) (
exp((h) ex))
(hh)
(h)
(55)
EK exp( e.)
E expl ei))
k
E exp( (h, ei))
Where h is vector input. Now, by taking the Taylor expansion of 4 near 2k. we get:
I (a(h(z))P(dx))
[ (#b(Zx)) (Z))()) (h(x) 2x) + 1 +2(((z)) 2x) (b(()) 2x)
Cx
2 /
C. (xx)))((d)))
since Cc, (Z)) (h(æ²) - 2²) = from equation 10 and the hessian of 4 is posilive semi-definite. The oquality holds when h(x) -h(x) E (pan[( 1)) since fo is strictly convex on the orthogonal complement
of (1). See Lemma 2.1, 3.1 in Wojtowytsch al. (2020) for the complete proof.
A.3 Gradient flow analysis of Mean Squared Error without regularization
Based on our analysis of the squared error without regularization, we show that the same line of reasoning holds true for MSE as well. At first glance, the scaling factor of & seems benign as (F, A, b) satisfying SNC for the squared error, minimize MSE as well. However, the purpose this analysis is to observe how the subspace and the rate of convergence of b to } 1k modified due to this scaling factor. We follow the same sketch as Mixon et al. 2020) and define the ERM for MSE
min (MSE(F,AA) (F, F.A.b
1 2N
AF + 61 Y1
(56)
The corresponding gradient flow equation for θ (F, A) b) is given by:
O'(1) = ViMSS(((())
V R RMSE = A T (AF + bbb Y)
N
1
(57)
A RASE N
(AF
bb Y)F
1
VORMSE N
(AF
+ b1 Y)1N
Assuming that F, A have small norms (leading to uncoupled bias), we analyse the following ODE:
1 F(11) N (b()) Y), A'(I)
L - (b(1)1 Y)F(!T, b'(1) - Y)1N
with initial conditions F(0) Fo, A()) Ao, b(0) 0. Let's begin by solving for the bias term:
b'())= = -000 Y)IN -
1
(Ix 1 b(1)))N 1
1 (1x Kb(() - Nb()) (1kk
N
K
39

---

Published in Transactions on Machine Learning Research (04/2023)
Based on the initial condition 6(0) 00 if consider b(t) 3(t)1k, then:
1 ((()) F( (1 - k((1)) /
Kg(t) /1<K(()) dt f 1dt ((1) 1 - e
()
1 -
b(t) (
K )
K
Let U= = (F,A) and U(tY (U(U,)), where:
A + [IK 81, ((1)111)), 1 N (Ix 61 -(t)1x1N) N
LFF A)
(58)
The self-adjoin property of Le is straightforward to check and is shown in Mixon et al. (2020). Next, we define the following sub spaces over which {L1120 are simultaneously diagonalizable.
E = [FF ((FA)): F= (A@ 1m) T, A -
Vn
ES ((F A): F 21 T
N: A = k , E }
Fg [FF, A): (IK s 1 F = = A
where c E {+1}. Now let's verify that these spaces are indeed the eigen spaces of Lt.
Case 1: (F,A)EL E Ej
1 N [IK @ 1 - -(11(11)
1 (A 81.)) V N
v N
A
v (1x 1 - =( N [Ix 1 - 3(t) (A 1,) = vn
1
N
This implies, {(F, A), x form the eigen-pait for L, in Ej.
Case 2: (F, Eg
A T 1 N
[K @ 1. 3())k1)) = I (1 & -(0
vn N =1k 8 -KB(111)
vn
N (1-K3(())=1) sxm (1-K((
1 N
Tx 2 1 (() FT
N (IK @ 11 (e.
1
(IIK @ 1,1(1x @ 1,) =())I=I]N))E
N
N (n1x= - -N8())kk22
ne
N [I 13(0)1K ==VV (1 K3(1)) A
N
This implics, ((F,)), syn (1 - KB(t))} form the cigen-pair for L, in E N
Case 3: (F E Eg
A 1 N
[IK 1 3(t)1k]])
N Ix 8 IT - 1()) = =-8(1)xx
40

---

Published in Transactions on Machine Learning Research (04/2023)
since the eigen value is 0 in Es, we can ignore it and represent the spectral decomposition of L
L. -1, (1- KB(())1 (1 K3(())11))
(59)
Where i the orthogonal projection onto E.- This allows us to solve U(I) I,(U(I)) by finding the orthogonal projection of (t) onto E. Thus, get:
(t)((() cvn (1)((() (1((t)) e ((()) N
(t) eyn (1 - KB(())15U(() = (1 cvn
1 l-e
cvn
(1 [1KK(-
(e
N
(t)(()) N
K
N
(t) = exp (c/m (1 -o-)))) (0)
As the final step in the analysis, we can apply the Pythagoras theorem and get:
U(()) - =
e x " (0) + exp
vm (-----)) -
[1; exp (=v=i - ett
11 10)))1u000
NN
|||ا (0)]le + exp
22
N (1 (1- [u5u(())
=
-2/n (1 (1-e-1)
(0))
+ exp
N
< ||uT (0)) + 2vm | + (0)
2ym <e N -I (U
Thus, by consider the subspace T = Et, we get the final result that (F, A, b) along the gradient flow satisfy:
-
L--ett
(F(1). V/ (F Ao) se ||T, A b(t)
1k, 1x.0
K
Where [[ A)l = || + Al and 1l the orthogonal projection onto the subspace:
T (F. (F. FF 1 (A (A 1 -
1
Note that the subspace T which in turn leads to subspace S is the same as the squared error case, but the rate at which b + K 1 has changed from e Nt to e- The empirical consequences of this modified setting on SNC would be interesting to observe (especially the role of initialization), which we defer to future work.
A.4 Applying the results for ReLU networks with batch-normaln normalization to UFM
As a follow-up of the role of normalization in attaining NC based on unconstrained features and gradient flow, we leverage the results of Ergen & Pilanci (2021) for RcLU networks with the popular batch-normalizationn (BN) layer to present different perspective on UFM based NC. Firstly, let the BN output of F be given as:
(IN 11N1)) (FBN);: = BN, (IN N 1IN1,)
IN VN
Uy Vj E [m]
(60)
Where the batch-normalization output for every row of unconstrained features F,. E R.N Vj [m] is denoted by (FEN),:- Here ?/ E is the scaling factor and vy E is the shift factor. For the sake of analysis, we consider the modified risk based the squared loss, similar to Ergen & Pilanci 2021 Ergen et al. (2021)
41

---

Published in Transactions on Machine Learning Research (04/2023)
2 + 1 A
Aa
m
min RSE BN F.A
( + vz + +1a111
(61)
2
Where (FBN)+ + indicates a ReLU non-lincarity on FBN. In our analysis till now, the non-linearity was implicitly assumed for sufficient expressivity of F. In this example, we take a step further and break down the role batch-normalization and ReLU on the optimal configurations for such expressive features. We can obtain a closed-form solution this optimization problem based on a convex dual formulation RNE-BN for RSE- BN (see Ergen & Pilanci 20021
for an elaborate formulation). Let be the j row of Y, then:
75 yj 1I AN vj
T5 VN 1yy12
Y
F
(62)
18 l2
((m-K)xN
The proof by Ergen & Pilanci [2021) was originally given for ReLU networks with batch normalization where strong duality was shown to hold true, i.e the global optimum for RSE-BN are the solutions for Rsg- BN as well. The values in equation 62 are obtained as a direct application of theorem 4.4 from Ergen & Pilanci (2021) the UFM, where the output of penultimate layer is now unconstrained. The optimal values can now be used to calculate FBN as:
(IN FF
IN
(FiN);:
1[ Fill 'f +
VN
(63)
K
Y
FBN
(m KYXN
For simplicity, if we consider K and center FøN around its global mean, we get:
1 J (N K / e DTN V
1 K ( K IN
IN N I
(64)
N
Thus, the features of class k have collapsed to their mean value of / / (ex 4x) and one can verify that the class means lie on the rotated and scaled version of the simplex ETF. Although this setting enables obtain closed form solution for RSE-BN using techniques such as interior points methods on R BN (Alizadch] 1995 Nemirovski & Todd 2008), this convexity doesn't come for free the convex program now consists of exponentially more terms to optimize (Ergen & Pilanci 2021 Ergen el al. 2021).
Interestingly, equation 62 shows that the first K rows of F* Fäx are just the scaled versions of Y. Does this indicate that batch--normalization layers in canonical deep neural networks facilitates collapse even in the earlier layers? Furthermore, can this intrinsic bias towards symmetric structtre batch- natchh-orrmllzztioon
layers explain its role in faster convergence (Toffe Szegody 2015 Luo al. 2018 Wei et al.] (019) A recent work by Poggio Liao| 2009 2020) shows an interesting relationship between norms of weights and margins of classification for squared error loss with regularization in ReLU networks. A consequence of this result is that batchh-normalization leads to weights with smaller norms which allows the network to learn large margins for classification. A comprehensive gradient flow analysis is presented in Poggio & Liao 2019 and is a valuable follow up of the UFM to canonical deep neural networks.
42

---

Published in Transactions on Machine Learning Research (04/2023)
A.5 Code availability
Table 3: List of NC related open-source implementations.
Model
Implementation
Reference
Neural Collapse Papyan al.
20201)
pytorch
neuralcollapse
LE-SDE (Zhang al. [2021b]]
pytorch
zjiayao/le_sde
SVAG Li et al. (2021)
pytorch
sadhikamallad//saag
Layer Peeled Model (Fang al.
2021))
pytorch
HornHehhf/LPM
Local Elasticity (He Su (2019
pytorch
hornhehhf/le
Max Margin (Lyu Li (2019))
tensorflow
vfleaking/max--arriii
Separation and Concentration (Zarka et al. 2020
pytorch
j
j-zarka/separation
Unconstrained Feature Model (Zhu et al. (2021)
pytorch
:dingl//lurral-Collapse
43


Reviews:
[{"review_id": "MU249zbMvF", "reviewer": "Reviewer_64hA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well-organized.\n- The manuscript does what it claims, and summarizes recent work on this topic within a unified framework. I am not an expert in this topic but I found the review relatively easy to follow at a high level. \n- The unified framework for studying NC was presented in a clear manner.\n\nWeaknesses: \n- I am skeptical about the prevalence of these neural collapse phenomena and their purported benefits. While the manuscript cites many recent papers to support these claims, I found myself wishing that there was a clear explanation for what exactly has been shown by this body of work, and why we should take it seriously. This review focuses more on summarizing the methods used for studying the phenomenon.\n- The discussion of NC on generalization and transfer learning didn't seem to have much substance. Defining the terminal phase of training (TPT) as beginning when classification error (01-loss) reaches zero appears to be a useful shorthand, but this is not based on anything fundamental since these models are trained to minimize a different objective (i.e. cross-entropy loss). So why is it important to define \"weak and strong test-collapse\" for generalization and transfer learning scenarios? I couldn't see how this was useful. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a review paper on the \"neural collapse\" (NC) phenomenon in deep neural networks. The authors summarize, compare, and contrast recent work within a unified framework. Implications of NC on generalization and transfer learning are discussed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some minor issues:\n- In Introduction, what are \"complexity based learning techniques\"?\n- In Section 2.1, I believe there is a typo where ||(r,s)||_E should be ||(r,s)||_F\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "02tgz0Jgku", "reviewer": "Reviewer_cr7x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n  - The paper consolidates a quickly-developing body of work into a single document, allowing the reader to easily compare results in a variety of settings. This is aided in particular by the parallel structure of the subsections found in section 3.\n  - The paper provides helpful intuitions to the reader in the interpretation of many technical results. The additional space provided by the TMLR format allows for greater discussion than would be permitted in a conference paper, allowing for a more didactic approach.\n  - The coverage of the literature seems quite thorough: while neural collapse is not my main research focus, I am somewhat familiar with the area and didn't notice any glaring omissions.\n\n**Weaknesses:**\nIn contrast to other TMLR submissions I have read, this paper does not offer any novel theoretical or empirical results: instead, it provides a didactic summary of existing work on neural collapse. While surveys can certainly be useful tools to help bring people up to speed on an emerging literature, this one in particular faces two principal limitations: the field of neural collapse is still very nascent and so the burden of reading the literature is relatively light, and the paper doesn't provide much significant insight above and beyond that provided by the works it discusses.\n\n  - While the paper claims to \"analyse NC modelling techniques by unifying them under a common set of principles,\", I was not able to determine what exactly this common set of principles was that was missing from the literature. Most of Section 3 seems to consist of summarizing the contributions of prior works according to their primary theoretical result, followed by a discussion of their interaction with the loss landscape. Aside from the shared notation, I don't see what additional principles are added by this paper on top of the contributions that already exist in the cited works.\n  - It is not always clear where a derivation is new (and is thus replicating prior results under a unified framework) vs where it closely follows the cited text. In most cases that I dug into the latter seemed to be the case.\n  -  Section 4 seems to be largely an expanded summary of two papers by Galanti (et al.), for which it takes up disproportionate space. This is particularly salient given that the practical relevance of the generalization bounds discussed in this section is quite limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of recent results concerning the phenomenon of neural collapse. It summarizes the findings of a number of recent works and contrasts their assumptions, problem settings, and resulting findings to give an over-arching view of the research landscape. Additionally, the paper discusses the implications of neural collapse on generalization and transfer learning, drawing from the findings of  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": " - Section 3 should more clearly delineate where the derivations are pulled from prior work and where the \"from the ground up\" modeling principle is being used to reproduce prior results.\n  - The tables of results are a nice way to concisely summarize the contributions of a number of papers, however simply showing whether or not the authors conducted a type of analysis (as in Table 1) is not a particularly helpful format. Instead, what would be more useful is to directly compare theoretical results -- e.g. what types of NC properties are guaranteed, or what dynamics are followed by the gradent flow.\n  - Section 4 should justify its large use of space for what appear to be relatively loose generalization bounds which hold in a relatively limited number of settings. Additionally, the discussion of transfer should be better-motivated: at the moment, the transfer result places very strong assumptions on the distribution of classes seen during training which makes the setting closer to a multitask iid setting than the generic distribution shift considered more broadly as the transfer learning setting.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7HxWwAy4XA", "reviewer": "Reviewer_sBKs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The work is on a relevant topic and in my opinion certainly satisfies the TMLR requirements for being of interest to the community.\n- The submission seems thorough in its review of literature that is directly related (however see weaknesses below).\n- Plenty of mathematical details are provided such that the conclusions can be derived without needing to read the reference materials.\n\nI reviewed the submission as a review article, that is with the assumption that the major scientific claims and contributions have been already submitted and peer reviewed independently. With this in mind, the article suffers from the following mild weaknesses.\n\n**Weaknesses**\n\n- Many interesting conclusions are buried underneath a lot of mathematics. A prospective reader needs to work through many lines of non-trivial math to get to the conclusions of each subsection. It would be very beneficial for the prospective reader if the authors had a section detailing all the conclusions (e.g. role of CE with or without regularization, role of normalization etc.). The reader can then delve into the mathematics in as much or as little detail as they would like.\n\n- As a review article with two major modeling approaches, the authors simply ask the reader to \"consider the best aspects of each for future research.\" A table is provided for simpler comparison but I believe it is a shortcoming that a more extensive comparison of the approaches - i.e. directly comparing their conclusions, whether or not the results are compatible etc. - is not provided. The reader could in principle read each section and draw their own conclusion, but the point of a review article is to facilitate these comparisons.\n\n- It is not always clear what parts of the article are summaries / reviews of previous work and which parts are new. For example, are sections 3.2.2 / 3.2.3 reviews based on published work? The authors provide some citations but it is not clear if these citations cover the idea behind the entire subsection or are just related. I would recommend being 100% clear about these, i.e. \"in this section we review ... following [...]\" or \"We summarize the results of [...] and [...] with different notation\" or \"These are novel conclusions based on the work of [...]\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission is a review of two different approaches of studying neural collapse: \n\n1. Unconstrained Feature Models, assuming that the penultimate layer is independent of the inputs and can be independently optimzed\n2. Locally elastic models, assuming that the output of the network on sample $x$ is not significantly affected when performing SGD updates on an unrelated sample $x'$.\n\nThe authors review a number of papers and scenarios in each modelling approach and summarize their findings. They conclude with future directions and comments on how neural collapse affects transfer learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.\n\nMinor point: some stylistic / grammatical corrections are in order. For example, \"it's\" is often used instead of \"its\". Informal language is used e.g. \"can't\" instead of \"cannot\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a summary of recent work on the phenomenon of neural collapse, collecting previous results under a unified framework that is clear and easy to follow. It summarizes the findings of these papers and compares their assumptions and problem settings, yielding a comprehensive view of the research landscape.

Some reviewers noted that it is not always clear when new perspectives or derivations are provided, or when previous work is being reproduced directly. They also mentioned that sometimes the high-level conclusions were not accessible and were only visible after a detailed read of mathematical derivations. These weaknesses have been at least partially addressed, and in any case do not render the manuscript unsuitable for publication. I think this paper will provide value to the community and recommend acceptance.
Decision: Accept (poster)