Memorization-Dilation: Modeling Neural Collapse Under Noise
Authors: Duc Anh Nguyen
Ron Levie
Julian Lienen
Eyke Hüllermeier
Gitta Kutyniok
Venue: ICLR 2023 poster (2023)
Keywords: Neural collapse
feature representation
label smoothing
cross entropy

Abstract: The notion of neural collapse refers to several emergent phenomena that have been empirically observed across various canonical classification problems.  During the terminal phase of training a deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation, and the features of different classes tend to separate as much as possible. Neural collapse is often studied through a simplified model, called the layer-peeled model, in which the network is assumed to have ``infinite expressivity'' and can map each data point to any arbitrary representation. In this work we study a more realistic variant of the layer-peeled model, which takes the positivity of the features into account. Furthermore, we extend this model to also incorporate the limited expressivity of the network. Empirical evidence suggests that the memorization of noisy data points leads to a degradation (dilation) of the neural collapse. Using a model of the memorization-dilation (M-D) phenomenon, we show one mechanism by which different losses lead to different performances of the trained network on noisy data. Our proofs reveal why label smoothing, a modification of cross-entropy empirically observed to produce a regularization effect, leads to improved generalization in classification tasks.

Published as a conference paper at ICLR 2023
MEMORIZATION-DILATIOO MODELING NEURAL COLLAPSE UNDER LABEL NOISE
Due Anh Nguyen*
Ron Levie*
Julian Lienen"
LMU Munich
Technion
Paderborn University
Israel Institute of Technology
Gitta Kutyniok
Eyke Hüllermeier
LMU Munich
LMU Munich
University of Tromsø
ABSTRACT
The notion of neural collapse refers to several emergent phenomena that have been empirically observed across various canonical classification problems. During the terminal phase of training deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation, and the features of different classes tend separate as much as possible. Neural collapse is often studied through a simplified model, called the layer-peeled model, in which the network is assumed have "infinite expressivity" and can map each data point to any arbitrary representation. In this work we study a more realistic variant of the layer-peeled model, which takes the positivity of the features into account. Furthermore, we extend this model to also incorporate the limited expressivity the network. Empirical evidence suggests that the memorization noisy data points leads to degradation (dilation) of the neural collapse. Using model of the memorization-dilae
(M-D) phenomenon, we show one mechanism by which different losses lead to different performances of the trained network noisy data. Our proof' reveal why label smoothing, a modification of cross-entrope empirically observed to produce regularization effect, leads to improved generalization in classification tasks.
INTRODUCTION
The empirical success of deep neural networks has accelerated the introduction of new learning algorithms and triggered new applications, with pace that makes hard to keep up with profound theoretical foundations and insightful explanations. As one of the few yet particularly appealing theo- retical characterizations of overparameterized models trained for canonical classification tasks, Neural Collapse (NC) provides a mathematically elegant formalization of learned feature representations Papyan et al. (2220).
To explain NC, consider the following setting. Suppose we are given balanced dataset D (k)
{a .2n) RE[K],EI[N]
C. X' x y the instance space IRd and label space y = [N] :=
(I...., N}. i.c. each class [N] has exactly K samples (1) ..... x! (K) We consider network architectures commonly used in classification tasks that are composed of a feature engineering part
g: X + M (which maps input signal E X' to its feature representation g(x) E RM) and linear classifier (-)) + b given by weight matrix W E IR NMM as well as a bias vector b E R- Let w, denote the row vector of W associated with class n E [N]. During training, both classifier components are simultancously optimized by minimizing the cross-entropy loss.
These authors contributed equally this work.

---

Published as a conference paper at ICLR 2023
Denoting the feature representations g global mean of the features by
(k)
of the sample (k) by hn the class means and the
(k)
K
h
(R)
h
has
K
N
==1
NC consists of the following interconnected phenomena (where the limits take place as training progresses):
(NC1) Variability collapse. For each class n E [N], we have
Ex=1 (
hn - 0.
K
(NC2) Convergence to simplex equiangular tight frame (ETF) structure. For any m, E [N] with m o n, we have
hn, hl12
|| hll2
+ 0, and
hn h
hm h
h2 hll2 ||mm h ll
N
(NC3) Convergence to self-duality. For any [N], it holds
hn h
0.
h/n h 12
1,,112
(NC4) Simplification to nearest class center behavior. For any feature representation u E R holds
max (wn, u. + b, - arg min Iu hallz RE[N]
ne[N]
In this paper, we consider a well known simplified model, in which the features h are not parameterized by the feature engineering network but are rather free variables. This model is often referred to as layer-peeled model or unconstrained features model, see e.g. Lu Steinerberger (2020); |Fang al. (2021) y Zhu et al.) (2021). However, as opposed those contributions, in which the features h can take any value in M we consider here the case h 2 0 (understood component- wise). This is motivated by the fact that features typically the outcome of some non-negative activation function, like the Rectified Linear Unit (ReLU) or sigmoid. Moreover, by incorporating the limited expressivity of the network to the layer-peell model, propose memorization dilation (MD). Given such model assumptions, we formally prove advantageous effects of the so-called label smoothing (LSS) technique SZzgedy et al. (2015) (training with modification of cross-entropy (CE) loss), in terms of generalization performance. This is further confirmed empirically.
model, called
RELATED WORK
Studying the nature of neural network optimization is challenging. the past, plethora of theoretical
models has been proposed to do so Sun (2020). These range from analyzing simple linear Kunin et al. Laurent & Brecht (018) non-linear deep neural networks |axe et al. (2014) Yun et al. 2018 As one prominent framework among others, Neural Tangent Kernels Jacot al. (2018 Roberts al. (202I), where neural networks are considered as linear models on top of randomized features, have been broadly leveraged for studying deep neural networks and their learning properties.
(2019) Zhu al.
2020
Many of the theoretical properties of deep neural networks in the regime of overparamettrrizatin are still unexplained. Nevertheless, certain peculiarities have emerged recently. Among those, so- called "benign overfitting" [Bartlett et al. (2019); Li et al. (2021). where deep models are capable of perfectly fitting potentially noisy data by retaining accurate predictions, has recently attracted attention. Memorization has been identified one significant factor contributing to this effect Arpit jet al. [Sayaal et al. (202I), which also relates to our studies. Not less interesting, the learning risk of eerraaaeeerpr models shows double-descent behavior when varying the model
2

---

Published as a conference paper at ICLR 2023
complexity Nakkiran et al. (2020) as yet another phenomenon. Lastly, the concept of NC Papyan et al.
(2020) has recently shed light on symmetries in learned representations of overparameterized models.
After laying the foundation of a rigorous mathematical characterization of the NC phenomenon by
Papyan et al. (020), several follow-up works have broadened the picture. As the former proceeds
from studying CE loss, the collapsing behavior has been investigated for alternative loss functions.
For instance, squared losses have shown similar collapsing characteristics Poggio & Liao (2020)
2021). and have paved the way for more opportunities in its mathematical analysis, e-g.. by an
NC-interpretable decomposition Han et al. (202T). More recently, Kornblith et al. (2021) provide an exhaustive overview over several commonly used loss functions for training deep neural networks
regarding their feature collapses.
Besides varying the loss function, different theoretical models have been proposed analyze NC. Most prominently, unconstrained feature models have been considered, which characterize the penultimate layer activations as free optimization variables Mixon et al. (2020); [uu & Steinerberger (2020); E & Wojtowytsch| (202I). This stems from the assumption that highly overparameterizee models can approximate any patterns in the feature space. While unconstrained features models typically only look at the last feature encoder layer, layer-peeling allows for "white-boxing" further layers before the last one for a more comprehensive theoretical analysis Fang et al. (2021). Indeed, this approach has been applied in Tirer & Bruna (022). which namely extends the unconstrained features model by one layer as well as the ReL.U nonlinearity. On the other hand. /hu et al. (202T), Ji al. (2021) and ZZou al. (2022a) extend the unconstrained features model analysis by studying the landscape of the loss function therein and the related training dynamics. Beyond unconstrained
features models. [rgenn & Pilanci (202I) introduce convex analytical framework characterize the encoder layers for a more profound understanding of the NC phenomenon. Referring to the implications of NC on our understanding of neural networks, Hui et al. (2022) and Galanti al.) (2021) discuss the impact of NC on test data in the sense of generalization and transfer learning. Finally, Kothapalli al. (2022) provides a multifaceted survey of recent works related NC.
LEEERD MODEL WITH POSITIVE FEATURES
As prerequisite to the MD model, in this section introduce slightly modified version of the layer-peeled (or unconstrained features) model (see e.g. Zhu et al. (2021): Fang et al. (202T)), in which the features have to be positive. Accordingly, we will show that the global minimizers of the modified layer-peeled model correspond to an NC configuration, which differs from the global minimizers specified in other works and captures more closely the NC phenomenon in practice.
For conciseness, we denote by H the matrix formed by the features (+) E [N]. E [K] columns, and define || [W| and ||| to be the Frobenius norm of the respective matrices, i.e.
| N |w,,.] 1 and || Ek
EN hn
(1) We consider the regularized version
of the model (instead of the norm constraint one as in e.g. Fang et al. (021
A [
min W. W.
La (W, (W (W. H) + Aw |
(Pa)
s.t.
2
where λw, AH 0 are the penalty parameters for the weight decays. By La we denote empirical risk with respect to the LS loss with parameter a E [0, 1), where a = 0 corresponds to the conventional CE loss. More precisely, given a value of a, the LS technique then defines the label assigned to class n E [N] as the following probability vector:
(a) = (1-a)e, - E 1],
yh
where en N denotes the n-th standard basis vector and 1N R denotes the vector consisting of only ones. petp: RM + R be the function that assigns to each feature representation E R
Note that for simplicity we assume that the last layer does not have bias terms, i.e. b b- 0. The result can be however easily extended to the more general case when the biases do not vanish. Namely, in presence of bias terms, the statement of Theorem 3.2 and also its proof remain unchanged.
3

---

Published as a conference paper at ICLR 2023
the probability scores of the classes (as probability vector in IF N),
e ())..))
Pw (z) = softmax W=)
E [0, 11
Ei=1 elwii
m=1
Then the LS loss corresponding to a sample in class n E [N] is given by
la (, 2, (a))) == yn (())
log
PW (z))
Σ
plon log (pw (z)m)
(1)
and the LS empirical risk La is defined as
(W H)
NK ΣE
to W (/ y
We will show that common settings, the minimizers of correspond to neural collapse (NC) configurations, which we formalize in Def. below.
Definition 3.1 (NC configurations). Let K, M,N E N, M 2 N. A pair (W, H) of a weight matrix formed by w, R and feature matrix formed by columns h R (with
(A)
n E [N], k E [ is said to be a NC configuration
(i) The feature representations h/ within every class n E [N] equal for all [K] and thus equal to their class mean h, == K E h
J. have equal norms and form (entry-wise) non-negative orthog- (ii) The class means { onal system.
(iii) Let PA_ be the projection upon the subspace RM orthogonal = 인2 Σ hn Then for every [N], holds w, C.,,' hn for some constant C' independent of n.
Our main theorem in this section can be represented as follows.
Theorem 3.2. Let M 2 N, E [0. 1). Assume that Ny + 2 2/(N 11) < 1. Then any global minimizer the problem (aa NC configuration.
Note that the NC configurations defined in Definition 3.1 above differ significantly from the ones specified in other works, c.g. |Fng et al. (2021); Zhu et al. (2021) Zhou et (2022b) Tirer & Bruna 2022), see Appendix B.1 for more discussion.
4 THE MENORIZATTON-DILATOON MODEL
4.1 EXPERIMENTAL MOTIVATION
Previous studies of the NC phenomenon mainly focus the collapsing variability of training activations, and make rather cautious statements about its effects on generalization. For instance. Papyan al. (202 report slightly improved
accuracies for training beyond zero training eтror. Going step further, Zhu et al. (202)) show that the NC phenomenon also happens for overparameterized models when labels are completely randomized. Here, the models seem to memorize by overfitting the data points, however, rigorous study how label corruption affects generalization in the regime NC is still lacking.
To fill the gap, we advocate to analyze the effects of label corruption in the training data on the (previously unseen) test instead of the training feature collapse. Eventually, tight test class clusters go hand in hand with easier separation of the instances and, thus, smaller generalization error. Following Zhu et al. (202I), we measure the collapse of the penultimate layer activations by the A',, metric. This metric depicts the relative magnitude of the within-class covariance Ew with respect to the between-class covariance ΣB of the penultimate layer features and defined as
1
NC₁ = N
trace Ew Σ (
(2)
4

---

Published as a conference paper at ICLR 2023
where
N K
Σw == NK
(h (k) hh,) (K) -ha)² E RMXM
n= k=1
1
ΣB Chhn (l) h)) ERMMM
and Ei denotes the pseudo-inverse of m. Here, we adopt the notations from Section | (k) E RA denotes the feature representation of k-th sample in class n, hn the class mean and h the global mean. Moreover, we distinguish NCtaain and NC be calculated on the training and test instances, respectively, We call NC dilation.
Let now turn the notion of memorization, which is not uniquely defined in deep learning literature. Here, we define memorization in the context of the NC setting and in global manner, different from other works, e.g. Feldman & Zhang (222)). Formally, suppose that label noise is incorporated by (independently) corrupting the instance of each class label n in the training data with probability 7) E (0, 1), where corruption means drawing a label uniformly at random from the label space y. We denote the set of corrupted instances by [K]. For a given dataset D (with label noise v7),
define memorization
h
KE[R]
where h denotes the mean of (unseen) test instances belonging to class n.
We call the original ground truth label of sample its true label. We call the label after corruption, which may be the true label or not, the observed label. Since instances of the same true label tend to have similar input features in some sense, the network is biased to map them to similar feature representations. Instances are corrupted randomly, and hence, instances of the same true label but different observed labels do not have predictable characteristics that allow the network separate them in a way that can be generalized. When the network nevertheless succeeds in separating such instances, we say that the network memorized the feature representations of the corrupted instances in the training set. The metric mem in (3) thus measures memorization. The above memorization also affects dilation. Indeed, the network uses the feature engineering part to embed samples of similar features (that originally came from the same class), to far apart features, that encode different labels. Such process degrades the ability of the network to embed samples consistently, and leads to dilation.
To quantify the interaction between mem and AC we analyzed the learned representations h in the penultimate layer feature space for different noise configurations. One may wonder whether one can see a systematic trend in the test collapse given the memorization, and how this evolves over different loss functions.
To this end, trained simple multi-layer neural networks for two classes (N = 2), which subsampled from the image classification datasets (NIST[LCuu et al. (1998). FashionMNIST Xiao & Hinton (2009) and SVHN Netzer al. (20II). The labels are corrupted with noise degrees 1) E [0.025, 0.4,. The network consists of 9 hidden layers with 2048 neurons each, thus, it represents a vastly overparameterized model. The feature dimension M is set to the number of classes N. We trained these networks using the CE and LS loss with a smoothing factor a = 0.1, as well as the mean-squared error (MSE). Moreover, we consider label relaxation (LR) & Hüllermeier (202T) as generalization to LS with relaxation degree a 0.1. The networks were trained until convergence in 200 epochs (where the last 50 epochs did not make any significant changes) using SGD with an initial learning rate of 0.1 multiplied by 0.1 each 40 epochs and a small weight decay of 0.001. Moreover, we considered ReLU as activation function throughout the network, well as batch normalization in each hidden layer. A linear softmax classifier is composed on the encoder. We conducted each experiment ten times with different seeds.
al. (017)) IIFAAR-10KKizzeviis
The results for the above experimental setting are shown in Fig. 1 in which one can observe the trends of VNCT per memorization for various configurations. As can be seen, the figure shows an approximately linear correspondence between /NCcle and mem for the CE derivatives (CE and LS) on all datasets when mem not large.
5

---

Published as a conference paper at ICLR 2023
Test Collapse Memorization (N
cletto
C2
0.70
CE
L.
L5
0.30
0.5
LR
LR
0.4
MSE
0.5-
MSE
0.25
0.4
0.0
9.25
0.3
Noboe
0.2
0.2
0.00
0.3
0.5
2.0
1.0
1.5
0.5
1.0
1.5
0.5
155
Test. Accuraty Memorization
100
CE
CE
CE
L.
15
L.
94-
L.
25
MSE
MSE
LA
88
LR MSE
1R
MSE
0.23
0.20
90
86
88
2.0
0.0
1.0
1.5
2.5
0.0
0.5
0.0 1.0 2.0
Σ
Σ
Figure 1: Feature collapse of test instances terms VNC per memorization (top row) and the resulting test accuracies (bottom row) averaged over ten random seeds. Comparing the markers of the same color, it can be observed that LS consistently performs better than CE across all datasets, with very few exceptions (the very low noise degrees in 1if
Diluaion Merrarization (NNIST, Phdd
Memorization (CIFAh10,
6,6
Mat
Ued
Ved
6.5
65
0.30
0.70
0.30
0.00
0.4
6.3
0.55
0.13
0.2
6.2
0.10
0.50
0.30
0.0
0.65
MSE
0.45
0.0
05
10
15
2.
0.5
1.6
15
20
0.
05
1.
15
20
6.5
20
200
-h2le (Memorizatin)
Figure 2: Feature collapse the test instances in terms of VNC per memorization the feature dimension M varies.
Moreover, as CE and LS share the same slope, these results suggest that the degradation of the test collapse (aka dilation) is function of memorization and the network expressitivity, and not of the choice of the loss. The loss only affects how the noise translates to memorization, but not how memorization translates dilation. Even though the different memorization values in CE and LS, the memorization-dilatio curve is nevertheless shared between CE and LS. Hence, since LS leads the network to memorize less, it results in improved performance (cf. Fig. D.. We can further see that MSE and LR show a different memorization-dilaan correspondence, which means that these losses affect the inductive bias in a different way than CE and LS.
amount of noise is mapped to
We repeated the experiments for different values of the feature dimension M and show the example results in Fig. [ Here, one can see the similar trends of dilation per memorization as before. In the appendix, we provide additional results showing the behavior in the multi-class case N 2 with different models for label noise. The results support our MD model, and show that the memorization- dilation curve is roughly independent of the noise model for low-to-mid noise levels.
4.2 THE MEMORIZATION-DIAE DILLAIOO MODEL
Motivated by the observations the previous experiments, we propose the so-called memorization- dilation (MD) model, which extends the unconstrained feature model by incorporating the interaction between memorization and dilation as a model assumption. By this, we explicitly capture the limited expressivity of the network, thereby modeling the inductive bias of the underlying model.

---

Published as a conference paper at ICLR 2023
This model shall provide a basis to mathematically characterize the difference in the learning behavior of CE and LS. More specifically, we would like to know why LS shows improved generalization performance over conventional CE, as observed in past works Müller et al. (2019). The main idea | can be explained as follows. We first note that dilation is directly linked to generalization (see also Korbliihh al. (202T)) since the more concentrated the feature representations of each class are, the casier it is to separate the different classes with linear classifier without having outliers crossing the decision boundary. The MD model asserts that dilation is linear function of memorization. Hence, the only way that LS can lead to less dilation than CE. is if LS memorizes less than CE. Hence, the goal in our analysis is to show that, under the MD model, LS indeed leads to less memorization than CE. Note that this description is observed empirically in the experiments of Section 4.1
Next we define the MD model the binary classification setting.
Definition 4.1. We call the following minimization problem MD, Minimize the MD risk
Rana (U,r) == (W, H,r) + G,, UU),
with respect to the noisy feature embedding U- [un, u2] E R and the standard deviation r 2 0, under the constraints
CMDE
7 11/b1 U2ll 5
(4)
h h2l
CNDD
" ||hh wills
(5)
h1 h2ll
Here,
H R2XM
M and W e Rx2 form an NC configuration (see Definition 3.1
CMDD 0 called the memorization-ddilaton slope, 0 S a < 1 called the LS parameter,
1) >> 0 the noise level, and À > 0 the regularization parameter.
Fx.a is the component in the (regularized) risk that is associated with the correctly labeled samples,
Fa (,,,H,,)) = / la (W. hi t.,yl (0) A ||,, + w// dpp((o)
/ la (ca(W, h2 v. v/ + 112 e1 dx²(()
where ni and μ² are some probability distributions with mean and standard deviation and la is the LS loss defined in (I.
Gx.a is the component in the (regularized) risk that is associated with the corrupted samples, defined as
(W., GW U,r) ta (W u1 vla)) + La W. u2 v5Si) + |u2l2 + A |22 ||uz||2
The MD model can be interpreted follows. First consider the feature representations of the correctly labeled samples in each class as samples from distribution (namely n4:2 in Def. 4.1 with standard deviation r, parameter that measures the dilation of the class cluster. In a natural way, the corresponding risk Fx., involves the loss average over all samples, i.c. the loss integral over the distribution. For simplicity, we assume that the class centers h1, h2 as well as the weight matrix W are fixed as described by the NC configuration. This is a reasonable simplification as it has been always observed in the experiments.
On the other hand, the feature representations of corrupted samples are and u2 The amount of memorization in the first class defined to be v/| uz ||. since the more noise ry there is, the more
*Certainly one can, instead two single points un and uz, two distributions centered around un and uz. similarly as before for uncorrupted samples. However, it is quite straightforward to see that the minimization of the MD risk over the dilation of these two distributions is independent of other variables (not like r). and thus the minimum should he attained in the case of collapsing into two single points. Thus, for convenience we assume directly here that GA involves only two single points.
7

---

Published as a conference paper at ICLR 2023
examples we need to memorize. The amount of memorization in the second class is defined the same way. The (normalized) dilation is defined to be ||A:=Az]|* which models a similar quantity to (2).
The constraints (4) and (5) tell us that in order to map noisy samples u1 away from f2z, we have to pay with dilation r. The larger r is, the further away we can map U1 from h2. The correspondence between memorization and dilation is linear with slope CAs D by assumption. There are two main forces in the optimization problem: 2s, would like be as close as possible to its optimal position h1, and similarly U2 likes to be close to h2. In view of the constraints (4) and (5). to achieve this, r has to be increased to "max == mil hall"
Instances h
Feature 2
Membrization
CarD On the other hand, the optimal r for the term Fx,o is r O, namely, the
NC configuration. An optimal solution hence balances between memorization and dilation. See Fig. 3 for visualization of the MD model.
Feature 1
Our goal in this section is to compare the optimal value r in case of LS and CE losses. We will distinguish between these two cases by setting the value of a in the MD model to for CE and to some og > 0 for L.S. This will result in different scales of the feature embeddings H, denoted by H and HLS for CE and LS loss respectively, with the ratio
Figure 3: Exemplary illustration of the MD model for MLP network trained on MNIST. The instances h are test images correctly la- beled as 1, with centroid h2. The centroid of the test images with cor- rect label 0 is h1. The centroid of training images which were orig- inally labeled as 1 but are misla- beled as 0 is U1. The memoriza- tion of uu moves it close to h.. and causes dilation of instances hg
= | H >
(6)
which holds under the reasonable assumption that the LS technique is sufficiently effective, more precisely ag > 2/
The main result in this section will be Theorem 4.3 which states informally that in the low noise regime, the optimal dilation in case of LS loss is smaller than that case of CE loss. Before presenting this theorem, we will first establish several assumptions on the distributions 2 and the noise 1j in Assumption 4.2 Basically we allow a rich class of distributions and only require certain symmetry and bounded supports in terms of r, well as require 1) to be small in terms of the ratio y.
Assumption 4.2.
1. Let org > O. We assume that the solution of
la W. hi, vi + la (W,2,,y5
22 (1)) + Aw [ 2 + An ll
min W.
s.t.
H H20. 0.
is given by (W, H (WCE, HCF) for = 0 and (W, = (W H for
a ag.
2. Assume that the distributions nt and ut are centered, in the sense that
/ (w2 wT, 1) du((v) = (wo - ww, U) d1f(()) = 0,
[ (hh1 o) = /h2 u) du²(v) 0.
Furthermore, we assume that there exists a constant A > 0 such that ||l||| < Ar for any vector v that lies in the support of μu or in the support of μ²²
3. Assume that the noise level and the LS parameter on satisfy the following, We suppose Og > Aw which guarantees 7 == [ H > 1. We moreover suppose that is sufficiently small to guarantee 1/ < Č(1 - where =
Cup
v2 A 4
Now main result in this section be formally stated as below.
8

---

Published as a conference paper at ICLR 2023
Theorem 4.3. Suppose that Assumption 4.2 holds true for M 2nd == du Let Le and T. J.S be the optimal dilations, i.e. the optimum r in the MD problem, corresponding to the CE and LS loss (accordingly 0 and ag), respectively. Then holds that
J.S
h CC 11
12 h
Theorem 4.3 reveals a mechanism by which LS achieves better generalization than CE. It is proven that LS memorizes and dilates less than CE, which is associated with better generalization. Note that in practice, the data often have noise in the sense that not all examples are perfectly labeled. More importantly, examples from different classes may share many similarities, a situation that is also covered by the MD model: the feature representations of samples from those classes are biased toward each other. In this case, LS also leads to decreased dilation which corresponds to better class separation and higher performance Korbllihh et al. (202T).
Interestingly, the concurrent work Zhou et al. (2022b) has shown that in the noiseless setting CE and LS lead to largely identical test accuracy, which seems to contradict the statement that LS performs better claimed by our work as well as many others, e.g. Kornblith et al. (202]); ); Müller et al. (211)). However, note that Zhou et al. (2022b) requires the network to be sufficiently large so that it has enough expressive power to fit the underlying mapping from input data to targets, as well as to be trained until convergence. While the latter is easy to obtain, it is difficult even to check if the first requirement holds. The difference between the two results is hence possibly caused by the effect of noise and by the network expressivity: while we aim model the limited expressivity by the MI) relation, Zhou et al. (2022b) focuses on networks with approximately infinite expressivity.
The MD model combines statistical term Fx,a: that describes the risk over the distribution of feature embeddings of samples with clean labels, and an empirical term r/G that describes the risk over training samples with noisy labels. One point of view that can motivate such a hybrid statistical- statisticllempirccal definition is the assumption that the network only memorizes samples of noisy labels, but not samples of clean labels. Such a memorization degrades (dilates) both the collapse of the training and test samples, possibly with different memorization is not limited to corrupted labels, but can also apply to samples of clean & Zhang (2020), by which the learner can partially negate the dilation of the training features (but not test features). The fact that our model does not take the memorization of clean samples into account is one of its limitations. We believe that future work should focus on modeling the total memorization of all examples. Nevertheless, we believe that our current MD model has merit, since 1) noisy labels memorized more than clean labels, and especially in the low noise regime the assumption of observing memorization merely for corrupted labels appears reasonable, and 2) our approach and proof techniques can be the basis of more elaborate future MD models.
cemmorizttion-dilatie
slopes. However,
5 CONCLUSION
In this paper, we first characterized the global minimizers of the Layer-Peeled Model (or the Un- constrained Features Model) with the positivity condition the feature representations. Our characterization shows some distinctions from the results that haven been obtained in recent works for the same model without feature positivity. Besides the conventional cross-entropy (CE) loss, we studied the model in case of the label smoothing (LS) loss, showing that NC also occurs when applying this technique.
Then we extended the model to the so-called cinnorzaation-iilatin (MD) Model by incorporating the limited expressivity of the network. Using the MD model, which is supported by our experimental observations, we show that when trained with the LS loss, the network memorizes less than when trained by the CE loss. This poses one explanation to the improved generalization performance of the LS technique over the conventional CE loss.
Our model has limitations, however, namely that it is limited to the case two classes. Motivated by promising results on the applicability of our model to the multi-class setting, we believe that future work should focus on extending the MD model in this respect. With such extensions, memorization- dilation analysis has the potential to underlie systematic comparison of the generalization capabilities of different losses, such as CE, LS, and label relaxation, by analytically deriving formulas for the amount of memorization associated with each loss.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was partially supported by the German Research Foundation (DFG) within the Collab- orative Research Center "On-The-Fly Computing" (CRC 901 project no, 160364472). Moreover, the authors gratefully acknowledge the funding of this project by computing time provided by the Paderborn Center for Parallel Computing (PC?).
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 2017.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. CoRR, abs/1906. 300, 2019.
Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers, 2021.
Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In Proceedings of the 38th International Conference Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 3004-3014. PMLR, 2021.
Cong Fang. Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences of the United States of America, 118, 2021.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Tomer Galanti, András György, and Marcus Hutter. On the role neural collapse in transfer learning. CoRR, abs/2112.15121, 2021.
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. CoRR, abs/2106.02073, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA. June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. CoRR, abs/2202.08384, 2022.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J. Su. unconstrained layer-peeled perspective on neural collapse. CoRR, abs/21 10.02796, 2021.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? CoRR, abs/2010.16402, 2021.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review modelling principles and generalization. CoRR, abs/2206.04041, 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Canada, 2009.
10

---

Published as a conference paper at ICLR 2023
Daniel Kunin, Jonathan Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp- 2560-35699 PMLR, 09-15 Jun 2019.
Thomas Laurent and James von Brecht. Deep linear networks with arbitrary loss: All local minima are global. In Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssaan, Stockholm, Sweden, July 1 10-15 (1-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2908-2913. PMLR, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, 2020.
Zhu Li, Zui-Hua Zhou, and Arthur Gretton. Towards understanding of benign overfitting in neural networks. CoRR, abs/2106.03212 2021.
Julian Lienen and Eyke Hüllermeier. From label smoothing label relaxation. In Thirty-Fifth AAAI Conference on Artificial Intellignce, AAAA 2021, Virtual Event, February 2-9, 2021, pp. 8583-8591, AAAI Press, 2021.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. CoRR, abs/2012.08465. 2020.
Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. CoRR, abs/2011.1166 2020.
Rafael Müllrr, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 4795, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NIPS, Granada, Spain, November 12-17, Workshop on Deep Learning and Unsupervised Feature Learn- ing. 2011.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. CoRR, abs/2008.00 186, 2020.
Tomaso A. Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square loss. 2020.
Tomaso A. Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. CoRR, abs/2101.00072, 2021.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. CoRR, abs/2106. 10165, 2021.
Amartya Sanyal, Puneet K. Dokania, Varun Kanade, and Philip H. S. Torr. How benign is benign overfitting? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.

---

Published as a conference paper at ICLR 2023
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Ruoyu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China, 8:249-294 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. CoRR, abs/2202.08087, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.07747, 2017.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Global optimality conditions for deep neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. CoRR, abs/2203.01238, 2022a.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. CoRR, abs/2210.02192 2022b.
Zhihui Zhu, Daniel Soudry, Yonina C. Eldar, and Michael B. Wakin. The global optimization geometry of shallow linear neural networks. J. Math. Imaging Vis., 62(3)279-222,, 2020.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. CoRR, abs/2105.023755 2021.
12


Reviews:
[{"review_id": "gF2uYh3kpHn", "reviewer": "Reviewer_ajzH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the phenomenon of neural collapse (NC), where representations of multi-class examples tend to collapse to the mean representation in a structured fashion.  The authors argue that the commonly used layer peeling model for NC is overly simplistic; in particular the assumption of \"infinite expressivity\" does not hold well with networks in practice.  They argue instead for a refinement of the layer peeling model that takes the signs of the features into account, and limits the expressiveness of the model to represent transformed inputs.  They use this model to study the interplay of memorization (which they define as the deviation from the expected NC structure of data with injected label noise) and dilation (which they define as deviation of the collapsed within-class mean structure that defines neural collapse) on the generalization ability of networks trained on both cross entropy and label smoothing.  \n", "summary_of_the_review": "The authors present a model that extends the Layer-Peeled model, introducing positivity conditions on the feature representations.  They develop the notions of memorization and dilation, which they use to explain the improved generalization of the label smoothing loss as compared to standard cross-entropy.  \n\nWhile i have some reservations about the wording of the definitions, and a more philosophical difference of opinion about memorization, I do not think either of these are serious impediments to the merit of the paper.\n\nI will say that I hope the authors do not stop at studying the effect of label noise on NC structures. If we are to consider the penalty to generalization that NC imposes on models that train until NC onset, then surely more common issues in generalization (e.g distribution shifts) should be considered as the centre of future work.", "strength_and_weaknesses": "### Strengths\n- The authors are attempting to make the study of neural collapse applicable to more real-world models, by refining the layer-peeling model to align with the properties of many commonly used DNNs (e.g non-negativity of features).  This is important work needed to understand the training dynamics of modern networks.\n\n- Though this paper unavoidably introduces a lot of notation, the authors do a good job in guiding readers thorugh its deployment in service of their arguments.\n\n- The authors also explain the properties of NC very clearly, and do a good job of succinctly summarizing the development of tools for the study of neural network optimization.  Though I would add I think the study on memorization by Feldman and Zhang (NeurIPS 2020) merits mentioning here, as they tackle long tailed natural data distributions (which combined with the rigidity implied by NC means that memorization would be necessary to achieve low training error). \n\n### Weaknesses\n\n- In section 1, the given definition for **NC2** has a typo.  I believe it should be that the inner product between any *different* orthonormal class means (centered at $\\mathbf{h}$) approaches $- \\frac{1}{N - 1}$.  As written, it suggests that $<x,x> \\rightarrow - \\frac{1}{N - 1}$\n\n- In section 3 just after the definition of $\\mathcal{P}_{\\alpha}$, it looks as if $\\mathbf{y}_{n}^{(\\alpha)} is defined by label smoothing, but it might be good to remind readers here, as there is a lot of notation introduced in this section.\n\n- Could the authors clarify element (iiii) of definition 3.1?  Definition 3.1 by inspection seems to agree with the original definition of NC properties,  with the exception of (iii).  Perhaps to aid the reader here, each component of Definition 3.1 could be annotated with the NC property that it entails?  E.g  component (i) seems to entail NC1\n\n- The first sentence in the last paragraph of section 3 is awkward (and contains a repitition \"in in\"). In the sentence, do the authors mean that def’n 3.1 (that includes condition (iii)) allows for Theorem 3.1 in Trier & Bruna 2022 to be established?   Could the authors please clarify.  I read this as suggesting \"Definition 3.1 and the orthogonal frame result in Theorem 3.1 in …\"\n\n- The definition of the dilation quantity $\\mathcal{NC}_1$ could use more motivation.  Why are we examining the trace of this product?  What intuition does this give us about NC test data? Zhu et al (which the authors cite as providing the definition) explain that $\\mathcal{NC}_1$ is intended to measure the within-class variability collapse.  I presume that this is because the trace of $\\Sigma_W$ will vanish as NC takes hold, while $\\Sigma_B$ will approach a constant.  Perhaps a footnote could help guide the reader in the absence of Zhu et al’s explanation? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "egvPmzCA1h", "reviewer": "Reviewer_7FKE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the phenomenon of neural collapse (NC) under several variants of the layer-peeled model.  Since features from modern networks are the outcome of some non-negative activation functions, such as ReLU, the paper first considers the case of non-negative features and shows that label smoothing also produces NC solutions in this case. The authors then propose a new memorization-dilation model on test data with randomly label-corrupted training samples, which shows a linear relation between NC degree (dilation) and overall distance between test samples and their corresponding class-mean (memorization).  Finally, in the label-corrupted setting, they formally prove the advantage of label smoothing over cross-entropy in binary classification tasks, and show some supporting experiments.", "summary_of_the_review": "This paper is well-organized and provides interesting theoretical and empirical results on neural collapse with label smoothing. The memorization-dilation model could provide further insight into the connection between neural collapse and generalization, though it is not clear how the performance with label noise can be used to understand the generalization performance in the standard setting without label noise. I look forward to the authors’ response to revise my decision. ", "strength_and_weaknesses": "## Strength:\n- This paper formally shows NC solutions for label smoothing and cross-entropy (CE) under nonnegative features. Nonnegative features are only considered for MSE loss in the previous work. \n- An interesting label-corrupted experiment is proposed, and the linear relation between dilation and memorization when corrupted levels are not large is very interesting. \n- The proposed memorization-dilation model may provide further insight into the connection between neural collapse and generalization.\n\n## Weakness:\n- As the nonnegative features model has already been studied in Tirer & Bruna (2022), though for the MSE loss, the authors may highlight the technical challenges in extending the results to label smoothing. \n- This paper shows that on training data with label noise, label smoothing could be more robust to label noise and achieve better performance on the original testing data. Based on this, the paper claims in many places that label smoothing leads to improved generalization in classification tasks. How does this better performance under label noise translate to better generalization in the standard case without label noise? \n- Following the above point, the recent work [A] shows that label smoothing and CE indeed produce similar performance when the network is sufficiently large and trained sufficiently long in the standard way without label noise. Since label smoothing and CE produce similar NC features under the unconstrained feature models, is the better performance of label smoothing on label noise because it has a different convergence speed than CE? It will be of interest to perform experiments with more iterations and see whether the results are the same. \n- It is interesting to note that MSE has better testing performance in Figure 1. Could the authors provide some comments on this? \n- Does the memorization-dilation model only consider one mislabeled training sample per class?\n- Could the memorization-dilation model be extended to the multi-class case?  \n\n[A] Zhou et al., Are All Losses Created Equal: A Neural Collapse Perspective; arXiv preprint arXiv:2210.02192; 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a-GVDvCnCu", "reviewer": "Reviewer_xSrW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper theoretically studies the notion of neural collapse (NC) -- the penultimate layer representations of all the examples in a specific class collapse to a single representation, and separate from other classes. Specifically, this paper studies this notion under uniform label noises, and proposed a model called memorization-dilation (M-D), in which it shows labels smoothing leads to less memorization and better generalization.", "summary_of_the_review": "This paper studies the phenomenon of neural collapse (NC) under label noises using a theoretical model to characterize the relation between memorization and NC. One of my major concern is that the theoretical model is too different from a real deep neural network, which is especially crucial for the topic of this paper (label noises and memorization). \n\n-------\nPost rebuttal: thanks to the authors for clarifying my questions. ", "strength_and_weaknesses": "**Strength**\n1. This paper theoretically studied neural collapse (NC) with a definition that describes more precisely the empirically observed phenomena in practice comparing to other papers.\n2. This paper formally study the NC under label noises with a memorization-dilation model, and make connection between memorization and NC.\n3. The theoretical model is motivated from empirical observations with deep neural networks on real data.\n\n**Weakness**\n1. The theoretical setup for studying neural collapse is quite different from what a real neural network behave in practice. In particular, it assumes \"infinite expressivity\" and allow the representation of each example to be freely trainable parameters. The resulting formulation looks like a matrix factorization problem with a classification loss. I think the most interesting part of neural network representation learning (and collapsing) would mostly happen jointly in the layers below the final classifier layer, and how those layers share weights when jointly computing the representations for all the training examples. This is especially important for the topic of this paper, where label noise is introduced to study memorization. In this case, the lower layers are forced to build different representations for visually similar inputs in some cases when they are assigned different training labels. But this interactions would be completely missing in the model proposed here.\n\n2. I appreciate that the empirical studies in this paper uses deep neural networks. However, since the theoretical models are so different, I would like to see empirical studies with a similar setup, using networks with approximately infinite expressivity. Or better, simply using a model where the representations are directly optimizable free variables, and see if the empirical observations are still similar and equally well for motivating the M-D models. One question here is how to get the representations for the test examples for computing Eq (3) in this case.\n\n3. The analysis is limited to binary classification problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8NN-arSInw", "reviewer": "Reviewer_ww8W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study gives the solution structure of the label smoothing (LS) loss with positivity constraint, which is an orthogonal variant of the original neural collapse configurations. The paper further explores why LS loss has a better generalization than CE loss when label corruption emerges based on the memorization-dilation model. It reveals that LS loss induces less memorization, so leads to less dilation, which explains its better generalization.  ", "summary_of_the_review": "The paper suffers from some issues, as listed in the weaknesses. Most importantly, the main claim lacks rigorous supports. Some settings in the model are confusing. ", "strength_and_weaknesses": "Strengths:\n\nThe study gives the solution structure of the LS loss, whereas previous studies mainly focus on the original CE loss. \nThe study gives an explanation of the better generalization of LS loss than CE loss under label corruption. \n\n\nWeaknesses:\n\n- Misleading description. The authors claim to address the model with noisy data. However, they actually only consider label corruption. Noisy data does not equivalently refer to label noise. A more precise description should be adopted. \n\n- The positivity constraint is confusing. Although the last layer will have positive feature when ReLU is performed, a neural network does not necessarily end with a ReLU activation in most cases. If an identity connection or a BN layer is appended, we can easily get rid of the positivity constraint. More importantly, I do not see a necessary connection between the positivity constraint and the later analysis of label corruption. Is your result (LS loss shows less memorization and dilation) valid only when the positivity constraint is accompanied? \n\n- The claim that less dilation leads to better generalization lacks rigorous support. As indicated by your definition, dilation reflects the compactness and separation of test features. But generalization ability seems to be more related to loss and accuracy on test set. So, a more rigorous relation between “dilation” and the “generalization” in your context should be constructed. Otherwise, the claim seems to be groundless. \n\n- The memorization-dilation model is confusing. First, it only considers two classes, which is unrealistic. Why do the authors only consider two classes? I do not think it would be a simple extension by generalizing the two-class result into multiple classes. Besides, why are H and W fixed as the optimal solution in the memorization-dilation model? Its motivation and rationality are unclear and need more discussion. \n\n- The second equation in (NC2) in page2 is wrong. \n\n- I suggest that the authors consider more cases in Theorem 3.2. It would be better if the authors could first give the solution structure of the LS loss without positivity constraint, and the CE loss with the positivity constraint, and then deal with the LS loss with positivity constraint as stated in Theorem 3.2.\n\n\n\n------ After rebuttal\n\nThe authors address most of my questions and concerns. I increase my score to 6.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster