Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?
Authors: Yibo Yang
Shixiang Chen
Xiangtai Li
Liang Xie
Zhouchen Lin
Dacheng Tao
Venue: NeurIPS 2022 Accept (2022)
Keywords: neural collapse
simplex ETF
imbalanced learning
machine learning
learning theory

Abstract: Modern deep neural networks for classification usually jointly learn a backbone for representation and a linear classifier to output the logit of each class. A recent study has shown a phenomenon called neural collapse that the within-class means of features and the classifier vectors converge to the vertices of a simplex equiangular tight frame (ETF) at the terminal phase of training on a balanced dataset. Since the ETF geometric structure maximally separates the pair-wise angles of all classes in the classifier, it is natural to raise the question, why do we spend an effort to learn a classifier when we know its optimal geometric structure? In this paper, we study the potential of learning a neural network for classification with the classifier randomly initialized as an ETF and fixed during training. Our analytical work based on the layer-peeled model indicates that the feature learning with a fixed ETF classifier naturally leads to the neural collapse state even when the dataset is imbalanced among classes. We further show that in this case the cross entropy (CE) loss is not necessary and can be replaced by a simple squared loss that shares the same global optimality but enjoys a better convergence property. Our experimental results show that our method is able to bring significant improvements with faster convergence on multiple imbalanced datasets.

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?
Yibo Yang', Shixiang Chen', Xiangtai Li?, Liang Xie Zhouchen Lin Dacheng Tao
ID Explore Academy, Beijing, China
Keey Lab. of Machine Perception (MoE), School of Intelligence Science and Technology. Peking University State Key Lab of CAD&CG, Zhejiang University "Institute for Artificial Intelligence, Peking University
Pazhou Laboratory, Guangzhou, China
The University of Sydney, Australia
Abstract
Modern deep neural networks for classification usually jointly learn a backbone for representation and linear classifier to output the logit of each class. A recent study has shown a phenomenon called neural collapse that the within-class means features and the classifier vectors converge the vertices of simplex equiangular tight frame (ETF) at the terminal phase of training on a balanced dataset. Since the ETF geometric structure maximally separates the pair-wise angles of all classes in the classifier, it is natural to raise the question, why do we spend an effort to learn classifier when we know its optimal geometric structure? In this paper, we study the potential of learning a neural network for classification with the classifier randomly initialized as an ETF and fixed during training. Our analytical work based on the layer-peeled model indicates that the feature learning with fixed ETF classifier naturally leads to the neural collapse state even when the dataset is imbalanced among classes. We further show that in this case the cross entropy (CE) loss is not necessary and can be replaced by simple squared loss that shares the same global optimality but enjoys better convergence property. Our experimental results show that our method is able bring significant improvements with faster convergence multiple imbalanced datasets.
Introduction
Modern deep neural networks for classification are composed of a backbone network to extract features, and a linear classifier in the last layer to predict the logit of each class. As widely adopted in various deep learning fields, the linear classifier has been learnable jointly with the backbone network using the cross entropy (CE) loss function for classification problems [19, 10, 13].
recent study reveals a very symmetric phenomenon named neural collapse, that the last-layer features of the same class will collapse their within-class mean, and the within- means of all classes and their corresponding classifier vectors will converge to the vertices of a simplex equiangu- lar tight frame (ETF) at the terminal phase of training balanced dataset [28]. A simplex ETF describes a geometric structure that maximally separates the pair-wise angles of K vectors in Rd and all vectors have equal f2 norm. As shown in Figure when d = K 1, the ETF
d > K reduces to a regular simplex. Following studies focus on unveiling the physics of such phenomenon based on the layer-peeled model (LPM) [5] unconstrained feature model [24]. They peel off the top- most layer, so the features are independent variables to optimize [41]. Although this toy model is im- practicable for application, inherits the nature of feature and classifier learning in real deep networks.
*: corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

It has been shown that the optimality of LPM satisfies neural collapse under the CE loss with constraints [22, 5, 41, 8]. regularization [50], or even no explicit constraint [15]. The other studies turn to analyze the mean squared error (MSE) loss and also derive the neural collapse solution as global optimality [24, 9, 31,38) However, all these theoretical results are only valid for balanced training.
Albeit neural collapse is an observed empirical result and has not been entirely understood from a theoretical point, it is intuitive and sensible. Features collapsing to their means minimize the within- class variability, while the ETF geometric structure maximizes the between-class variability, so the Fisher discriminant ratio [6, 33] is maximized, which corresponds to an optimal linearly separable state for classification. Several works have shown that neural networks progressively increase the linear separability during training [26, 27, 463. Therefore, classifier that satisfies the ETF structure should be a "correct" answer for network training. However, as pointed out by [5], in the training on an imbalanced dataset, the classifier vectors of minor classes will be merged, termed as minority collapse, which breaks up the ETF structure and deteriorates the performance on test data. So a learnable classifier does not always lead to neural collapse when training on imbalanced data. With the spirit of Occam's 's Razor [36], we raise and study the question:
Figure 1: An illustration of simplex equiangular tight frame when d = 3 and K = 4. The black spheres are the ver- tices of the ETF, The "+" and "x" signs in different colors re- fer to features and classifier vectors of different classes, re- spectively. Neural collapse in- dicates that the features and classifier vectors are aligned with the same simplex ETF.
Do we really need to learn a linear classifier at the end of deep neural network?
Since a backbone network is usually peeepparameterized and can align the output feature with any direction, what makes classification effective should be the feature-classifier geometric structure, instead of their specific directions. In this paper, we propose initialize the classifier using randomly generated ETF and fix it during training, i.e., only the backbone is learnable. It turns out that this practice makes the LPM more mathematically tractable. Our analytical work indicates that even in the training on an imbalanced dataset, the features will converge to the vertices of the same ETF formed by the fixed classifier, i.e., neural collapse inherently emerges regardless of class balance.
We further analyze the cross entropy loss function. We point out the reason for minority collapse from the perspective of imbalanced gradients with respect to learnable classifier. As comparison, our fixed ETF classifier does not suffer from this dilemma. We also show that the gradients with respect the last-layer feature are composed of a "pull" term that pulls the feature into the direction of its corresponding classifier vector of the same class, and a "push" term that pushes it away from the other classifier vectors. the pull-push mechanism in the CE loss that makes the features collapsed and separated. However, in our case, the classifier has been fixed as a "correct" answer. As a result, the "pull" term is always an accurate gradient towards the optimality, and we no longer need the "push" term that may cause inaccurate gradients. Inspired by the analyses, we further propose simple squared loss, named dot-regression (DR) loss, which shares a similar "pull" gradient with the CE loss, but does not have any "push" term. It has the same neural collapse global optimality, but is proved to enjoy a better convergence property than the CE loss.
The contributions of this study can be listed as follows:
We theoretically show that the neural collapse optimality can be induced even in imbalanced learning as long the learnable classifier is fixed as simplex ETF,
Our analysis indicates that the broken neural collapse in imbalanced learning is caused by the imbalanced gradients w.r.f. a learnable classifier.
We point out that our fixed ETF classifier no longer relies on the "push" gradient w.z. feature that crucial for the CE loss but may be inaccurate. We further propose a new loss function with only "pull" gradient. Its better convergence property is theoretically proved.
Statistical results show that the model with our method converges more closely to neural collapse. In experiments, our method able to improve the performance of classification on multiple imbalanced datasets by a significant margin. Our method enjoys a faster convergence compared with the traditional learnable classifier with the CE loss. As an extension, we find that our method also works for fine-grained classification.
2

---

2 Related Work
Neural Network for Classification. Despite the success of deep learning for classification [18, 35, 10, 13, 45, 11]. theoretical foundation that can guide the design of neural architecture still remains an open problem and inspires many studies from different perspectives [7, 23, 20, 44, 1, 14, 42]. The last-layer linear classifier has been relatively transparent. In most cases, it is jointly optimized with the backbone network. In long-tailed classification, two-stage method of training backbone and classifier successively is preferred [2, 16, 48]. Some prior studies have shown that fixing the classifier as regular polytopes [29, 30] and Hadamard matrix [12] does not harm the classification performance. Zhu et al. [50] tried the practice of fixing the classifier as a simplex ETF in experiment, but does not provide any benefit except for the saved computation cost. We also study the potential of using fixed classifier throughout the training. But different from these studies, our proposed practice is proved to be beneficial to imbalanced learning by both theoretical and experimental results.
Neural Collapse. In [28], neural collapse was observed at the terminal phase of training on balanced dataset. Albeit the phenomenon is intuitive, its reason has not been entirely understood, which inspires several lines of theoretical work on it. Papyan et al. [28] proved that if features satisfy neural collapse, the optimal classifier vectors under the MSE loss will also converge to neural collapse based on [40]. Some studies turn to simplified model that only considers the last-layer features and classifier as independent variables. They prove that neural collapse emerges under the CE loss with proper constraints regularizations [41, 5, 22, 50, 15, 8J. Other studies focus on the neural collapse under the MSE loss [24, 9, 31, 38, 49, 32]. In [4], a convex formulation is proposed for particular network to explain neural collapse. However, current results are only valid for balanced training. Inspired by neural collapse, the study [43] modifies the CE loss for imbalanced learning, but still cannot rigorously induce neural collapse. Our work differs from these studies in that theoretically show that neural collapse can inherently happen even in imbalanced learning. We also derive a new loss function that theoretically enjoys better convergence property than the CE loss.
Preliminaries
3.1 Neural Collapse
Papyan et al. [28] revealed the phenomenon that the last-layer features will converge to their within- class means, and the within-class means together with the classifier vectors will collapse to the vertices a simplex equiangular tight frame at the terminal phase of training on a balanced dataset.
Definition (Simplex Equiangular Tight Frame) A collection of vectors m; E Rdd 1,2 K, d 2 K 1, is said to be simplex equiangular tight frame if:
M
V K U
K
(I
1K
where M = [m,,... mx] E RAxk U RAxk allows rotation and satisfies U = IK. Ix is the identity matrix, and 1 K an all-ones vector.
All vectors simplex ETF have an equal l norm and the same pair- angle, i.e.,
K
m my =
K - ô,g K K- Vi,j E [1,
(2)
where d.,y equals to 1 when and 0 otherwise. The pair-wise angle K-I is the maximal equiangular separation of K vectors in Rd [28].
1
Then the neural collapse (NC) phenomenon can be formally described as:
(NC1) Within-class variability of the last-layer features collapse: Σw - O. and Ew == Avgra ((be,i (b,, h&) "T, where hg. is the last-layer feature of the i-th sample in the k-th class, and h2 Avg, (bg,( is the within-class mean of the last-layer features in the k-th class;
(NC2) Convergence to simplex ETF: h4 (h+ hc)/||b hell, & e [1. satisfies Eq. (2). where hg is the global mean of the last-layer features, i.e., hG = Avg., (ha.:)
(NC3) Self duality: hig wx w where We is the classifier vector of the k-th class;
(NC4) Simplification to the nearest class center prediction: argmax, (h, ws) = argmin -hll.
where h is the last- feature of sample to predict for classification.
3

---

3.2 Layer-peeled Model
Neural collapse has attracted a lot of researchers to unveil the physics of such an elegant phenomenon. Currently most studies target the cross entropy loss function that is widely used in deep learning for classification. It is defined as:
exp(h" we)
Co W) - log
(3)
R=1 exp(h sxp(h)w::)
where h E Rd is the feature output by a backbone network with input x, W [W WK] E R4xK K a learnable classifier, and is the class label of x.
However, deep neural network as highly interacted function is difficult to analyze due to its non- convexity. A simplification is always necessary make tractable analysis. For neural collapse, current studies often consider the case where only the last-layer features and classifier are learnable without considering the layers in the backbone network. termed as layer-peeled model (LPM) [S], and can be formulated as
K
mB
min W.H. H
LCe (hg,5, W),
s.t. || & Ew, A1 < k < K
(4)
lhs S En, VI < k < K, sis
where h is the feature of the i-th sample in the k-th class, W is a learnable classifier, La is the cross entropy loss function defined in Eq. (3), le norm constraints for feature h and classifier vector w, respectively.
the number of samples, and En and Ew are
Albeit the LPM cannot be applied real application problems, serves as an analytical tool and inherits the learning behaviors of the last-layer features and classifier in deep neural network. Actually, the learning of a backbone network is through the multiplication between the Jacobian and the gradient with respect the last-layer features, i.e., -W
8H 0C where W W.t.11 denotes the parameters in the backbone network, and H the collection of the last-layer features.
It has been shown that the global optimality for the LPM in Eq. (4) satisfies neural collapse in the balanced case [5,8] 8]. The CE loss with regularization also has similar conclusion [50]. However, the results in current studies are only valid for training on balanced dataset, i.e., n& = n, v1 S k S K.
Main Results
4.1
ETF Classifier
From the neural collapse solution, NC1 minimizes the within-class covariance Ew, and NC2 maximizes the between-class covariance Eg by the ETF structure. So the Fisher discriminant ratio, defined as En1' EB, is maximized, which can measure the linear separability and has been used to extract features to replace the CE loss [37]. So we deem an ETF as the optimal geometric structure for the linear classifier. Considering that backbone network is usually over- parameterized and can produce features aligned with any direction, in this paper, we study the potential of learning a network with the linear classifier fixed an ETF, named ETF classifier.
Concretely, we initialize the linear classifier W as a random simplex ETF by Eq. (1) with a scaling of / /Ew as the fixed length for each classifier vector, and only optimize the features H. In this case, the layer-peeled model (LPM) in Eq. (4) reduces to the following problem:
min
Co W*),
||ba < En, V1 v1 K, ng.
Note that the sample-wit wise constraint of H and the class-wir constraint of W in Eq. (4) are more strict than the overall constraints in [5], but are still active with the same global optimality. The model is also known as unconstrained feature model [24, 50] when the norm constraints omitted replaced by regularizations.

---

where W is the fixed classifier as simplex ETF and satisfies:
K
1
Wik w Eww
K-1 1
wk, W/ E [1, [],
(6)
K- K
where og equals to 1 when h: K' and 0 otherwise.
We observe that this practice decouples the multiplied learnable variables hy. and we of LPM Eq. (4), and makes the model in Eq. (5) convex problem that is more mathematically tractable. We term the decoupled LPM in Eq. (5) as DLPM for short. We have the global optimality for DLPM the imbalanced case with the ETF classifier in the following theorem.
Theorem No matter the data distribution balanced or not among classes (it is allowed that ak, k' E [J, k + k,, such that >> nyy) ) any global minimizer H [ha 1 k S K, S i S nu] of Eq. (5) converges to simplex ETF with the same direction as W and a length of vEH. ie..
his T w V
(K-1 (
K OR
1 ..I. -
& M
which means that the neural collapse phenomenon emerges regardless of class balance.
Proof 1 Please refer to Appendix A for our proof.
Remark As observed in [5], LPM in the extreme imbalance case would suffer from "minority collapse", where the classifier vectors of minor classes close merged into the same vector, which explains the deteriorated classification performance of imbalanced training. As comparison, Theorem shows that DLPM with our ETF classifier can inherently produce the neural collapse solution even the training on imbalanced data.
Although practice of using a fixed ETF classifier simplifies the problem, it actually brings theoretical merits that also get validated in our long-tailed classification experiments.
4.2
Rethinking the Cross Entropy Loss
In this subsection, we rethink the CE loss Lov from the perspective of gradients with respect to both feature and classifier to analyze its learning behaviors.
4.2.1
Gradient w.r.f Classifier
We first analyze the gradient of LeE w.r.f learnable classifier W = [W1, wx] E Raxk
OLCE
78k
-
(1- (bx,i) h +
Σ
Pke (bw he
Ow
where P& (h) is the predicted probability that h belongs to the k-th class. It is calculated by the softmax function and takes the following form in the CE loss:
exp(h" wx)
Pk (h)
1 1 h: S.K K.
(9)
Σn exp(h" wer)
We make the following definitions:
OLCE (w))
F(W) push?
(10)
Ow: pull
where
K Ag
F (w)) pull
Σ (1 -1-28 Pk (hz,i) hg.
F(w)
Ps (bg he
(1I)
push
Ae y=1
reveals that the negative gradient w.el We is decomposed into two terms. The "pull" term F F(w) pulls Wt towards the directions of the features of the same class, iee, hg while the "push" term
pull
F F(W) pushes wa away from the directions of the features of the other classes, i.c. hy VK' o k.

---

Remark 2 Note that F and F have different magnitudes. When the datasel is in extreme imbalance, the direction of OCCE. for minor class is dominated by the push term F because ny is small, while N nk is large. In this case, the classifier vectors of two minor classes are optimized towards nearly the same direction and would be close or merged after training. So the deteriorated performance of classification with imbalanced training data results from the imbalanced gradient w.r. a learnable classifier in the CE loss. As comparison, our proposed practice of fixing the classifier as an ETF does not suffer from this dilemma.
4.2.2 Gradient w.r.t Feature
The gradient of CE loss in Eq. (3) with respect to h is:
K
(1- -
=
we Σ
(12)
8h
where c is the class label of h, and pk (h) is the probability that h belongs to the k-th class as defined in Eq. (9).
It is shown that Eq. (12) has similar form to that of Eq. (8). The negative gradient &Ccx be decomposed the addition of "pull" and "push" terms defined as:
can also
8h
K
F Fthh pull = (1 - (-)(h)) Wes
F F(h) push
Σ (h)he lep
(13)
The "pull" term F(h)
pull pulls towards the classifier vector of the same class, iee, Wes while the "push" term push pushes h away from the other classifier vectors, i.e., Wk. Vk # c.
F(h)
We note that Eq. (11) and Eq. (13) have a similar form and are very symmetric. The "pull" terms F F(m) and F [(h) Duliffrce and h of the same class to converge the same direction, which in line with (NCI) and (NC3). The "push" terms p(w) and (b)) push make them of different classes separated, which is in line with (NC2). Generally, (NCI)-(NC3) can lead to (NC4). We remark that is
Remark 3
the "pull-push" " mechanism in the CE loss that leads to the neural collapse solution in the case of training balanced data.
The "push" gradient push in Eq. (13) is crucial for learnable classifier. However, in our case where the classifier has been fixed as an ETF, the "push" gradient F(h) push is unnecessary. As shown in Figure 2, the "pull" gradient ((h) in our case is always accurate towards the optimality wa, which corresponds to the neural collapse solution. But the "push" gradient does not necessarily direct to the optimality. So, we no longer rely on the "push" term F(h) that may cause deviation. It inspires us to develop a new loss function specified for our ETF classifier.
4.3 Dot-Regression Loss
We consider the following squared loss function:
1
2
LDR(h, W*) 2/EWEn (w:" / Ew E
(14)
where is the class label of W is fixed ETF classifier, and Ew and Eh are the norm constraints (predefined and not learnable) given in Eq. (5). It performs regression between the dot product of h and w and the multiplication of their lengths. We term this simple loss function as dot-regression (DR) loss. Its gradient with respect to h takes the form:
0LDR = (1 - cos L(h, w:)) w
(15)
Oh
where cos L(h, w") denotes the cosine similarity between h and w

---

02
h
Ipuit
W
dh (1 cos wh)ws
8. (-
pb)wii
W2
(CE)
(DR)
Figure 2: An empirical comparison of gradient directions w.rt an h (belongs to the 3-rd class) of the CE loss (left) and our proposed DR loss (right), Because h close to wi. the gradient of the CE loss dominated by - (*Cc=), whose direction deviates from the optimality wa. while the gradient of DR only has a "pull" term and always directs to w..
We that the gradient has similar form to the first in Eq. (12), which plays the role of "pull", but has no "push" term. is easy to identify that if we replace the CE loss in the decoupled layer-peeled model (DLPM) defined in Eq. (5) with the DR loss defined in Eq. (14), the same global minimizer Eq. (7) still holds. The global optimality happens when CDR(h", W*) = 0 and cos 2(h", w accordingly. Then we give formal analysis of the convergence properties of both CE and DR loss functions in the DLPM.
Definition 2 Given å O, for any h satisfying | h-ll S å and ||b||² Eh, the 17- regularity number of function L(h) is defined by the convergence rate of the projected gradient method. That is, there exists Th E [0, 1] such that:
Oc
|| PrOjER (h 1h--- 0h
-
7hh
where y is the learning rate such that 7/h is as small as possible.
Note that 7h is decided by h. For many problems, we cannot find its uniform upper bound S f < 1. The smaller 7h is, the better property the loss function has.
Theorem Assume that given a small 5 > O, when || 1h hill 6, p(h) defined in Eq. (9) satisfies that P& (h) (1 (e))))/(K - 1). vk # c, where is the label h. When optimizing the DLPM defined in Eq. (5) with the CE and DR loss funcitons, for any fixed learning rate ", we have:
Th
(CE)
+ cos L(h, w") (DR)
Th
(16)
2
(DR) (CE) and 1h
where 1h
are the rrregularity numbers of the CE and DR loss functions, respectively:
Proof 2 Please refer to Appendix B for our proof.
Eq. (16) indicates that the DR loss has better convergence property when h is close to h". In implementations, we train a backbone network with our ETF classifier and DR loss. To induce balanced gradients w.r.f the backbone network parameters, we define the length of each classifier vector according to class distribution. Please refer to Appendix C for the complete implementation details. In experiments, we also compare our method with the CE loss weighted by class distribution.
Experiments
In experiments, first make empirical observations of neural collapse convergence the imbalanced training with and without our method, and then compare the performances on long-tailed classification. As an extension, we surprisingly find that our method also able improve the performance of tne-graines classification, which can be deemed as another imbalanced problem where majority of features are close to each other. The datasets and training details are described in Appendix D.
Its rational lies in that h² aligned with w has equal dot products with wł. vk & and h is close h..
7

---

0.501
Irai
Loctll Meeved
020
Figure 3: Averages and standard deviations of cos (h. hg: he hg) (two columns the left), and cos (h. - hy, Wi)) (two columns on the right) with (red) and without (black) our method on train set (a) and test set (b). hg the global mean. he, h& are within-class means, where e and & all pairs of different classes. The models are trained on CIFAR-100 with the imbalance ratio Aimin of 0.02, where Tmin and Remax are the minimal and maximal numbers of training samples in all classes.
Table 1: An ablation study with ResNet on CIFAR-10 10 [17] using different classifiers and loss functions. The numbers in the second row denote the imbalance ratio = Amin. where Mmni and Remax' mmax are the minimal and maximal numbers of training samples in all classes. Results are the mean of three repeated experiments with different seeds. * indicates that the CE loss is weighted by Kng to induce class balance, where is the number of classes and ng is the number of samples in class k.
without Mixup
with Mixup 0.02
Methods
0.005
0.01
0.02
0.1
balanced 0.005
0.01
0.1
balanced
Learnable Classifier CF
166.1 2o.3 71.0 77.1 20.2 20.2 93.4 66.3 20.4 72.8 20.3 78.6 87.7 10.1 93.6
Learnable Classitter CE
66.8+
-2. 77.6
87.4
93.1
68.5 73.9 +0.3 79.3 +0.2 87.8 37.8 93.2
RTF Classifier CF
60.4
72.9 79.5
87.2
92.6
60.6
67.0 77.2 87.0+o.2 93.3
ETF Classiler DR
68.4+0.2 73.0 78.4 86.9 92.9 71.9
76.5 81.0 87.7 92.0+o.2
5.1 Empirical Results
Following [28], we calculate statistics during training to show the neural collapse convergence. We first compare the averages and standard deviations of two cosine similarities, cos L(h,- hg h2 hg) and cos Z(h,- - hg W&). where h, is the global mean, for all pairs different classes (c, k), c + k. with and without our method. As shown in Figure 3, the averages of both cos L(h,- hg. h; h₂) and cos Z(h.- hy, w&) converge to a negative value near zero. is consistent with neural collapse that the feature means or classifier vectors of different classes should have a cosine similarity of
" However, their standard deviations are much smaller when our method is used. It indicates that the models with our method converge to neural collapse more closely.
We further calculate the averages of cos Lhh hg, we), VI S K, and || H in Figure 4 and 5 in Appendix E. It reveals that the model using our method generally has a higher cos <hh, - hg. w.) and lower || H I/:, which indicates that the feature means and classifier vectors of the same class are better aligned. We observe no advantage of ResNet on STL-10 and DenseNet on CIFAR-100 in Figure and 5. In Table 2, we see that the two cases are right the failure cases, which shows consistency between neural collapse convergence and classification performance.
5.2 Performances on Long-tailed Classification
We conduct an ablation study with ResNet on 10. As shown in Table 1, when we replace the learnable classifier with our fixed ETF classifier, the performances get improved for the imbalance ratio of 0.01 and 0.02 without Mixup [47]. They also achieve comparable results for = 0.1 and the balanced setting (r = 1). However, only using the ETF classifier does not work for the extreme imbalance case where 0.005. Besides, it is not compatible with Mixup, which is
8

---

Table 2: Long-tailed classification accuracy (%) with ResNet and DenseNet on four datasets. Results are the mean of three repeated experiments with different seeds.
CIFAR-10 [17] 0.005
CIFAR-100 100 [17]
SVHN [25]
STL-110 [3]
Methods
0.01 0.02
0.005
0.01
0.02
0.005
0.01
0.02
0.005
0.01
0.02
ResNet
Learnable Classifier CE
67.3
72.8
78.6
38.7
43.0
48.1
40.5
40.9
49.3
33.1
37.9
38.8
ETF Classifier + DR
71.9
76.5
81.0
40.9
45.3
50.4
42.8
45.7
49.8
33.5
37.2
37.9
Improvements
+4.6
+3.7
+2.4
+2.2
+2.3
+2.3
+2.3
+4.8
+0.5
+0.4
-0.7
-0.9
DenseNer
Learnable Classifier CE
71.1
77.7
84.1
40.3
43.8
49.8
39.7
40.5
46.4
38.5
41.2
44.9
ETF Classifier + DR
72.9
78.5
83.4
40.1
44.0
49.7
40.5
44.8
48.4
39.5
42.9
46.3
Improvements
+1.8
+0.8
-0.7
-0.2
+0.2
-0.1
+0.8
+4.3
+2.0
+1.0
+1.7 +1.4
Table 3: Long-taile classification accuracy (%) on ImageNet-LT [21] with ResNet-50 backbone and different training epochs.
Table 4: Fine- Fine-grained classification accuracy (%) CUB-200-2011 [39] with different
ResNet backbones pre-trained on ImageNet.
Training details are described in Appendix D.
Epoch Methods
Acc. (%)
Learnable Classifier + CE
34.6
Backbone
Methods
Acc. (%)
90
ETF Classifier DR
41.8
Learnable Classifier + CE
82.2
ResNet-34
Learnable Classifier + CE
41.9
ETF Classifier DR
83.0
120
ETF Classifier DR
43.2
Learnable Classifier + CE
85.5
ResNet-50
Learnable Classifier + CE
42.5
ETF Classifier DR
86.1
150
ETF Classificr DR
43.8
Learnable Classifier + CE
86.2
ResNet-100 101
Learnable Classifier + CE
44.3
ETF Classifier DR
87.0
180
ETF Classifier DR
44.7
strong augmentation tool alleviate the bias brought by adversarial training samples. When the DR loss that is specifically designed for the ETF classifier is used, we achieve significant performance improvements for T of 0.005, 0.01, and 0.02 both with and without Mixup. Generally our proposed ETF classifier with the DR loss has better performances on multiple long-tailed cases than the learnable classifier with the original or weighted CE loss.
In Table 2, the advantage of our method further verified on more datasets with ResNet and DenseNet. Concretely, achieve an average improvement of 2.0% for ResNet and 1.1% for DenseNet. We also test our method on the large-scale dataset, ImageNet-LT [21], which is an imbalanced version of the ImageNet dataset [34]. As shown in Table 3, we compare our method with baseline by training the models for different epochs. We observe that the superiority of our method is more remarkable when training for less epochs. It can be explained by the fact that method directly has the classifier in its optimality and optimizes the features towards the neural collapse solution, while the learnable classifier with the CE loss needs a sufficient training process to separate classifier vectors of different classes. So our method can be preferred when fast convergence or limited training time is required. The accuracy curves in training for the results in Table 3 are shown in Figure 6 in Appendix E.
5.3 Performances on Fine-grained Classification
Fine-grainer classification can also be deemed as an imbalanced problem as the features of multiple classes are close each other. We surprisingly find that our method is also helpful for fine-grained classification even though most of the analytical work is conducted under the case of class imbalance. As shown in Table 4, our method achieves 0.7%-0.88 accuracy improvements on CUB-200-2011
Conclusion
In this paper, we study the potential of training network with the last-layer linear classifier randomly initialized as simplex ETF and fixed during training. This practice enjoys theoretical merits under the layer-peeled analytical framework. We further develop simple loss function specifically for the ETF classifier. Its advantage gets verified by both theoretical and experimental results. We conclude that not necessary to learn a linear classifier for classification networks, and our simplified practice even helps to improve the long-tailed and fine-grained classification performances with no cost. Our work may help to further understand neural collapse and the neural network architecture for classification. Code address: https://githhbb. wwwp:////wwooe.eoolololepppeeeae wtwpraiiiiiinnnaaanneeeeeeeiiiia

---

Limitations and societal impacts are discussed in Appendix F. As suggested by reviewer of this paper, we compare with [5, 30, 50] more details Appendix G.
Acknowledgments and Disclosure of Funding
Z. Lin was supported by the NSF China (No.s 62276004 and 61731018). NSFC Tianyuan Fund for Mathematics (No. 12026606), and Project 2020BD006 supported by PKU-Baid Fund.
References
[1] S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In ICML, pages 322-332. PMIR, 2019.
[2] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-distriuttion
aware margin loss. In NeurIPS, volume 32, 2019.
[3] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks unsupervised feature learning. In AISTATS. pages 215-223. 2011.
T. Ergen and M. Pilanci. Revealing the structure of deep neural networks via convex duality. In ICML, pages 3004-3014, PMLR, 2021.
[5] C. Fang. H. He, Q. Long, and W. Su. Exploring deep neural networks via jayer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021. [6] R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2):79--188)
1936.
[7] R. Ge, J. D. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. In ICLR,
2018.
[8] F. Graf, C. Hofer, M. Niethammer, and R. Kwitt. Dissecting supervised constrastive learning. In ICML, pages 3821-3830. PMLR, 2021.
X. Han, V. Papyan, and L. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. ICLR, 2022.
[10] K. Нe, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages
770-778, 2016.
[11] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, Seybold, al. Con architectures for large-scale audio classification. In ICASSP, pages
131-135. IEEE, 2017.
[12] E. Hoffer, I. Hubara, and D. Soudry, Fix your classifier: the marginal value of training the last weight layer. In ICLR, 2018
[13] G. Huang. Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[14] A. Jacot, C. Hongler, and F. Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In NeurlPS, pages 8580-8589, 2018.
[15] Ji, Y. Lu,, Zhang, 7., Deng, and W. Su. An unconstrained layer-peeled perspective on neural
collapse. In ICLR, 2022.
[16] B. Kang. S. Xie, Rohrbach, Z. Yan, Gordo, Feng, Kalantidis. Decoupling representation and classifier long-tailed recognition. ICIR, 2020.
[17] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[18] Q Le and T. Mikolov. Distributed representations of sentences and documents. In ICML, pages 1188-1196.
PMIR, 2014.
[19] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
[20] H. Li, Y. Yang, D. Chen, and Z. Lin. Optimization algorithm inspired deep neural network structure design.
In ACML, 2018.
[21] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in open world. In CVPR, pages 2537-2546. 2019.
[22] J. Lu and S. Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXi::2012.08-65,
2020.
[23] S. Mei, A. Montanari, and P.M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, (3133))67665-77677 2018.
[24] D. G. Mixon. Parshall,
Pi. Neural collapse with unconstrained features. arXiv preprint
urXiv:2011.11612 2020.
[25] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits natural images with
unsupervised feature learning. 2011.
[26] E. Oyallon. Building regular decision boundary with deep networks. In CVPR, pages 5106-51 5114, 2017.
10

---

[27] V. Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Revearch, 21(252):1-64, 2020.
[28] V. Papyan. X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training- Proceedings of the National Academy of Sciences, 11(40):24652-24665 2020.
[29] F. Pernici, M. Bruni, C. Baecchi, and Del Bimbo. Maximally compact and separated features with
regular polytope networks. In CVPR Workshops, pages 46-53, 2019.
[30] F. Pemiici, M. Bruni, C. Baecchi, and A. Del Bimbo. Regular polytope networks. IEEE Transactions on
Neural Networks and Learning Systems, 2021.
[31] T. Poggio and Q. Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.0000 2020.
[32] A. Rangamani and A. Banburski-Fahee Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP, pages 4243-4247. IEEE, 2022.
[33] C. R. Rao. The utilization of multiple measurements in problems of biological classification. Journal of (122):199220. 1948.
the Royal Statistical Sociery. Series (Methodological),
[34] O. Russakovsky. J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla,
M. Bersstein, al. Imagenet large scale visual recognition challenge. IJC,, 115())211-2522 2015.
[35] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[36] A. F. Smith and D. J. Spiegelhalter. Bayes factors and choice criteria for linear models. Journal of the Royal Statistical Society: Series B (Methodological), 42(2):213-22) 1980.
[37] A. Stuhlsatz, J. Lippel, and Zielke. Feature extraction with deep neural networks by a generalized discriminant analysis. IEEE Transactions on Neural Networks and Learning Systems, 23(4):596-605
2012.
[38] T. Tirer and J. Bruna. Extended unconstrained features model for exploring deep neural collapse. In ICMI.,
2022.
[39] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 1 dataset.
2011.
[40] A. R. Webb and D. Lowe. The optimised internal representation of multilayer classifier networks performs nonlinear discriminant analysis. Neural Networks, 3(4):367-375, 1990.
[41] E. Weinan and S. Wojtowytsch, On the emergence tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint 2rXi::20/2.05422, 2020.
[42] L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. ICMI., pages 10462-10472. PMIR, 2020.
[43] L. Xie, Y. Yang, D. Cai, and X. He. Neural collapse inspired attraction-repukiio--aaaaa balanceed loss for imbalanced learning. arXiv preprint arXiv:2204.N0 00835,, 2022.
[44] Y. Yang. J. Wu, H. Li, X. Li, T. Shen, and Z. Lin. Dynamical system inspired adaptive time stepping controller for residual network families. In AAAI, 2020.
[45] Y. Yang, Zhong, Shen, and Lin. Convolutional neural networks with alternately updated clique, In 2018.
CVPR, pages
[46] J. Zarka, F. Guth, and S. Mallat. Separation and concentration in deep networks. In ICLR, 2020.
[47] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
[48] Z. Zhong, Cui, S. Liu, and Jia. Improving calibration for long-tailed recognition. In CVPR, pages 16489-16498, 2021.
[49] J. Zhou, X. Li, T. Ding, C. You, Q. Qu, and Z. Zhu. On the optimization landscape of neural collapse under
mse loss: Global optimality with unconstrained features. arXiv preprint arXiv:2203.01 2022. [50] Z. Zhu, T. DING, J. Zhou, X. Li, C. You, J. Sulam, and Q Qu. A geometric analysis of neural collapse with unconstrained features. In NeurlPS, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the last paragraph in Section 1.
(b) Did you describe the limitations of your work? [Yes] See Appendix F.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Ye]] We have read the guidelines and ensured that our paper conforms to them.
2. If you are including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theorem and 2.
(b) Did you include complete proof' of all theoretical results? [Yes] See Appendix A and
B.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplementray material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tabel 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes) We cite their papers, as shown in Table 2, 3, and 4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "w5cpEtCbfT", "reviewer": "Reviewer_MaCE", "summary": "The paper proposes a loss function which projects the features to a pre-defined simplex and computes the gradients w.r.t it. It claims that neural networks use linear classifiers at the end and this is not needed if features are computed for each class and projected to a simplex. A theoretical explanation for this approach is provided and experimental results are presented.", "questions": "Line 23:\nThe paper starts by citing a reference [28] where a phenomenon called neural collapse is mentioned. For example, it is not clear why would features collapse to within-class mean? What is the loss function being used to do so? Are class labels even used during training which leads to this equiangular tight frame? The reader needs to be given a high level/intuitive introduction to ETF at this stage of the paper even though these details may be described in [28]. It could be like, once a neural net is trained, [28] observes 4 properties NC 1,2,3,4, which are the properties for an ETF and we can describe what is NC 1/2/3/4.\n\nPlease clarify how the work is different from the papers which I mentioned or if it would be better compared to these in a given scenario.", "limitations": "yes, it has been addressed.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea to try to solve classification using a simplex is interesting but I believe there are several papers in the deep learning community for large-scale classification problems (like face/few-shot recognition) which have evaluated similar loss functions which have not been cited or compared with in this paper. I am mentioning a few of these papers and they use similar kinds of projection functions but also present more variants of the version presented in this paper. For example, Wen et. al. (ECCV 2016) and Snell et. al. Neurips 2017, use class means/prototypes as a loss function (called center/prototype loss) to improve recognition performance. The class means are then constrained to a sphere by Liu et. al in SphereFace (CVPR 2017) which further improves the performance (like the ETF proposed in this paper). Deng et al. in ArcFace (CVPR 2019) further add an angular penalty between points on a sphere. There have been several papers after this (look for papers citing these 4) which explore ways to do classification without a linear layer followed by softmax+CE which are based on per-class features and their margins.\n\n[1] A Discriminative Feature Learning Approach for Deep Face Recognition, Wen et. al, ECCV 2016\n\n[2] Prototypical networks for few-shot learning, Snell et. al, Neurips 2017\n\n[3] SphereFace: Deep Hypersphere Embedding for Face Recognition, Liu et. al, CVPR 2017\n\n[4] ArcFace: Additive Angular Margin Loss for Deep Face Recognition, Deng et. al, CVPR 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "In0NLdUnayg", "reviewer": "Reviewer_Nj4c", "summary": "On balanced datasets and when using cross-entropy loss for learning, it is observed that [28] the last layer features from the same class converge to their within-class means and the weights the network in the last layer collapses to the vertices of an equiangular tight frame (ETF). Thus, a natural question to ask is: if this empirical observation can be used to train better deep learning models? The current paper uses this insight to design a network where instead of training for a cross-entropy loss, a loss that regresses the deep features against  a randomly sampled set of vectors (corresponding to the vertices of an ETF) is used for classification. In situations when the data is imbalanced, the paper claims that this prior knowledge of ETF allows better learning of the model. Experiments are provided on small scale datasets (such as CIFAR, STL, etc.) and demonstrate promise of the method in imbalanced settings. ", "questions": "Please see above. ", "limitations": "The paper does not describe any limitations as such. However, I believe experiments on larger datasets such as ImageNet could bring out the limitations of the approach better. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the insights underlying the paper are encouraging and the application of the ETF model for imbalanced datasets is reasonable. The experiments show improvements on a variety of datasets under imbalanced settings. \n\nWhile, the insights are interesting, I do not think the contributions are compelling for the following reasons.\n\n1.  The key question the paper asks is L58: \"Do we really need to learn a linear classifier at the end of a deep neural network for classification?\". While, in the context of ETF, it is an interesting question, however when thinking of this question in the context of learning in general, it is obviously not needed to learn a linear classifier at the end of a DNN. The one-hot encoding typically used on labels in cross-entropy loss based training is one way of encoding with its own geometric meaning, while the one proposed using ETF in the paper is yet another encoding. In that sense, I do not really see a significant contribution the paper brings in. Of course, when using an LPM model, we may get rid of learning of the classifier weights, but when using the method in an end-to-end learning framework, doesn't one still need to learn a method that maps the features into one of the vertices of the ETF? \n\n2.  While ETF is a structure that evolves from the use of CE loss, why not consider potentially other structures if we do not assume any CE loss? Why not a regression into the columns of an arbitrary orthogonal matrix (assuming there is no need to interpret the classification as probabilities any more, and thus there is no need to use a softmax)? \n\n3. There are standard approaches for dealing with the problem of imbalanced datasets, such as choosing weights during training, focal loss, etc. The paper appears to have not considered or compared the proposed approach to any of such alternatives and thus limits the understanding of the benefits to a narrow scope (e.g., Figure 3, Tables 2,3,4, etc.).\n\n4. More experiments, for example in settings when there are very large number of classes as in ImageNet, can make the analysis significantly better. I wonder if in such settings a cross-entropy loss provides better \"push\" and \"pull\" gradients (due to the log term) than the proposed linear alternative (14).\n\nOther comments:\n1. For the layer-peeled model in (4), it is not clear how precisely are the features produced? If you optimize over W and H, then how do you associate the features to be from the data x? When optimizing for h, are you optimizing the parameters of the underlying network? If so, rewriting h to h_\\theta(x) and optimizing over \\theta would make the math better. If it is being optimized only on W and H, then how is H conditioned on the data X, and how do you prevent H from being arbitrary?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HIzVMfDQhM", "reviewer": "Reviewer_amWi", "summary": "The paper studies neural collapse in the case in which the final classifier is not subject to back-propagation and is initialized with the vertices of a simplex equiangular tight frame (ETF). In particular the paper studies the unbalanced learning regime and shows that the neural collapse properties hold also in this case. The proof is based on the layered-peeled model assumption.\n", "questions": "See above.", "limitations": "No noteworthy limitations or potential negative societal impacts. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\n1) The paper is very interesting as it provides a theoretical analysis of the class imbalance problem with a fixed classifier inspired by the recent neural collapse phenomenon.\n\n2) The push-pull mechanism from the gradient analysis is novel and it is a clear contribution of the paper.\n\n3) The idea may also have practical implications related to some recent works about learning compatible features representations [C, D]. Compatible features are learned so that they can be directly compared even if they are learned from different time instants or from different network architectures. The method [D] specifically takes advantage of a fixed classifier exactly defined as the one proposed in [47] and in this submission.\n\nWEAKNESSES:\n\n1) The paper contributions should be improved. It seems that the main contribution of the paper is about the combination of the following three aspects: neural collapse, unbalanced data and fixed classifiers. The title does not convey a direct relationship with these three aspects. In its current shape it seems more referring to the fixed classifier aspect, like in [29] and [12], rather than to the imbalanced Neural Collapse training regime. This is mostly due to the fact that if a learnable classifier is not “really needed” it is supposed that a fixed one would be preferred. In the reviewer's opinion the title should be improved reflecting more the main content of the paper.\nIn this respect, the differences with [47] and [29] should be better highlighted. In particular a question that should be answered is: what is the difference between an ETF simplex and the classic d-simplex regular polytope? The question is valid either in the fixed case and in the learnable one. Finally, as the paper [5] is very related with the imbalance aspect of the paper, it should be discussed in the related work section and possibly differences highlighted.\n\n2) It seems that in many papers (including the seminal one [28] and [47]) the relationship between the number of classes K and the dimension d required to observe the neural Collapse phenomenon is d≥K while in the submitted manuscript and in [47] d≥K-1. Is there any specific motivation for these two cases?\n\n3) Some important references are missing: [A,B]. Although not directly related to neural collapse, the paper [A] proposes a similar gradient analysis of the final classifier. A brief discussion highlighting differences and contributions should be given. The paper [B] seems to be one of the first published papers on maximal separation and fixed classifiers with the simplex geometry.\n\n4) Is the random ETF initialization needed to avoid possible biases in its choice? Is there any evaluation quantifying this dependency?\n\n5) Eq. 15 is referring to the optimization of the angle between features and classifier prototypes. What is the difference between that loss and the classic entropy loss in which both features and prototypes are normalized? Both minimize the angles between features and prototypes.\n\nReferences\n\n[A] P2SGrad: Refined Gradients for Optimizing Deep Face Models Xiao Zhang, Rui Zhao, Junjie Yan, Mengya Gao, Yu Qiao, Xiaogang Wang, Hongsheng Li; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019\n\n[B] Maximally Compact and Separated Features with Regular Polytope Networks Federico Pernici, Matteo Bruni, Claudio Baecchi, Alberto Del Bimbo; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019, pp. 46-53\n\n[C] Shen Y, Xiong Y, Xia W, Soatto S. Towards backward-compatible representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 (pp. 6368-6377).\n\n[D] Biondi N, Pernici F, Bruni M, Del Bimbo A. CoReS: Compatible Representations via Stationarity. arXiv preprint arXiv:2111.07632. 2021 Nov 15.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept