Neural Collapse To Multiple Centers For Imbalanced Data
Authors: Hongren Yan
Yuhua Qian
Furong Peng
Jiachen Luo
zheqing zhu
Feijiang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Collapse
Imbalanced Data
Classification Learning
Classification Rule
Primary area: learning_theory

Abstract: Neural Collapse (NC) was a recently discovered phenomenon that the output features and the classifier weights of the neural network converge to optimal geometric structures at the Terminal Phase of Training (TPT) under various losses. However, the relationship between these optimal structures at TPT and the classification performance remains elusive, especially in imbalanced learning. Even though it is noticed that fixing the classifier to an optimal structure can mitigate the minority collapse problem, the performance is still not comparable to the classical imbalanced learning methods with a learnable classifier. In this work, we find that the optimal structure can be designed to represent a better classification rule, and thus achieve better performance. In particular, we justify that, to achieve better classification, the features from the minor classes should align with more directions.  This justification then yields a decision rule called the Generalized Classification Rule (GCR) and we also term these directions as the centers of the classes. Then we study the NC under an MSE-type loss via the Unconstrained Features Model (UFM) framework where (1) the features from a class tend to collapse to the mean of the corresponding centers of that class (named Neural Collapse to Multiple Centers (NCMC)) at the global optimum, and (2) the original classifier approximates a surrogate to GCR when NCMC occurs. Based on the analysis, we develop a strategy for determining the number of centers and propose a Cosine Loss function for the fixed classifier that induces NCMC. Our experiments have shown that the Cosine Loss can induce NCMC and has performance on long-tail classification comparable to the classical imbalanced learning methods.

Neural Collapse To Multiple Centers For Imbalanced Data
Hongren Yan, Yuhua Qian', Furong Peng, Jiachen Luo, Zheqing Zhu, Feijiang Li
Shanxi University
Abstract
Neural Collapse (NC) was a recently discovered phenomenon that the output fea- tures and the classifier weights of the neural network converge to optimal geometric structures the Terminal Phase of Training (TPT) under various losses. However, the relationship between these optimal structures at TPT and the classification performance remains elusive, especially in imbalanced learning. Even though it is noticed that fixing the classifier to an optimal structure can mitigate the mi- nority collapse problem, the performance still not comparable to the classical imbalanced learning methods with learnable classifier. In this work, we find that the optimal structure can designed represent a better classification rule, and thus achieve better performance. In particular, we justify that, to achieve better classification, the features from the minor classes should align with more directions. This justification then yields decision rule called the Generalized Classification Rule (GCR) and we also term these directions as the centers of the classes. Then study the NC under MSE-type loss via the Unconstrained Features Model (UFM) framework where (1) the features from a class tend to collapse to the mean of the corresponding centers of that class (named Neural Collapse Multiple Cen- ters (NCMC) at the global optimum, and (2) the original classifier approximates surrogate to GCR when NCMC occurs. Based the analysis, develop a strategy for determining the number of centers and propose Cosine Loss function for the fixed classifier that induces NCMC. Our experiments have shown that the Cosine Loss can induce NCMC and has performance on long-tail classification comparable to the classical imbalanced learning methods.
Introduction
Deep neural networks are popular choices classification tasks I demystify the deep representations learned from data [8] 9.. recent paper 10
5 6
Researchers try to
observed "Neural Collapse" (NC) phenomenon: all the backbone network output features from each class converge into their corresponding vertices of equiangular tight frame (ETF) and the within-class variability
collapses.
The
model (LPM) and unconstrained feature model (UFM) [12 are the simplified model to study NC, in which the backbone output feature and the classifier weights are assumed free variables optimize. NC phenomena occur with different loss functions. The optimality of
UFM satisfies NC under the CE loss with constraints 13 14, regularization [5] or even no
explicit constraint [16]. MSE objectives also induce NC global optimality [12 17 18 19). There
another line of works that extend UFM or LPM on deeper linear layers [19 20
22
Data imbalance has been recently considered in NC literature U1 23 20
In particular, Fang et al.
|| originally observe the "minority collapse" phenomenon that the classifier weights of the minority
Corresponding Author. Email: jinchenggyh@ 126.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classes will approach each other when the imbalance level goes high. Thrampoulidis et al. 123 theoretically study the Unconstrained-feauee features SVM problem, whose global minima take the form of Haaplex-Ennoded---e Labels Interpolation (SELI), a more general structure compared to the ETF. Dang al. [20 inspects the Rel.U-activated output features of deep linear network collapse to a general orthogonal frame for imbalanced data, where the orthonormal vectors of the frame are rescaled.
There a line of works that connect the NC the DNN performance 10 24 25 Some researchers
treat NC as a tool to alleviate minority collapse problem in imbalanced learning 226 27 288] We
reproduce these methods with backbone network ResNet50 (29] and datasets 10/eifr 100, Table 3) shows NC-inspired methods ETF and ARB only outperform the plain model (ResNet50 with CE loss) slightly (or even worse in few settings), which indicates that minority collapse is one but not the only problem that harms the generalization of learning model. One possibly important issue is that NC on training sample does not necessarily imply the NC on the distribution, as pointed out by Hui et al. [0]. This inconsistency can lead to severe performance degeneration when the sample is too scarce to represent certain classes.
Neural Collapse implies "maximal separateness" between classes, which inspires some works to consider fixed classifiers in the training [31 33
However, these methods do not display advantages over the classical imbalanced learning methods equipped with learnable classifiers.
35
In this paper, we study the connection between the optimal structure induced by neural collapse and its corresponding classification rule, and propose a MSE-type loss function that improves the imbalanced learning with fixed classifier. Specifically,
1. Through the analysis of hard-to-predict feature distribution (the features that are distributed randomly around the mean of the classifiers), we find that the classification accuracy is improved if the features from the minor classes align with more directions and those from major classes with less directions, which corresponds to a decision rule called the Generalized Classification Rule (GCR) discriminated from the Regular I-Nearest Neighbor Classification Rule (RCR) induced by general NC in literature, and we also term these directions the centers of the classes.
2. We design an MSE-type objective that describes the average distance between the centers and a given feature. We show in the theoretical framework of Unconstrained Feature Model (UFM) that, for balanced or imbalanced data and fixed or learnable classifiers, the output features collapse but skew from the classifiers at terminal phase of training (TPT), which is different from the original Neural Collapse phenomenon and termed "Neural Collapse to Multiple Centers" " (NCMC) (Theorem 3.3 and 3.4); moreover, we find that RCR (with respect to classifiers) becomes surrogate of GCR at NCMC (Remark 3.9 and Proposition C.I.
3. We design practical loss function for fixed classifiers and class-aware strategy for determining the number of centers for each class. The loss induces the NCMC which is justified in theory and experiments and achieves comparable performance on several datasets with varying imbalance ratios the classical imbalanced learning methods such as LDAM (36,, KCL (37), ARBLoss (27). which indicates that NCMC can improve generalization in imbalanced learning.
2 Preliminaries
2.1 Neural Collapse
Consider a classification task with K classes and ny training samples per class, i.e., overall N = Σk Mg samples. DNN-based classifiers generally have the form
the (x) Wxho(x) b
where h-(-) : DRD + Rd is the feature mapping (d 2 K), Wo = [wI.... wx] E Rxd with Wa E Rd the weight vector of class k and b E R the bias of classifier, respectively. O = (W, b, 0) is the set of the trainable network parameters, which includes the parameters 0 of nonlinear compositional feature mapping (e.g., hx(x)
(W,, (... (W2a (Wix))... where a(.) is an element-wiss nonlinear function). Let [A] be the set 21, ... A} for positive integer A.

---

The network parameters are optimized by minimizing an empirical risk
K
nk
M L(Wohg (xk,.. + b. yk) + R(O), N
min
-=1
where ((.,-) is a loss function (e.g., cross-entropy or MSE) and R(-) is a regularization term (e.g., squared Lg-norm). Let us denote the feature vector of the ię-th training sample of the k-th class by hg, (i.e., hg. = he (Xk.i. ), with i;; E [n Denote the centralized mean of feature from class k by h& h& Mc where hg := D Th H h1, h2,..., hx).
hR and Mg R Σ K
k=1 hg; let
Recently noticed NC phenomenon [10 shows the weight vectors align with the class-mean features
H Wow. HoWW, x
K
IK
1 1x1k
K K-1
with
h h
hg
for all [K] at the terminal phase of training. where we use I to denote the K K identity matrix, 1 to denote the all-ones vector of size K x 1. The alignment may have alternative shapes for other problem settings such as h is ReLU-activated feature or the data class is imbalanced.
2.2 NC for Unconstrained Features Model with Regularized MSE Loss
To understand the emergence of symmetric structures, recent papers study the "unconstrained features model" (UFM), where the features {h)) and Wa are treated as free variables. The rationality behind this simplification is the powerful expressivity of a trained neural network. Some use UFM to study the NC phenomenon under MSE loss.
Let H = [hi,):-... hi h2,1,... hxnz] E IR dx In balanced case where n1 = 122 =.. 7K- H associated with the one-hot vectors matrix Y = I & 1 E KxKK, where @ denotes the Kronecker product. The optimization of the following problem
Aw 2
min H. b 2N
WoH+ b1
b
(5)
gives the NC to ETF, where Awo λи. and Å are positive regularization hyper- parameters and || denotes the Frobenius norm. closely related model the Bias-Free models
||
W.H Y + Awo
+ Aн
min Wo.H 2Kn
|Wol
|
(6)
2
2
which proves to converge to an Orthogonal Frame [19]. In the next section, we study a bias-free variant of 6
3 Main Results
In this section we show why the regular classification rule is not optimal, and propose the generalized classification rule and its surrogate losses, then offer UFM analysis the NC phenomenon under these losses. Since our paper focuses on bias-free model, we will simply call Wo the classifier.
3.1 Nearest-Neighbor Classification Rule Revisit: A Toy Example
Let X' C Iddo (dg is the input dimension) be the underlying data population, and := h(X)= Rd represent the feature population the trained backbone network h; Assume the trained classifier W1, w....., Wx align with training samples drawn from X' perfectly and thus achieve zero training enror.
Additionally, assume the classifiers are orthonormal and equally normed. Meanwhile, let the classes of training data C1, C2,.C.,.. Cx arranged such that the class sample sizes follow a descending order, i.c. n1 n22 ... nk.

---

According to NC, when the neural network arrives at the terminal phase of training, the regular classification rule (RCR) with respect wo
c argmax {w 2
(7)
i.e,, the 1-nearest neighbor decision rule.
Assume 7 ~ P1 + 0. the mixture of the subpopulation Pi which is correctly classified by the decision rule and the subpopulation P/2 that classified at random, where an and ag are the positive weights with 01 + 02 = 1.
In the analysis, let P2 have the form = u p. with p (N and jr :== w where is a small positive number and w the global mean of the classifier. For convenience we denote events E = {2 C2l2 P2} for all
Wk. which is termed Tard-it- feature distribution near
K
E [K] and note that Σ P(Ex)= =1
By virtue of the expressivity of the deep neural network, the population of the major classes will be more concentrated to the classifier than that of the minor classes, so that populations trained by major classes have less probability falling in P2, i.e.
p & P(F2) s... P(EE).
(8)
Then by the gaussianity of p and the orthogonality of the classifiers WI; W2, ......y WK, the proba- bilities of correctly classifying data from P2 are the same, that is,
P(W]=
max w & 2/2 #
E
E&} P(WIY-
> max w WorlD (((0 kiH
(9)
K'
since WT. 2 > whz = w I( + p) Therefore, by the total probability
w + = wHp > wTE given w 1r = wHA
K Σ P[Ex] K K
1
PRCR == (( correctly classified
|2 P2]
(10)
This calculation inspires to come up with different classification rule that outperforms RCR on Pz. Indeed, we can assign the label = to the set (w, Jy=1 of vectors, for all k E [K]: denote
(R) fa
F Ek fu and West
(k)) also similar the RCR, assume all these F vectors are
equally normed and mutually orthogonal. These vectors correspond to Generalized Classification Rule (GCR)
argmax max (w(A) (k)
,2)},
(11)
kE|K] jElfe)
or equivalently, fie-nerrett neighbor classification rule, where number of nearest neighbors, fs. of an example depends on which class it belongs to. Again by the properties of normal distribution and total probability formula, we obtain
1 F E S.P(E))
Peen P(2 is correctly classified ls P2)
(12)
Recall the ascending sequence of P(E.) ... P(EK), so that any choice of fi f2 < ... < fx gives higher correctly-classife probability for z ~ Pz. If we in addition consider pairwise comparison instead of the one-vs-all fashion (9), we have for any pair ls + k2
P P( (w 1 Ex U+UE 2'
(13)
for RCR. and
(k) Pf max (w) 2) JELf2)
max (w) (k') f' J'Elsel 3' .2) 2 Es UK)
ft fk + fw
(14)
for GCR. this case, we require fa < S for < k'' for all pairs of k' E [K] or equivalantly,
fi f2 < < fk. The analysis whole has yielded the following proposition.

---

Proposition 3.1. assume (1) all the classifiers, {wk) in RCR or ) in GCR are orthonormal frames; (2) Z ~ 0r 22 the mixture of the subpopulation Pi which is correctly classified by the decision rule and the subpopulation P2 where on and ag are the positive weights with on 02 = (3) P2 has the form z =
p or Hext + +p depending on which classification rules used, with p ((O,,saa where $ is a small positive number; (4) P(E,} S P{E2] S ... P(Ex). Then
POCR 2 for fi S f2 ......... S fk.
The Hard-To-Predic features in the proposition 3.1 are considered be drawn randomly around the mean of the classifiers. Thus, we are motivated to use more orthogonal directions to classify the minor class. However, the GCR in the analysis does not apply the practical training of the neural network easily. Indeed, the loss vanishes quickly when training directly with this rule. For this reason, we consider finding surrogate loss that a) induces neural collapse an orthogonal frame, and b) the classification rule at the neural collapse approximate GCR.
3.2 Center and Multi-Center Frame
We first define the "centers" that resemble W
(k)-s "s in GCR 11 (in definition 3.2
Let
f. f1. 12....,,f fK Z+ be preset factors such that fK Ek=1 Jk. N
Σ T4e and
= EE feng and E [0, 3J be angle constant. Let the linear classifier Wo satisfies
Wi Wk > O, and W WK Oif
(15)
and the data features H [h1.1. him h2.1.... hg. 1 E Rd
Definition 3.2 (Center of Class k). Let IJK, (K)
(1)
(1)
(2)
(K)
is
f1
fx
matrix consisting of fl rows of d-dimensional vectors vj 's and satisfies equality
[V Wo "ı |wo 1
(16)
diag(I) [org(//)/l' I ||
12 f....... wxl fix || 11 ||wll?
wx
12)
(17)
where(-l.) :) is the column augmentation of the matrix, and diag(,,.... is the diagonalization of the block matrices. Then center of class is defined as
(8)
(k) cos W. sin 0, j [fa].
v
(18)
A multi-center frame is the matrix consists of f K rows of w (k) i.e.
(K)
T
W
w
(19)
fx
Let denote the constraint of the tuple (V, Wo) such that [V Wo T satisfies 16 and || are positive for all h E [K].
By the definition 3.2 (8)
w =... w (k) for all l E [K]. verbally, the centers of each class
(k)
are equally-normed, and v w/ 0 for all tuple (k: k.j)) E [K] x [K] x [fx] with k' * k. Note 2 (ʃ + 1)K is a necessary condition for the existence of (ʃ + 1)K mutually orthogonal d-dim vectors in equation (16). Figure illustrates the centers of Class and Class
(k) Let (k) = +/2 Σ/1 w (k) be the mean of the centers class k. Since all w w E [fx] are equally
normed and equi-angular for each k; we can denote of = ((w(k) (k) w (k) and pi = ((w(k) wx) with
no ambiguity. It is easy check that cos of
sin cos" 0 sin? and cos p = V cos 0+ sin
fx
Then we define the bias-free regression loss for the UFM w.r.t the feature h and the multi-center frame by
K
Wha
(K) hg
1)2
(w(
(K)
be )²
(20)
25
2S
2S
j=1
J
k k
align centers
seperate with the centers of other classes
5

---

to measure the average extent to which feature hg,4, collapses to its class centers w) w while stays away from centers of other classes, where Va,i, is the "fa-hot coding":
Yk.1 = Yk. Sk.l = 0,...,0, O,
O, O..
fu 1's
Σ fm U's
Let Y : V1...... .2. KK,,g E RFXN the minimization of the regression loss over features subject to C turns into our prototype optimization problem in this paper
AWo 2
A
P: min
WH
Woll
H 2
s.L. (V, Wo) EC.
(21)
H.V.W. 25
3.3 Neural Collapse to Multiple Centers (NCMC)
The following theorem (proved in appendix D characterizes the global solutions of the optimization when the data is balanced.
Theorem 3.3. Given ni 112 _... ak and d 2 (s + 1)K. K vndudww. S cos cos then any global minimizer (V, W H of satisfies
ha _.. -hi = hi x (k), Wk E [K].
(22)
(hr hî) = O, w (k)a* he w by Vj.j' E [fx] and k W E [K]
(23)
Awo nAw
wgh K, and ||wil| = =...= [w - -AAw. +
COS o
1h w" I''h" =
(24)
Awo ndn
cos? a
Awo 1/ 1 = ndH || and | =... ||b l12
(25)
or otherwise, the objective P is minimized by (V, Wi. H') = 00,
For imbalanced data and non-identical expansion factors fk. the following theorem shows the
relationship between the optimal conditions and the parameters fk. 0, and ny (proved in appendix [)
Theorem 3.4. If cos cos 1 satisfies
AWOAR
for all & E n
[K], then the global optimizer (V, W. H") of
h =... hi = h x (+), (t) Vk E [K].
(26)
w (k)**
h -w (K)++ hi Vj.j' E [fkls and k [K]
(27)
(hir, hī) = 0, + k'
(28)
SAwo + V
Awo cos cos aj
|| Jenk
nxAH
and |
Awo naAH
||
(29)
Awo cos of ngAn
or otherwise the objective P is minimized by (V, W H 0).
Compared to the Theorem 3.3 Theorem 3.4 indicates that || ||- |/ || and the ratio between them depends on all the expansion factors fj's and the size of the class n2j. Both theorems show the features of class l converge in the direction of (k), the mean of the centers of class A. We term this type of collapse "Neural Collapse to Multiple Centers (NCMC)".
Remark 3.5. The optimality of V is controlled by the centers.
Remark 3.6. NCMC differs from UFM analyses in existing literature since h and w are not aligned at optimum, and the norm of the optimal classifier depends on the expansion factors of the classes fk-

---

Corollary 3.7 (Corollary of Theorem 3.4). The optimality condition of P as V and Wo are both fixed satisfies
hi,1 = 1k = h x w (b) Ak E [K].
(30)
(w)) 3 hīg = (h hg) = 0 Ak * ke [K],j' E [sv]
(31)
wi (R)T
hE -... w (R) h vk [K]
(32)
fix cos a
[ l2 fk cos² at SHS
(33)
Remark 3.8. According to the proof of the corollary, the fixed classifier does not require the condition w.r.t the lower bound of cos a". Moreover, is also clear that although h aligns with (((), the length of it relies on the value of fR nonlinearly. Indeed, from the Corollary,
fkr cos a SE cos ot + SHS
h 22
COS o
S Jn cos OZ
has the global maximum cos oj Remark 3.9. NCMC induces an approximate rule GCR the following two aspects:
S if A < fk. It
(1) the centers are "almost orthogonal" to each other: two centers from different classes are orthogonal to each other. The angle between two enters from the same class is arccos(
(w (R);
||
arccossin's for j + j'. j'. as small, the angle close to 2"
(2) Under NCMC of our problem setting, the RCR w.r.t. the Wo (see 7 can be considered surrogate of GCR correctly, then can be classified correctly by GCR with probability larger than 5.2 We will discuss this more formally in the proposition C.1
extent: if hard-to-predict feature can be classified by RCR with a margin
3.4 NCMC for Fixed Classifier
In the Theorem 3.3 Theorem 3.4, we present the NC conditions for P. However, solving P requires optimization of scaled orthonormal frame non-euclidean manifold, which is computationally expensive for overparameterized models. We hope the classifier can be fixed while not losing its performance severely. We first analyze the fixed elassifier via UFM (proof in the appendix D). and later in the next section we propose practical loss function for the fixed classifier.
3.5 Class-Aware Strategy for Determining the Number of Centers
The proposition3.1 3.1 indicates the extra dimensions help improve the classification of "hard-to-predict" samples in the distribution, and basically the expansion factors should satisfy fi < J2 <... < JK when the class size decreases, i.c., n1 n2 ... Rg. The principle of generating these fa 's's is twofold: (1): fi 2 1; (2): the ratio of ascending {fx}) shall approximates the ratio descending of {nz}. i.e., Js -x-4 for all E [K].
TE+1 PK-t--
Concretely, we use the Class-Aware Strategy to generate the expansion factors fs when f 2 2: [n R.............. . , nk] to
Step 1: Given the descending list [2,,72...... n and scale
(1) [N 22 ".....,".
"
Step 2: Calculate [a1, "1,,a...... "2..a.,,aa" agl where as ($ () 1, to ensure each element is positive;
Step 3: Reverse the order of the list to [ax, akk, an then add from the left until the sum of the elements in the list equals JK.
For example, when (n, n22 n3) = (1,3,3) and f then Step outputs (4, 7. 7); Step outputs (1, 3, 3); Step 3 outputs (fi, f2. fa) 44,4,
7

---

Experiments
In this section, we (1) propose a cosine loss function for fixed classifier; (2) verify NCMC induced by the cosine loss through experiments; (3) show how f and 0 influence the learning performance; (4) Compare long-tail tail classification performance to SETF method with fixed classifier and other classical methods with learnable classifier.
4.1 Datasets and Training Schedule
We set long-tailed classification tasks on five datasets, CIFAR-10 38
CIFAR- 100 (38), SVHN 39
STL-I0 [40], and large dataset ImageNet [41]with two architectures ResNet-50 and densnet- 150 (details in appendix F Let = Pane represent the imbalance ratio of the long-tailed sampling from the dataset, where mmaz and Flmin are the size of the largest class and the smallest class, resp. The accuracy results are the average of three repeated experiments with different seeds. The best and second-best results are boldfaced and underlined.
Ammn
4.2 The Cosine Regression Loss
The Cosine Regression Loss. Motivated by the toy example 3.1 and the theoretical justification of NCMC, we propose the regularized loss for the fixed unit-norm multi-center frame that satisfies definition 3.2
L(W, he.se) = B > Cos(w) (k) .x.ie) + A (1 1)².
(34)
where Cos(w, h) = (1 [[h]] 112 is termed Cosine Loss, À and / are regularization coefficients (we relegate the selection of the coefficients to appendix ). This is called Cosine loss discard all
Wj (kk) terms of loss (20) for features from class k, where k' * k. since in practice h2 aligns with ((k)
(KK)
only if (w/?),
(K) .h) 0. The derivative of the loss with respect to some feature h then is given by
a Cos(w,h)
(w. h) |h]
(w,h)h |h|
(1
(w
(35)
dh
[h|
(the derivation is postponed the Appendix It shows the gradient changes both the magnitude and the direction of the features with scale |h| of the feature, compared to the dot-regression loss in (26). The regularizer guarantees the gradient does not vanish for the features with large norm and explode for the features with small norm since Cos(w, h) is scaled by ||h||-?
Note that, the loss uses the Multi Celti--eeeerr Frame W the training while, due to Remark 3.9 we still use Wo as the classifier. In the context, we term the objective (34) as fi J2 the "Average Loss" (AL.), and that with fs selected by class-aware strategy the "Class-Aware Loss" (CAL).
JK
4.3 Neural Collapse
We design experiments to verify loss P. AL and CAL induce NCMC. The collapse is measured
by ACC hx.i./ [ Il - w((k)/ with) || For simplicity, calculate the mean and standard deviations the vector hg.a b w (k)) (/()) when the mean and standard deviation tends
zero, we can show that NC occurs. We f = 20 and 0.2; we pick the tuple [f1 fe..... .......fkt fx] by the Class-Aware Strategy. Figure [ shows that both AL and CAL induce NC. The change of mean is rapid and the standard deviation first increases and then converges to zero slower. Variability is defined as the average of the sum of a =
ba.ig
[wall over all l and it. and measures the
b
alignment of H and W, and we observe the in all three objectives the variabilities stay away from zero. Figure 4 plots the NCMC of loss with or without regularization on the norm of features. The regularization results in heavier NC.

---

We also provide plots of NCMC on VGG and LeNet for CAL that demonstrate the universality of the phenomenon (refer to Table 5) We also draw heatmaps of the neural collapse for better visualization. h is the normalized 6 the heatmap of H H where H [ h2 ... hg] and h3 == Th I|
class-mean features. Fig is is the heatmap for the normalized features in class 9 of CIFAR-10; 10; is noted that all features in this class stay close to each other from the initialization to the end of training.
NC
Variabiity
Cifar100 (00 x = 0.01
Cifar10
Cifar100
Cifar10
0.01
= 0.01
0.01
0.72
0.52
043
0.50
0.50
0.25
0:25
Figure 1: An illustration of the Neural Collapse curves of AL (a and b. on CIFARI00 with T 0.01 and CIFAR 10 with r = 0.01 respectively), CAL (c and d, on CIFARI00 with r = 0.01 and CIFAR 10 with 7 0.01 respectively). The AL is equipped with f=20 and CAL uses Class-Aware Strategy.
4.4 Long-Tailed Classification
We conduct an ablation study with ResNet50 CIFAR-100. Table presents the performance of ResNet50 on both balanced CIFAR- and its imbalanced sample with or without Mixup training under CAL at imbalance ratio T = {0.005, 0.01, 0.02, 1(balanced case)). ]. We do not use objective P since it is inferior to the Cosine Regression Loss (see Table P is incompatible with mixup training and it may be degenerated by the myriad of distraction terms in the objective.
As shown in Table the performances get improved for almost all imbalanced ratios when the CE is replaced with our designs, especially CAL. And method is compatible with Mixup training. However, the SETF and our method have lower accuracy than the CE in the balanced case. The analysis in the Proposition tells us that in the balanced case where fi 's identical, our theory will not have a significant positive effect on the classification. Our method has a similar form to SETF
Table 1: An ablation study of Mixup method for training ResNet50 on CIFAR-100 using different classifiers and loss functions. The numbers in the second row are imbalance ratios. The parameters are fixed with f = 20, and 0- = 0.2
without Mixup
with Mixup
Methods
0.005
0.01
0.02
balanced
0.005
0.01
0.02
balanced
ResNet50
CE
35.6=0.3
37.5.0.4
43.310.2
79.5-0.3
42.410.55
46.7.0.3
52.710.3
81.810.1
SETF
38.1=0.4
42.610.2
48.410.3
78.7-0.2
43.0.1.11
48.3
52,510.3
79.710.2
CAL
40.6=0.3
44.7+0.2 50.2+0.5 78.5=0.4
46.5+0.5
50.1 0.3 54.3+0.4 79.4+0.4
who fixes the ETF classifier and emphasizes the gradient norm balance among the classes. On the contrary, CAL concentrates the fitting to the multiple centers. We compare our method CAL to CE and SETF on four small datasets CIFAR-10, CIFAR-100, SVHN, STL-10 (Table and Table 5). It shows the stability of our Class-Aware Strategy and displays significant improvement the original networks.
The hyper-parameter have influences on the performance: if 0 is small, the centers of class very close to each other, else if 0 gets larger, the margin of the classifier will be smaller; parameter f encodes partial imbalance information of the classes, when f is small all fa are close to each other and cannot offer useful supervision for the hrrr-tt-- samples. Refer to appendix for more information about hyperparameter selection.
We summarize in the performance of CAL as f and vary. The result empirically demon- strates: (1) fixing J. the accuracy roughly peaks at some 0 bounded away from 0 and л 7/2. (2) at 0 = 7/2, The frame collapses to the classifier, resulting an accuracy rate similar to SETF [26] where the norm of the gradient is weighted by the class imbalance ratio; (3) Frame that is orthogonal

---

Table 2: Long-tailed classification accuracy (%) with ResNet and DenseNet on CIFAR-10 and CIFAR-100.
CIFAR-10 10 0.01
CIFAR-I00 0.01
Methods
0.005
0.02
0.005
0.02
ResNet
CE
72.3:10.1 78.610.2
84.010.1
42.4.0.55
46.7.0..
52.7.0.3
SETF
74.2+10.5 79.740.44
838840.
43.0.0.11
48.3+10.5
52.5.0.3
CAL
80.010..5
84.1+0.4
85.910.2
50.11 46.5
0.3
5..310.4
DenseNet
CE
71.1+00.5
77.7+0.3
84.140.1
42.9+00.2
47.4+0.2
03.3+0.2
SETF
72.9+0.4
78.5+0.3
83.4+03
42.3+0.2
46.3+0.3
52.6++0.2
CAL
78.1+0.2 81.1+0.2 84.5+0.2
46.3+0.3 50.1+0.2 54.0+0.2
to the classifier does not learn anything in the training since the representation is irrelevant to the classifier, thus providing no useful discrimination information. We compare our method to several
Table 3: A comparison of several recent methods of long-tail tail classification trained on ResNet50. = 20 and 0 = 0.2 are fixed. The values without are that we did not reproduce.
Cifar-10
Cifar-100 100
ImageNet
Methods
0.005
0.01
0.02
0.1
0.005
0.01
0.02
CE (Mixup)
1
78.6+02
84.0+0.1
41,
46.7=0.3
52.7+0.3
67.9
$7..=0.1144.2+00 04.
LDAM-DRW
74.6=1..3
80.1
84.1
90.0
39.44...
44.2=0.2
50.0
62.5#0.2
47.7
KCL
75.0-10.3 80.910.2
84.510.3 40.:::0.4:4003::44
44.8:10.3
50.2:1.2 63.0:0.2
51.5
SETF
74.2.10..5
83.8.1..3 79.710.4
1..31...433.01100
48.3.0..5
$2.510.33 66.110.3
44.7
ARBloss
79.5++0.7
83.8+-0.4
86.1+0.3
1..54+0.3
47.1 49.7+00.2 64.4--0.4
52.8
CAL
80.0++0.5 84.1+0.4 85.9 +0.2 92.0+-0. 46.5+0.5 50.1+0.3 54.3+0.4 65.9+-0. 49.7++0.2
classical methods including NC-inspired methods ARB loss and SETF, margin-based LDAM-DRW, contrastive learning method KCL and original CE. The Jable the methods in comparison. We observe that our method is comparable to or even better than the others. The comparison indicates: (1) Our method has some advantages for heavily imbalanced cases. One of the underlying mechanisms is when the minor classes are underestimated due to the lack of sample, they are likely to display the gaussianity such that the proposition 3.] and class-aware strategy work fine; as the imbalance ratio T rise, the minor classes lose their randomness during training, where method fails. (2) Our method does not compete with the KCL and ARBLoss on ImageNet, reflecting the limitations of method in the flexibility of the direction and magnitude of the classifier weights when learning large datasets. We also compare our method to a recent work RBL [42] in the Appendix H that demonstrates the potential limitation of the fixed classifier and the advantage of CAL. (3) two classical methods LDAM-DRW and KCL do not compete with other NC-inspired methods trained with Mixup training, demonstrating the effectiveness Mixup training strategy. (4) Our method with the parameter chosen in the experiment outperforms CE negligibly or performs worse than CE when the imbalance ratio approaches 1. However, we find picking f 10, 0.5 gives accuracy rate 92.5 + 0.2 on Cifar-10 10 with 0.1, which shows the significance of hyperparameter selection for our method.
5
Conclusion
In this paper, we rethink the regular 1- 1-Nearest Neighbor Classification Rule (RCR) in imbalanced learning; an analysis of the Hard-To-PPeed feature indicates under certain circumstances the general- ized classification rule (GCR) is superior to RCR, which implies that minor classes should compare to more "neighbors" in the classification. Then we introduce neural collapse to multiple centers (NCMC) under an MSE-type loss, where the centers play role similar to the neighbors in GCR. According to the framework of the Unconstrained Features Model, the features of each class collapse the class mean of the centers in balanced or imbalanced settings for learnable or fixed classifiers. We notice that at NCMC. RCR resembles GCR in terms of the hard-to- -ordot----eeettt feature distribution. We then propose the cosine loss, a surrogate regression objective of the MSE-type loss called Cosine Loss, that applies to the fixed classifier; and develop the class-aware strategy for determining the number of centers of each class, inspired by the analysis of the Hard-To-Predice Feature. The cosine loss practically induces the NCMC at the terminal phase of training; the combo of the class-aware strategy and the loss with the fixed classifier demonstrates its effectiveness in long-tailed classification. Our work shows the possibility of obtaining task-specific classification rule by designing the optimal structure at neural collapse under customized losses; it provides connection among the optimal structure of the feature-classine aligument, the classification rule, and the generalization in the learning problem.
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China (Nos. 62136005, 62276162. 62476160, 62306170) 170). the National Science and Technology Major Project (No.2021ZD01 12400), the Science and Technology Major Project of Shanxi (No. 202201020101066, the Special Fund for Science and Technology Innovation Teams of Shanxi Province (No. 202304051001101)
References
[1] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6088)5333-336 1986.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffry E Hinton. Imagenet classification with deep convolutional neural networks. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
[5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708. 2017.
[6] Yibo Yang, Zhisheng Zhong, Tiancheng Shen, and Zhouchen Lin. Convolutional neural networks with alternately updated clique. CoRR. abs/1802. 10419, 2018.
[7] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort E. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seyhold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. CNN architectures for large-scale audio classification. CoRR, (bs//609,094300 2016.
[8] Andrew R Webb and David Lowe. The optimised internal representation of multilayer classifier networks performs nonlinear discriminant analysis. Neural Networks, 3(4)3367-375, 1990.
Daniel Soudry, Flad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srehro. The implicit bias of gradient descent on separable data, 2022.
[10] Vardan Papyan, X. Y. Han, and David I. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. In Proceedings of the National Academy of Sciences (PNAS), volume 117, 2020.
[11] Cong Fang, Hangfeng He, Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences (PNAS), volume 118, 2021.
[12] Dustin Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint 20Xiv:2011.11619, 2020.
[13] Janneng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224 221-241, 2022. Special Issue on Harmonic Analysis and Machine Learning.
[14] Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors. Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 270-290. PMLR, 16-19 Aug 2022.
[15] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. CoRR, abs/2105.02375, 2021.
[16] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J. Su. An unconstrained layer-peeled perspective on neural collapse. CoRR, abs/21 2021.
[17] X. Y. Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference Learning Representations (ICLR), 2022.
[18] Tomaso A. Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. CoRR, abs/2101.0002,, 2021.

---

[19] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Muchine Learning (ICML), 2022.
[20] Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The, Nhat Ho, and Tan Nguyen. Neural collapse in deep linear networks: From balanced to imbalanced data, 2023.
[21] Peter Súkenfk, Marco Mondelli, and Christopph Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model, 2023.
[22] Like Hui and Mikhail Belkin, Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. CoRR, abs/2006.07322,, 2020.
[23] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[24] Tomer Galanti, András György, and Marcus Hutter. On the rolc of neural collapse in transfer learning, 2022.
[25] Tomer Galanti, András György, and Mareus Hutter. Improved generalization bounds for transfer learning neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
[26] Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[27] Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction-repulsion ballacee loss for imhalanced learning. In Newrocompating. 2023.
[28] Xuantong Liu, Jianfeng Zhang. Tianyang Hu, He Cao, Lujia Pan, and Yuan Yao. Inducing neural collapse in deep long-tailed learning. 2023.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings the IEEE international conference on computer vision (ICCV), 2015.
[30] Like Hui, Mikhail Belkin, and Prectum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning, arXiv preprint 22X:::2002.08884 2022.
[31] Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo. Maximally compact and separated features with regular polytope networks, 2023.
[32] Federico Pernici, Matteo Bruni, Claudio Baccchi, and Alberto Del Bimbo. Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems, 3379)4373-44887, 2021.
[33] Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and decoupling neural collapse via hyperspherical uniformity gap, 2023.
[34] Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last 2018
weight layer. CoRR,
[35] Tong Liang and Jim Davis. Inducing neural collapse a fixed hicrarchy-aware frame for redueing mistake severity, arXiv preprint arXiv:2303.05688 2023.
[36] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datascts with -ibel-iistribution aware margin loss. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Alu, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[37] Bingyi Kang, Yu Li. Xie, Zehuan Yuan. and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021.
[38] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
[39] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
12

---

[40] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.
[41] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image datahase. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[42] Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang, Huiyang Shao, and Qingming Huang. Feature directions matter: -ailed learning via rotated balanced representation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27542-27563. PMIR, 23-29 Jul 2023.
[43] E Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning 2022.
[44] Jianfeng Lu and Stefan Steinerherger. Neural collapse under cross-entropy loss. In Applied and Computa- tional Harmonic Analysis, volume 59, 2022.
[45] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML), 2022.
[46] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[47] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeles perspective on neural collapse. In International Conference on Learning Representations (ICLR), 2022.
[48] Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. arXiv preprint arXiv:2210. 16658, 2022.
[49] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold, 2023.
[50] John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks. In International Conference on Learning Representations (ICLR), 2021.
[51] Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and 12 normalization with improved -uttr--diittiuutti detection in deep neural networks. Transactions on Machine Learning Research (TMLR), 2022.
[52] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Topological, Algebraic and Geometric Learning Workshops, 2022.
[53] Xiao Li, Sheng I Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez- Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arxiv: 12. /2206, 2022.
[54] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Un- derstanding imbalanced semantic segmentation through neural collapse. arXiv preprint arXiv:2301.01 100, 2023.
[55] Zexi Li, Xinyi Shang, Rui Hc, Tao I. Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv preprint arXiv:2303. 10058, 2023.
[56] Yongyi Yang, Jacob Steinhardt, and Wei Hu. Are neurons actually collapsed? the fine-grained structure in neural representations. I Can't I Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification (NewrIPS 2022 Workshop), 2022.
[57| Jiawei Ma, Chong You, Sashank J Reddi, Sadeep Jayasumana, Himanshu Jain, Felix Yu, Shih-Fu Chang, and Sanjiv Kumar. Do we need neural collapse? Learning diverse features for fine-grained and long-tail classification. Openreview preprint, 2023.
[58] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng, Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10795 10816, 2023.
13

---

[59] Nitesh V Chawla, Kevin W Bowyer. Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[60] Chris Drummond. Class imbalance and cost sensitivity: Why undersampling beats oversampling. In JCML-KDD 2003 Workshop: Learning from Imbalanced Datasets, volume 3, 2003.
[61] Hui Han, Wen- Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing, pages 878-887. Springer, 2005.
[62] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem convolutional neural networks. Neural Networks, 106:249 259, 2018.
[63] Chen Huang. Yining Li, Chen Change Loy, Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and puitern recognition, pages 5375-5384, 2016.
[64] Samuel Rota Bulo, Gerhard Neuhold, and Peter Kontschieder. Loss max-pooling for semantic image segmentation. In Proceedings the IEEE conference computer vision and pattern recognition, pages 2126 2135, 2017.
[65] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[66] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662 11671, 2020.
[67] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in Neural Information Processing Syxtems, 33:4175-41888 2020.
[68] Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2980-2988, 2017.
[69] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Punect Dokania. Calibrating deep neural networks using focal loss, Advances in Neural Information Processing Systems, 33, 2020.
[70] Zitai Wang, Qiangian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re- weighting and ldjit-adjustment for imbalanced Iearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[71] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbu: Bilateral- branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 9719-9728, 2020.
[72] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. CoRR, abs/1910.09217, 2019.
[73] Zhisheng Zhong. Jiequan Cui. Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489-16498, 2021.
[74] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
[75] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Liu, and Dilip Krishnan. Supervised contrastive learning, 2021.
[76] Jicquan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning, 2023.
[77] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang. Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long tailed recognition, 2022.
14

---

[78] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[79] Changyao Tian, Wenhai Wang. Xizhou Zhu, Jifeng Dai, and Yu Qiao. VI-Itr: Learning class-wise visual-linguisic representation for long-tailed tailed visual recognition. 2022.
[80] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts test-agnostic taigetailed recognition. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 34077-34099 Curran Associates, Inc., 2022.
[81] Zhiyong Yang. Qianqian Xu. Zitai Wang, Sicong Li, Boyu Han, Shilong Bao, Xiaochun Cao, and Qingming Huang. Harnessing hicrarchical label distribution variations in test agnostic long-tail recognition. CoRR, abs/2405.07780, 2024.
[82] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. CoRR, abs/17 1710.0941, 2017.
15

---

APPENDIX
A Related Work
A.1 Neural Collapse
Since the original paper 10, neural collapse has been intensively studied, with 12 11 introducing the now widely accepted unconstrained features model (UFM), and layer peeled model. The literature of NC can be loosely categorized into: (a) the study of the geometry of NC [4] 44 45] [ 46
47
12
33
48
50
(b) finding connection between NC phenomena and NN performance
27
25
51
52
26
53
54
35
55
and (c) empirical observations concerning NC [66] 57
A.2
Long-Taile Tailed Learning
Long-tail learning refers to the training from data with long tailed-class distributions 158 We list several categories of methods that are related to our work. Re-sampling. Early works [59 60 61 62 re-sampling is a sampling strategy that balances the sample used for learning by over-sampling the samples from minor classes or under-sampling the samples from major classes. Over-sampling may make the model overfit these classes and harm the generality. Under-sampling removes some samples of major classes, which might remove the key data for representation learning and bring the performance drop.
Re-weighting,
Another idea is to assign different weights for different classes, even instances. 66 64 65 66] re-weight the loss according to the sample size of the class. (7] introduce the prior probabilities from a Bayesian view and balance the exponential logits via the class sample size. Focal loss 668 [69] re-weight each instance according to their hard-to-lean
level, i.e., making the model take more care of the wrong- recognied samples. Wang al. [70] provide a data-dependent generalization bound that explains the success of re- re-weighting and logit adjustment strategy.
Two-stage Learning. The representations learned from iistance-balancinn sampling are believed to be general. Many researchers attempt to separate learning process to two stages. (36) trains the model in a normal approach in stage 1 and then uses deferred resampling strategy tune with class-balanced resampling or uses deferred re-weighting to re-weight different classes in stage 2. 77] disentangles the basic feature learning and re-balancing learning via two separated decouple the representation learning and classification. They claim that instance- balanced sampling gives more general representation and originally use balanced sampling to learn the representations at first stage, then fix the feature and retrained the classifier (cRT) by techniques such as label-aware smoothing (LAS) and learnable weight scaling (LWS) in the second.
fine-
branches. 72, 73
Contrastive Learning. Contrastive learning is a learning paradigm that learns representation that maximizes the similarity between positive and negative samples (74). Khosla et al. (75 initiated Supervised Contrastive Learning (SCL) paradigm by leveraging class labels. For imbalanced classification tasks, Contrastive learning usually faces imbalanced positive/ncgative pairs of samples. To balance the feature space, KCL (37] balances feature space by using the same number of positive pairs for all the classes; class complement methods 76 77) are proposed to construct positive and negative pairs for the rebalance of the class. Recent advances in multi-mode foundation models such as CLIP (8) and VLLTR (79] have displayed remarkable performance on the downstream long-tail classification tasks.
Learning to Mitigate Minority Collapse. Yang et al. 26 propose the method (we name it "SETF") that learns the representation from imbalanced data with last-layer classifiers fixed as a simplex ETF during training and prove the optimal last-layer features converge to ETF structure; Gao et al. [42] propose the method that optimize the ETF structure under rotation and the uses post-hoc logit-adjustment for prediction. Liu et al. (2 use NC Regularization to minimize the within-class variability and maximize the between- separateness of the output features; Inspired by neural collapse phenomenon in balanced case, Xie al. (27 design class- balsncad CE loss (termed "ARB loss") that aims to balance the gradient among classes. Zhong et al. [54] discover the minority collapse phenomenon semantic segmentation and use neural collapse as regularization to improve the discriminate ability of the network. Learning Test Agonistic Label distribution. In 80,, the authors consider an imbalanced learning task where the training data is long-tailed while the distribution shift between training data and test data is unknown. They deal with the problem by applying diverse
16

---

expert networks in training to handle different class distributions and aggregate the experts in the test time. Yang et al. [8]] propose another expert-mixing strategy that tackles the mild changes in the label distributions.
Learning with fixed classifiers. Apart from SETF, other symmetric structures, such regular polytopes [31 32 33), Hadamard matrix [34], and hierarchy-awarr frame (35) have the classifier fixed in the network training process. Additionally, fixing the classifier also computationally friendly strategy since the classifier does not need backpropagation.
B Illustrations
B.1 A Toy Example of Hard-to-Predict Sample
Fig Fig2is is illustration of Hard-To- To-Predict sample (the purple dots). The points simulated from the 3-dim multivariate normal distributions, where colors indicate different covariances and means. The red, green, and blue represent the unseen sample from the distribution (Call it P1) that can be well-classified by the trained model. The purple points are drawn from the distribution (P2 in the proposition) randomly classified by the trained model. The whole feature space of the underlying data distribution is assumed to be = on Pi + Q2
05
0.0
-0.5
-10
-15
-15
-10
0.0
Xaxis
805-8.5
00
431
10
15
-10
Figure 2: An illustration of the toy example
17

---

100
Class
0.75
0.50
Vector w1
0.25 Z
Vector vll
0.00
Vector v12
-025
-0.50
wl1
0.75
w12
1.00
Vector w2
Class 2
00
Vector v21
5
50
Vector v22
25
w21 75
00
500.2 2...25.00 2.000....222.. 2.0
0 Y
25
w22
X
0.50 0.75 1. 100 00 - 11.00 00
02..500..70
50
Figure 3: The 3D illustration of multiple centers for class 1 and class 2. The dashed blue vectors are the two centers of class and the dashed green are the two centers of class 2. And Wiy cos f 0ij + sin 0 wi where i,j (1,2).
B.2 Multiple Centers of Two Classes
C Regular Classification Rule v.s. Generalized Classification Rule at NCMC
In the following proposition C.I let z be the hard-to-predicc feature sampled from P2 at NCMC, ie. = = μ² 1/'+2 where p ~ (0, Ia), and μ' 7 Ex= (M), Denote 7 == max (v) (8) 2) ,)). The difference here to proposition 3.1 is the mean jí' induced by NCMC, instead of original NC without the centers under the regular MSE loss. Then we prove that in one-vs-one setting, the original RCR
( (k)
arggaax We 2}. kE(K]
(36)
with small margin implies correct classification with probability over 1 by GCR. Proposition C.1. Let W be defined as in definition
3.2 with unit-norm centers and classifiers assume cos?
fu fw. Then there exists an eo
cos2
such that
J&K
J.cK
for any co. (Wk,2) (Ww, 2)
cos
fn -
- implies
K sin
for
sin
Pfmas (w/+) (k)
2))
max (w (k') ( 1 y'Elful
felfal
for any e' ep. (Wk,2) (WAr z)
cos"
Ter +
sin implies
d'
K Kind
max (w (k) JE
12)
max (w)/ (k))
f'Elfe)
Proof. Due to the orthogonality of V and Wn. (v())) 2) or (v 2) are still independent isotropic
Gaussian, conditioning on the value of
(Wa, 2) (Ww, 2) (Wr.P) (ww.p) .p)
(37)
18

---

since
W&H = -arj
(38)
Moreover, the maximum of i.id gaussians has continuous (differentiable) density, so {(x) == ply > 3) is continuous decreasing function, where f(-oo) = ((0)) = 0, and
cOS 0 E( fuK
cos² A
= max (v
(k)
(k') J'El .p)}
max (k')
fx
.P)
LwK
Je[Sal
SK + Sw
Therefore there exists only one
cos a J.K
ens J.kk k
such that,
ED COS
and
"
for 60.
(39)
cos 0
d'
<
for d' > co.
(40)
cos
Then with probability over } (1) the last few conditions implies
max (w .2) max (w (k') JELSa
J'ElNN] 3' .2)
(41)
max (v) (k) jElSal
cos 0, 2) + (W sin 0, 2)
max (v(P) (K) jElfal 8'
cos fos0,25 2) - (Wer sin 0, 2)
(42)
cos?
cos²
=y cos 0 + sin ((W..P) - (ww.p))
(43)
fuK
fw K
20
(44)
for Co-
(2) Similar to (1), using formula 440 and (38) we deduce that with probability over 1::
max (k) ,2)
max (w) jE
.2
(45)
J'E(
cos2
cos? a
=y cos 0 + sin 0((w,((w)) - -(w.. +
(46)
f
f
<
(47)
for d' > co-
D Proof of the Theorems and Their Corollaries
Proof of Theorem 3.3 this proof we consider the balanced case where n, = n2 = ... ngg and
S = fNN Y = Ix & 1 8 1T The proof is to lower bound the
Aw 2
Ah 2
f(W, H) = 2JN
WH
Y
+
W
H
19

---

by a column of inequalities. First, observe that
Awo
WH 11
+
Wo
2
+
|
(48)
2/N
2
2
K
n
AW.
K
K
n
Whax' yxel2
|w
+
hx
(49)
2K
[n
2
2
k= i=1
k=1
K
T h - 1)=
1
( he.
(50)
2K fn
2K fn
Kith
K
Xwo
A 2
|
1 +
1baill
(51)
2
k=1
k=
1=1
K
f
1
n1
(4)
hgg
1)
2
AW
Wk.
A
n
h
112
fm
11
2
2
72
k=
{=j
k=
(52)
(0)
f
(A)
2
Aw
K
(6)
2
n
h
+
W
+
n
h
2K
72
72
k=1j
k=1
k=1
(53)
The inequality (a) follows from setting
(k')
h
(54)
for all k' * k and i [n]]. In (b) we used Jensen's inequality, which (due to the strict convexity of ( 1)² and || ||2) holds with equality
h =.. ha.
(55)
for all & E [K].
Since all features in each class are identical, h2 h for all is [ng]. Continuing from the last inequality, we have
K
K
RHS
ns
(R)
1 +
Xwa
K
1
[well2
+
ndH
K
1
|
ll
2K
2
K
2
K
k=1
k=1
(56)
We get (c) by Jensen's inequality, which holds with equality iff
w(+) h w (8) h w2
(1)
h4 =.. =w/
w/
(K)
h Vj E [f]
(57)
[w] 1 .... wwkl22
(58)
| 2l2 _...= [bxll2,
(59)
then continuing from the RHS of the last inequality
(d)
K Σ (8) -1)
RHS
K /nddAww [[wll2 |lball2
(60)
In (dl we use Jensen inequality for the first term, the equality holds when
W h = (2) h = kK)) hx:
(61)
20

---

and young's inequality 2 + 2 vab for second and third term, with Awo ( ( E /|wol12)
)
and -ndH (# Σ [ball2) 2
It holds with equality iff
Xw. w: 22 = nAw |
(62)
Note the sequel of equality conditions are satisfied by null solution (V, W. H²) (0. so remains to show when the solution is not trivial.
According the equality conditions and the symmetry w.r.t. le E [k]] and j E [ʃ], the RHS of the last inequality turns into the expression
(k) hi 1) + K +KVnnh VndHHW /|wall2 ||hal|2
(63)
2
ww/ (k)
cos cosaa- 1)
+ K VnVH nd //well2 |ball2
(64)
2
Xwo | cosa -
V
+ KAw, ||wal12
(65)
2
ndH
Awo cos2 a [Iwall 2 ndæ
KAwa
Awa ndæ
cos a
2
K\wo
Awa a nAH
Awo nAH
cos? a
2
2
Awa cos? a nAH
(66)
which obtains minimum at |w&l1 O if Kλw.
Awa cos
>
0 and
||wal]?
nAs
Awa
K KAw +V
if KAwo
Awa
cOs < O, where a = L(w) (R) hx), and for simplicity we RAH
Wo cos nAw
denote
KAw,
Awa
cos
f(a) 2
(67)
Awa, cos? nAH
In the first case, the minimum is 1; in the second case, have to find the smallest value of fa). which is equivalent to find the lower bound (or upper bound) a (or cos a) since f(a) increasing
in a if KAw.- V Aww
cos < O.
nxw
We observe that a is non-zero, otherwise by the equality conditions from above, h& has to align with all w (k) which is impossible.
since (w/hh are equiangular and equally normed, for any norm- norm-fixed vector q RRd satisfying
=...= (w( (w(()) v) =
(k) v), have
w (k) q
min <> max c min
(68)
which has its optimum when aligns with ((k according to the convexity of the last mini- mization problem, and thus cos a* c 0+ the optimal direction also form an angle f siu
arccos
with Wi-
oos 9+ sin²
21

---

We have bound a E [K AnAddwWo V coos2 0+ 0j. the minimum of the objective is achieved at the boundaries. It is easy to see that the value of the objective function is lower than 2- which
is the value at the right boundary. Therefore, at the optimum, cos or cos 0+ f siu and
sin 0 oos 4 sin
cos p
For the orthogonality of h 's and wy 's, since let b2 = "1 (()) and h2 7/2 P2w( since w(k) h2 0, (hx. h 7 Ej=( (w(+') .ba) = O. Then we use definition of w to extend the 0 = (hg, h2) = 7zz (Wk: wir sin 0 which implies (Wis, Wxr) = O. We also have learnt that
Awo |w nAH
Kdw. + V
Awo nAw
cos a
1
(69)
cos α²
The imbalanced case is proved with a slightly different strategy.
Proof of Theorem 3.4 the imbalanced case
AH
Awa
WH-
2
+
+
Woll
(70)
25
2
K
Tkk ThE
1
(k)
h 1)
XH
11
1
h
Xwo
Wx
l
25
2
2
Am
1=1
(71)
K
K
1
Aw
Tik
hg 1
b
l1
25
Tik
Tikk
i=1
k=1
i=1
(72)
(c)
1 K
Rk (N) h& -1 11
+ 2 km
K
|
12 +
Awo
K
||
/
(73)
25
2
km
min H.W
K
TKk fi (wilk) h&s 1)²
+ SH 1
Aw.S
l|wxl12
(74)
25
7kk
k=1
fkr (w())" h - 1) 2 SHS (ashaall
Aw.
Tok min ha
1
|wall2
(75)
25
(c)
m& min
Sk (w(N) he - The
-)² + 2.5
AHdWo
[hell [|wall
(76)
28
we
where (a), (b), and (c) are the same as the last proof. The equality of (e) holds only when
Xw.l/ = /
by Young's 's Inequality. We decompose the objective in (e) diffeernt from that in the proof of theorem 3.3 because the existence of non-identical fx fails optimal condition (61). Now we minimize
2 +2 Sk (wrlk) h - 1)
AndWo n
V
||b||]
[[wa|l
for each k where the method in the last proof is applicable, which result in
-SWwG Jane
Awo maAH
+
cos
lw 12=
(77)
Awo
cos2
oz
AgA
when cos of > S.
Fo /
Awa Au n1
22

---

The reader can check that the result of the imbalanced setting contains the balanced one as a special case.
When W and Wo are fixed, the optimal conditions become simpler.
Proof of Corollary 3.7
Similar to the proof of Theorem 3.4
1
d
WH
YII
+
H
(78)
28
K
4
K
1
(k)
he
+
XH
T1kk
b 1
(79)
28
TIK
2
k=
1=1
K
K
(R)
1
AH
TA
hisi
1
+
(80)
25
Tkk
2
Tkk
k=15=1
iml
Σ
K
ni.ff w(+)T he 1
2
+
AH
K
71.
|he
(81)
28
2
k=1
So (w(k) hR - 1) + S 1
min
(82)
25
h
where (k) h& w
h = [hella cos og for all j E [fi]] and ag is the angle between h2 and
w for every j E [fk]. from (a) to (c), the equality holds iff
h = h = h Ak E [K]
(83)
(wir, hī) = =0,Vk' * (K)
(84)
(he hig = =0, V. # k
(85)
(k)) h
(A)) fn
h AK E [K]
(86)
Since each summand is the minimum of a convex function of || 1: P attains its minimum when all of the summands are minimized separately in (d), that is, by the analogous analysis to Theorem 3.3 |
ft cos oj Te cos" ag S
Since we only use the information of the end of the proof, this generalized NC can hold also true given 0₂ 's have different values for each l E [K] in the construction of Multi-Center Frame, demonstrating the generality our method. However, in this paper we are limited to the case of identical 0 for all classes.
E Derivative of the Cosine Loss
In this section we calculate the derivative of the cosine loss Cos(w, h). For Cos(w, h) =
h (w h 11..
d Cos(w,, h) = 2- (1 a)
dh
where is the Jacobian of h [h] w.r.t. and a = (w.h) [[]]]]
It is straightforward to calculate J
Thj ([hb] I hh hhT). |[h]] I - hh has eigenvalue ||h|| ||b]|2 on the direction [b] and [h] on all
23

---

other directions orthogonal to
h
so that
Jw
(87)
h
h
=J J
(w
+
(88)
[h|
[h]
(w
a
+
(89)
||b]|]
h
h
h
[b|
(w ah).
(90)
[|]]]
F Implementation Details and Results
F. Long-Tail Classification on Four Datasets
We run experiments with backbone ResNet50 and DenseNet150 on the four datasets CIFAR-10. CIFAR-100, SVHN and STL-10) by 1 A100 GPU, and run ResNer50 on ImageNet by A100 GPU with an extra linear layer that expand the dimension of the backbone feature to be larger than fK. We use DenseNet150 with reduction 0.5; the growth rate is set 12 on CIFAR-10 and CIFAR-100 and 32 on SVHN and STL-10. The reason we adjust growth rate to make the dimension of output feature higher than fK) which is necessary for our method. To display the neural collapse phenomenon, an extra linear layer is added between the backbone and the classifier.
We the code released by 73 produce the imbalanced datasets. We train the model on the four dataset for 200 epochs, with step learning rate initialized to 0.1 decaying to 0.01 and 0.001 at epoch 160 and epoch 180, batch size of 128, momentum 0.9, and a weight decay of 2e We train on ImageNet for 200 epochs, with a ConealAn learning rate initialized to 0.1, batch size of 512, a momentum of 0.9, and weight decay of 5e 4.
Given imbalance ratio 5 = Imin where Mmin and Mmaax are the minimal and maximal numbers of training samples all classes, the numbers of training samples are decayed exponentially from Tmxa to among classes. We take the canonical data normalization and augmentation for the five datasets. We use both the regular method and Mixup method [82] to demonstrate the effectiveness of proposed structure. The distribution when Mixup is used. We fix the classifiers as orthonormal vectors for every setting.
Tmaax"
1 by default controls the shape of the symmetric
We train the network by weighted cosine loss with a norm regularization which has the form
cos2 0 + fx sin? C(bg, W)
0 fk Cos(w/
(R)
> ((/ba, 1)², (91)
fk
where hg. is an example in class k. and À = 6c - 4.
The classification rule of method, BN(h) argmazse(k] [|]N())]] w where BN is the batch normalization of h; we use BN since the data may have non-zero mean; For SVHN and STL-10 datasets, we first normalize the backbone outputs h, then train and predict by the same loss and decision rule. For Densenet, we h use the BN [h ) as the backbone output, apply the original loss to train, and predict by the same decision rule for ResNet50. The code is modified from (26), and can be found in the supplementary
suggested by remark 3.9 is
24

---

material. Then the upper bound of the gradient norm of a class & becomes
ThE
cos²
fk sin?
fu
d
Cos
(K)
hg.in
(92)
fN
dha
Ft
heg ||
cos
fk sin?
Σ
fn
w
(k)
(93)
fk
lhg ||
(cos² + sin" 0).
(94)
Compared with the upper bound of the gradient norm of the unweighted cosine loss
nle
d Cos(w) (K)
(95)
dha,iz
71g
fe
Σ
(8)
(96)
R;
2
V Su cos 0 + f2 sin
(97)
Our weighted loss gives the linearity between the upper bound fx and the gradient norm.
Figure presents the NC phenomenon under objective P with without regularization on the feature norm.
Figure shows NCMC for different architectures, including DenseNet, ResNet, VGG, and LeNet.
Table 4: Loss P vs Cosine Regression Loss on CIFAR-100
0,005
0.01
0.02
P w/o mixup
4..
43.4 0.3
43.5=:0.2
P W mixup
36.5+:0.6
40.1
49.0+10.1
CAL
46.3-10.3
50.1
54.0+10.2
Table
presents long-tail classification results on SVHN and STL-10, complementary Table 2
Table shows how the accuracy of imbalanced learning is changed by the parameters f, and on CIFAR- 100.
F.2
Selection of Hyperparameterss
When picking fa according to the class-aware strategy, we hope fm ss fin; for L, m E [K]. This condition, combined with the equality fK = Ex fung refuses small f when there is a heavy class imbalance. For example, when the K classes 1, m E [K], and thus f& ~ f by the equality; on the other hand, for extreme imbalance, say the ratio T v 5π with large constant,
balanced, the strategy requires f.. s for
have fK fK 2 bK, thus /> b. As f increases from 10 to 20, imbalanced ratio 0.005 (the heaviest ratio in the imbalanced settings), and = 0.2, we achieve 80.6 0.5 on cifar10. and 46.6 + 0.4, both better than the reported Table 2.
Moreover, we suggest small E (0, r/4) in practice (e.g., cos 0 2 since large 0 generates concentrated centers of the classes, which is more likely to overweight and overfit the minor classes (consider the case of cos 1 where the centers collapse one direction, then the gradient directions will be severely skewed).
25

---

0.005
0.005+reg
0.01
0.00
mean
0.0
meen
0.00
mean
120
-0.2-
-0.25
-0.4
-0.6
MI
-0.75
-1000
-.25
-1.0
-1.25
-1500
-12-
-1.75
-1.4
-2.00
-1.6
-1.75
50
100
150
200
50
100
150
200
100
150
200
0.01+reg
8.0
mean
0.00
mean
0.00
mean
std
std
std
-025
-0.25
-044
-06
-08
-1.00
-10
-1.25
-1.50
-14
-1.75
-1.75
50
100
150
200
100
150
200
150
200
0.1+rre
80
mean
mean
mean
-02
0.6
-2
-5
100
350
150 200
Figure 4: NCMC phenomenon under Loss P different epochs. We draw the mean and standard deviation of the neural collapse metric used in the paper CIFAR-10 for different imbalanced ratios w/ or w/o regularization. The horizontal axis 1o NC metric value. The regularization coefficient is 5e-4.
82-
102
-33
-0.
-3.0
-3.0
-15
-1.
-375
-2.
-1o-
-29
-25
-2.
-50
-35
-35
Z00 200
120 150
200
50
100
150
300
Figure 5: NCMC phenomenon Different Backbones at different epochs. We draw the mean and standard deviation of the neural collapse metric used in the paper for four different backbones that trained on CifarI0-IT with - = 0.005: 550, ResNet50, LeNet, and, VGG1T, The horizontal axis is logio-scales NC metric value.
26

---

Table 5: Long-tailed classification accuracy (%) with ResNet and DenseNet on SVHN and STL-10.
Methods
SVHN 0.01
STL-10 0.01
0.005
0.02
0.005
0.02
ResNet
CE
39.410.2 40.610.2
46.4.0.2
40.1
35.0-0.3
36.310.3
SETF
41.3-10.2 45.4.0.11
49.6
374404
38.2-0.7 42.0 40.3
CAL
43.5+-011
47.4+0.2
49.1
40.5+02
42.8+0.4
45.4+0.3
DenseNer
CE
38.9+0.4 40.8+0.4
3
41.2 +0.3 44.9+0.2
SETF
40.5.10.1 44.8
48.4-02
39.510.3
42.9-0.3 46.3-10.2
CAL
45.410.1 46.610.11 48.810.2
42:10.3
43.3=0.4 47.4=1.11
Table 6: ResNet50's accuracy changes with the parameter tuple (f.0) CIFARIOO,5 0.005
(f.0)
0
0.2
0.4
0.6
0.8
T/2
1.8
41.8
42.2
43.1
41.8
41.6
44.0
1.1
45.2
45
44.4
41.6
43.2
43.7
10
1.3
45.7
46.1
44.4
43.5
42.6
43.0
13
2.3
46.0
46.2
44.5
42.7
41.8
41.5
16
0.9
46
44.9
42.7
41.1
39.6
38.6
20
1.6
47.0
44.7
41.3
39.1
38.2
37.9
G Heatmaps of the Neural Collapse
Figure&shows shows the inner product of normalized class-mean features during training. We observe that (a) records the contracted mean features at initialization; then the training separates the class means gradually be orthogonal.
We also capture the feature collapse of the most minor class (class size=25) for CIFAR-10 dataset in It is interesting note that the collapse occurs almost at the beginning.
Fig 7
H Impact and Limitations
Impact of our work. Our topic Imbalanced Classification which is a general concern in machine learing.
2. We consider classification rule that work better than regular classification rule under certain theoretical assumptions in the imbalanced setting. The analysis of "hard-to-predict feature" novel and can be further developed in general machine learning theory.
We have studied imbalanced learning through the lens of Neural Collapse, which provides insight into the connection among the optimal fcature-classifier alignment, the classification rule, and the performance of DNN.
4. We proposed loss and the strategy for fixed classifier that has comparable performance to methods with learnable classifier; the loss and strategy can apply to general machine learning models.
Limitations.
1. The theoretical analysis of our motivation only considers Gaussian case;
2. Due the computationally expensive optimization the Stiefel manifold for large dimension fK, the UFM analysis on learnable classifier in Theorem 3.3 and 3.4 are not justified in practical networks;
Although the proposed loss applicable to general classification models, we only conduct experiments two architectures, ResNet and DenseNet;
The proposed method for imbalanced learning does not have impressive performance on high- dimensional datasets (large number of classes, large number of features), one of the possible reasons the use of global expansion factor to expand the backbone feature dimension imposes negative effect on the representation of an deep architecture;
27

---

0.25
020
6C
A39
330
D00
RGL
0:0
6:4
027
4:2
340
GCD
SUN
0:00
SKN
603
AM
0.90
840
R13
0:30
C:0
GSP
920
BGC
60
4:30
013
4m
44
4.00
900
820
023
4:30
48
C0B
D81
ECE
6E
Figure 6: The heatmap of Meat-fe Separateness.
Figure The Within Class Feature Collapse.
5. Although the class-aware strategy is novel idea, the implementation is far from optimal. The example given in section 3.5 shows fin, 4, f2 = 12, and fanz = 3 which will result in an emphasis on the middle class instead of the minor class. How to create a strategy that is more "class-aware" will be our future work.
6. We compare our work with RBL (42 in Table 7 It is remarkable that the post-hoc logit adjustment significantly improves RBL. Our method outperforms RBL in two settings for cifar We conjecture that in setting of a small imbalanced ratio and large number of classes, the hard- to-predict distribution may dominate the performance. The reason that our method CAL has lower accuracy compared to RBL in other settings, is three-fold: 1. Imbalanced learning with MSE loss less effective than CE loss general: 2. Our experiment setting is chosen as close as possible to that where the theoretical analysis (proposition B.1 and C.I) is conducted, for example, to match the isotropy of the Gaussian, we normalize/batch normalize the feature ensure it is unit norm and centered before the classification, and the weight are unit vectors through training. This setting
28

---

possibly harms the learnability and flexibility of our model. 3. Under the class- MSE loss, we use the original classifier be the surrogate classification rule of our general classification rule. However, the rule is designed especially for "hard-to- tot--t--peedtt unseen data and thus is not necessarily optimal for the classification of other unseen data. When the hard-to-predict unseen data takes very small portion of the population, our design may lose its effectiveness. The success of RBL and PLA inspires us to find an optimal classifier for the loss P and CAL.
Table CAL vs RBL in long-tail classification trained on ResNet50. 20 and 0 0.2 are fixed. The values without + are that we did not reproduce. "-" represents a missing value.
Cifar- 10
Cif
Methods
0.005
0.01
0.02
0.1
0.005
0.01
0.02
0.1
RBL w/ PLA
73.6
0.3
84.3
90.7
RBL
81.8+0.55
84.9=0.3
92.5+ 87.6+0.2
41.7+004
51.7+0.2
68.4+0.1 54.310.4 65.910.3
52,440.2
CAL
80.0.10.5
84.110.3
85.9 10.2
92.0.0.3
46.5.0.55
50.110.3
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to sectior section 4.4 and the appendix
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
Refer to Theorem 3.3 and Theorem 3.4 and appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: appendix and supplementary material
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
See and the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiiicgguie//eeeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to section appapndii
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Table Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////uipp/eeeeeeeee httppppcccppppp..t.tccccccccleea
Answer: [Yes]
Justification:
The research conform with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss impact in appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
33

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use the datasets and refer the papers, see section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
34

---

Answer: [Yes]
Justification: We have describe the experiment details in section 4.4 appendix F and the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "HZ5t7oFrZZ", "reviewer": "Reviewer_FHuh", "summary": "This paper studies the Neural Collapse phenomenon under the imbalanced training data. The authors extend the optimal structure of neural collapse classification to a multiple center setting to enhance the model performance. Specifically, the authors propose to leverage the Generalized Classification Rule to make the minor classes align with more directions. Moreover, a practical MSE-type objective function has been proposed to train a \"neural collapse to multiple centers\" model. The proposed method achieves comparable performance with existing baselines.", "questions": "Please refer to the weakness section.", "limitations": "The authors have discussed the limitation of the proposed method, including the assumption made in the theoretical analysis, the computational cost and the not impressive performance on high-dimensional datasets.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors have conducted a thorough theoretical and experimental analysis on a list of SOTA neural collapse-related methods.", "weaknesses": "- The performance improvement of the proposed CAL in Table 4 is marginal in comparison to other baselines. \n\n- The writing of the paper can be further improved. \n  - The challenge and motivation of this work hasn't been well addressed in the introduction. Content in line 43-63 is more likely to appear in related work rather than introduction. \n  - It is hard for readers to understand the meaning of a sentence when too many abbreviated words come together, e.g., at line 73-77.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n$\\textbf{Response to weakness 1}$:\n\nAs far as we see, the proposed CAL has marginal improvement on cifar10 compared to ARBloss, but clearly outperforms other classical methods in comparison; it also has non-negligible improvements on cifar100. The experiment shows that CAL is effective for the fixed classifiers under MSE loss, which matches our theoretical analysis from the perspective of Neural Collapse. \nThere are quite a few SOTA results in imbalanced learning that outperforms our method, but our primary contributions are more than the performance improvement: \n\n(i) We formalize a novel principle we call “generalized classification rule” that focuses on the hard-to-predict subpopulation of the underlying data distribution, which is a different perspective compared to the popular margin theory and minority collapse. This rule claims that more directions are needed for minor classes in the imbalanced classification. The analysis is inspired by the recently discovered neural collapse phenomenon that for a highly expressive neural network, the classifier and penultimate layer feature has symmetric structures. \n\n(ii) We design the loss that leads the model to a state where the analysis in (i) is effective. In particular, since directly using GCR rule results in gradient vanishing and quick overfitting, we propose to study the surrogate objective P. We perform a theoretical analysis based on unconstrained features model to show that P leads to a neural collapse where the original classifier can be considered a representative of the multiple directions. In practice, we use Cosine Regression loss with a fixed classifier for data training. In contrast to P, Cosine regression loss has similar neural collapse types according to our theory (corollaries for fixed W), but is simpler and easier to train.\n\n(iv) The theoretical analysis is justified. The baseline SETF has nearly the same setting except that we use multiple directions for a class and normalize the features. Our method outperform SETF under most settings, which indicate that the information from multiple directions is useful for the imbalanced classification. This paper demonstrates the power of neural collapse analysis: through Unconstrained feature model we are able to design specific convergence pattern of the classifier and feature.\n\nIn short, our work can provide several insights in imbalanced learning and general model design, and we hope the reviewer can re-evaluate our contributions.\n\n$\\textbf{Response to weakness 2}$:\n\na) Indeed, the paragraphs in line 43-63 refer to several closely related works to our paper, but they are also the inspiration of our work and serve as the intro to the presentation of our motivation and contributions in the face of the issue that NC does not offer enough information of the underlying data distribution. Since our narrative is not effective according to the reviewer, we will highlight our challenge and motivation more concisely in the revision and offer more discussions about these references in “Related Work” section. \n\nb) We are sorry that there are 4 abbreviated words between line 73 and line 77: UFM (unconstrained feature model), NCMC (Neural Collapse to Multiple Centers), RCR (Regular Classification Rule) and GCR (Generalized Classification Rule). We will reduce the frequency of abbreviate names per sentence in the revision. The whole paper will be carefully polished for high readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6s71YdO7rk", "reviewer": "Reviewer_HFbj", "summary": "This paper studies the Neural Collapse (NC) phenomenon in imbalanced learning. Specifically, the authors find that the minor classes should align with more directions to achieve better classification results. Such finding yields the Generalized Classification Rule (GCR). The authors study NC under UFM. They find that the features of a class $k$ tend to collapse to the mean of centers of class $k$ which is termed Neural Collapse to Multiple Centers (NCMC), and RCR (the original classifier) approximates GCR at NCMC. Based on the above studies, the authors propose Cosine Loss and show from experiments that Cosine Loss can induce NCMC and has comparable performance to classical long-tail learning methods.", "questions": "One question: why study under UFM? Is this generalizable?", "limitations": "See *Weaknesses*.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall, this paper is well-written and quite novel. Here is a detailed assessment:\n\n1. **Originality**: The originality of the paper is commendable. This paper proposes a new classification rule named Generalized Classification Rule (GCR) for imbalanced learning, introduces Neural Collapse to Multiple Centers (NCMC) within UFM framework, and shows that the traditional RCR classification rule resembles GCR at NCMC. Based on such theoretical findings, this paper introduces a new type of loss termed Cosine Loss. Extensive studies show the effectiveness of Cosine Loss.\n2. **Quality**: This paper is in good quality. The theoretical study is rigorous and quite convincing. Backed by the theoretical study, the effectiveness of proposed Cosine Loss has been verified by extensive experiments, providing strong evidence for their conclusions.\n3. **Clarity**: The writing is clear and concise. From GCR to NCMC, then the resemblance of RCR to GCR at NCMC, finally the Cosine Loss and extensive experimental verification, the paper is well-organized and quite natural.\n4. **Significance**: This paper proposes a new phenomenon called NCMC which deepens the understanding of Neural Collapse especially under imbalanced settings. Also, the proposed Cosine Loss is an effective long-tail learning method.", "weaknesses": "This paper is generally well-written without much weaknesses. Here are a few possible points.\n\n1. The introduction of $\\widetilde{\\boldsymbol{w}}_j^{(k)}$ is a bit abrupt at line 121-124. I would suggest more explanations including in the corresponding appendix section. Also, the tilde symbol is missing in Eq.(8).\n2. Some minor typos. Line 95 “denote the (mean?) of features of class $k$…”. Line 96 “$\\boldsymbol{h}_k:==$” double =’s. Line 104 “$\\{\\boldsymbol{h})\\}$” redundant ‘)’. End of line 138 “satisfies” to “satisfy”. Line 197 “approximates” to “approximate”.\n3. Citation recommendations. I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study. Specifically, i recommend citing the following articles:\n  -  A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning (NeurIPS 23)\n  - Understanding imbalanced semantic segmentation through neural collapse (CVPR 23)\n  - Deep long-tailed learning: A survey (TPAMI 23)\n  - Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition (ICML 24)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our contribution.\n\n$\\textbf{Response to the weakness}$:\n\n1. thank you for your advice. We think it is a good idea to move proposition B.1 to the Main Result section and add more explanations of $\\tilde{w}_j^{(k)}$. In particular, the set of $\\tilde{w}_j^{(k)}$’s form an orthogonal frame (vectors are mutually orthogonal and have unit norm). \n\n2. you are right about the typos, we will check over the formulas and words to guarantee the readability of this paper.\n\n3. Thank you for the recommendations. The citations are recent literature on imbalanced learning and neural collapse. They are closely related to our work, and we will discuss their methods in the introduction and related work.\n\n$\\textbf{Response to Question 1}$:\n\nWe study UFM for the following two reasons: \n\n1. Deep models are generally hard to analyze due to their intricacy and diversity. UFM offers an intuitive explanation of what representation has been learned and how the classification is carried out. \n\n2. Because of the simplicity of UFM, it can be used to help design special representation structures and decision rules of interest, and the conclusions based on UFM are easy to verify.\n\nThe underlying assumption of the effectiveness of UFM is the high expressivity of the deep neural networks. The conclusions such as the optimal structure of the classifier and the large margin analysis at the terminal phase of training based on UFM can be generalizable when the model is overparameterized and easily interpolates the data.\n\nDespite the power of UFM, it is no more than an expedient approach. Completely understanding the neural collapse and its connection to other deep learning phenomenons such as benign overfitting and grokking demands deeper theories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48jW2vY9dQ", "reviewer": "Reviewer_vNc6", "summary": "This paper addresses the issue of minority collapse in imbalanced learning, finding an optimal structure to represent a better classification rule. The authors induce a new definition called NCMC and design an MSE-type loss to alleviate the minority collapse phenomenon.", "questions": "- Could the authors provide more visualizations of multiple centers?\n\n- In the experiments, what is $\\tau$ and $\\theta$ refer to? How is the degree of imbalance discussed?\n\n- Is the conclusion only applicable to MSE loss, or could it also apply to CE loss or other loss functions designed to handle imbalanced data?\n\n- Is the NCMC phenomenon, like NC, model-agnostic? Would different backbones affect the conclusion?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is well-written and has a clear logic. It discusses an interesting phenomenon where features in the minor class contribute to mitigating the minority collapse, providing a novel perspective.\n\nThe authors also designed a practical loss function to induce NCMC and improve generalization.", "weaknesses": "The experimental results did not show significant improvement; the accuracies of these loss functions are quite similar. For example, in Table 4's CIFAR-100 experiments, the results might all be within the margin of error, suggesting that the newly proposed loss function may not be very effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Response to weakness 1}$:\n\nThe main purpose of this paper is to assign a novel classification rule to the imbalanced classification. The classification rule focuses on the hard-to-predict subpopulation, which is different from the popular margin theory.\n\n In order to justify the usefulness of our theoretical result of the generalized classification rule, we compare the proposed class-aware MSE loss to several baselines. The most useful baseline is SETF, which has nearly the same settings as ours: SETF and CAL both consider fixed classifiers, use only the within-class alignment part of the MSE loss function, and apply the same training schedule. They are comprehensively compared on (four datasets $\\times$ three imbalanced ratios $\\times$ two backbone networks) (24 settings in total). CAL outperforms SETF in most settings, indicating that assigning more directions for minor classes is a useful strategy aiming more than alleviating minority collapse, and the generalized classification rule is effective. In particular, under either loss, all samples from a class collapse to only one vector, so CAL and SETF should have the same capability of reducing the minority collapse. The significant difference comes primarily from the number of directions: In the supervised training, SETF only has one available direction (the classifier vector) for the feature to align while CAL uses information from multiple directions (the centers of each class). \n\nWe further examine the different choices of parameter $f$ and $\\theta$; it is worth noting when $\\theta = \\frac{\\pi}{2}$, i.e. the centers are all identical to their corresponding classifier vector, and the performance is no better than SETF. This observation shows reweighting alone does not improve the classification, thus our method is different from reweighting methods. \n\nThere are comparisons to other classical methods, the comparable performance (or the marginal improvement) implies fixing classifier has the potential to be developed further and applied in practice.\n\n$\\textbf{Response to question 1}$:\nThe 3D illustration (Fig.1) of the multi-center frame is presented in the attached pdf. Imagine that all solid-line vectors are orthonormal vectors, among which the red and yellow ones are the $w_1$ and $w_{2}$ in the definition 3.2 which corresponds to class 1 and class 2. The solid green and solid blue are the augmented vectors associated with $w_1$ and $w_2$ resp. Then the green dashed-line vectors are the centers of class 1 and the blue ones are the centers of class 2. Since the classes has more than one center in general, we call them multiple centers.\n\n$\\textbf{Response to question 2}$:\nWe apologize that $\\tau$ is not claimed in the main article before being used. We consider long-tail image data which is generated from imbalance sampling from the data including cifar10, cifar100, SVHN, and STL10. $\\tau:= \\frac{n_{min}}{n_{max}} \\leq 1$ is the imbalance ratio that describes the exponential decay of class sample size $n_k$, more precisely, let $n_1>n_2>\\ldots>n_K$, then $n_k = n_{k-1}\\tau^{\\frac{1}{K-1}}$; $\\theta$ is defined in definition 3.2, which is the angle between $\\mathbf{w}_{k}$ and its associate centers $\\mathbf{w_j^{(k)}}$. Please see more discussions of the experiment settings in Appendix F and the 3D illustration in the attached pdf.\n\n$\\textbf{Reponse to question 3}$:\nThe logic here is, that the paper finds that a model needs more directions for minor classes (General Classification Rule, GCR), and then proposes loss P and Cosine Loss that lead to a surrogate of the GCR. In particular, we design the MSE-type loss to fit the purpose. It is highly possible to find an analogous conclusion under CE loss or other popular loss [1] if we can figure out the global optimal solution to the corresponding UFM under such losses. Our work demonstrates the effectiveness of the UFM analysis and the methodology can be useful to the model design: given the specific classification rule or other purposes, the joint structure of classifier and feature can be designed.\n\n$\\textbf{Response to question 4:}$\n\nThe Discovery of NC phenomenon [2] and the Classic analysis of UFM [3] tell us that NC is model-agonistic conditional on the high expressivity of the deep neural networks. What we know is NC will not happen if the network does not interpolate the data. We show NCMC occurs through UFM analysis, and it can be verified on loss P and the Cosine Loss. Fig.2 in the attached pdf shows the NC on different backbones. It is observed that ResNet, Densenet, and VGG have more severe collapse than LeNet, a relatively small model that hardly interpolates cifar10 in 200 epochs. However, formulating a complete theoretical justification of the occurrence of either NC or NCMC during SGD training is a hard yet ongoing problem in deep learning theory.\n\n[1] Zhou et al. Are All Losses Created Equal: A Neural Collapse Perspective. NeurIPS 2022.\n\n[2] Papyan et al.. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, volume 117, 2020.\n\n[3] Zhu et al.. A geometric analysis of neural collapse with unconstrained features. NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtgNqeIeSk", "reviewer": "Reviewer_sPeJ", "summary": "This paper explores Neural Collapse (NC) in the context of imbalanced data, proposing the concept of Neural Collapse to Multiple Centers (NCMC). It establishes that aligning features from minor classes with more directions improves classification accuracy, introducing the Generalized Classification Rule (GCR). The authors design an MSE-type objective and a practical loss function that induces NCMC, achieving performance comparable to classical imbalanced learning methods. Their findings are validated both theoretically and experimentally, offering new insights into the application of NC in imbalanced learning scenarios.", "questions": "see Weaknesses.", "limitations": "see Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ The paper is clearly written and easy to follow. Complex concepts are systematically explained, and the logical flow is maintained throughout the document, making it accessible to readers with varying levels of expertise in the field.\n+ The findings have significant implications for imbalanced learning, offering a new approach that achieves performance comparable to classical methods, thereby advancing the field of deep learning and classification.", "weaknesses": "+ Figure 1 should be larger for better visualization for readers.\n+ Theorems 3.3 and 3.4 prove that the optimization problem $\\mathbf{P}$ can induce the \"Neural Collapse to Multiple Centers\" solution, but the loss function in $\\mathbf{P}$ does not appear in the experiments. It might be better to include the loss function $\\mathbf{P}$ in the comparison experiments and compare, analyze, and explain the results. \n+ If multiple centers are better than a single center for each class in hard-to-predict feature distribution, why don't the authors directly use the loss function in $\\mathbf{P}$ instead of designing a Cosine Regression Loss? The authors should explain this intent.\n+ [1], as a study about long-tailed learning and NC, should be compared within this study. \n+ I personally think that Proposition 3.1 deserves more space and discussion in the main text, as it serves as a key motivational component for the paper. Feel free to adopt this suggestion; this point does not affect my rating. \n\n[1] Feature Directions Matter: Long-Tailed Learning via Rotated Balanced Representation. Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang, Huiyang Shao, Qingming Huang Proceedings of the 40th International Conference on Machine Learning, PMLR 202:27542-27563, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our contribution to the field of deep learning and classification. To convey the meaning of our work more fluently, we answer the questions in the following order.  \n\n$\\textbf{Response to weaknesses 1 and 5 }:$\n\nWe will enlarge the Fig.1 for a better illustration.\n\nWe would like to move Proposition B.1 to the Main Result section (section 3) and introduce the notion of $\\tilde{w}_j^{(k)}$ more carefully. \n\n$\\textbf{Response to weakness 3}:$\n\nWe are sorry that our narrative creates a misunderstanding here. \nIn the analysis of the classification rule we are assuming the classifiers and features are well-trained, the optimal symmetric final structure can be seen as the guarantee of the isotropy of the hard-to-predict populations. However, when we use GCR to supervise the model, the training will quickly encounter a gradient vanishing. Indeed, at initialization let $s = argmax_j\\lbrace \\langle w_j^{(k)},h_{k,i}\\rangle\\rbrace$, the gradient descent will take $h_{k,i}$ to the direction closer to the specific center $w_s^{(k)}$. Then during training $h_{k,i}$ approaches $w_s^{(k)}$ even faster since $s = argmax_j\\lbrace \\langle w_j^{(k)},h_{k,i}\\rangle\\rbrace$ from the beginning. So the model ends up with a structure far from optimal; each class will have several clusters scattered randomly nearly orthogonal to each other. In this case, the model overfits the rule easily. This is the drawback of directly using GCR as a practical objective and the reason we try to use the average of centers instead of the maximum of them.\n\nBoth P and Cosine loss are approximations to the Generalized Classification Rule (GCR) while cosine loss is a concise version of P; Both of them use multiple centers, please recall that a center is a vector associated with a certain classifier. The difference between P and Cosine loss is: P uses all centers available while Cosine loss uses centers of one class (the class to which the feature belongs). Unlike P which uses both within-class alignment and between-class separateness (refer to line155 in the main article), Cosine loss uses the within-class alignment only which has faster NC and an advantage in Mixup training (refer to the empirical results in the next Response). \n\n$\\textbf{Response to weakness 2}:$\n\nThank you for your question. We did not pay enough attention to the empirical results w.r.t the loss P, since our major concern is the final state of the neural collapse and cosine loss is obviously simpler than P. We now provide a few experimental results w.r.t P, where the classifier is fixed. The table below shows the performance of P on cifar100 with different imbalance ratios. \n\n|$\\tau$|0.005|0.01|0.02|\n|--------|--------|--------|--------|\n|P w/o mixup|  41.9$\\pm$0.2|  43.4$\\pm$0.3  |  43.5$\\pm$0.2|\n|P w/ mixup|  36.5$\\pm$0.6  |  40.1$\\pm$0.4  |49.0$\\pm$0.1|\n|CAL mixup|$\\textbf{46.5}$$\\pm$0.5| $\\textbf{50.1}$$\\pm$0.3| $\\textbf{54.3}$$\\pm$0.4|\n\nThe performance of P is lower than CAL, and P starts to be incompatible with Mixup training when the ratio decreases (to 0.01). One possible explanation is that the Cosine loss do not have distraction terms from the centers of other classes, while P needs to handle the between-class separateness, which can become a complicated procedure when there are a myriad of centers presented.\n \n\n\n$\\textbf{Response to weakness 4}:$\n\nWe add the comparison to the method Rotated Balanced Representation (RBL) of [1] to our paper, and the literature will be discussed in the introduction or related work. It can be observed in the following table (the first four columns are cifar10 test accuracy and the last four are cifar100) that the learnable orthogonal layer is effective when the post-hoc logit adjustment (PLA) is applied (the ablation shows how powerful the post-hoc logit adjustment is). \"-\" indicates no results recorded (due to time limit). \n\n|$\\tau$|0.005|0.01|0.02| 0.1 |0.005|0.01|0.02| 0.1 |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|\n|RBL w/o PLA|73.6|78.5|84.3|90.7|-|-|-|-|\n|RBL w/ PLA|$\\textbf{81.8}$$\\pm$0.5|$\\textbf{84.9}$$\\pm$0.3|$\\textbf{87.6}$$\\pm$0.2|$\\textbf{92.5}$$\\pm$0.3|41.7$\\pm$0.4|$\\textbf{51.7}$$\\pm$0.2|52.4$\\pm$0.2|$\\textbf{68.4}$$\\pm$0.1|\n|CAL|80.0$\\pm$0.5| 84.1$\\pm$0.3| 85.9 $\\pm$0.2| 92.0$\\pm$0.3| $\\textbf{46.5}$$\\pm$0.5| 50.1$\\pm$0.3| $\\textbf{54.3}$$\\pm$0.4| 65.9$\\pm$0.3|\n\nOur method outperforms RBL in two settings for cifar100. We conjecture that in a setting of a small imbalanced ratio and a large number of classes, the hard-to-predict distribution may dominate the performance. The reason that our method CAL has lower accuracy compared to the method of [1] in other settings, is three-fold: 1. Imbalanced learning with MSE loss is less effective than CE loss in general; 2. Our experiment setting is chosen as close as possible to that where the theoretical analysis (proposition B.1 and C.1) is conducted, for example, to match the isotropy of the Gaussian, we normalize/batch normalize the feature to ensure it is unit norm and centered before the classification, and the weight are unit vectors through training. This setting possibly harms the learnability and flexibility of our model. 3. Under the class-aware MSE loss, we use the original classifier to be the surrogate classification rule of our general classification rule. However, the rule is designed especially for “hard-to-predict” unseen data and thus is not necessarily optimal for the classification of other unseen data. When the hard-to-predict unseen data takes a very small portion of the population, our design may lose its effectiveness. The success of RBL and PLA inspires us to find an optimal classifier for the loss P.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to the reviewers for their patience and time.

According to the questions from the reviewers, we add a few experiments w.r.t loss P and the generalized classification rule. The attached contains three figures:

$\textbf{Figure 1}$: The 3D illustration of multi-center frame;

$\textbf{Figure 2}$: NCMC phenomenon in Different Backbones. We draw the mean and standard deviation of the neural collapse metric used in the paper for four different backbones that are trained on Cifar10-LT;

$\textbf{Figure 3}$: The NC Analysis of Loss P on different $\tau$ w/ or w/o regularization on the feature norm

Two of the four reviewers recognize our work as excellent, and the rest believe that the weakness of this paper is the marginal performance improvement to some baseline in our experiments. Although performance is a fairly important dimension of our work, we still hope that the other contributions including the theoretical novelty can be correctly evaluated. Let me restate here the contributions of this paper, which conform to the presentation in the main article.

(1) We formalize a novel principle we call the “generalized classification rule” that focuses on the hard-to-predict subpopulation of the underlying data distribution, which is a different perspective compared to the popular margin theory and minority collapse. 

(2) We design the loss P, and perform a theoretical analysis based on unconstrained features model (UFM) to show that P and its variants (including the Cosine Regression Loss) lead to a neural collapse where the original classifier can be considered a representative of the multiple directions.

(3) The effectiveness of our loss function for fixed classifier can be justified by the empirical results.

Meta-review: This paper studies the Neural Collapse (NC) phenomenon in imbalanced learning and finds that minor classes should align with more directions for improved classification results.

Initially, this paper received mixed scores, but the authors effectively addressed these concerns in their rebuttal. Consequently, I recommend accepting the paper. Nonetheless, the authors are required to follow the reviewers' suggestions and integrate their rebuttal content into the original paper.
Decision: Accept (poster)