Do We Need Neural Collapse? Learning Diverse Features for Fine-grained and Long-tail Classification
Authors: Jiawei Ma
Chong You
Sashank J. Reddi
Sadeep Jayasumana
Himanshu Jain
Felix Yu
Shih-Fu Chang
Sanjiv Kumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Collapse
Diverse deep learning features
Finegrained transfer learning

Abstract: Feature extractors learned from supervised training of deep neural networks have demonstrated superior performance over handcrafted ones. Recently, it is shown that such learned features have a neural collapse property, where within-class features collapse to the class mean and different class means are maximally separated. This paper examines the neural collapse property in the context of fine-grained classification tasks, where a feature extractor pretrained from a classification task with coarse labels is used for generating features for a downstream classification task with fine-grained labels. We argue that the within-class feature collapse is an undesirable property for fine-grained classification. Hence, we introduce a geometric arrangement of features called the maximal-separating-cone, where within-class features lie in a cone of nontrivial radius instead of collapsing to the class mean, and cones of different classes are maximally separated. We present a technique based on classifier weight and training loss design to produce such an arrangement. Experimentally we demonstrate an improved fine-grained classification performance with a feature extractor pretrained by our method. Moreover, our technique also provides benefits for the classification on data with long-tail distribution over classes. Our work may motivate future efforts on the design of better geometric arrangements of deep features.

Under review as a conference paper at ICLR 2023
Do WE NEED NEURAL COLLAPSE? LEARNING DI- VERSE FEATURES FOR FINE-GRAINED AND LONG- TAIL CLASSIFICATION
Anonymous authors
Paper under double-bling blind review
ABSTRACT
Feature extractors learned from supervised training deep neural networks have demonstrated superior performance over handerafted ones. Recently, it is shown that such learned features have a neural collupse property, where within-class fea- tures collapse the class mean and different class means maximally separated. This paper examines the neural collapse property in the context of fine- fine-grained classification tasks, where feature extractor pretrained from classification task with coarse labels used for generating features for a downstream classifica- tion task with fine- fineinrained labels. We argue that the within-class feature collapse is an undesirable property for fine-grained classification. Hence, we introduce a geometric arrangement of features called the texximal-epprrating-oo cone,, where within-class features lie in cone of nontrivial radius instead of collapsing to the class mean, and cones of different classes are maximally separated. We present a technique based on classifier weight and training loss design to produce such an arrangement. Experimentally we demonstrate an improved fine-grained classifi- cation performance with feature extractor pretrained by our method. Moreover, our technique also provides benefits for classification on data with long-tail distri- bution over classes. Our work may motivate future efforts the design of better geometric arrangement of deep features.
INTRODUCTION
The extraction of features representations from image, language, and speech data in their raw forms is a problem of fundamental interest in machine learning. A standard approach in classical machine learning is handeraft feature extractor that maps from the input to the feature space (Lowe, 2004), which often requires meticulous and onerous work of human and domain experts. With the success of deep learning, methods based on learned feature extractors have become very popular. Such an approach not only requires less human expertise, but also exhibits better empirical performance compared handerafted feature extractors (Krizhevsky et al., 2017). Moreover, deep features extractors obtained from pre- pre-training task can be used for downstream tasks by simply training a task-specific classifier, which offers good empirical performance (Yosinski al., 2014).
The great success of learned features naturally leads to the following question: Are learned features optimal for various application scenarios? In this paper, we show that learned features in a standard deep classification model are not optimal and can be improved by handcrafting better geometric arrangement of the features. We are motivated by a recent line of work that shows that the geometric arrangement of features in a standard deep classification model has simple and elegant form:
Within each class, the features are maximally concentrated and collapse to the class mean.
Across different classes, the features maximally separated. Namely, the distance between each pair of class means is the same and that distance is at the maximum.
This phenomenon called Neural Collapse (ACC) (Papyan et al., 2020). The occurrence and preva- lence of NC is verified empirically through experiments with variety of datasets and network architectures (Han et al., 2021). Motivated by such an observation, recent work provided theoreti- cal analysis of deep features under the so-called unconstrained feature models (Mixon et al., 2020;

---

Under review as a conference paper at ICLR 2023
CE
(b) HingeCE
85-
54
13
-2-
-1-
-04
31-
-28 -60 -02
06 04 36
-04 10 32 06
Figure 1: Fine-grained classification with NC vs. diverse features. We train ResNet32 CIFAR- 20, which contains 20 coarse classes, and use the features for classification on CIFAR-I00, which contains 100 fine-grained class labels of the 20 coarse classes. The plots provide a visualization of randomly sampled features from three selected coarse classes (enclosed in dashed circles), each with two selected fine-grained classes (shown with different colors). Our visualization uses the technique of Müller et al. (2019). (a) With cross-entrops (CE) loss, within-class features collapse and fine- grained classes are less distinguishable. (b) With our method (i.e., Hinge CE), within-clas features are diverse and fine-grained classes are more distinguishable.
Fang et al., 2021), where the features are considered to be free optimization variables for the reason that deep neural networks have sufficient expressive power to produce any features for any train- ing dataset. In particular, Lu & Steinerberger (2020); Weinan & Wojtowytsch (2020); Mixon et al. (2020); Graf et al. (2021); Fang et al. (2021); Ji et al. (2021); Tirer & Bruna (2022) show that NC solutions are global optimal solutions for the nonconvex objective functions. Moreover, Zhu et al. (2021); Zhou et al. (2022) show that such nonconvex problems have a benign global landscape hence NC can be easily obtained by standard training algorithms. The study of NC arguably brings us a better understanding of the properties of the deep features. Nonetheless, existing study does not provide an answer to the following question:
Is Neural Collapse the desired property for deep features?
Contribution. We provide study of whether NC is the desired property for deep features by considering a specific learning task. In particular, we consider the fine-grained classification task, where deep learning model first pretrained on a classification dataset with coarse class labels, and use the features extracted from such model for downstream classification task with fine fine-grained classes. For example, the pretraining task contains coarse class, say flower, that contains all variants of flowers, while the fine-grained task may give each variant of flower different class label. NCC occurs hence all features for the coarse class collapse to the class mean, then such features do not distinguish between different fine- fine-grained classes hence cannot be used to successfully perform fine-grained classification. This simple thought experiment demonstrates that NC not a desired property for fine-grained classification.
Motivated by the observation above, we argue that deep features extracted during pretraining should not have the NCC property for the task of fine- ane-grained classification. Instead, the features for each pretraining class should be diverse to a certain extent, so that variation within the class is preserved and can benefit fine-grained classification. To realize this idea, introduce geometric arrange- (MSC), where features of each class lie
ment of deep features called the maximaal-epprrratnn--oom inside cone with a nontrivial radius (i.e. instead of collapsing to class mean as in ACC), and the axes of the cones are maximally separated. See Figure 2 for an illustration. To obtain such an arrange- ment, present a technique based on the simple idea that, a feature lies inside its cone, then its loss is set to be constant by hinge function. Figure 1 provides a comparison of NC features and for fine- -raiine classification with CIFAR-20 and CIFAR- 100. The contribution of this work is summarized as follows.
our MSC features
We design MSC arrangement of deep features where within-class features are diverse within cone instead of collapsing to the class mean (as in ACC.
We present generic technique based on the design of classifier weight and training loss function to obtain MSC features.
We conduct experiments on fine- fine-graine grained classification to demonstrate the effectiveness of our method. We also provide ablation study to justify the design of each component in our method.
2

---

Under review as a conference paper at ICLR 2023
(a)
(b)
0g 9x
Figure 2: Conceptual illustration of features obtained from deep learning based classification mod- els. Each ball represents a feature vector for an input in the training set. (Left) Neural collapse features obtained by standard training ( Secciion 2.2), where within each class, all features collapse to the class mean. (Right) saximall-eaaratingccom cone features that our method obtains (Section 3.1), where within each class, the features lie in cone and the axes of the cones are maximally separated.
Aside from fine-grained classification, we demonstrate that our method also improves the perfor- mance for classification with training data that has imbalanced number of samples for different classes (i.e., long-tail data).
BACKGROUND
In this section, we start by reviewing the basics of deep learning based classification in Section 2.1, where a feature extractor parameterized by deep neural network and linear classifier on the fea- tures are jointly learned from training dataset. Then, Section 2.2 explains A'C which characterizes the features and classifier obtained from training deep learning based classification model.
2.1 DEEP LEARNING BASED CLASSIFICATION
Consider the classification task where the goal to predict the label y [ (1,..., K} for an input æ Ih" where K is the number of classes. With deep learning based classification, this is typically achieved by training a model with the following optimization objective:
N
min O.W. b N
Σ C(ee...... C(e v.k), s.t. u, f(z,, 0), Usk = (u, wk) + bk.
(1)
where {æ;, 2/ LN is a training dataset of input-label pairs. In above, f(: 0) R² - R the feature extractor implemented u, of the feature extractor is referred to as the fcature of Φ,. The other two optimization variables in (1), namely W [w....., wx] E RR and b = [b,..... bb] IR" are parameters of linear classifier. The vector [Uat.... Usk] is often referred to the logit vector for data æ;, The loss function L(.))) :) is defined the logits and the label. One of the most popular choice of L(.,)) is the cross-entropy (CE) loss, but other choices such as the mean squared error loss may also induce NC.
deep neural network with trainable parameters O. The output
2.2 NEURAL COLLAPSE (NC)
NC a phenomenon for the features obtained terminal phase of training a deep learning based classification model (Papyan et al., 2020). Let (0", W bb) "y*,b" that the learned features {u. f(x,, 0*) P have the following properties (see also Figure 2(a)).
be the solution to (1). Then, ACC states
NC1: Within-class Variability Collapse. For each class E [K], the features {ui | i y = k}
E.
collapse to the class mean μ
i AC2: Between-class Maximal Separation. Let μa ΣK - m be the global mean. Then, the class means after subtracting the global mean, i.c. {Mr - M k E [K]} are maximally distant and located a sphere centered origin. More precisely, they form simplex equiangular tight frame (Simplex ETF) up to a rotation and scaling:
[n
Po. Pk Mgl sUW ETT;
3

---

Under review as a conference paper at ICLR 2023
where 0 is arbitrary scaling and U E RRm* is an arbitrary rotation (i.e., orthonormal matrix) of the Simplex ETF. In above, W EIP is a matrix that defines a Simplex ETF:
W WEFF V 1
K
1 k -
[IK
(3)
where Ix E RRK*K is an identity matrix and 1 RK a vector of ones. can be verified that the angle between any pair of class means after subtracting the global mean is the same:
f PGg p - -Mae
(-x-1) vk, k K
arccos
1 Hella 1 Hal2 = 0xx- arccos (-
K 1K-
(4)
Aside from the learned features, NC also states the following two properties hold for the classifier.
(iii) NCC: Self-duality between Feature and Classifier. The classifier weight =*= [w .....
kk
satisfies w = C- (ui pg)), where is a scalar independent of
(iv) AC4: Nearest Class Mean Classifier. The class label for test data æ is determined by a nearest class mean classifier: arg minke:]] [] [ 0*) 1
Theoretical studies of the NC phenomenon are often conducted with the unconstrained feature mod- els (UFMs). The idea of UFMs is that, since performing an amalysis of a multi-layer nonlinear neural network f(.) is difficult, we treat f(.) as a model that can produce any set of features given any set of inputs. Hence, the features fu, ) can be treated free optimization variables, which allows us to peel off the network (() and consider the following optimization problem²
1 E(mn......
min (w )
u.] v.), s.t. ua = (u., ws) bb.
(5)
W N
Under certain assumptions, it can be shown that all global solutions to (5) satisfy NC and that such an optimization problem has benign global landscape (Zhu et al., 2021).
3 METHOD
In this section, we start by presenting an arrangement of features described by their geometric prop- erties in Section 3.1, where within-class features diverse within cone. Then, Section 3.2 intro- duces our method for obtaining such an arrangement by the design of classifier and loss function.
3.1 MAXIMAL-SFPARATIIN CONE (MSC) ARRANGEMENT
As explained in Section 2.2, a standard deep learning based classification produces features with an arrangement described by NCC, where within-class variability converges to zero (i.e., NC1) and between-class separation is maximized (ie.,.CC) (see Figure 2(a)). Here we introduce an arrange- ment of features where within-class features are allowed to be diverse. As briefly explained in the introduction, within-class diversity is a desirable property for fine-grained classification.
Definition MMaximal-SSpparatig--CCo Cone Arrangement), The set of features { S. IR" is said (MSC) arrangement with respect to (W.r..) the set of labels (y C [K] with angle T > if there exist a collection of vectors {B&] C I!" with HK forming Simplex ETF, such that for all i, k that satisfies y/ = k, it has
to have a maximal-separating
1K
{ui, 3x) || 2 cos T.
(6)
|uull2
The MSC arrangement has simple geometric description (see Figure 2(b)). In words, it requires that features associated with each class lies in a cone. Moreover, the cones have the properties that 1) their axes form Simplex ETF, hence different classes sufficiently separated, and that 2) their angular radius is 7 > 0, hence within-class features do not collapse to a single point as in NC.
We assume that the feature space dimension m is greater or equal to the number of classes K. A regularization term is often added the optimization variables. We omit to simplify the discussion.
4

---

Under review as a conference paper at ICLR 2023
3.2
INDUCING MSC FEATURES VIA CLASSIFIER AND LOSS DESIGN
We obtain MSC features by training a deep neural network with the following optimization problem:
N C (
min 0
Bi). s.t. u, f 0), V/R h(u,, Vis was by):
N
W Wi..... wk] W BTF, and [b_..................... .g] 0,
where h(-) is defined in (8) and W ETF is a Simplex ETF matrix defined in (3), The problem in (7) differs from (1) in two aspects, which we explain below.
Fixed Classifier. Instead of learning the classifier weights W and b from training data, we fix them as constant model parameters in (7). In particular, W is fixed as a Simplex ETF matrix W WETe The idea is that we would like the columns of W to be the axes of the cones in MSC arrangement, which are required to form a Simplex ETF according to Definition
For the parameter b, we simply fix to be zero vector, which makes the design of method for obtaining MSC features easier.
Hinge Loss. The second difference in (7) is that the logits are set as 0ak h(us, ye, WK, bja) with h()) being a hinge function defined below:
1/+ (u: Wa: ba) = min((wys, uz), (dop-grad(( (1,11111)]]))) (21l)))llxxx ll2 cos r+} } + bb, if k = Vi:
h (u, Ve, wk, bx) h (uq wn by) = max (wmz, uz), stop-grad (I [2)] wx|12 cos r } bx. if h: Vi:
(8)
where -t and J are two pareerrrr.meerss... I = 0 and r = we obtain h h(u,, Vi, WR. bn) (wa; u:) + b; which is the logit function used in (7). The idea of designing (8) is explained below.
For the logit ty corresponding to the positive class, i.c., when & = y/. we define eis = (wx, 4 which the typically used logit in (7), when the angle between wt and u, is greater than ++ Otherwise, we define the logit to be UiA = stop-grad([Iur (|u:l))) cos r with stop-grad() denotes an operation to stop the backpropagation of gradient; thereby making the logit independent of network parameters O.
For logits Usk corresponding to the negative classes, i.e., when k o yr. we define was = (wx, u4). which is the typically used logit in (7), if the angle between ww and u, is smaller than ; Otherwise, we define Usk stop-grad ([ cosT so that the logits becomes independent of network parameters 0.
By setting -+ r, the logit eu for the positive class is independent of network parameters 0 when u, lies in its corresponding cone in an MSC. Similarly, using the fact that the pairwise angular distance of vectors in a Simplex ETF with K vectors is the same and is given by 0x (see (4)), by setting -- = 0g T, the logits Usk for the negative classes are independent of @ when u, lies in its corresponding cone in an MSC. Combining the cases above, we readily obtain the following result.
Theorem 1. Consider (7) with -+ 0k - =. If the set of features {ua } satisfy the MSC arrangement w.r.l. {y: N with angle v, then the gradient of the objective w.r.l. 0 is zero.
From Theorem 1, if the features already have an MSC arrangement, then the network f( will no longer be updated and hence the features will not collapse to their corresponding class centers.
4 EXPERIMENTS
In this section, we demonstrate the empirical performance of our approach. In particular, we focus on two applications: (i) Fine-grainee classification, where the goal is illustrate the benefit of diverse features Iearned during pre-training for fine-grained downstream classification, and (ii) Long-tail classification, where we apply our approach to improve the performance on tail classes.
For our approach, we train feature extractors f() by solving the optimization problem in (7) with L being the CE loss and f() being different variants of ResNet (He et al., 2016) depending on the
"This design choice was previously made in Zhu al. (2021), with the idea that since NC occurs con- vergence, simply fixing W to be a a Simplex ETF should not hurt performance while may accelerate training. More broadly, Hoffer et al. (2018) also noted that the classifier need not be trained. 'Here we use the fact that |m ||2 1 by the constraint in (7).
5

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparison of transfer learning using linear evaluation on nine downstream image classification datasets. We report the Top-1 accuracy (%) where the first and the second high results are highlighted. Our approach, HingeCE, improves the transfer learning accuracy consis- tently and reaches the highest mean accuracy on several datasets. Symbols & here indicate the results reported Chen et al. (2020) and Kornblith et al. (2021) respectively.
Approach
Food
Cifar10
Cifar100
Birdsnap
SUN
Car
Aircraft
Pet
Flower
Mean
SimCLR
68.4
90.6
71.6
37.4
58.8
50.3
50.3
83.6
91.2
66.9
CE
74.6
92.4
76.9
55.4
62.0
60.3
60.7
92.0
94.0
74.3
Smooth*
72.7
91.6
75.2
53.6
61.6
54.8
58.9
92,9
91.9
72.6
CE+ETF
73.4
91.0
73.0
55.1
65.3
60.1
59.9
92.0
95.8
74.0
Smooth+ETF
70.5
90.6
71.4
54.6
64.7
58.8
59.0
90.2
94.3
72.7
HingeCE (Ours)
74.9
92.3
76.6
55.8
68.2
62.2
61.2
92.2
97.8
75.7
dataset and application. We refer to our method as HingeCE. Since the last two layers of (all variants of) ResNets is a ReLU activation followed by pooling, the extracted features are constrained to have nen-negative entries, hence cannot satisfy an MSC arrangement. Therefore, we add fully con- nected layer with the same input and output dimensions" on top of ResNets as the feature extractor f0, such that the output feature is theoretically defined over RR... The choice of typer-parameten Tt and T well as additional analysis, are discussed in Section
4.1 FINE-GRAINED CLASSIFICATION
Experimental Setup. For this setup, we follow Kornblith et al. (2021) and first pre-train a ResNet- 50 model (He et al., 2016) on ImageNet ILSVRC 2012 dataset (Russakovsky et al., 2015). The outputs of ResNet-50 are then used as features for the downstream tasks. More specifically, after pre-training, we fix the pre-trained feature extractor and only learn linear classifier for each of the nine different downstream natural image classification tasks through Le-regularized multinomial lo- gistic regression. The downstream datasets are Food (Bossard et al., 2014), CIFARIO & CIFARI00 (Krizhevsky et al., 2009), Birdsnap (Berg et al., 2014), Sun (Xiao et al., 2010), Car (Krause et al., 2013), Aireraft (Maji et al., 2013), Pet (Parkhi et al., 2012) and Flower (Nilsback & Zisserman, 2008). Most of these datasets concern recognition on fine fine-grained classes (see Appendix A for more details). The primary goal of this experiment is to demonstrate that diverse features in pre-training enables better downstream fine-grained classification.
Results. The results for fine-grained classification are summarized in Table We consider several baselines to evaluate our performance gain. We use CE to denote the baseline which jointly learns the classifier and the feature extractor from scratch by minimizing CE loss LCg during pre-training. Another natural baseline is CE+ETF, where the learnable classifier in CE is replaced with fixed Simplex ETF. To have a fair comparison with our HingeCE which also uses fixed Simplex ETF as the classifier but uses an additional linear layer, we also add such layer for CE+ETF. Hence, HingeCE and CE+ETF are the same in terms of architecture of feature extractor and using fixed Simplex ETF classifier, with the only difference being the choice of loss function for network training.
Additionally, following Kornblith et al. (2021), we use label-smoothed variants of the basclines above as additional baselines (e.g. Smooth+ETF). In particular, we modify the target distribution t as follows t' = (il-a)+a// where a determines the weighting of the original and uniform targets and K is the number of classes in Eq. 7. We set as 0.1 and mean linear evaluation accuracy drops from 44.3% to 92.6% (Row of Table 1) when a classifier is jointly learned and from 94.0% to 92.7% (RoW(1.5) of Table 1) when a Simplex ETF used. We first note that CE+ETF achieves sim- ilar or competitive accuracy in linear evaluation as well as the accuracy on ImageNet (pre-training) test set when compared to CE (Row of Table 3 ), indicating that using Simplex ETF as classifier does not impact the transfcrability of deep learning features. Finally, with our approach HingeCE, we mitigate the degree of collapse by maintaining diversity within each class, and observe substan- tial performance boost over the baselines on both transfer learning (Row(s.o) of Table 1) and test accuracy on ImageNet (23 of Table 3). Also, in contrast to self-supervision approaches (e.g.
'For CIFAR100-LT LT classification using 322, the number of class K = 100 is greater than the feature dimension m 64 which violates the constraint of m 2 K for constructing a Simplex ETF in (3). In this particular case we set the output dimension be 128.

---

Under review as a conference paper at ICLR 2023
Table Performance comparison on long-tail datasets. We report Top-1 accuracy (%) and the best accuracy is highlighted. Our implementation is based upon Menon et al. (2020). Symbol ": the accuracy on is 38.4% when the feature dimension m 64 and is 39.5% when m 128. To ensure Simplex ETF constraint holds, for CIFAR 100-LT, we increase the feature dimension to 128.
Approach
CIFARIO-LT
CIFARIO0-LT ImageNet-LT
Focal (Lin et al., 2017)
70.4
38.4
LADM (Cao al., 2019)
73.4
39.6
M2M (Kim et al., 2020)
79.1
43.5
OLTR (Liu al., 2019)
32.2
CRT (Kang et al., 2019)
47.3
FCL (Kang et al., 2020)
49.8
KCL (Kang et al., 2020)
77.6
42.8
51.5
TSC (Li et al., 2022)
79.7
43.8
52.4
Logit Adjusted (Menon al., 2020)
77.7
43.9
51.1
CE (Menon et al., 2020)
72.8
38,4
46.9
CE+ETF
77.3
42.6
48.1
HingeCE (Ours)
80.4 (1 2.9)
46.2 (1 3.6)
50.8 (1 2.7)
SimCLR) which aim to learn diverse features, our method can provide both discriminative power and feature diversity, thereby, yielding better transfer learning performance.
4.2 LONG-TAIL CLASSIFICATION
Experimental Setup. For long-tail classification, we conduct experiments on three popular long- tail benchmarks: CIFARIO-LT, CIFAR 100-LT (Krizhevsky et al., 2009), and ImageNet-LT (Rus- sakovsky et al., 20. CIFARTO-LT and CIFAR100-LT are derived from CIFAR1O and CIFAR100 with the number of samples per class ranges from 5000 to 50 and 500 to 5, respectively. ImageNet- LT is derived from ImageNet and contains 1000 classes. The number of samples per class ranges from 1280 to 5. Based the number of training samples per class, group the classes into two categories: major class which have more than 50 samples, and minor classes with less than 50 sam- ples. In our method, we tune r+ for the minor classes and simply use 7 0 (which is equivalent to using (w), u:) as logits) for the major classes.
Results. In addition to standard baselines. also compare approach with state-of- -fate-of-the-ar long- tail classification methods. The results are summarized in Table 2. Using Simplex ETF as classifier, CE+ETF already yields improved performance when compared to CE (Row(10,1)) of Table 2) as class centers are predefined and maximally separated while the centers of minor classes may not be well-learned (Fang al., 2021). For I00-LT, even though the test accuracy improved from 38.4% to 39.5% by increasing feature dimension from 64 to 128, larger gain in test accuracy (from 39.5% 42.6%) was obtained through the use of Simplex ETF. Note that the performance gain using CE+ETF on ImageNet-I.T is relatively minor compared to CIFAR-related datasets. On the other hand, by preserving the diversity of each class, our HingeCE considerably improves the performance on all three datasets yielding state-of-the-art performance on a few datasets and is competitive on the rest. Note that TSC (Li et al., 2022) and KCL (Kang et al., 2020) use two- stage training involving supervised contrastive learning (SCL) to improve the performance on long- tail classification, which complementary to our approach. Nonetheless, even without SCL, we obtain better or competitive results. We believe our results can be further improved using SCL and additional hyperparameter tuning.
DISCUSSION
++ and -- are two hyper-parameterss of our method introduced in Equation 8. Here we plot the cosine similarities between each feature on ImageNet train set with its positive class center and rest of the negative class centers during the training process, and present the results in Figure 3(a) and (b), respectively. It can be observed that the mean cosine similarity with positive class centers around 0.46, which is perhaps larger than expected (since all features collapse completely to class mean, then this value should be one). Moreover, the similarity with
Choice of

---

Under review as a conference paper at ICLR 2023
(with negttive class centert
[with positive class centerl
%
Covine Similrity
Cosine Feritty
0.4
Accuracy
0.3
80
70
0.2
60
0.1
Training steps
100k
Training steps
50
0.125 0.375m
Threshold
(a) Cosine similarity with positive class centers.
(b) Cosine similarity with negative
class centers.
(c) Transfer learning accuracy.
Figure 3: (a, b) Cosine similarity of each feature with its corresponding positive (in (a)) and negative (in (b)) class centers averaged over all features obtained with CE+ETF. (c) Transfer learning accu- racy on Flower dataset as function 7+ E [0, 17m with T = æ during ImageNet pretraining with HingeCE.
36
negative class centers remains close zero during the entire training process. Hence, the negative logits have small impact on the value of the CE loss and for simplicity we set T- a (i.e., use (w), u.) as logits). In the following we examine the effect of r+.
Figure 3(c) demonstrates the effect of -+ for fine-grained classification on Flower dataset. It can be seen that with a small value of 7+ HingeCE does not have a significant effect. This is aligned with the observation in Figure 3(a). When r is increased to the range of 11 36 S T I 12 the accuracy increases considerably and is not sensitive to the choice of T within that range.
For all results in Table & 3, the value of F- is fixed 13 36 R. Since this choice works fairly well across various settings, hyper-parameter tuning for our method is very easy. Finally, while this fixed choice works well, we find empirically without reporting the details here that additional tuning can further improve the final performance.
Numerical Rank
(a)
(b)
Approach CE
Accuracy (%)
76.5
CE+ETF
76.3
HingeCE 77.4
0
20
40
60
80
0
20
60 Threshold r*
40
80
Threshold
Figure 4: The (a) numerical rank and (b) NCI metric among features on the ImageNet training set with different thresholds -+ for ht,
Table Top-1 test accuracy on ImageNet.
Measuring Diversity. To measure the diversity of features, we use NC1, defined through between- class and within-class covariance matrices, and numerical rank, defined the ratio of squared nu- clear norm and squared Frobenius norm for features of each class averaged over all classes. The definition of these two measures can be found c.g., Zhou et al. (2022). In Figure 4, we plot NC1 and numerical rank as a function of r+ It can be seen that both measures increases with r+, showing that our method indeed improves feature diversity.
Pre-training Performance. While designed for improving the performance of downstream fine- grained classification task, we show in Table that our method improves the performance on the test set of ImageNet as well.
Per-class Analysis for Long-tail Classification. We evaluate the performance gain of our method on major vs minor classes in long-tail classification, and report the results in Figure 5. Specifically, Figure 5(a) & (b) shows that the benefit of our method mainly on the minor classes. Figure 5(c) demonstrates the effect of r on major and minor classes separately. It can be seen that setting 7+ to be in the range of 36 < + < f provides a notable gain for minor classes. addition, setting + to be in the range of 12 ST+ < 11 50 (which is a range with lower values than that of the minor classes) provides a gain for major classes. This may imply that the optimal threshold 7 should be dependant upon the number of training samples in each class. Hence, while the results in Table 2 are obtained from using the same + in all minor classes and another - for all major classes, which is a choice made to simplify the study, the performance may be improved with a more careful choice of r for individual classes.
8

---

Under review as a conference paper at ICLR 2023
%
Mingr Major
45
100
Accurccy
80
WV Mo
Hingect
Relative Gain
m wn
Accuracy
50
60
35
40
20
100
20
40
60 Class Index
80
100 Minor
0.125
0.25m 0.375M if
Major
Class Index
Minar
Major
Threshold:
(a) Test Accuracy.
(b) Relative gain.
(c) Transfer learning accuracy.
Figure 5: On CIFARIOO-LT, for each class, we calculate (a) test accuracy and (b) relative perfor- mance gain (the class with a larger index x-axis has less training data) and the performance on minor class are mostly improved. Then, we apply our approach on the major and minor classes separately and adjust the threshold 7+ In (c), we observe that the best testing accuracy is reached when our approach is applied on minor classes and the threshold is around 5 S + S 5
18 n
6 RELATED WORK
Transfer Learning is a pipeline where feature extractors trained using pretraining task are used to perform downstream tasks (Kornblith et al., 2019). Existing literature on transfer learning abounds and most related to ours include Kornblith et al. (2021) which show trade-off between pretrain- ing and downstream performance. Specifically, many techniques such as label smoothing produces more collapsed within-class features (Müller et al., 2019) hence improves the performance Ima- geNet pre-training task, but such features lead to worse downstream performance. Our result shows that by introducing diversity, the performance classification tasks can be improved. While other methods for obtaining diverse features exist, they either require an additional k-nearest neighbor search (Dwibedi al., 2021; Touvron et al., 2021; Feng et al., 2022), or require that different classes share the same feature distribution (Xu et al., 2021). On the other hand, within-clat compactness is found benefit face recognition (Wen al., 2016; Liu et al., 2017: Deng et al., 2019; Wang et al., 2018; Meng et al., 2021).
both pretraining and the downsteam fine- ine-grained
Neural Collapse is phenomenon reported in Papyan et al. (2020) for deep models fully trained on balanced dataset in classification tasks. It simplifies the deep model by considering the last-layer fea- tures and the classifier as independent variables, and subsequent works theoretically demonstrated phenomenon of NC under Cross-
(Wojtowytsch & Weinan, 2020; Lu & Steinerberger, 2020; Graf et al., 2021; Fang et al., 2021) or Mean Squared Error (Mixon et al., 2022; Rangamani & Banburski-Fahey. 2022: Poggio & Liao, 2020) losses. Since the classifier weight forms a Simplex ETF in AC, Zhu et al. (2021) shows that fixing the classifier as a Simplex ETF during training can achieve similar performance. While the minimized diversity may hurt the adaptation performance to new classes (Kornblith et al., 2019), the Simplex ETF has been shown to be useful for continual learning (Wu et al., 2021) and long-tail learning (Li et al., 2022; Yang et al., 2022).
Long-tail Learning with imbalance of data distribution a common real-world scenario. Deep learning models tend to perform well on major elasses (Zhang et al., 2021) with a lower accuracy on minor classes. To address the issue, an intuitive strategy is to explicitly re-sample the training data (Ando & Huang, 2017) which usually hurts performance on major classes (Kang et al., 2019). Alternative methods include re-weighting the loss for different classes Byrd & Lipton (2019); Cui al. (2019) or adjusting the logits (Menon al., 2020; Cao et al., 2019) during training. Mean- while, several regularization techniques such as weight-normallzztiio (Kang et al., 2019) and loss modification (Iranmehr et al., 2019) can be applied, but they can be sensitive to the selection of optimizer and may not reach a balanced error (Menon et al., 2020).
CONCLUSION
In this paper, we ask the question of whether Neural Collapse is a desirable property of deep features and argue that the lack of within-class diversity may hurt the performance fine-grained classifi- cation task. To address that, we introduce a geometric arrangement of features with within-class diversity, and present method called HingeCE based on the classifier and loss function design. Empirically we show that HingeCE is substantially better than or competitive to the state-of the-art art methods for fine-grained classification and long-tail classification tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp- 770-785. Springer, 2017.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pp. 201 1-1018, 2014.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-4646 Springer, 2014.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pp. 872-881. PMLR, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbll-distribution-aware margin loss. Advances in neural information processing systems, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. blass-balanced loss based on effective number samples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp.
2019.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9588-95999 2021.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(4))=e103009111 2021.
Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre- training for better downstream transferring. In International Conference on Learning Represen- tations, 2022.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised con- strastive learning. In International Conference on Machine Learning, pp. 3821-3830. PMLR, 2021.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint 1rXii:2106.02073 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value training the last weight layer. In International Conference on Learning Representations, 2018.
Arya Iranmehr, Hamed Masnadi-Shne Shirazi, and Nuno Vasconcelos. Cost-sensitive support vector ma- chines. Neurocomputing, 343:50-60 64, 2019.
10

---

keypoints. International journal of Under review as a conference paper at ICLR 2023
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXi::2110.02796, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09221, 2019.
Bingyi Kang, Yu Li, Sa Xic, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference Learning Representations, 2020.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13896-13905, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661- 2671, 2019.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 34: 28648-28662, 2021.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine- grained cars. 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Tianhong Li, Peng Cao. Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6918-6928, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 212-220, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
David G Lowe. Distinctive image features from computer vision, 60(2):91-110, 2004.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint 1rx2v:20122008655. 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5157 2013.
Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recognition and quality assessment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14225-1422 2021.
Aditya Krishna Menon, Sadcep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint ar 1rXiv:2011.11611 2020.
11

---

Under review as a conference paper at ICLR 2023
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-13, 2022.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in neural information processing systems, 32, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Vardan Papyan, XY Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint Siki::2001.00072,, 2020.
Akshay Rangamani and Andrzej Banburski- Fahee.. Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4243-4247 IEEE, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):2112255 2015.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural col- lapse. arXiv preprint arXiv:2202.080077 2022.
Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning grained image representations with coarse labels. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 884, 2021.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018.
E Weinan and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint 2012::001..05420, 2020.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Stephan Wojtowytsch and E Weinan. On the emergence of tetrahedral symmetry in the final and penultimate layers neural network classifiers. arXiv preprint arXiv:2012.05440,, 2020.
Ziyang Wu, Christina Back, Chong You, and Yi Ma. Incremental learning via rate reduction. Pro- ceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp- 1 1125- 1133, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Jingyi Xu, Hieu Le, Mingzhen Huang, ShahRukh Athar, and Dimitris Samaras. Variational feature disentangling for fine-grained few-shot classification. Proceedings of the IEEF//VF Interna- tional Conference on Computer Vision, pp. 2021.
Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203.099: .9088, 2022.
12

---

Under review as a conference paper at ICLR 2023
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class- imbsllnebd learning. Advances in neural information processing systems, 13:19290-19301. 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-taile aag-tailed learning: A survey. arXiv preprint arXiv: 10.04596, 2021.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You. Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. arXiv preprint arXiv:2203.0/238, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A ge- ometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
13

---

Under review as a conference paper at ICLR 2023
A FINGRAINED TRANSFER DATASET
The downstream datasets are Food (Food-101) (Bossard et al., 2014) with 101 finegrained classes of food, CIFARIO & CO (Krizhevsky et al., 2009), Birdsnap (Berg et al., 2014) with 500 finegrained classes regarding bird species, Sun (SUN-397) (Xiao et al., 2010) with 397 scene cate- gories, Car (Stanford Cars) (Krause et al., 2013) with 196 finegrained class of car models, Aircraft (Maji et al., 2013) with 120 finegrained classes of aircraft models, Pet (Oxford IIIT Pets) (Parkhi al., 2012) with 37 finegrained classes of dog & cat, and Flower (Oxford IITT Flowers) (Nilsback & Zisserman, 2008) with 102 finegrained classes regarding flower species. Food(Bossard et al., 2014) contains 101 finegrained classes For all datasets, we follow Kornblith et (2021) and use the official train-test split for linear evaluation training. Then, for evaluation, on the test set, we report the top-1 accuracy for Food, CIFARIO, CIFARI00, Birdsnap, SUN and Cars; and mean per-class accuracy for Aircraft, Pets, Flower.
HiqpeCE
CE
(c) HingCCE Trandee
Fre-Trai
Trander
Figure 6: Visualization comparison ImageNet (pre-train) and Flower (Transfer learning). Compar- ing with baseline CE, by applying HingeCE, the features are more diverse on pre-train dataset ((b) vs. (a)). the features for finegrained classes more separable ((d) vs. (c)), which then facilitate the finegrained recognition in transfer learning.
Visualization on fingrained classes. As shown in Figure 6, we provide visualization on three coarse classes for ImageNet and three Flower separately. All of the classes are randomly selected. Comparing Figure 6(a.b), applying our approach HingeCE can preserve more diversity within each coarse class. Then, from Figure 6(c,d), the features of finegrained classes can be more separable.
14

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
We use the ResNet for all experiments. For each dataset, we follow the same configuration of ResNet used the baseline approach for fair comparison. Meanwhile, as ResNet has ReLU activation before average pooling He et al. (2016), the feature vector (after pooling) cannot satisfy the constraint of the defined Simplex ETF in an MSC arrangement, i.e., the weights simplex ETF maximally separated over the full feature space (always have negative entries) while the averaged feature vectors always have non-negative entries and its distance with with the weights then cannot be zero. As such, we add fully connected layer, serving as projector, top of the resnet structure such that the value of each entry in the output features is theoretically defined over 1R.
As the thresholds (r+ r-) are the only hyper- prperpparaeeters introduced in approach where r and 7 is usually set as O, we summarize them the Table 4 and explain the details other hyper- parameters below.
Table 4: Summary of threshold selection
Dataset
T
++
Range
The classes on which HingeCE are applied.
ImageNet
13
S r+
5
All 1000 classes
36
ImageNet-LT
I8
T S --
<
36
11
383 minor class
CIFARIO-LT
18 π S + S 12
I minor class
CIFAR100-LT
7 -T-T S T
50 minor class
36
The class with < 50 training data are treated minor classes.
B. PRETRAINING ON IMAGENET
We use the standard Resnet-50 as feature extractor and follow the configuration defined in He al. (2016). For the projector, we set the output dimension the same as the input dimension 2048, We set the batch size as 512, use SGD optimizer with nesterov, and train for 90 epoches. The coefficient of weight decay is 0.001. We set the initial learning rate as 0.4 with linear warm-up of 100 steps and scheduled according to cosine annealing function.
B.2 TRANSFER LEARNING FOR FINE-GRAINED CLASSIFICATION
During transfer learning, we fix the feature extractor learned by our HingeCE to extract the 2048- dim feature vector for each image. Then, we learn linear classifier on top of the extracted features from images of train set. As the dataset considered in transfer learning are abount fine-gratd ane-grained clas- sification, we use the performance fine-grained classification indicate the quality & diversity of the learned features.
Note that the classifiers in transfer learning trained by minimizing vanilla cross-entropy loss and we do not apply our approach in transfer learning. During training, we set the batch size as 128, use SGD optimizer, and train the classifier for 100 epoches. As only linear classifier is learned, we set a large learning rate 1.0 in the whole training process. The information of dataset is summarized in Section
B.3 LONG-TAIL LEARNING
We use ResNet-32 as backbone for experiments on CIFARIO-LT and CIFARI00-LT, and ResNet- 50 for experiments on ImageNet-LT. For each dataset, all of the approaches being compared the same architecture as ours and the comparison is fair regarding network architecture. The implemen- tation details for each dataset can be found below.
We use the standard ResNet32 architecture in experiments and exactly follow the configuration in Menon et al. (2020); Yang Xu (2020). As the spatial resolution of image is 32 32, the first convolution layer is with stride and kernel of size 3x3. The number of channels
CIFARIO-LT
15

---

Under review as a conference paper at ICLR 2023
in the feature output is 64 and the projector linear layer has the same input and output dimension (i.e., 64). We set the batch size as 128, use SGD optimizer, and train for 256 epoches. We set the coefficient of weight decay as 0.0001. We set the initial learning rate as 0.4 without any warm-up and scheduled according to a cosine annealing function.
CIFARI00-LT We use the same network (i.e., ResNet32) feature backbone and follow the experiment configuration described in CIFARIO-LT except for the configuration of projector. As the number of classes in CIFARIO0-LT'S is K 100 and larger than the feature dimension 64, we thus set the output dimension changed.
128 to meet the constraint have m 2 K. To note, the
structure of ResNet32
ImageNet-LT We use the same network (i.e., ResNet50) feature backbone and experiment setup described in Section B.1 but we use 0.0001 as the coefficient of weight decay.
16


Reviews:
[{"review_id": "MEXbxi4y7D9", "reviewer": "Reviewer_tRGD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method that tries to rectify feature collapse for classification, arguing that this is an undesirable property for the case where coarse labels are used for training, but we are (also) interested in a finer-grained downstream task. They advocate to fix the classifiers and use a hinge loss. They experiment on coarse-to-fine as well as long-tail classification.", "summary_of_the_review": "The paper has a number of weaknesses detailed above. I encourage the authors to provide answers to the concerns and questions.", "strength_and_weaknesses": "### Strengths\n\n* The paper dealing a very interesting property of learning with cross-entropy, that of class collapse\n\n* the paper is  experimenting on interesting tasks like long-tail recognition and coarse-to-fine learning.\n\n### Weaknesses\n\nW1) Novelty of the paper is questionable: There are some missing references and comparisons that are very related to the proposed. [A] refers to the same issue tackled here, class collapse for transfer learning. Feng et al in [A] do in some sense provide an answer to the question asked in the intro and propose a loss that avoids collapse. Grafit from Touvron et al [B] is even closer to the task and formulation of this paper: it presents an approach for learning with coarse labels and the loss proposed in [B] is a kNN loss therefore also avoiding collapse. The authors should not only discuss but also compare to such methods.  Another missing reference that should be discussed and compared to is wrf fixed classifiers is [C]. \n\n\nW2) Experimental validation is lacking in many regards: \n\n*  It is unclear how this supervised learning method affect the training task of ImageNet in 4.1 The authors compare against SimCLR, a self-supervised method, but not supervised methods like [A,D] in Table 1. They also do not report performance on the training task of imagenet (one more column should be added). As [Kornblith et al 2021], [E] and other papers have mentioned, there exists a tradeoff between training and transfer accuracy.  What is the performance of all methods at the end of training? \n\n* For long-tail, the method is not outpuerforming multiple methods of Table 2 on the larger and less toy-scale ImageNet-LT. Also Fig 5c shows that setting the main hyperparameter \\tau is far from trivial. Moreover it is unclear what model is used in each dataset and method in Table2 - are all methods using the same backbone?\n\n### References\n[A] Feng, Yutong, et al. \"Rethinking supervised pre-training for better downstream transferring.\" ICLR 2022\n\n[B] Touvron, Hugo, et al. \"Grafit: Learning fine-grained image representations with coarse labels.\"CVPR 2021\n\n[C] Hoffer, Elad,et al  \"Fix your classifier: the marginal value of training the last weight layer.\" ICLR 2018.\n\n[D] Khosla, Prannay, et al. \"Supervised contrastive learning.\" NeurIPS 2020\n\n[E] Sariyildiz et al \"Improving the Generalization of Supervised Models.\" arXiv 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G9Ezzad_gG", "reviewer": "Reviewer_ur4C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses NC - intra-class features collapse to the class mean and different class means are maximally separated. The authors argue that NC is an undesirable property for fine-grained classification. Thus they constrain the features of the same class should lie in a cone instead of collapsing to the mean. A feature extractor pre-trained by this method improves fine-grained classification on multiple datasets.", "summary_of_the_review": "Overall, I am on the fence. The paper is simple and the method seems to work in some cases. The numbers are not substantially strong. The explanation and motivation aren't clear. I am not an expert in this area so please fell free to correct me.", "strength_and_weaknesses": "Strengths:\n- The paper presents a simple idea and is easy to follow. \n- The method is appropriately designed following the main idea. \n\nWeaknesses:\n- My main concern here is that the paper doesn't offer an explanation of why NC is undesirable for fine-grained classification, i.e., the main idea isn't well justified. I think NC should be characterized carefully in order to support the main idea.  In particular, I disagree that \"collapsing to the class mean\" is a good enough reason to say NC is undesirable for fine-grained classification. The authors can convince me by quantifying the collapse and demonstrating how it correlates with the fine-grained classification performance. As of now, I need to believe that it's the case and without any proof of concept. Personally, I think \"collapsing to the class mean\" is quite vague and unclear. For example, one possible interpretation is that the distances between features of the same class are small w.r.t the distance between classes. In this case, it doesn't mean that the feature doesn't encode enough fine-grained information for each instance since intra-class variance can still be high. \n\n- Likewise, since the method is based on a weak motivation (to me), I am not sure how rearranging features into a cone helps with fine-grained classification. There are methods that propose feature space re-arrangement before (Mag-face, arc-face) and they are very clear about why their geometrical constraints boost their performance. I don't see a similar explanation here. Is there any relation between the proposed  MSC and intra-class variance [1]? If there is a connection here that would add depth to the paper and offer an explanation for what is happening.\n[1] Variational Feature Disentangling for Fine-Grained Few-Shot Classification - ICCV21\n\n- The paper introduces some hyperparameters that need to be tuned. \n\n- The paper doesn't improve the performance on the most challenging case of the imagenet-LT.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FC_WnK0rSXk", "reviewer": "Reviewer_6L41", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the undesirable Neural Collapse (NC) phenomenon for specific classification tasks, including fine-grained classification and long-tailed classification. The authors then propose to learn deep features that have the maximal-separating-cone (MSC) property by optimizing towards their designed hinge loss function. Empirical results verified their motivation and the effects of the proposed approach.", "summary_of_the_review": "This paper is decent research towards learning deep features with MSC property, and hence tackles certain specific classification problems where NC is undesirable. ", "strength_and_weaknesses": "Pros:\n\n+ Well-motivated approach: Investigates an intriguing phenomenon of NC, and its undesirable properties for fine-grained classification adaptation.\n+ Technically sound solutions with a novel hinge function design.\n\nCons:\n- It is unclear to me why learning MSC features can also benefit long-tailed classification tasks. Experimental results in Table 2 also seem to show weaker performance gain on long-tailed datasets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject