Where is the bottleneck in long-tailed classification?
Authors: Zaid Khan
Yun Fu
Venue: ICLR 2022 Submitted (2022)
Keywords: fairness
bias
long tailed learning
imbalanced learning

Abstract: A commonly held belief in deep-learning based long-tailed classiﬁcation is that the representations learned from long-tailed data are ”good enough” and the performance bottleneck is the classiﬁcation head atop the representation learner. We design experiments to investigate this folk wisdom, and ﬁnd that representations learned from long-tailed data distributions substantially differ from the representations learned from ”normal” data distributions. We show that the long-tailed representations are volatile and brittle with respect to the true data distribution. Compared to the representations learned from the true, balanced distributions, long-tailed representations fail to localize tail classes and display vastly worse inter-class separation and intra-class compactness when unseen samples from the true data distribution are embedded into the feature space. We provide an explanation for why data augmentation helps long-tailed classiﬁcation despite leaving the dataset imbalance unchanged — it promotes inter-class separation, intra-class compactness, and improves localization of tail classes w.r.t to the true data distribution.

Under review as a conference paper at ICLR 2022
WHERE ARE THE BOTTLENECKS IN LONG-TAILED CLASSIFICATION?
Anonymous authors Paper under double-blind review
ABSTRACT
A commonly held belief in deep-learning based long-tailed classification is that the representations learned from long-tailed data are "good enough" and the per- formance bottleneck is the classification head atop the representation learner. We design experiments to investigate this folk wisdom, and find that representations learned from long-tailed data distributions substantially differ from the represen- tations learned from "normal" data distributions. We show that the long-tailed representations are volatile and brittle with respect the true data distribution. Compared to the representations learned from the true, balanced distributions, long-tailed representations fail localize tail classes and display vastly worse inter-class separation and intra- compactness when unseen samples from the data distribution are embedded into the feature space. We provide an expla- nation for why data augmentation helps long-tailed classification despite leaving the dataset imbalance unchanged - promotes inter-class separation, intra-class compactness, and improves localization of tail classes w.r. to the true data distri- bution.
INTRODUCTION
Long-tailed distributions are those in which samples from small number of head (majority) classes vastly outnumber the samples for large number of tail (minority) classes. Such distributions can occur naturally, such rare diseases in medical contexts minority ethnic groups in face recogni- tion. Learning from long-tailed data is challenging because it combines three problems: imbalanced learning across the head/body/tail, few-shot learning on the tail classes, and label distribution shift at test time (long-tailed training set, balanced test set).
In practice, most research focuses on the data imbalance label distribution shift problem, which reflects a communal belief that the bottleneck in long-tailed classification lies in the classifier rather than the representation belief and provide evidence against it through a series of experiments. First, we show that the representation not good enough (TI.T) and may be a larger bottleneck than the classifier. We show that the long-tailed representations and "normal" representations have substantial differences in their second moments ($2.I) and that long-tailed representations have significantly worse inter-class separation and intra-class compactness ($2.2), and poorly localize tail classes in feature space ($8). Finally, we explain the reason why data augmentation boosts performance in long-tailed learning (Zhong et al.; [Zhang al. (021b) ($ 1.2g) despite having effect the imbalance factor, showing that it significantly improves localization of classes ($8), and confers robustness to distribution shift by improving inter-class separation and inter-class compactness ($ 2.2
the representation is though to be "good enough". We question this
1.1 IS THE REPRESENTATION OR CLASSIFIER THE BOTTLENECK?
There is a tension between representation learning and classifier learning in long-tailed classifica- tion, due to their opposite locations in the bias-variance trade-off spectrum. Representation learning thought to suffer more from data scarcity rather than data imbalance (Yang & Xu 2020). while classifier learning is though to suffer more from data imbalance. Decoupled training (Zhong al.)
Kang et al.] 2020) methods attempt to address this tension by training the classifier and representa- tion separately (left side of Fig 1]. It is common belief that the representations are "good enough"

---

Under review as a conference paper at ICLR 2022
Standard Decoupled Training (cRT)
Experimental Configuration
Stage
Stage
Stage
Stage
D LT
Class Balanced
DL
DT,,
( LT. D*
Resampling
Backhone
Freeze
Backbone
Backbone
Freeze
Backbone
Classifier
Classifier:
Classifier
Classifier
LCE
CC
LC
Figure 1: The standard classifier retraining (cRT) setup the left, and the experimental configu- ration we use on the right. Instead of resampling, we swap datasets between stages to simulate an "ideal" classifier or "ideal" representation.
Accuracy
Dataset
DLT + DLT
DDT + D
D' - DDT
D -> D
Backbone
CIFAR-IO LT 0.7212
0.8261
0.9106
0,9302
ResNet-32
CIFAR-100 LT
0.409
0.5157
0.662
0.7109
ResNet-32
ImageNet-LT
0.3531
0.4586
0.5383
0.6393
ResNet-10
Table 1: Models with classifiers trained on long-tailed distributions but representations trained the true distribution are less-biased than models with representations trained on the long-tailed dis- tribution and classifiers trained the true data distribution, A high quality representation can resist a highly biased classifier.
and the bottleneck is the classifier Zhang al. 2021a Yang & Xu 2020 Kang et al. 2020). thus many works focus on interventions in the classifier learning phase (Menon et al. 2020).
We test this belief by carrying out altered form of decoupled training (right side of Fig 1. The three most commonly used long-tat long-tailed datasets - CIFAR 10/100 LT and ImageNet-LT L et al. 2019) and ImageNet-LT (Krizhessky) and ImageNet-Ik (Deng et al.] 2009). Let DD.T denote the long-tailed version of dataset, and D. denote the normal, balanced version. To assess the relative impact of the representa- tion and classifier, we apply decoupled training without resampling. Instead of resampling, we swap between the DLr and D* versions of the dataset between Stage I and Stage 2. If the classifier truly the bottleneck, we expect to see that a representation trained on D with a classifier retrained on DLT should perform worse on the balanced test set than a representation trained on DLr with classifier retrained on D*
all have larger corresponding balanced datasets, namely CIFAR 10/100
We find that this not true (Table D. expected, the D* + D* model (with a representation and classifier trained on the true, balanced dataset), performs the best. However, across all three datasets, the D. - DLr model outperforms the Drr >> D model by a substantial margin. This indicates that contrary to popular belief, the bias in the representation dominates the bias in the classifier, and not the other way around.
1.2 WHY DOES DATA AUGMENTATION HELP?
At a first glance, it unclear why class agnostic input data augmentation should improve long-tailed tailed classification. Data augmentation is incapable of addressing data imbalance it leaves the imbal-
2

---

Under review as a conference paper at ICLR 2022
CIFAR-10
CIFAR-
ImageNet-LTT
D*
D* D
Dur D
DIr DDT
0.0
0.2
0.4 0.6
0.8
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Accuracy
Accuracy
Accuracy
Figure 2: Table in graphic form, showing that biased representation is very difficult to overcome, but a biased classifier can be mitigated if the representation unbiased.
ResNett32
w/ MixUp
w/ Auto
w/ AutoAuggment + Mp
Empicioal Bound
[1
79.48
$1.73
13.44
Stage II
Stage
13.05
65
70
75 Accuracy
50
65
70
75 Accursy
80
85
65
70
75 Accuracy
80
85
65
70
80
85
Accurcy
Figure 3:
Data augmentation improves fair classification performance in long-tailed settings (CIFAR-10 with an imbalance factor of 100)
ance factor of dataset unchanged. Nevertheless, input data augmentation such MixUp (Zhang jt al.] 2018). Manifold Mixup (Verma et al.) 2019) and AutoAug (Cubuk et al. 2019) have proven to be highly effective for reducing bias in long-tailed classification (Zhong al. Zhang al. 202 Tan et al., (020). When combined with decoupled training (Kang et al. 2020). data augmenta- tion results in even larger increases (Fig However, the mechanism by which data augmentation improves performance and reduces bias is not known. It cannot address data imbalance, so data augmentation must be affecting another quality of the representation.
1.3
WHY ISN'T REBALANCING ENOUGH?
Resampling and reweighting strategies have been an essential part of long-tailed classification. How- ever, as first observed by Zhang et al. (2021)), rebalancing does not seem to be enough. Following the methodology of/Zhan al. (2021a) Ta), we construct an empirical classifier bound by first training representation on the long-tailed ed DTT dataset, in this case CIFAR-10 10 with an imbalance factor
of 100. Next, we follow the standard decoupled training startegy for cRT (Kang et al. 2020) and retrain the classifier version of DLr that has been balanced by class- resampling ling Kang al. (202)) after freezing the representation. To obtain the empirical classifier performance bound, we then take the frozen representations from the first stage and train a new classifier the true dataset D* The performance of the classifier trained on D can be viewed as an empirical upper bound on the performance obtainable through any resampling strategy, and is the ideal performance aim for. There is consistent gap between the class- resampling strategy and the upper bound (Fig 3). suggesting that the resampling strategy alone leaves performance on the table. Moreover, applying data augmentation again in the second stage boosts the performance even further, which is suprising in light of the fact that the classifier is primarily thought suffer from imbalance so why should applying data augmentation in the classifier learning phase have such big impact performance?
2 EXPERIMENTS
In the previous section, we showed results that provoke three questions.
Why are the representations learned from the long-taie tailed data so much poorer than the representations learned from the true dataset?
3

---

Under review as a conference paper at ICLR 2022
2. Why does data augmentation help in the representation learning phase of decoupled- training for long-tailed classification?
3. Why isn't 't rebalancing enough to fix the linear classifier's bias?
Covariance in the
Variance in the Head Tail
Per Representation
Dir Regresentation
Representation
Representation
10
norm varancee
14
CIFARIO-LT
F-Norm of2
8-
6-
Class Index
Class Index
Covariance in the
Variance Head
20
Dir Representation
Dy. Representation
D' Representationn
D' Representation
18:
16
L1 narm of variance
50
CIFAR1OI-LIT
12-
10
60
100
20
60
100
Class Index
Class Index
Figure 4:
The second moments of features learned from long-tailes tailed distributions are substantially different from those of representations learned from the true distribution. Variance for the tail classes higher, and the covariance matrices have larger norms for the body and the tail.
We provide answers to these questions by analyzing the differences between the representations learned from the long- tailed data, the representations learned from the dataset, and representa- tions learned from the long-tailed data with augmentation.
2.1 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA HAVE DIFFERENT 2ND MOMENTS
We train a ResNet-32 on a true, balanced dataset D and a ResNet-32 on a long-tailed dataset DLT. We then extract features from both models for the entire dataset (D*), to reflect the true data distribution. We use 10FAR10/100-LT with an imbalance factor of 100.
The representations learned from long-tailed data differ from the representations learned from the true dataset in two significant ways. First, there are notable differences in the variance and co- variance of the classes that belonged to the head or tail in the long-tailed dataset within the DLT representation, but not in the D representations (Fig 4). Specifically, the variance of the features belonging the tail and body classes (the I.l-norm of the covariance matrix diagonal) diverges

---

Under review as a conference paper at ICLR 2022
CIPARIO-LT
CI
Soon
daa mode
indor
Explained Companent
Veriance of Thus Data Distribution Explained by be Principal Companent
Lampttakll hapooelatton
Long inted Novrssetatton
Hdee 10 Seprosrtation
0.5
0.0
<bes intec
23
60
80
100
ImageNet-LIT
of
2002
200
600
400
803
1000
acrted dlasssa
Varience of True Data Distribution Exp wined by 1st Pninc Component
Lang Thiled Represeniation denl Represertarion on
% Saverra Warrinn eotunee
0.20
0.15
0.10
600
E03 1003
Figure 5: The feature space of long-tailed representations is biased their variance is concentrated in fewer dimensions than those of the representations trained on the true class distributions, This can be seen by the relatively higher proportion of variance explained by the first principal component.
substantially from the variance of the features belonging to the head classes. This is not only true of the variance, but of the covariance. The Frobenius norm of the sample covariance matrix computed from the features of a class is also different between the D and DLT learners. For the learner trained on D* there are no significant differences in variance or covariance between the head, tail, and body, and differences are dominated by noise among the classes. For the learner trained on DLT, the variance of the head is systemically lower than the variance of the head, and this is also true of the covariance.
Next, we examine the proportion of variance explained by the first principal component of the data matrix for each class. Concretely, we take all feature vectors belonging the class from the com- plete dataset D* and compute the first principal component of the data matrix of features for the class. We then plot the proportion of variance explained by the first principal component of each class (Fig. 5). thus obtaining an estimate of the relative complexity of the subspace each class resides the
result: the first principal component explains more of the variance in the representations learned from DITT than the representations learned from D*.
in. Across all three datasets,
Intuitively, these results suggest that the representations learned from long-tailed data are substan- tially different than the representations learned from the true, balanced dataset. The differences are most marked for tail and body classes, and least for the head classes. Geometrically, the long-tailes representations appear to occupy a less complex subspace than the D" representations, yet exhibit greater variance, suggesting that the feature space is biased fewer dimensions in the long-taid tailed feature space are relevant to all classes, and thus the variance concentrates in fewer dimensions of the feature space.
2.2 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA ARE MORE DIFFUSE AND LESS SEPARABLE
We now examine the representations from the angular perspective. Recall that the score for the class k logit in standard linear classifier for feature vector
W
|/&ll ||x|| cos 0
5

---

Under review as a conference paper at ICLR 2022
Differences in intra-class compactness between representations
Representations on Der with augmentation
class in Drr
unseen samples added (
80
degrves
50-
angular spread
true anguler spread
Representations trained on Dur
Representations trained on D
80
80
anguler spread
angul spread true angular spread
70-
true angular spread
70-
deggeees
50
50
40-
40-
Figure 6: More unseen samples from the true data distribution fall outside the angular maximum of the tail class distributions for long-tailed representations, than for representations trained the true distributions. Augmentation can mitigate this effect.
function of the angle 0 between x and Wa when ||wa|l |x|] In practice, the norm [[ of the logits of a class & are strongly correlated with the cardinality of the class (Menon et al. 2020 Kang al. 2020 Zhong et al.) tail classes have smaller norms and vice versa, reflecting the class priors. However, decoupled training roughly equalizes the logit norms Kang et al. (2020), allowing us to ignore the || ||x|| term and focus only on the angle 6. In this setting, the classifier reduces to an angular classifier, and sample x will be assigned to the logit wr to which it has the smallest angular distance. Thus, we proceed to an examination
the angular distribution created by the features of the long-tailed representation learner and the D* representation learner.
The representations of the tail classes learned by the long-tailed representation learner are signifi- cantly less compact than the representations learned by the D* representation learner with respect to the true data distribution (F While the long-tailes per-class representations are compact w.r.t to the long-tat tailed distribution, when the unseen samples making up the true data distribution of D* are added, the angular spread significantly increases for the tail classes. This means that any classifier boundaries learned using only the samples available in the DrT distribution will fail to generalize, as the majority of unseen samples for tail will escape the classification boundaries. Interestingly, augmentation appears significantly reduce the compactness gap between the long-tailed and true data distribution for the tail classes. This may partly explain the success of data augmentation in long-tailed classification. However, data augmentation (MixUp) also significantly increases the an- gular spread of all classes, in contrast to the representation trained on D* which addresses the classes
distribution shift without compromising the angular spread
The representations of the tail classes learned by the long-tailed representation learner also display worse inter-class separation with respect to the true data distribution (Fig 7 The angular distance to the nearest incorrect class center significantly shrinks when the unseen samples from the true data distribution are considered, Again, augmentation makes a significant difference: the angular dis- tribution shift of representations trained the long-tailed dataset with augmentation is far smaller than the angular distribution shift of representations trained on the long-tailed dataset without aug- mentation.
Thus, two major differences between the representations learned from sentations learned from the D* data is the effect of distribution shift on intra-class compactness and
data and repre-

---

Under review as a conference paper at ICLR 2022
inter-class separation. When unseen samples from the true data distribution added, the angular distributions of the tail classes balloon for the representation trained on long-tailed data. The poor intra-class compactness also leads poor inter-class separation w.r. to the true data distribution, as the expansion of angular distributions results in each class eating into the margin between it's nearest neighbor. This means that classifier boundaries drawn the long-tailed distribution will likely be optimistic and fail to generalize due to unseen samples from the true distribution mostly escaping the convex polytope of the long-tailed classifier boundaries. However, data augmentation seems to help, because the representations learned from long-tailed data with data augmentation exhibit much better inter-class separation and intra-class compactness under distribution shift.
Differences in inter-class separation between representations
(under long tailed data distribution)
(under true data distribution)
class A in 9gr
class B in 9
class A in 2*
class B in 2*
Representations trained on Dar
40
degrees
20
0
20
40
60
80
100
class index
Representations trained on D
40
degrees
20
20
40
100
class index
Representations trained on D.y with augmentation
degrees
20
20
40
60
80
100
Figure 7: When unseen samples from the true data distribution are embedded into the long-taid tailed feature space, inter-class separation significantly drops. This is not true for the toilet distribu- tion, which sees relatively small shifts inter-class separation. Augmentation mitigates this effect, and augmented representations experience small change in inter-class separation.
2.3 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA POORLY LOCALIZE TAIL CLASSES
Intra-class compactness and inter-class separation have known fixes: apply a margin. However, margins will only be minimally helpful if the class centers are very poorly localized, which we now show occurs for long-tailed representation learnes. Proceeding with the experimental setup of the previous sections, we examine the angular shift induced in the class centers by the distribution shift between the loigetailed distribution and the true D' distribution. Specifically, let μu be the center of class l with respect to the long-tailed dataset DDT, and let n²g be the center of class k with respect to the true data distribution (the D* dataset). We show that for the tail classes the angular change 0A, = #4 14 in the location of the class center for class k induced by knowledge of the true data distribution is substantial for the long-tailed learner, nearly equivalent to the angular width of the class itself. Knowledge of the true data distribution also significantly expands the variance of the angular distribution proportional to the inverse class frequency for the tail classes the long-tailed representation learner.
Again, augmentation has an ameliorating effect and helps to repair this problem in the long- representation. The angular shift in the class centers of Dr. induced by adding the unseen sam-

---

Under review as a conference paper at ICLR 2022
Knowledge of the true data distribution shifts class centers
Angular shift class
Mean angle between instance and class center in long-tniled sample
Mean angle berween instance and true class center
Representations Dir augm entation
class Lt
unseen samples added (5
20-
10
0
20
40
60
80
100
class index
Representations trained
Representations trained
35-
25-
degress
25
20-
15-
15
10-
5
5
40
80
100
20
60
100
class Index
dass Index
Figure 8: Tail classes are poorly localized w.r.t to the true data distribution for long tailed represen- tations. The class centers experience a large angular shift, nearly equal the width of the classes for classes in the tail.
ples of DD* to Dr.r is significantly lower for the augmentation representation than the unaugmented representation, In the augmented representation, the increase angular variance induced by knowl- edge of the true data distribution is also lower, meaning the augmented representations both better localize the class centers and are more resistant to distribution shift and thus likelier to generalize.
3 IMPLEMENTATION DETAILS
For experiments, create artificially long-tailed versions of the 1IFAR-10/100 (Krizhevsky) datasets by following the procedure of Cui et al. Specifically, the number of samples n/ in the i-th class is set to be n' naJL', where is the original number of samples in the class, and E [0. 1] is used to control the imbalance factor, defined as the count of samples in the largest class divided by the count of samples in the smallest class. For all experiments,
long-tailed ver- sion of SIFAR-100/11 with imbalance factor of 100. Following Zhang et al. (2021b) by; Cui et al., we train a ResNer-32 He et al. (2155) for 200 epochs using the SGD optimizer with an initial learning rate of 0.01, with 5 epochs of warm start Goyal et al. 2018). weight decay of 2c 4, momentum of 0.9, and divide the learning rate by 100 at the 160th and 180th epochs using a batch size of 128. For ImageNet-LT, we follow the setting of Liu et al. (2019) and train ResNet- 10 for 100 epochs with a weight decay of x 10-4 using SGD with a momentum 0.9 and an intital learning rate of 0.2 with batch size of 512. The learning rate is divided by 10 at epochs 30, 60, and 80. For the classifier learning phase, we follow (Zhong et al.) and use a cosine learning rate scheduler with batch size of 512 for 10 epochs. For both CIFAR 10/100-LT and ImageNet LT, we use initial learning rate of 0.05 and a weight decay of 2e - in the classifier learning phase. We found that the value for MixUp had minimal effect over the values 0.6, 0.8, 1.0, and fixed it at 1.0.
RELATED WORK
Feldman & Zhang 2020) develop an intriguing theory for long tailed learning in particular, which they claim that the primary obstacle learning from natural datasets is not label noise, but insuf-
8

---

Under review as a conference paper at ICLR 2022
ficent samples for rare and atypical examples. They provide experimental evidence in Feldman (202I). In the case of decoupled learning, Zhou et al. (020)) makes an attempt to identify why decoupled learning works. They conclude that class-level rebalancing strategies harm the repre- sentation learning. Specifically, class-level rebalancing results better separability of classes, but causes the within-class distribution of samples become more diffuse and spread, instead of com- pact. Kornblith et al. (2020) notes that all loss fuunctions which improve over the standard softmax do so by producing greater class separation in the final representations. Starting with CenterFace Wen et al. (016) and continuing on to SphereFace Liu et al. (2017) and ArcFace Deng et al. 20119, large-margin loss functions have driven significant performance improvements face recognition. To achieve perfect accuracy when metric learning. it is necessary that the maximal intra-class dis- tance is smaller than the minimal inter-class distance to allow a nearest-neighhor classifier to work correctly. This requires the metric to push class centers apart, while compacting the "feature clouds" of each class. The results of Zhou et al. (2020) suggests that only separability is enhanced by class rebalancing, while intra-class compactness Is harmed. Liu et al. (2020) notice that in the long-tailed setting under a cosine classifier, the angles made by the head classes with respect to the class cen- ter exhibit larger variation than the angles made by the tail classes. Khan et al. (2019) show that Bayesian uncertainty estimates (equivalent to the variance of a Gaussian process) are negatively correlated with the number of samples tail-classes have higher uncertainty (variance) estimates. Yang et al. (2018) point out that CNNs trained with this loss only learn a linearly separable space, and therefore have no reason to be intra-class compact. Both Zhang et al. (202Ib) and Zhong et al. test MixUp Zhang et al. (2018) and find that it results in substantial performance gains. Yang & Xu (2020) leverage unlabeled additional data with semi-supervised learning and a pseudo label- ing loss to train a classifier, showing that semi/self-supervisee representations may have advantages supervised representations do not.
5 CONCLUSION
We probe the fundamental differences between representations learned from long-tailed data and representations learned from the true data distribution, and use those differences to explain why data augmentation improves long-tailed classification. Contrary to popular wisdom, we show that representation has greater potential to be the bottleneck in long-tailed classification than the clas- sifier. Our characterization illustrates the primary differences between long-tailed representations and those learned from the true class distributions: biased feature space, less separable and more diffuse classes, and poor localization of tail classes. The results suggest that the primary problem in long-tailed classification may in fact be the few-shot learning problem on the tail classes, rather than data imbalance problem.
REFERENCES
Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 113-123. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/C/RR.20190... 000200 URL http:/7 Aut oAuqment.
openaccess. thectf. com content Learn www.au/eeennttioottteteeeeeeeeee From BatteDaata______ Saattta__VR__2199_ htmaaerr.mm
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-Balanced Loss Based on Effective Number of Samples. pp. 10.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 248-255, 2009. doi: 10. 1OW.CC/R.2009 22045488800
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. arXiv: 07699 [cs], February 2019.
Vitaly Feldman. Does Learning Require Memorization? A Short Tale about a Long Tail. arXiv:/906.000 05271 (es, stat], January 2021.

---

Under review as a conference paper at ICLR 2022
Vitaly Feldman and Chiyuan Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. arXiv:2008.0377 [cs, stat], August 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate. Large Minibatch SGD: Training Ima- geNet in Hour. 1rXiv:1706.0266 [cs], April 2018.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv: 1512.03388 [cs], December 2015.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long-Tailed Recognition. arXiv:1910009217 [cs], February 2020.
Salman Khan, Munawar Hayat, Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the Right Balance with Uncertainty. arXiv:1901.07590 [cs], April 2019.
Simon Kornblith, Honglak Lee, Ting Chen, and Mohammad Norouzi. What's in Loss Function for Image Classification? arXiv:2010. 16402 [cs). October 2020.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. pp. 60.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep Representation Learning on Long-Tailed Data: A Learnable Embedding Augmentation Perspective. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2967-2976, Seattle, WA, USA, June 2020. IEEE. ISBN 978 177781-728111 1778177-668-..
doi: 10. 10.11009C2442200..O 202000000000000000....
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep Hypersphere Embedding for Face Recognition. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6738-6746, Honolulu, HI, July 2017. IEEE. ISBN 978-1- 5386-0457-1. doi: 10.11009C2...
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in open world. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546. Computer Vision Foundation IEEE, 2019. doi: 10. 1..100//CVPR.201.00022
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. +-Xiv:2007.07314 [cs, stat], July 2020.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang. Changqing Yin, and Junjie Yan. Equalization loss for loigetailed object recognition. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11659- 11668. Computer Vision Foundation / IEEE, 2020. doi: 0211199CCVPR2220022000..oo 168. URL httpp:://ppeaccesss thecvf. com cotp:/coneett..om CVB.__CVVVR_C Equalizaa wtt.atiioonoss__or________ Taainn--aalleee htt.a//o.....oooooo......oo..oo. paper. html
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6438 6447. PMLR, 2019. URL http::/proceet
mww..rr.......mm.....mmm..mmmm.. press / v97 veeeeeeeraaaaeaaaeeeeeeeeeee 9a. html
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A Discriminative Feature Learn- ing Approach for Deep Face Recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016, volume 9911. pp. 515- Springer doi:
International Publishing, Cham, 2016. ISBN 978-3-319-46477-0 10. 00007/978-3331944644 31.
10

---

Under review as a conference paper at ICLR 2022
Mlong-Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolu- tional prototype learning. In 2018 JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 3474-3482. IEEE Computer Society, 2018. doi: 10. 109/CV9/ 201.V........03666666 8.00366.
Yuzhe Yang and Zhi Xu. Rethinking the Value of Labels for Improving Class-Imbalanced Learning. Stt1:20006.05529 [es, stat], September 2020.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empiri- cal risk minimization. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution Alignment: A Unified Framework for Long-tail Visual Recognition. arXiv:2103. 16370 [cs], March 202
Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3447-3455 May 2021b.
Zhisheng Zhong. Jiequan Cui, Shu Liu, and Jiaya Jia. Improving Calibration for Long- Tailed Recog- nition. pp. 10.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In 2020 IEEE/CVF Conference Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp- 9716-9725, IEEE, 2020. doi: 10.1 N0.11099CCV44200000000000000
A APPENDIX
You may include other additional sections here.
11


Reviews:
[{"review_id": "jmg-PcyNAMI", "reviewer": "Reviewer_B7am", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n1. This paper seeks to empirically investigate the importance of representation learning, which may provide a new understanding of deep long-tailed learning to the community.\n\n\n2. This work shows the effectiveness of intra-class compactness and inter-class separation on long-tailed representation learning.  \n\n\n3. This work also analyzes the influence of data augmentation on long-tailed representation learning, which provides a better understanding of data augmentation in deep long-tailed learning. \n\nNegative points:\n1. This paper mentioned that \"a commonly held belief in deep long-tailed classification is that performance bottleneck is the classification head atop the representation learner\". However, such a belief may not be common. Note that many recent long-tailed learning studies focus on improving representation learning [1], e.g., KCL [2], Hybrid [3], PaCo[4] and DRO-LT [5].  Moreover, in the conclusion, this paper stated that \"the results suggest that the primary problem in long-tailed classification may in fact be the few-shot learning problem on the tail classes, rather than the imbalance problem\". However, this argument is too strong and the obtained results cannot support it. It would be better if the authors had written all the arguments more rigorously and verified them more completely.  \n\n\n2. As mentioned by the above question, there are many representation learning-based long-tailed studies, e.g., [1-5]. Therefore, it would be better if authors can review mire representation learning based long-tailed learning methods in related work.\n\n\n3. The vital problem in this paper is the used balanced set: i.e., CIFAR-10/100 and ImageNet-1K. Please note that the data number of  CIFAR-10/100 or ImageNet-1K is much more than their long-tailed variants, i.e., CIFAR-LT and ImageNet-LT. Considering using more training samples will lead to significant improvement in representation learning and model performance, most empirical comparisons in this paper (especially Table 1 and Figure 2) are unfair and the corresponding arguments are unpersuasive. The experiments would have been more persuasive, if the balanced training set is a variant of the long-tailed training set with a similar total data number but each class has the same/similar data number, like [1,2]. For example, a balanced set of ImageNet-LT can be obtained at https://github.com/Vanint/Awesome-LongTailed-Learning/tree/main/resources/data_txt/ImageNet_LT. \n\n\n4. Please discuss more of Figure 5. On ImageNet-LT and CIFAR100-LT, the variance trends of long-tailed representations and ideal representations are quite consistent. Such observations seem different from the conclusion of Sec 2.1. Since I am confused about the results, I guess other readers may also do so. Therefore, I suggest the authors explain them more. \n\n\n\n\nMinor suggestions:\n1. Figure 3 is not clear enough. It would be better if the authors can explain it more in the captions. Moreover, what are \"degrees\" in Fig.6-8? Please make them more clear.\n\n2. In Lines 4-5 of page 2, ImageNet-LT appears twice. In Line 1 of page 6, there should be a \"full stop\" before In practice. \n\n\nReferences:\n\n[1] Deep long-tailed learning: A survey. ArXiv, 2021.\n\n[2] Exploring balanced feature spaces for representation learning. In ICLR, 2021.\n\n[3] Contrastive learning based hybrid networks for long-tailed image classification. In CVPR, 2021.\n\n[4] Parametric contrastive learning. In ICCV, 2021.\n\n[5] Distributional robustness loss for long-tail learning. In ICCV, 2021.\n\n", "summary_of_the_paper": "This paper seeks to study what is the bottleneck in long-tailed learning. Based on extensive experiments, the authors propose that representation learning is the bottleneck in long-tailed classification. Also, this paper analyzes representation learning from the perspectives of intra-class compactness and inter-class separation, as well as the influence of data mixup on long-tailed representation learning.", "summary_of_the_review": "Overall, I like the goal of this paper, i.e., analyzing the bottleneck of long-tailed learning. However, I cannot champion this paper since the data number of the used balanced set is much larger than the long-tailed set, which makes the empirical comparisons unfair and the corresponding finding unpersuasive. Moreover, the arguments in this paper should be written more rigorously. I am glad to see the response of the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jk3FNRzY1l", "reviewer": "Reviewer_eREq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really enjoyed reading the paper. It has a very clear direction from the beginning with good experiments to back it up. The writing is clear as well. I believe long-tailed classification is an interesting problem with clear real-world applications, so studying it in-depth is necessary for the community.\n\nOverall, I don't see any major drawbacks or shortcomings as the experiments and ablations combined with the analysis are solid. That said I have few questions:\n\nThe difference in representation between D* and D_{LT} is clearly visible. However, apart from difference in the shape of distribution (long-tail vs balanced) there is also difference in the amount of data between those two which might play an important role, especially when considering learned representations. It would be good to see what is the difference as well between two datasets that have equal number of examples, but different distributions: normal ad LT. Since otherwise the authors conclusion might raise a question.\n\nAdditionally,  Yin et al. [1] performed a related analysis of classifier magnitude difference which was depending on the distribution of classes. The authors analysis seems deeper here, however it would be good to address any similarities/differences. On top of that, few methods [2, 3] used adversarial examples in order to modify the learned representations instead of the classifiers in LT task. It would be good to see what the authors think about such direction and what impact on the feature space and measured statistics it would have.\n\nAnd finally, apart from the analysis, what are the conclusions here for the future researchers - any thoughts on proposed directions/approaches that could originate from the performed analysis?\n\n[1] Yin, Xi, et al. \"Feature transfer learning for face recognition with under-represented data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[2] Kim, Jaehyung, Jongheon Jeong, and Jinwoo Shin. \"M2m: Imbalanced classification via major-to-minor translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[3] Kozerawski, Jedrzej, et al. \"BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images.\" Proceedings of the Asian Conference on Computer Vision. 2020.", "summary_of_the_paper": "The authors study the long-tail dataset problem in order to determine the true bottleneck for the task. After performing many ablations and experiments on 3 benchmark datasets they establish that contrary to common belief the bottleneck is in data representation rather than the classifier itself.", "summary_of_the_review": "Good analytical paper on interesting subject of long-tailed classification. It would be good to see authors thoughts on impact of the amount of data in training, impact of adversarial augmentations, and proposed directions stemming from the analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6LGNfTrb7h", "reviewer": "Reviewer_gifg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The weakness: \n1) The second objective of this paper is to discuss \"why data augmentation helps in representation learning\". However, in the paper, only positive effects from data augmentation were shown, the reasons and mechanisms were not fully discussed. \n\n2) The overall paper is based on the unserious term \"good enough\". What is this term defined? How good is good enough? Good enough in terms of what? Generalization and robustness compared to full balanced data sets or in terms of knowledge transfer? If it is the first one, of course, long-tailed representations are less generalized and robust compared to balanced representations. It is not a new idea and is already discussed in [1]. If it is the second one, then the later experiments don't make any sense. And I think when people say long-tail representations are \"good enough\" in studies like [2], it is more like it is good enough for long-tail learning rather than comparing it to balanced learning. \n\n3) All the experiments seem unfair to me. For example, D_LT are representations from long-tailed data sets, and D* are representations from balanced data sets. Balanced data sets always have much more training samples compared to corresponding long-tailed counterparts. How do you know these inferior results were not caused by the lack of training samples? \n\n4) In the \"adding unseen samples\" experiments (e.g., Fig 6, Fig 7, Fig 8), only results on D_LT were reported. I want to see results when unseen samples are added to D* as well. Only by doing this can you prove D* is less diffused and better localized. \n\n5) Fig 7 needs a more detailed legend. So many components don't have explanations. \n\n6) By looking at Figure 5, I don't see a significant difference between D_LT and D* in Cifar100-LT and ImageNet-LT.\n\n[1] Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., & Yu, S. X. (2019). Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2537-2546).\n[2] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., & Kalantidis, Y. (2019). Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217.", "summary_of_the_paper": "This paper tries to prove that there is a bottleneck in feature learning for long-tailed classification and data augmentation can help relieve the issues in long-tail feature space. Three major experiments were done to prove that feature space 1) is more biased than balanced feature space, 2)  is more disused and less compact than balanced feature space, and 3) less localized in terms of feature centroids. And data augmentation can help alleviate all three issues. ", "summary_of_the_review": "I think the intuition of this paper is not clear and the experiments are not persuasive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-oWQzglOEuq", "reviewer": "Reviewer_jXk1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1. The topic is interesting and the papers pose some unique observations after extensive empirical analysis and experiments\n2. The paper defines several simple mathematical and statistical metrics to measure the differences between representations.\n\nWeakness：\n1. The posed questions are not well addressed\nThe paper shows some observations but did not provide a concrete and reasonable solution such that the long-tailed classification issue can be addressed. The useful insight from this paper is limited.  \n\n2. The empirical observations are not solid and rigorous \nThe paper only provides some simple metrics but did not explain why the metric is necessary and what's the high-level intuition. I did not get the motivation why the authors come up with these metrics to show the differences between representation. In addition, there is no rigorous mathematical proof or statistical analysis.\n\n3. Lack of careful related work discussion \nThe related work section is hard to follow and the authors did not explain their contributions and differences from existing works.  \n\n4. The writing needs to be improved. \nMany typos result in additional difficulties to read.   The citation format is not consistent. For example,  Cui et al.. in section 3 does not have a   year, and \"He et al (2015)\" followed by Resnet-32 should be \"(He et al, 2015)\".  The \"google enough\" and \"Normal\" in the abstract should be corrected. \n\n\n", "summary_of_the_paper": "This paper poses an interesting and important question - where are the bottlenecks in long-tailed classification. The authors use empirical experiments to show their observations: (1) representation is more critical than classifier, (2) data augmentation is helpful.  Three datasets (CIFAR-10 LT, CIFAR-100 LT and ImageNet-LT) are employed to work with ResNet-32 and ResNet-10 models to demonstrate their observations.  ", "summary_of_the_review": "An overall feeling is that the paper is an ongoing work and needs to be carefully written and improved. My recommendation is to reject it in the current form.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the role of representation learning when the distribution over the feature space has a long tail. The main motivation is to determine how much of the overall learning, in this case, is bottlenecked specifically by representation learning. The main findings are that vanilla learning gives brittle long-tailed representations, harming overall performance. The paper suggests a form of data augmentation to remedy this. Reviewers acknowledge that this investigation is worthwhile. However, many concerns were raised as to whether experiments support the drawn conclusions. A more principled approach to the data augmentation methodology is also needed. The authors address some of these, providing further experiments, but these were not enough to sway reviewers. Since results are fundamentally empirical in nature, this shortcoming indicates that the paper is not ready to share with the community just yet. Stronger experiments with clearer evidence are needed to fully support the thesis of the work.
Decision: Reject