Exploring Balanced Feature Spaces for Representation Learning
Authors: Bingyi Kang
Yu Li
Sa Xie
Zehuan Yuan
Jiashi Feng
Venue: ICLR 2021 Poster (2021)
Keywords: Representation Learning
Contrastive Learning
Long-Tailed Recognition

Abstract: Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artificially balanced datasets (e.g., ImageNet). It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t. the classes. Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one. Our findings are quite intriguing. Different from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. This motivates us to explore the balanced feature spaces learned by contrastive learning, where the feature representations present similar linear separability w.r.t. all the classes. Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings. Inspired by these insights, we develop a novel representation learning method, called $k$-positive contrastive learning. It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced. Extensive experiments demonstrate its superiority on multiple recognition tasks. Remarkably, it achieves new state-of-the-art on challenging long-tailed recognition benchmarks. Code and models will be released.

Published as a conference paper at ICLR 2021
EXPLORING BALANCED FEATURE SPACES FOR REP- RESENTATION LEARNING
Bingyi Kang', Yu Li Zehuan Yuan', Jiashi Feng'
National University of Singapore, "Institute of Computing Technology, CAS, 'ByteDance Al Lab kang@u. Long@u.nus.cem n edu, liyu@ict. en, cww..uaanneeuuunnbeeeeneee.e com, elefjia@nus edu. sg
ABSTRACT
Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artificially balanced datasets (e.g. ImageNet). is un- clear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t. the classes. Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to long-tailed one. Our findings quite intriguing. Dif- ferent from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heav- ily imbalanced. This motivates us to explore the balanced fettineceefaauuree spaces learned by contrastive learning, where the feature representations present similar linear separability w.r.t. all the classes. Our further experiments reveal that a represen- tation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings. Inspired by these insights, we develop novel representation learning method, called k- k-positive contrastive learning. It effectively combines strengths of the supervised method and the con- trastive learning method to learn representations that are both discriminative and balanced. Extensive experiments demonstrate its superiority on multiple recog- nition tasks, including both long-tailed ones and normal balanced ones. Code is available https: //gi thub. com
INTRODUCTION
Self-supervised learning (SSL) has been popularly explored as it can learn data representations with- out requiring manual annotations and offer attractive potential leveraging the vast amount of unla- beled data in the wild obtain strong representation models (Gidaris et al., 2018; Noroozi & Favaro, 2016; Hc et al., 2020; Chen et al., 2020a; Wu et al., 2018). For instance, some recent SSL meth- ods (Hénaff et al., 2019; Oord et al., 2018; Hjelm et al., 2018; He al., 2020) use the unsupervised contrastive loss (Hadsell et al., 2006) to train the representation models by maximizing the instance discriminativeness, which are shown to generalize well across various downstream tasks, and even surpass the supervised learning counterparts in some cases (He et al., 2020; Chen et al., 2020a).
Despite the great success, existing SSL methods focus learning data representations from the artificially balanced datasets (e.g. ImageNet (Deng et al., 2009)) where all the classes have similar numbers of training instances. However in reality, since the classes in natural images follow the Zipfian distribution, the datasets are usually imbalanced and show long-tailed distribution (Zipf, 1999 Spain & Perona, 2007), i.e., some classes involving significantly fewer training instances than others. Such imbalanced datasets are very challenging for supervised learning methods to model, leading to noticeable performance drop (Wang et al., 2017; Mahajan et al., 2018; Zhong et al., 2019). Thus several interesting questions arise: How well will SSL methods perform on imbalanced datasets? Will the quality of their learned representations deteriorate as the supervised learning methods? Or can they perform stably well? Answering these questions is important for understanding the behavior of SSL in practice. But these questions remain open as no research investigations have been conducted along this direction so far.

---

Published as a conference paper at ICLR 2021
Supervised CE loss
Contrastive Logs
K posilive Contrastive Loss
Feature spaces learned with different losses given an imbalanced dataset. The supervised cross- entropy (CE) learns space biased to the dominant class. The space learned by unsupervised contrastive loss balanced but less semantically discriminative. Our proposed k-positive contrastive loss learns a balanced and indicates the decision boundary of each class.
Figure 1:
discriminative fcature space. The shadow area
Our work is motivated by the above questions to study the properties of data representations learned with
methods in a practical scenario. We start with two representative losses used by these methods, i.e., the supervised cross-entropy and the unsupervised contrastive losses (Hadsell et al., 2006; Oord et al., 2018), and investigate the classification performance of their trained representation models from multiple training datasets where the instance distribution gradually varies from a balanced one to long-tailed one. We surprisingly observe that, different from the ones learned from supervised cross-entropy loss where performance drops quickly, the representation models learned from the unsupervised contrastive loss perform stably well, no matter how much the training instance distribution is skewed to be imbalanced. Such a stark difference stably.
between the two representation learning methods drives us explore why SSL performs We find that using the contrastive loss can obtain representation models generating a balanced feature space that has similar separability (and classification performance) for all the classes, as illustrated in Figure 1.
Such balanced property of the feature spaces from SSL is intriguing and provides new perspec- tive to understand the behavior of SSL methods. We dig deeper into its benefits via a systematic study. In particular, since pre-trained representation model often used initialization for down- stream tasks (He et al., 2020; Newell & Deng, 2020; Hénaff et al., 2019), we evaluate and compare the generalization ability of the models that produce feature spaces of different balanced levels (or 'balancedness'). We find that more balanced model tends to generalize better across variety of settings, including the out-to-distributton recognition as well as the cross-domain and cross-task applications. These studies imply that feature space balancedness is an important but often neglected factor for learning high-quality representations.
Inspired by the above insights, we propose new representation learning method, the k-positive con- strastive learning, which inherits the strength of constrastive learning in learning balanced feature spaces and meanwhile improves the feature spaces' discriminative capability. Specifically, different from the contrastive learning methods lacking semantic discriminativeness, the proposed k-positive constrastive method leverages the available instance semantic labels by taking k instances of the same label with the anchor instance to embed semantics into the contrastive loss. As such, it can learn representations with desirable balancedness and discriminativeness (Figure 1). Extensive ex- periments and analyses clearly demonstrate its superiority over the supervised learning and latest contrastive learning methods (He et al., 2020) for various recognition tasks, including visual recog- nition in both tailed setting (e.g., ImageNet- LT, iNaturalist) and balanced setting.
This work makes the following important observations and contributions. (1) We present the first systematic studies on the performance of self-supervise contrastive learning on imbalanced datasets which are helpful to understanding the merits and limitations of SSL in practice. (2) Our studies re- veal an intriguing property of the model trained by contrastive learning the model can robustly learn balanced feature spaces-that has never been discussed before. (3) Our empirical analysis demonstrates that learning balanced feature spaces benefits the generalization of representation mod- els and offer a new perspective for understanding deep model generalizability. (4) We develop a new method to explicitly pursue balanced feature spaces for representation learning and it outperforms the popular cross-ente cross-entropy and contrastive losses based methods. We believe our findings and the novel k-positive contrastive method are inspiring for future research representation learning.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORKS
Self-supervised learning is a form of unsupervised learning. Recently there has been a surge of self-supervised data representation learning methods developed alleviate the demand for manual annotations by mining free supervision information through specifically designed loss functions and pretext tasks. The contrastive loss measures the similarities of sample pairs in feature space and is at the core of several recent SSL methods (Chen et al., 2020a;b; He et al., 2020; Chen et al., 2020c). Adversarial losses that measure the distribution difference are also exploited for self-supervised rep- resentation learning (Donahue et al., 2016; Doersch & Zisserman, 2017). A wide range of pretext tasks have been developed including image inpainting (Jenni Favaro, 2018; Pathak et al., 2016), image colorization (Larsson et al., 2016; 2017), context prediction (Doersch et al., 2015), jigsaw puzzles (Carlucci et al., 2019; Noroozi & Favaro, 2016; Wei et al., 2019), rotation prediction (Gi- daris et al., 2018). Though very successful, the behavior of SSL largely remains mystery. Recently Wang & Isola (2020) analyze contrastive learning from the perspective of uniformity and alignment of learned representations. However investigations balanced datasets are still absent. We present the first study on this problem and our investigation methodology is also applicable other SSL methods.
the behavior of contrastive learning on im-
In practice, the visual data usually follow a long-tailed distribution (Zipf, 1999; Spain & Perona, 2007), challenging supervised learning methods. Due the imbalance in the number of training instances for different classes, conventional methods tend to perform much more poorly on instance- rare classes than on instance-rich ones. To alleviate this performance bias, existing approaches either re-balance the data distribution through sampling (Chawla et al., 2002; Han et al., 2005; Shen et al., 2016; Mahajan al., 2018) or the loss for each class (Cui et al., 2019; Khan et al., 2017; Cao et al., 2019; Khan et al., 2019) by reweighting. Kang et al. (2020) first propose to decouple representation learning from classifier learning to boost performance, and demonstrate that learning good feature spaces is crucial for long-tailed recognition. Along this direction, SSP (Yang & Xu, 2020) is among the first methods that introduce SSL pretraining into learning the long-tailed recognition models. More specifically, instead of directly training randomly initialized model from scratch as conventional supervised learning methods, SSP uses a model pretrained with SSL on the same dataset for initialization, which is observed be able to alleviate the label bias issue in imbalanced datasets and boost long-tailed recognition performance.
In contrast, we conduct a series of systematic studies to directly compare SSL with supervised learning on representation learning. We show that SSL can learn stably well feature spaces robust to the underlying distribution of a dataset. Moreover, inspired by our findings on the benefits of a balanced feature space for generalization, we introduce the k-positive contrastive learning method to explicitly pursue balancedness and discriminativeness for representation learning, which has been shown through experiments to benefit not only long-tailed recognition but also normal recognition tasks.
BALANCED FEATURE SPACES FROM CONTRASTIVE LEARNING
In this section, we systematically study the performance of representation models trained by SSL from a collection of training datasets with varying instance number distributions, in contrast with the models learned by supervised learning methods, to explore how SSL performs when the training datasets are not artificially balanced. Furthermore, we investigate the generalization performance of these learned representation models under multiple settings, in order explore the relationship between the representation model's 's generalizability and the property of its learned feature space.
Notations We define the notations used in this paper. Representation learning aims to obtain a representation model fo that maps sample into a feature space V such that its corresponding representation is E V encapsulates desired features for target applications. Let D Drep-tmin = {x,, y4 ) i = 1,...,, N be the dataset for training the representation model, where y; is the class label for sample r; Let C denote the number of total classes and n, denote the number of instances within class j. We use (g1.....4c' with q; n to denote the discrete instance distribution over the C' classes. An imbalanced dataset has significant difference in the class instance numbers, e.g.,
q > gc. We use multi-layer convolutional neural network fo(+) : Ia + o; to implement
3

---

Published as a conference paper at ICLR 2021
the representation model. The final classification prediction is given by linear classifier arg max[W + b]. where W denotes the classifier weight matrix and b denotes the bias term.
3.1 METHODOLOGY OF OUR STUDY
Representation learning methods Various loss functions have been developed for learning the representation model fe on the training dataset. Among them, the most popular one is the supervised cross-entropy (CE) loss:
N Σ N
1 LCE
log Pyr
where Py. softmax(W, V/ + b) is the normalized probability prediction of sample i belonging to its ground truth class 74- Using the semantic labels directly as supervision signal (y: in Equation I), the representation model trained by the CE loss can have strong semantic discrimination ability but its generated feature space is easily biased by the imbalance the training instance distribution-if some classes have significantly more training instances than the others, their data representations will occupy dominant portion of the feature space (Figure 1) and get higher classification accuracy than the instance-rare classes (Kang et al., 2020; Wang et al., 2017).
y,
Different from the supervised learning ones, self-supervised learning methods adopt semantic-free loss functions to learn representations from unlabeled data (He al., 2020; Gidaris et al., 2018). For example, the contrastive loss' (CL) (Oord et al., 2018) learns representations via maximizing the instance-wise discriminativeness::
1 N
exp(U: - vt /T) exp(e: of /r) + En EV- exp(u: - 2 /T)*
CCL
log
where 7 is a temperature hyper- hyper- paremeeerreete v; is a positive sample for the anchor instance i (typically produced by data augmentation), v/ E V is the negative sample randomly drawn from the training samples excluding instance This contrastive loss encourages the feature representations from positive pairs to be similar, while pushing features from the sampled negative pairs apart.
We take the two loss functions (LeE and fcI.) as representatives to study how the representation models (and the corresponding feature spaces) trained with supervisedd/sll-sspervvi methods are affected by the training instance distribution [c.]:
Balancedness of feature spaces Since semantic labels are not involved in the contrastive loss (Equation 2), hypothesize may lead representation models yielding feature spaces that less biased by the imbalance of the training dataset, compared with the ones from the supervised loss (Equation 1). To verify this, we introduce a metric to characterize such an "unbiased" or "balanced" property of feature space at first. A feature space V is balanced if the representations {2;} from different classes within have similar degrees of linear separability. As the linear separability degree of the representations is usually evaluated by the accuracy of linear classifier over them (Vapnik, 2013), we follow this criterion to develop the balancedness metric. Specifically, let a1...., ag denote the classification accuracy of a linear classifier (W, b) over the representations {wr} C V from C' classes. We take the following uniformity of these accuracies as the balancedness of the feature space V:
C exp
a,1²) la=); σ
|a;
_.p²).
where a, #(u, = j.
j. =.
B(V)
A
(3)
C
#( E
Here is fixed scaling parameter. This metric achieves its maximum when all the class-wis: accuracies are equal, i.e., there being no separability bias the learned representations to any class. Note that this metric developed provide a quantitative measure of the balancedness of a feature space, but it has certain limitations such as it can be easily hacked. We leave developing more rigorous metrie that can better characterize balanced feature spaces as future work.
'The term of contrastive loss has heen used to refer to various loss functions over positive and negative samples. This work focuses on the specifie form in Equation 2 that widely used in modern SSL methods.
4

---

Published as a conference paper at ICLR 2021
Overall Performance
Visual zodon of Balancing Degree
Topl
Top5
50.6
27.2
77.0
761
Accuracy (ho
Accuracy (%)
70.0
CE
74.7
CR
CL
LTO LT2
LT
LTQ LTG
LTB
LT0 LT2 LTQ
LT6
LTU
LT
L10 172
L14
LT0
L18
LT
Classification accuracy (left) and bal- ancedness (right) of the representations learned from cross-entropy (CE) loss and contrastive loss (CL) datasets (LTTO to LTT) with increasing imbalance.
Figure 2:
Figure 3: Nut-of-distribution generalization on Ima- geNet. Top and Top testing accuracy the model are learned from datasets LTO to that are increas- ingly more imbalanced.
Experimental protocol We adopt a multi-stage protocol for learning and evaluating the feature spaces. (1) Representation learning: pre-train the representation model fo on the provided training set Drep-trai using the above training losses LCR and La. (2) Classifier learning: train a linear classifier (W, b) on top of fo with A fixed using another training dataset Dimin' and the supervised CE loss; (3) Representation evaluation: evaluate the classification accuracy of the learned classifier on the test dataset D test with the representations from fe and compute the above balancedness, 3(V).
To thoroughly investigate sensitiveness of different representation learning methods to the imbal- ance level of training datasets, we construct six datasets from the long-tailed benchmark ImageNet- LT (Liu et al., 2019) (Dur) by varying its instance distribution (9I..... 9c) from a to uniform
one gradually, while keeping the total instance number similar. The generated datasets, denoted as Diro..... Drrs. DLr (which are increasingly more imbalanced), are used as Drep train for representation learning in the following experiments. See appendix for their details.
3.2 CONTRASTIVE LOss HELPS LEARN BALANCED FEATURE SPACES
We first investigate classification performance of the representation models trained with the CE and CL losses on the above six datasets D DLTS. Dur that are increasingly more imbalanced. Since linear classifiers are easily biased by skewed training dataset distribution (Kang et al., 2020), it is necessary to eliminate the imblancedness of the evaluation datasets for reliable representation evaluation. Thus, we use the (balanced) training and test sets of ImageNet as Durin and Dieat to learn classifiers and evaluate their classification accuracy, following the above protocol.
The results are summarized in Figure 2, from which we make an important observation: compared with the supervised cross-entropy loss, the model trained with the unsupervised contrastive loss generates a more balanced feature space, even in presence of highly imbalanced training instance distribution. As shown in Figure 2 (left), the classification accuracy of representation models learned with the CE loss drops quickly when the dataset becomes more imbalanced- quality of repre- sentations from these models is very sensitive the imbalance of training datasets. In contrast, the classification accuracy of CL-trained models remains stable even when the training dataset transits heavily long-tailed one. Such surprising performance robustness imbalance of the training datasets implies that using contrastive learning can consistently learn balanced feature spaces. To see this, we also visualize the balancedness scores (Equation 3) the Iearned feature spaces from CE and CL in Figure 2 (right). Even when the training set is heavily long-tailed, the feature spaces learned with CL loss are highly balanced as the ones learned from uniform training distribution, while the blancedness score of the feature spaces from CE loss is lower and drops quickly. Such balanced feature space offered by CL loss much desired for the representation learning prac- tice, where the training instance distribution is usually long-tailed. Certainly, using an unsupervised loss will sacrifice semantic discriminativeness of the representations, leading to the accuracy gap between the CE and CL-trained models.
3.3
MORE BALANCED REPRESENTATION MODELS GENERALIZE BETTER
The above studies reveal that the representation models trained with the contrastive loss can produce more balanced feature spaces. A natural question is what are the benefits from a balanced model for
Note Drnin and D Drip he the same dataset in recent SSL works (e.g., He al. (2020)).
5

---

Published as a conference paper at ICLR 2021
Table I: Results on Places365, VOC and COCO, APse is the default metric for VOC, while AP and AP denote the bounding-bor -ox and mask AP for COCO respectively. Black / gray numbers correspond to results of the representation models trained on ImageNet-LT ImageNet respectively. See appendix for complete results.
crosy-domain
cross-task
Places365 (Topl)
VOC (APsi)
COCO (AP"))
COCO (AP*))
CB
38.50 / 46.06
76.45 / 81.26
38.13/ 40.08
33.29 / 34.85
CL
41.24 / 46.16
78.19 / 82.28
39.67 / 40.41
34.73 / 35.14
ÁCI.CE
+2.74 40.10
+1.64/ +0.02
+1.54/ 40.33
+1.44 / +0.29
recognition? Since a pre-trained representation model is often used to facilitate downstream tasks (He et al., 2020; Newell & Deng, 2020; Hénaff' et al., 2019), we here conduct extensive experiments to study its potential benefits on model generalization performance under the following settings.
Out-of-distr generalization We first study the relationship between the balancedness of representation models and their generalizability for recognizing new classes. To thoroughly evalu- ate performance of representation models with different balancedness, we evenly divide the 1,000 classes into two splits (500 vs. 500 classes) on ImageNet, referred to as the source and tar- ger split respectively. We use the subsets (corresponding to the source class split) of the above Drm...., DITT. Dir datasets to construct six different D repp-min for training the representation model So. To obtain models with different balancedness, we use the CE loss for training, since the above studies reveal using the CL loss will always produce models with similar balancedness (Figure 2). We use the subsets (corresponding to the target class split) of the training and test sets of ImageNet as Dirain and Diea for classifier learning and evaluation, with the representation model fixed.
The testing performance of the models with different balancedness on the target classes is presented in Figure 3. It is observed that the source dataset becomes increasingly more imbalanced (from DuuU to Dur) and the corresponding models become more imbalanced, their generalization perfor- mance degrades correspondingly. Such positive correlation between balancedness of the models and testing accuracy on the target classes clearly demonstrate that more balanced representation models tend to generalize better for recognizing unseen classes. More details and results about the out-of out-of-disttre distriitriutonnn generalization studies are deferred to the appendix.
Cross-domain and cross-task generalization We then explore whether learning balanced repre- sentation models is able to benefit model's generalizability to new domains and tasks. We use the ImageNet-LT as Deep-trin to train the models with the CE and CL losses, obtaining imbalanced and balanced representation models respectively. For the cross-domain setting, we train linear classifier on the Places365 dataset (Zhou et al., 2017) with the representation model fixed. From the results in Table 1, it can be clearly observed that the balanced representation model (from CL) surpasses the less balanced one (from CE) significantly, in terms of the top-1 accuracy (by 2. 2.74%). For the cross- take train/test splits of the PASCAL VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) datasets as Dust for evaluating detection performance. The results are given in Table 1. Again, the balanced model (from the CL loss) outperforms the less balanced one (from the CE loss) significantly (up to 1. 1.64%). In comparison, the improvement from the CL-trained model over the CF--trained model moderate (around 0.3%) when using the full ImageNet for training, as CE can learn relatively balanced feature space from balanced dataset. This clearly shows that the generalization performance boost for the cross-domain and cross-task settings brought from CL-trained models does not simply stem from using self-supervised pre-training, but indeed come from learning more balanced feature spaces.
task setting,
LEARNING BALANCED FEATURE SPACES FOR RECOGNITION
The above studies demonstrate the representation models trained with the contrastive loss can gen- erate balanced feature spaces showing strong generalizability. Here explore how to effectively leverage these findings in practice. We introduce a new method that inherits the strength of the contrastive loss in learning balanced feature spaces and enhances the feature spaces' semantic dis- crimination capability simultancously. We thoroughly study its superiority via two application cases, Le., the long-tailed recognition and pre-training representation models for downstream tasks.

---

Published as a conference paper at ICLR 2021
4.1 K-POSITIVE CONSTRASTIVE LOSS
Though balanced, the feature spaces from contrastive learning have limited capability of semantic discrimination, as shown in Figure 2 (left). This is because the contrastive loss blindly encourages instance-level discriniantiveness. Every two instances, even if they are from the same class, are forced to be apart from each other in the learned feature space. To embed semantic discriminative- ness into the representations while maintaining desired balancedness, we develop new method to leverage the provided semantic labels to adaptively compute the instance contrastive loss.
Concretely, given an anchor training instance with its semantic label Vi. our proposed method draws & instances from the same class form the positive sample set V instead of only using its augmentation as in Equation 2. Thus, gives new loss called k- k-positive contrastive loss (KCL):
1 LKCL= N(K + 1)
Σ log
exp(u: 23 /T)
(4)
exp(1: - i//T) + +., exp(e; ((ll)uu/T)))) uy/T)²
of offECCJJVtt
where 0g generated by augmenting U4 Vi the current batch of examples excluding 8is and VA C V; is positive set containing k instances randomly drawn from the same class as uy The proposed KCL loss purposely keeps the number of positive instances equal, which is crucial for balancing the learned feature spaces. It brings two benefits. First, helps learn representations with stronger discriminative ability as it leverages the label information as supervised learning. Secondly, uses the same number of instances (i.e., k) for all the classes in positive pair construction which further balances the learned feature space. Note our proposed KCL. is different from the supervised contrastive learning (Khosla et al., 2020) that leverages all the instances from the same class to construct the positive pairs, which cannot avoid the dominance of instance-rich classes in the representation learning. This is also evidenced by our following experiments on long-tailed recognition. In the following experiments, choose i = 6 via validation and use ResNet50 as the backbone. Other hyper-parameter choices and implementation details are given in the appendix.
4.2 LONG-TAILED RECOGNITION
KCL provides feature spaces with desirable balancedness and semantic discriminativeness, which makes it naturally fit for addressing the challenges of long-tailed recognition, i.e., severe perfor- mance bias to the instance-rich classes and poor generalization the instance-rare classes (Mahajan et al., 2018; Zhong et al., 2019). Here we implement and evaluate KCL for long-tailed recognition, following the two-stage training strategy from Kang et al. (2020): 1) train the representation model with the KCL loss; 2) learn a linear classifier with cross-entropp loss and claas-balanced sampling.
Baselines Besides well established state-of-the-aars,, we consider following three kinds of base- lines for justifying the advantages of KCL. (1) Classifier balancing methods, i.e., 7-norm and cRT (Kang et al., 2020), that re-train classifiers with class-balanced sampling as KCL but learn the rep- resentation models by supervised cross-entropy loss. Comparison with them helps understand the effectiveness of learning balanced features in long-tailed recognition.
(2) Methods that train the rep- resentation model and classifier jointly with cross-entropy loss (SL) and various data re-sampling strategies, including instance-balanced (SL-i), class-balanced (SI--c), progressively-balae -balancy (SL.-p) and square-root re-sampling (SI--) (Kang et al., 2020). Comparison with them will show advantages of KCL over these data-enriching strategies in feature space balancing. (3) A fuull-positive variant of KCL, named full-positive contrastive learning (FCL), that uses all the available same-class samples in the current batch to construct positive pairs for computing the contrastive loss, which is similar to the supervised contrastive learning (Khosla al., 2020). Comparing KCL with FCL will show the benefits of keeping the number of positive samples equal for all the anchor instances in KCL.
Results We evaluate KCL and compare it with the above strong baselines on two large-scale benchmark datasets, ImageNet-LT (Liu et al., 2019) and iNaturalist 2018 (iNatrualist, 2018). For comprehensive evaluation, following (Liu et al., 2019), we split the classes of ImageNet-LT into many-shot (> 100 images), medium-shot (20~100 images) and few-shot (<20 images) groups. The results are summarized in Tables 2 and respectively, along with the balancedness of these methods on ImageNet-LT in Figure 4. We make the following observations.
More balanced feature spaces give better performance. We first compare KCL with cRT and T- norm, the latest state-of oftheoo-the-arsss with feature spaces learned by supervised cross-entropy loss and
7

---

Published as a conference paper at ICLR 2021
Table 2: ImageNet-LT results
Table 3:
iNaturalist 2018 results
Method
Many
Medium
Few
Method
Top1
OLTR (Liu et al., 2019)"
35.8
32.3
21.5
32.2
CB-Focal (Cui et al., 2019)
61.1
Joint (SL-i) (Kang al., 2020)
64.9
35.2
6.8
42.5
LDAM (Cao et al., 2019)
64.6
7-norm (Kang et al. 2020)
56.6
44.2
27.4
46.7
LDAM+DRW (Cao al., 2019)
68.0
cRT (Kang et al., 2020)
58.8
44.0
26.1
47.3
cRT (Kang al., 2020)
65.2
7-norm (Kang al., 2020)
65.6
FCI. KCL
61.4
47.0
28.2
49.8
BBN (Zhou al., 2020)
66.3
61.8
49.4
30.9
51.5
FCL
66.4
'Reproduced by re-running their code with ResNet50.
KCL
68.6
Balancrdness(%)
Accuracies ImageNet-LI
Figure 4: Comparison of different methods on their feature space balancedness (left) and class-wise accuracy (right). Here the linear clas- sifier is fine-tuned on the full ImageNet for representation evaluation (see Sec. 3.2).
(96) Saraaa
MoCo
100
KCL
Class Index
thus less balanced as demonstrated in Sec. 3. Compared with them, KCL improves the overall accu- racy by a large margin (4.2% on ImageNet-LT and %% on iNaturalist), demonstrating the importance of learning more balanced feature spaces for long-tailed recognition.
KCL more effective at learning balanced and discriminative feature spaces. Data re-sampling is widely used straightforward approach alleviate performance bias for long-taill long-tailed recognition (Kang et al., 2020). We compare the feature space balancedness of KCL and SL methods with different data re-sampling strategies ImageNet-LTT in Figure 4. Clearly, data re-sampling cannot effectively improve balancedness of the feature space as KCL. Besides data re-sampling, Figure also shows the balancedness of the feature spaces learned by the latest contrastive learning method MoCo (He et al., 2020) on ImageNet-LT, MoCo can balance the feature space but has lower accuracy, due to the lack of semantic discriminativeness in the learned feature space. KCL performs the best, demonstrating its effectiveness at learning both balanced and discriminative feature spaces.
Equalizing the number of positive instances in KCL is important. To further justify the design of KCL loss in keeping the number of positive instances to be equal, we compare it with its variant FCL. From Tables 2, and Figure 4, though FCL. outperforms other baselines, its performance is inferior to KCL, in terms of both the overall accuracy and the balancedness of the learned feature spaces. Equalizing the number of positive instances as KCL is crucial for learning balanced feature spaces and improving recognition performance.
4.3 PRE-TRAINING REPRESENTATION MODELS FOR DOWNSTREAM TASKS
The effectiveness of KCL is not limited the cases where training datasets imbalanced. In this section, we study KCL as general representation learning method, i.e., we apply KCL for pre- training a representation model on balanced datasets which is later fine-tuned for downstream tasks, including the
(OOD) recognition and detection.
Out-of-distre distribution Generalization Similar to Sec.3.3, we evenly divide the 1000 classes in Im- ageNet into two splits, use one split to learning representation backbone and the thoer one to learn a linear classifier with the backbone fixed. We adopt two different splitting strategies. Split-overlap (split with semantic overlap) allows the classes within the two splits to share the same super class (e.g., dog and wolf from canidae are put into different splits) in the ImageNet ontology. As such, though the target classes are all novel to the representation model, some of their attributes have been seen by the model before from the source classes. In contrast, Split- intii-ddedeedent (split without se- mantic overlap) strictly avoids classes from the same super classes to be distributed into different presents more challenging case for model's generalization ability as all the target classes (and attributes) recognize are novel.
splits.
8

---

Published as a conference paper at ICLR 2021
Table 4: OOD generalization results (top-1 accuracy) on balanced datasets. We use source classes of origninal ImageNet to learn representation network (ResNet50), and use the target classes and all classes respectively to learn linear classifiers for evaluation.
Split-overlap
Split-independent
source
target
all
source
target
all
CE
81.2
70.7
67.2
82.8
50.3
62.4
CL
67.0
60.1
58.3
68.2
54.8
58.2
KCL
81.4
74.8 (+4.1)
70.8 (+3.6)
83.2
58.1 (43.3)
67.2 (+4.8)
The generalization performance comparsion of the representation models with different methods given in Table 4. When the source classes and target classes share similar semantics (on split- overlap), the CE-trained model surpasses the CL-trained model on both the source and target classes. But when looking into the generalization gap (i.e., the difference between the source and target accuracy), the CL-trained model suffers larger generalization gap than the CL-trained model (10.5 vs. 1.8). When there is not semantic overlap between source and target (on split-independent), the CL-model outperforms CE-model on the target classes by 4.3% with much smaller generalization gap (13.4 v.s. 32.5). By comparing the "full" performance from split-overlap to split- pplit-independent, one can observe that CL loss performs consistently well (58.3 and 58.2), but CE drops as large as 5%. This implies that CL. is robust to imbalanced training distribution used for representation learning, while CE is extremely sensitive to it. These results clearly demonstrate the consistent superiority of balanced representation learning in terms of generalization for various training dataset distribution. Notably, our proposed KCL loss surpasses both CE and CL loss on all the four different settings by large margin (more than 3 points). These results clearly demonstrate that KCL is able to learn balanced and discriminative feature space, and balancedness is a general property that benefits both balanced and imbalanced datasets.
Table 5: Comparison of different representation learning methods for the downstream tasks.
repr
VOC (APs))
COCO (AP"))
COCO (Ap
SL
76.6
81.26
40.08
34.85
MoCo
60.6
81.28
40.41
35.15
KCL 76.8
82.32
40.79
35.45
Cross-domain and cross-task generalization In this part, we first pretrain model on ImageNet then further finetune it for downstream object detaction tasks (including PASCAL VOC and COCO). representation learning method, rather than
Note aim to study the generalizability of KCL aiming at state-of-the-art performance. Hence we compare with the vanilla supervised cross- entropy loss (SL.) and MoCo (which the KCL. is built on) (He et al., 2020). The results are summa- rized in Table 5. We also evaluate the discrminativeness of the learned representations (the "repr" in the table) from their classification accuracy by learning linear classifier on the pretraining datasets. Clearly, KCL outperforms SL and MoCo for both downstream tasks. This is because KCL learns more balanced feature spaces than SL with similar discriminativeness, and learns more discrimina- tive features than MoCo. For more results, Please refer to the appendix.
CONCLUSIONS
This work piloted studies on performance of the self-supervised learning methods for imbalanced datasets, and made several intriguing findings. At the heart of these findings is the balanced feature space, which is identified to be an inherent property of the representations learned by the contrastive learning and bring stronger generalizability. It provides new perspective for understanding the behavior of the contrastive learning. This work further developed a new representation learning method to leverage the benefits of balanced feature spaces. We believe the findings and method developed here are inspiring for the future research on representation learning. However, theoretical understandings balanced feature spaces are not mature yet and worthy of future exploration.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENT
We would like to express our deepest gratitude to Saining Xie for his comments and suggestions throughout this project and the writing of the paper, to Yu Sun for his insightful discussion the begining of this project.
REFERENCES
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-disributtionawwwss margin loss. In Advances Neural Information Processing Systems, 2019.
Fabio M Carlucci, Antonio D Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2229-2238 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 20Xx::2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint wrXiv:2003.0429) 2020c.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pp- 1422-1430, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1rXiv:1605.09782. 2016.
Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journaa of computer vision, 88(2): 303-338, 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0778 2018.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR'OL (.))) volume 2, pp. 1735-1742. IEEE, 2006.
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference intelligent computing, pp. 878-887. Springer, 2005.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint FrXXiv:9055002722 2019.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1xx:::808.06000 06670, 2018.
Karan Grewal, Phil Bachman, Adam
iNatrualist. The inaturalist 2018 competition dataset. https: thub. com/visipeddaa/int
htmp:/tree/master22000 2018.
Simon Jenni and Paolo Favaro. Self-supervised feature learning by learning to spot artifacts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2733- 2742, 2018.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-3587, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. / 1362, 2020.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European conference computer vision, pp. 577-593. Springer, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6874-6883, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European Conference on Computer Vision (ECCV), pp. 181- 196, 2018.
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7345- 7354, 2020.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 007448 2018.

---

Published as a conference paper at ICLR 2021
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pp. 467-482. Springer, 2016.
Merrielle Spain and Pietro Perona. Measuring and predicting importance of objects in our visual world. 2007.
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprint arXiv:2005. /0242, 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Chen Wei, Lingxi Xie, Xutong Ren, Yingda Xia, Chi Su, Jiaying Liu, Qi Tian, and Alan L Yuille. Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsu- pervised representation learning. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1910-1919, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving dass-imbalanced learning. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Yaoyao Zhong, Weihong Deng, Mei Wang. Jiani Hu, Jianteng Peng, Xunqiang Tao, and Yaohai Huang. Unequal-training for deep face recognition with long-tailed noisy data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1422 1464, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bhn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 9719-9728, 2020.
George Kingsley Zipf. The psycho-biology of language: introduction to dynamic philology, volume 21. Psychology Press, 1999.
12

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
Representation Learning For supervised cross-entropy (CE) loss, we adopt the standard PyTorch distributed training implementation' For unsupervised contrastive loss (CL), we use the official im- plementation" of MoCo (He et al., 2020) with default hyper-parameters. Our k-positive contrastive loss is implemented based on MoCo by randomly selecting k positive examples from the key mem- ory for each of the query example. When KCL is applied for a long-taile tailed dataset, there might be less than positive examples in the memory for some of the classes. In such cases, we use all the positive examples. Besides, we keep all the hyper-parameters (e.g., data augmentation, learning rate and batch size) the same as supervised learning. We only carefully tune the number of training epochs for KCL to make it achieve similar performance supervised learning a balanced dataset (full Imagenet) for fair comparison (see Table 6). As result, throughout the paper our KCL is trained for 200 epochs while its supervised counterpart is trained for 90 epochs. This is reasonable as contrastive learning usually takes much longer to converge (He et al., 2020). We are using = for KCL throughout the paper, which is carefully tuned the validation set of ImageNet-LT, as of different loss functions are given in Table 7.
shown in Fig. 5. The detailed
Table 6: Results on full ImageNet.
Top1
TopS
CE (90 epochs)
76.616
93.090
SCL (200 epochs)
76,976
92.972
KCL (200 epochs)
76.814
92.936
Accuracies ImageNet-LT (val)
hps
CE
CL
KCL
52
epochs
90
200
200
batch size
256
256
256
learning rate
0.1
0.03
50
learning rate schedule
cosine
step
cosine
49
data augmentation memory size encoder momentum feature dimension softmax temperature
default
moco v1
default
65536
65536
0.999
0.999
128
128
0.07
0.07
A-
6
Table 7: Elyper-paramedd used by different loss functions. "default" means the standard data augmentation strategies used by supervised lcarning.
Figure 5: Validation accuracies of KCI. on ImageNet-LT as the value of & varies.
Classifier Learning We need to train linear classifiers in two cases. 1) representation evalauation (Sec. 3 and Figure 4). For CE-learned representations, we train a linear classifier using the same parameters as representation learning in Table with smaller number of epochs (10). For CL and KCL, we adopt the classifier training protocol introduced by MoCo (He et al., 2020) with default hyper-parameters (i.e., learning rate 30 and weight decay 0). 2) Long-tailed recognition (Sec. 4.2). We obtain re-balanced classifier for CE representations following Kang et al. (2020), and adopt the MoCo classifier learning strategy with class balanee sampling by setting the learning rate to 10 on ImageNet-LT and 30 on iNaturalist 2018
Detection model training We use exactly the same setting and evaluation metrics He et al. (2020). R50-C4 backbone is used with BN tuned. The image is rescaled to [640, 800) during training and 800 inference. All layers are fine-tuned end-to-end with batch size = 16. For Pascal VOC, we train Faster R-CNN (Ren et al., 2015) on trainval07+1) set with 24k schedule and evaluate on test07 set. For COCO, we train Mask R-CNN (He et al., 2017) on train2017 set with x2 schedule and evaluate on va12017 set.
13

---

Published as a conference paper at ICLR 2021
B DATASET CONSTRUCTION
Datasets for studying balancedness of feature spaces We here explain the details on the con- struction of the series of datasets used in our study Sec. 3.
In particular, we take the standard taig-tilled training set from ImageNet-LT (Liu et al., 2019) whose instances follow the Pareto distribution as the base dataset, denoted PLr. We vary its training instance distribution (g1.....ec) (c) gradually to obtain different datasets as follows,
1
n2 ND X Σ.Sr o
(5)
where No is the total number of training instances in Dr., a E [0, controls the dataset balanced- ness. When = 0, it corresponds to fully balanced dataset; when 1, it becomes a heavy long-tailed ones. In total, we generated 6 datasets with a E 0.2 0.4, 0.6, 0.8, 1.0). denoted as PLTO.... ................................ DLTS Pur respectively, different examples of rrep-train for representation learning. The detailed statistics and visualization of the datasets Prio.... DLIS Drr are summarized Table and Fig. 6.
Dataset
Max
Min
Total
Duro
115
115
115,000
Pinstance
Dur
204
115,885
Dura
343
115,801
LTD
Duso
553
20
115,836
Dirs
857
10
115,852
17
Dir
1,280
115,846
Class Index
Table 8: Dataset statistics on training instance numbers, including the maxi- mal and minimal instance number per class and total number.
Figure 6: Training instance num- her distributions of the datasets we use in our empirical studies.
Datasets for generalizability studies We carefully choose the proper datasets to construct the Duain and Deet for evaluating generalizability the representation models under multiple settings. The choices summarized in Table 9.
Table 9: Summary on used datascts for representation model pre-training and evaluation in our studies.
study
Deepstaio
Duran
Drest
Balancedness (Sec. 3.2)
Diro. DLTS. Dur source split of above ImageNet-LT ImageNet-LT
ImageNet (train)
ImageNet (val)
-ut-of-distribution (Sec. 3.3)
target split above
Cross-domain (Sec. 3.3)
Places 365 (train)
Places 365 (test)
Cross-task (Sec. 3.3)
VOC/MSCOCO (train)
VOC/MSCOCO (test)
C ADDITIONAL RESULTS ON MODEL GENERALIZATION PERFORMANCE
Cross-domain and Cross-task Generalization We evaluate the generalization ability of the rep- resentation models trained on the balanced full ImageNet datasets, for cross -ross-domain and cross-task applications. The results are given in Table 10 (cross-domain) and Tables 11 and 12 (for detection) respectively. From Table 10, when the training datasets are balanced, the models trained with CL. and CE achieve comparable performance. While when the training datasets are not balanced, the CL model significantly outperforms the CE model (Table 1). This demonstrates that the CL loss can consistently produce balanced representation models and the model generalization performance indeed benefit from being balanced.
Similar conclusion can be drawn for the cross-task generalization. From Tables 11 and 12, when training the model on the full ImageNet dataset, using sulf-super CL loss can produce the model performing slightly better than using the supervised CE loss. On PASCAL VOC, the performance
14

---

Published as a conference paper at ICLR 2021
advantage is marginal 0.02% in 50 In contrast, when training the model the ImageNet- LT dataset, using CL loss can boost the model performance over using the CE loss much more significantly. The improvement is large 1.64% in APso- Thus the performance benefit on the generalization to detection brought by CL. does not simple stem from using self-supervised pre- training, but indeed come from learning more balanced feature spaces.
Similar to OOD generalization, the model learned with KCL gives better downstream performance on both VOC and COCO, which mean enforcing feature space balancedness with KCL is indeed able to help learning better representation.
Table 10: Results on Places 365. The encoder is trained on ImageNet.
Top1
Top5
CE
46.06
77.11
CL.
46.16 (+0.1)
76.27 (.084)
Table 11: Object detection Results on PASCAL VOC. The representation model trained on ImageNet and ImageNet-I.T. We report results in 5o VOC metric; AP: COCO-style metric.
ImageNet
ImageNet-LT
AP50
AP
AP
APso
AP
AP33
CE
81.26
53.66
59.19
76.45
48.53
51.01
CL
81.28 (+0.02)
56.10 (+2.44)
62.71 (+3.52)
78.19 (+1.4)
51.52 (+2.99)
56.48 (+5.47)
KCL
82.32 (+1.06)
55.51 (+1.85)
62.05 (+2.86)
79.70 (+3.25)
52.63 (+4.10)
57.89 (+6.88)
Object detection Results on COCO. The representation model trained on ImageNet and ImageNet- LT. We report results bounding-boo box AP (APth) and mask AP (APm)).
Table 12:
ImageNet
ImageNet-LT
AP
APs0
AP75
AP
APs0
AP75
Ap
CE
40.08
59.76
43.29
38.13
57.38
41.15
CL
40.41 (+0.33)
60.05 (+0.29)
44.09 (+0.80)
39.67 (+1.54)
59.40 (+2.02)
42.73 (+158)
KCL
40.79 (+0.78)
60.63 (+0.87)
43.99 (+0.70)
39.43 (+1.30)
59.08 (+1.70)
42.56 (+L41)
Apnk
CE
34.85
56.60
37.02
33.29
54.24
35,38
CL.
35.14 (+0.29)
56.88 (+0.28)
37.56 (+0.54)
34.73 (+1.44)
56.07 (+1.83)
37.13 (+175)
KCL
35,45 (+0.60)
57.40 (+0.80)
37.80 (+0.78)
34.38 (+1.09)
55.81 (+1.57)
36.39 (+101)
15


Reviews:
[{"review_id": "hGrTeZU2u7V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n- This paper made a key observation that the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. As for the method this paper proposes a classed-balanced version of supervised contrastive loss (Khosla et al., 2020). Extensive experiments demonstrate its superiority on multiple recognition tasks.\n\nPros:\n- The paper is well written with nice figures.\n- The paper conducted experiments on extensions other than image classification, which provides some useful insights.\n\nCons:\n- The advantage of using self-supervised learning to learn a representation to combat label bias issue in imbalanced problems has been discovered and validated by a previous work [1]. I understand that [1] was accepted just a few days before the DDL of ICLR. Unfortunately I still have to devalue the contribution of this paper as it seems to me that the major contribution of this paper relies on the finding that SSL can help to alleviate the issue of label bias. In terms of large-scale experiments on ImageNet-LT and iNatualist, it seems that this paper has no advantage over [1].\n- It seems the runtime of the proposed method is much worse than supervised based methods.\n\nAdditional Questions and Concerns:\n- Why does the author use cosine lr schedular?\n- How to interpret accuracys on imagenet in fig. 4? Why is there only 10 classes?\n\n[1] Yang, Yuzhe, and Zhi Xu. \"Rethinking the Value of Labels for Improving Class-Imbalanced Learning.\" NeurIPS 2020\n\n----\npost-rebuttal update\n\nI appreciate the discussions between the authors. I plan to keep my original score, for the reason that, at least in my point of view, the difference of the two methods is subtle and it is not clear whether the subtle difference results in drastic improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WczXyHM2dzC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview:** The paper presents experiments showing that the contrastive learning losses produce better embeddings or feature spaces than those produced by using binary cross-entropy losses. The experiments show that embeddings learned using contrastive learning losses seem to favor long-tailed learning tasks, out-of-distribution tasks, and object detection. The paper also presents an extension of the contrastive loss to improve the embeddings. The experiments in the paper  use common and recent long-tail datasets as well as datasets for object detection and out-of-distribution tasks. \n\n**Pros**:\n*Interesting problem and approach*. I think the paper tackles a hard and important problem, i.e., learning from a long-tailed dataset. Overall, I think that learning a feature space improving the learning from these imbalanced datasets is an interesting idea.\n\n*Clarity of the paper*. Overall the clarity of the paper is good. The motivation is clear and the narrative is clear overall. However, I think the clarity in the experiments is insufficient, see below.\n\n**Cons**:\n*Insufficient clarity in the experiments*. I have several concerns with the experiments:\n1. The *balancedness* metric in Eq.(3) may not be a robust metric for measure performance. The reason I am not convinced about this metric is that if the accuracies of the classifier are low but equal, then the metric will say that the *balancedness* is good. I think a good metric for a classifier learning from an imbalanced dataset is one that indicates if the overall accuracy is high, maintains the many-shot the classification accuracy high, and increases the accuracy of the classes in the tail. I think this metric does not indicate if the overall accuracy is high.\n\n2. I am not convinced about how classifiers are trained in experiments in Sec. 3.2. The paper trains and tests using a balanced set after learning a feature space. To my understanding, the challenge of learning from a long-tailed dataset is to test whether a classifier can generalize well for classes with few training samples while maintaining a good performance on classes with more training examples. Thus, by training a linear classifier with a balanced set using the learned feature space does not really comply with learning from an imbalanced dataset. I think if the experiments would've been stronger if they included results of a trained linear classifier on the learned embedding and still showing good results, then I would be more convinced about the impact of a contrastive loss. From the practical point of view, what matters is the classifier performance. In practice, it is challenging to have a balanced dataset as the paper used. The main question is about the performance when training a linear classifier from a long-tailed dataset using the learned representation.\n\n3. Datasets derived from ImageNet-LT. While I value the goal of using different datasets varying the imbalance in a dataset, I am not convinced that ImageNet-LT is the dataset to use. The reason is that ImageNet-LT is a synthetic long-tailed dataset. In fact, while the dataset shows imbalance, it does not necessarily follow a power-law distribution. I think the generation of these datasets in the experiments should be done using a power-law distribution. From the text, it is unclear how the datasets from ImageNet-LT were generated for the experiments.\n\n\nMinor concerns:\n*Plots lack information*. What is the y-scale in Fig. 2? The figure is missing y-scale information and it is hard to interpret the gap in accuracy  between CE and CL in the left plot. Same comment for Fig 4, what is the scale in y-axis?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Uyc9Y3Rrf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new loss function to learn feature representations for image datasets that are class-imbalanced. The loss function is a simple yet effective tweak on an existing supervised contrastive loss work. A number of empirical tests are performed on long-tailed datasets showing the benefits of the proposed loss in beating state of the art methods.  Some specific questions are listed below:\n\n1. Is Figure 1 hypothetical (\"desired outcome\") or real (based on actual observations)? \n\n2. Minor style comment: please don't call your own contributions \"important\" :-) - that is for others to decide.\n\n3. Eqn. (3) is not clear at all - please provide an intuitive explanation and motivation. It seems to appear out of thin air.\n\n4. Is there a tradeoff between accuracy and balancedness? Figure 2 seems to suggest so. For example, if we did not train CE loss to maximal accuracy, would be automatically get the balancedness property? Was this tested? How about a combined loss CE + CL to get the best of both worlds? And similarly, how about CE + KCL?\n\n5. Table 1 the delta for VOC seems to be computed wrongly.\n\n6. Figure 7: it's strange that there are no numbers on the y-axis.\n\n7. What is the ImageNet accuracy of KCL?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
 This paper studies the difference between cross-entropy and contrastive learning losses in the feature representations that they learn, specifically looking at class-imbalanced datasets. The authors show that contrastive losses result in a more "balanced" representation, as measured by the balance of accuracy across the classes when a linear classifier is learned mapping from the feature representation to the class labels. They also show that empirically this tends to result in better generalization to downstream tasks. Inspired by this, they devise a simple modification of the prior supervised contrastive loss method and show that it can improve performance on ImageNet-LT and even generalization performance when trained on balanced datasets and applied to downstream tasks. 

  The reviewers identified several weaknesses, including some clarity issues (R1), limitations of how balancedness is measured and lack of theoretical/statistical rigor in terms of the resulting claims (R2), and differences with respect to concurrent work (R4). A lengthy discussion occurred between reviewers and authors, as well as input from a co-author of the concurrent work. In the end, the reviewers were not fully satisfied both in terms of the balancedness measure and relationship to the concurrent work. 

  Overall, despite this and the valid limitations of the work, I recommend accepting this paper as I believe the contributions outweigh the limitations, and that the findings would be interesting to the community. First, the paper provides some interesting analysis of balancedness and differences across these two loss functions, as well as connections to generalization, which even the concurrent work does not provide. The resulting method, while being a simple modification of the supervised contrastive loss work, is effective both for long-tailed datasets and generalization to downstream tasks (even when trained in a balanced manner) which is nice. In the end, we should not use [3] to reject this paper since it was accepted right before the ICLR deadline. 

  However, I **strongly** recommend that the paper address the valid limitations mentioned in the discussions. Specifically: 
  1) While I agree that [3] is concurrent work, this paper should none-the-less tone down its claims of being the first in exploring balance for the camera-ready version and clearly address differences between this paper and that one (even if mentioned as concurrent work). It is important to give credit when it is due, and while I think [3] is a different perspective it should be mentioned. Further, the claim that their methodology is not correct is highly arguable, so this should not be mentioned; rather the differences in perspectives and what each paper shows should be emphasized. Even without [3], self-supervised pre-training (initialization) should arguably be included as a baseline given that it is the logical first choice for incorporating self-supervised learning.

  2) Like R2, I do not believe the balancedness metric shows uniformity of the feature space. This would have to be shown through methods such as t-SNE or in some other way. Being linearly separable in a balanced way across classes (which is what you showed) is not sufficient to show that feature space "uniformity". One can draw many feature space distributions that do not have the intuitive meaning of this (which isn't precisely defined by the authors) but still be linearly separable. I recommend authors remove this type of characterization (unless they can define/show it) and instead include a discussion of the limitations of the current methodology for measuring balancedness. Figure 1 should also emphasize that it is notional (not from real data). 
Decision: Accept (Poster)