Subclass-balancing Contrastive Learning for Long-tailed Recognition
Authors: Chengkai Hou
Jieyu Zhang
Haonan Wang
Tianyi Zhou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Long-tailed Recognition
Supervised Contrastive Learning
Representation Learning

Abstract: Long-tailed recognition with imbalanced classes naturally emerges in practical machine learning applications. Existing methods such as data reweighing, resampling, and supervised contrastive learning enforce the class balance with a price of introducing imbalance between instances of head class and tail class, which may ignore the underlying rich semantic substructures of the former and exaggerate the biases in the latter. We overcome these drawbacks by a novel "subclass-balancing contrastive learning (SBCL)'' approach that clusters each head class into multiple subclasses of similar sizes as the tail classes and enforce representations to capture the two-layer class hierarchy between the original classes and their subclasses. Since the clustering is conducted in the representation space and updated during the course of training, the subclass labels preserve the semantic substructures of head classes. Meanwhile, it does not overemphasize tail class samples so each individual instance contribute to the representation learning equally. Hence, our method achieves both the instance- and subclass-balance, while the original class labels are also learned through contrastive learning among subclasses from different classes. We evaluate SBCL over a list of long-tailed benchmark datasets and it achieves the state-of-the-art performance. In addition, we present extensive analyses and ablation studies of SBCL to verify its advantages. 

Under review as a conference paper at ICLR 2023
SUBCLASS-BALANCHIE CONTRASTIVE LEARNING FOR LONG-TAILED RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Long-tailed recognition with imbalanced class distribution naturally emerges in practical machine learning applications. Existing methods such as data reweighing, resampling, and supervised contrastive learning enforce the class balance with a price of introducing imbalance between instances of head class and tail class, which may ignore the underlying rich semantic substructures of the former and exaggerate the biases in the latter. We overcome these drawbacks by a novel "subclass- balancing contrastive learning (SBCL)" approach that clusters each head class into multiple subclasses of similar sizes as the tail classes and enforce representations capture the two-layer class hierarchy between the original classes and their subclasses. Since the clustering is conducted in the representation space and updated during the course of training, the subclass labels preserve the semantic substructures of head classes. Meanwhile, does not overemphasize tail class samples so each individual instance contribute the representation learning equally. Hence, our method achieves both the instance- and sosclass-balance while the original class labels are also learned through contrastive learning among subclasses from different classes. We evaluate SBCL over list of long-tailed benchmark datasets and achieves the state-of-the-ar performance. In addition, we present extensive analyses and ablation studies of SBCL to verify its advantages.
INTRODUCTION
In reality, the datasets often follow the Zipfian distribution over classes with a long tail ZZipi 2013 Spain & Perona 2007). i.e., a few classes (head classes) containing significantly more instances than the remaining tail classes. Such tail classes could be of great importance for high-stake applications.
egg. patient class in medical diagnosis or accident class in autonomous driving (Cao et al. 2019
Shen et al. 2015). However, training on such class-imbalanced datasets can result in severely biased model with noticeable performance drop in classification tasks (Wang et al., 2017 Mahajan et al.
2018 Zhong et al. 2019 Ando & Huang 2017 Buda et al.) 2018 Collobert et al. 2008 Yang et al.
2019
To overcome the challenges posed by long-tailed data, data resampling (Ando & Huang 2017 Buda
let al.
2018 Chawla et al.] 2002 (Shen et al. (0166) and loss reweighing (Byrd & Lipton 2019 Cao
al. 2019 Cui al.] [019 Dong et al., 2018) have been widely applied but they cannot fully leverage all the head-class samples. Very recent work discovered that supervised contrastive learning (SCL) (Khosla al. 2020) can achieve state-of-the-art (SOTA) performance on benchmark datasets
of tailet recognition (Kang et al. 2020; Li et al. 2022). Specifically, the k-positive contrastive learning (KCL) (Kang et al. (020) and its subsequent work targeted supervised contrastive learning (TSC) (Li et al. 2022) revamp SCL by encouraging the learned feature space to be class-balanced and uniformly distributed. However, those methods enforcing Mlas-balance often come with price of eestance-imbalanee i.e., each individual instance of tail classes would have much greater impact on model training than that head classes.
Such instance-imbalanee can result in significant degradation of the performance long-tailed recognition for several reasons. On the one hand, the limited samples in each tail class might not be sufficiently representative of the whole class. So even small bias of them can be enormously exaggerated by balancinn methods and result sub-optimal learning of classifiers or repre- sentations. On the other hand, head classes usually have more complicated semantic substructures,

---

Under review as a conference paper at ICLR 2023
e.g., multiple high-density regions of the data distribution, simply downweighing samples of head classes and treating them equally can easily lose critical structural information. For example, images of a head class "cat" might be highly diverse in breeds and colors, which need to be captured by different features but downweighing or subsampling them may easily lose such information, while tail class "platypus" might only contain few similar images that unlikely cover all the representative features. Therefore, it is non-trivial to enforce both class-balance and instance-balance simultaneously in the same method.
Can we remove the negative impact of class-imbalance while still retain the advantages of instance- balance? In this paper, we achieve both through a novel supervised contrastive learning defined on subclasses, which are the clusters within each head class, have comparable size as tail classes, and are adaptively updated during the training. Instead of sacrificing sestance-balanee for class-balance, our method achieves both instance- and dabelass-balanee by exploring the head-class structure in the learned representation space of the model-in-training In particular, we propose bi bi-granalarity contrastive loss that enforces sample (I) to be closer to samples from the same subclass than all the other samples; and (2) to be closer to samples from different subclass but the same class than samples from any other subclasses. While the former learns representations with balanced and compact subclasses, the latter preserves the class structure on subclass level by encouraging the same class's subslasses be closer to each other than to any different class's ' subclasses. Hence, it can learn an accurate classifier distinguishing original classes while enjoy both the instance- and pibolass-bbllnce.
contrastive learning (SBCL),
In this paper, we apply SBCL for several visual recognition tasks to demonstrate SBCL superiority over other previous works (e.g., /CL/Kang et al. paper makes the following contributions:
(Li al., 2022 To summarize, this
(a). We provide new design principal of leveraging supervised contrastive learning for long- tailed recognition, i.e., aiming at achieving both instance- and jubclass-balance instead of class-balance at the expense of instance-balance.
(b). We propose novel instantiation of the aforementioned design principal, balaniis- contrastive learning (SBCL), which consists of two major components, namely, subelass- balancing adaptive clustering and bi-granularity contrastive loss.
(c). Empirically, we compare the SBCL against state-of-the-art methods on three visual tasks: image classification, object detection, and instance segmentation demonstrate its effec- tiveness on handling class imbalance. We also conduct a series of experiments to analyze the efficacy of SBCL..
2 BACKGROUND AND NOTATIONS
Long-tailed recognition. Long-tailed recognition aims to learn a classifier from a training dataset with long-tailed class distribution, i.e., few classes contain many data (head classes) while most classes contain only a few data (tail classes), where the major challenge to require model recognizing all classes equally well. Let D {z,, 34 J [e[n]] be long-tailess training dataset, where x; denotes sample and yı E [C] denotes its label. Denote by Dr D the set of instances belonging class k: and n, = |Dxl the number of samples in class k The total number of training samples over C' classes is EE=1 74. Without loss of generality, to assume that the classes are sorted by cardinality in decreasing order (t.c., if < J. then n.g 2 my). and n, ng. In addition, we define the imbalance ratio maX (n+) minne(c) (ng) = n, / Finally, let fo(-) be deep feature extractor, e.g., neural network, parameterized by 0 and wiis the linear classifier of class c, then the classifier we aim to learn h(x;; = arg max,e(o) w. fo(xz).
follow prior work (Kang et al. 2019 Hong et al. 2021
Supervised contrastive learning. Recent studies have shown that supervised contrastive learning (SCL) Khosla et al. 2020) provides a strong performance gain for long-tailed recognition and its variants have achieved state-of-the-aa (SOTA) performance (Kang et al., 2020 Li et al.) 2022). Specifically. SCL learns the feature extractor f.). via maximizing the discriminttivenns of positive instances, i.e,, instances from the same class, and the learning objective for single training data
2

---

Under review as a conference paper at ICLR 2023
Long-talled
"Medinm Cln Dog
Medid
Goldte Sarriner
vall
Hapd
Carmen Shephard
portoe
nel Casa
pansko
[X] Subdass balancnng www.cnggaaappeeee.. accaoovv clantring Seecion 2.3)
l-granddaitt contrastiva low (Sect 2.20
Figure 1: Illustration of subclass- contrastive learning (SBCL).
(o,, yr) in a batch B = {(zs, 94) kE(N). is
N
exple: : 2 /T) log
ESCr
Σ SpeR,
P.
Eneev, exp(2: 2 /T)
where is the temperature hyperparameter, 2i fo(2;) is the feature generated from V; (z (eeNe \ {z is the current batch of features except for 2,, P. = {zz E V, Y/j = =/: is set of instances with the same label as Ts. Finally, let & be the feature of 2;, the augmented version of 2;, and for any set S; indexed by i, we use S; = S; U (=.), e.g., V. = V; U ) However, for the long-tailed datasets, the feature spaces is dominated by head classes and thus have limited capability of semantic discrimination (Kang et al. 2020). To address this, the k- k-positive contrastive learning (KCL) (Kang et al. 2020) attempts balance the feature space by keeping the number of positive instances in P. equal for each class, leading to the following loss
exp(2i 2 /T)
LRCL
Σ log ApEPP
ExEeR. exp(2: 2,_ (T)
where Ph: is subset of P. with k randomly drawn instances. Finally, the learned feature extractor fa(.) is exploited in sequel stage of training the classifier for long-tailed recognition (Kang et al.
2020 Li al. 2022).
3
METHODOLOGY
As mentioned above, KCL, and its sequels (Kang et al. 2020 Li et al. 2022 balance the learning objective of SCL by picking the same number of positive instance for each class, i.e., | = k in argue that such a
Eq.2 2 no matter which class belongs to, however, would inevitably introduce instance-imbalance: the instances of tail classes have much more chances to be engaged in the training than that of head class. Specifically, assume each class has no less than l instances, then the probability of an instance class being selected as positive instance is p(c) no have p(1) TODE 0.01 while p(C) = 1. We can see that when the instances of head class are selected once, that of tail class may already be trained 100 times. Thus, the training is immensely biased towards the few samples in each tail class. Besides, as tail classes only have very few instances that are not necessarily representative, the learned feature space might be unsatisfactory and sensitive to the training data of tail classes.
approach
; if the tail class ne has only instances and the imbalance ratio is An 100, then nc
&
Here, we provide new prospective of handling class-imbalance issue by contrastive learning: instead of aiming at class-balance at the expense of instance- instance-imbalance we propose to achieve both instance- and nubclass-balance. We argue that head classes typically contain more diverse instances and thus have richer semantics in the training dataset. Therefore, it might be wise to break down the head classes into multiple semantically coherent subclasses, each of which consists of similar number of instance as tail classes. Built this spirit, we develop subclass-balancing contrastive learning (SBCL), a new contrastive learning framework for long-tailed recognition (visualized in Figure that achieves both instance- and subclass-balance
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Subclass-balancing Adaptive Clustering
Require:
Sample set S {x:) A threshold M; The number of iterations K; Cluster assignments for samples in S
Ensure:
for to K do
Update the cluster centers = 1 n: 2i=1 Te.
P Ty is the number of samples in a cluster b m is the number of cluster centers Assign samples to centers y:
Construct the cluster center set C = {15 1==1"
while S # Φ de Sclect the most similar pair (2%, UU)
arg max cosmaaaoooiii--imilaititt.t y). zES.yEC
Assign the sample z; to the center y5. Delete the assigned sample from the sample set S = S/ {x;}.
if 2 M then
b Sample number in cluster exceeds the threshold M
Delete the cluster center y) from the cluster center set C C/ {yz}.
end end while
end for
3.1 SUBCLASS-BALANCING ADAPTIVE CLUSTERING
We break down the class into several "subclasses" to attack the imbalanced phenomenon. Particularly, given a class and the associated set of data De. employ clustering algorithm of choice based on the features extracted by current feature extractor f-(-) to divide D. into m. subclassescllsster. We use (zx( to denote the cluster label of instance I; of class To ensure that the number of samples for each subclass is roughly the same, we propose a new cluster algorithm to divide the unit-length feature vectors, i.., the features output by fo(-) with additional unit-length normalization. The new proposed cluster algorithm is described in Algorithm[] | In the process of assigning vectors their centers, we threshold M an upper limit of sample size in cluster, which guarantees clusters of balanced sizes. We show the distribution of the sample size in clusters on the benchmark dataset in Appendix A.I Specifically, the threshold M is
M = max (ncx 6)
where nc is the size of tail class and the hyperparameter controls the lower bound of sample size in clusters to prevent overly-small cluster. Note that we only apply clustering algorithm to classes which contain multiple instances while the tail classes remain unchanged. As a consequence, the size of each resultant cluster is similar to that of tail class ng. In addition, instead of only clustering once the beginning, we update the cluster assignment adaptively based on the current feature extractor fa(-) during training process and empirically show that such adaptive clustering outperforms only-once clustering in Section 4.5
Then, by replacing the class labels of head classes used SCL/KCL with the finer-graines cluster labels, we ensure the instance-balane balanced, i.e., each instance has similar chance of being selected regardless of its class. By breaking down the head classes, which typically contain more diverse instances, into multiple semantically coherent subclasses, we achieve subelass-balanced (instead of blass-balanced) while maintain the rich semantics rendered by head classes in training dataset.
3.2 BI-GRANULARITY CONTRASTIVE LOSS
We now have types of label for instances in head classes from different granularities: the coarse- grained class label and the fine-grained cluster label. A direct consequence of replacing class label in SCL/KCL with cluster label is that we no longer distinguish instances from different head classes, and therefore the boundaries between classes might be blurry, leading to optimal feature space. As remedy, we combine the contrastive loss of both class label and cluster label into the following one and reuse the notations of Eq. [
NV LSBOL
exp(x: 2/ /TI) Enee, exp(2 z /TT))
Σ
log
M.M.l
1m1
SpEAt,
+
1
Σ
log Σ Ex.ER/ exp(2; 2
explz: : 22 /T2)
MM
SpEP./M;
4

---

Under review as a conference paper at ICLR 2023
where Mr E P:: -
is a set of instances with the same cluster label 2;- a hyperparameter that balances these two loss terms. The first term corresponds to the SCL loss with cluster labels, while the second term leverages the class label but does not consider the instances of the same cluster, Le., the instances in M, are removed in the second term. Such a design choice reflects the two types of positive instances for 2;: (1) the instance in the same cluster and (2) the instance of the same class but in different clusters.
According previous studies (Wang & Liu 2021 Hoffmann et al. 2022 Li et 202 the temperature 7 in contrastive loss is critical in controlling the local separation and global uniformity of the feature distribution. Specifically. low temperature forees the features to concentrate, while the temperature increases, the features would distribute more uniformly. Although the above objective explicitly considers the two types of label from different granularities, still treats class and cluster label similarly. Intuitively, we expect instances of the same subclass to form more concentrated cluster in feature space than those of the same class, since subclass naturally indicates finer-grained semantic coherence. To achieve this, ensure the temperature T2 > Ti and dynamically adjust 72 for each class according to its current level concentration the instances' feature. Following al. 2021a), for class define (cc) as
Eis || tell2
6(c)
(5)
Ne loglno e + a))
where ta is the centroid for the class c, is hyperparameter to ensure that ((c) is not overly-large, and 2; corresponds to instances of class c. From the formulation, we can see that if the current averaged distance the class centroid large or the class contains fewer data, thus the temperature will be set large to adopt the feature distribution of class e during the training process. Then we define the temperature of class
o(c) 1 EE 6(1)
(c) exp
(6)
such that (2(c) for class label is always larger than Tj for cluster label (since ((c) 0) and could reflect the current level of concentration of the instances in a class. In particular, the proposed T2 (2(c) encourages the features of instances in class to form less tight cluster than that of a subclass (by Te(c) > T1) while adaptively adjust the temperature to prevent an overly-loose/dense cluster.
3.3 TRAINING ALGORITHM
Here, we describe the overall training process of babbelass-balancie contrastive learning and the algorithm can be found in Algorithm 2 First, the adaptive clustering (Section 3.1 could be noisy at the early stage of training (Li et al. 202 Wang et al. (021b). Thus, we warm-up the feature extractor (-) (-) by few epochs training on ordinary SCL or KCL loss. In addition, our algorithm parts, namely, the cluster assignment and the temperature 72(c) for
involves two adaptively-adjustin each head class c. Instead of updating these every epoch, we use a hyperparameter K as the update interval, i.e., we update the cluster assignment and the temperature based the current learned fo(-) every K epoch.
Algorithm 2 Training Algorithm
Require: Dataset = {Zt, ys heeen) The update interval of cluster assignment K: The number of warm-up
epoch Ta: The total number epoch T; The hyperparameters 3 and
Ensure: A trained feature extractor fo(-)
1: Initialize the model parameters 0
2: Train fo(.) with SCL/KCL for To epochs
Warm-up stage
3: for To T do
b Update cluster assignment and termperature 5: Update the cluster assignment based on the current feature extractor fe(
4: t///K = 0 then
6: Update the temperture 72 for each head class using Eq. 5 and Eq.
7:
end if
8:
Train fo using Eq.
Subelass-balancing contrastive learning
9:
end for
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We consider three commonly used long-tailed recognition benchmark datasets: CIFAR-
100-LT (Cao et al. 2019), ImageNet-LT (Liu et al. 2019), and iNaturalist 2018 (Van Horn et al. 2018).
The CIFAR-I00-L3 and ImageNet-LT datasets are artificially generated long-tailed datasets from class-balanced datasets (Krizhevsky et al. 2009, Russakovsky et al. 2013), and the iNaturalist 2018 imbalance.
dataset is a large-scale real-world dataset that exhibits
Baselines. We consider baseline methods of the following three categories: (1) class-balancing
classifiers, including 7-norm, LWS and cRT (Kang et al.) 2019), which fixes the representation which trained by cross-entropy loss and trains the classifier with baas-balanced sampling; (2) one-stage balancing loss, including CB loss (Cui et al., 2019). Focal loss (Lin et al. 2017), and LDAM loss (Cao)
al. 2019). These supervised distribution-awann loss makes the model pay more attention on the minority class during training. (3) contrastive learning methods, including SCL (Khosla et al. 22020),
KCL (Kang al. 2020), SwAV (Caron al., 22020), PCL (Li al.
2021a) and TSC (Li al. 2022)
which train feature extractor with the contrastive loss and then learn classifier given the trained feature extractor.
Li al. 2022), we implement SBCL, as well as other contrastive learning methods, two-stage framework. In the first stage, we train the feature extractor with contrastive learning method, while in the second stage, train linear classifier on top of the learned representation. Specifically, for CIFAR-I00-LE dataset, the linear classifier is trained with LDAM loss and class re- weightittng (Cao et al. 2019). For ImageNet- LT and INaturalist 2018 datasets, the linear classifier is trained with CE loss and class- balanced sampling (Kang et al. 2019). All results are averaged over trials with different random seeds. We mainly report the overall top-1 accuracy. For the two large datasets, ImageNet-L.T and iNaturalist 2018 datasets, following the previous work (Liu al. 20T9), also report the accuracy of three disjoint subsets: Many-shot classes (classes with more than 100 samples), Medium-shot classes (classes with 20 100 samples), and Few-shot classes (classes under 20 samples). We leave the implementation details and additional experiments to the appendix.
Evaluation protocol. Following (Kang et al., 2020 2019
4.2 MAIN RESULTS
Table 1: Performance comparison on ImageNet-LT and iNaturalist 2018 datasets. Top-1 accuracy of ResNet-50 (He et al. 2016) reported. The "Many", "Medium". "Few" and Alll" denotes different
groups.
Backbone
ImageNet-LT
iNaturalist 2018
Methods
Many
Medium
Few
All
Many
Medium
Few
CE
64.0
33.8
3.8
41.6
72.2
63.0
57.2
61.7
Focal loss
Lin et al.
2017
$1.0
40.8
20.8
43.7
CB-Focal
61.1
LDAM-DKW (Cao
2019
60.4
46.9
30.7
49.8
64.6
OLTR
Liu al
9
35.8
52.3
21.5
32.2
59.0
64.9
7-norm
Kant
2019
56.6
44.2
27.4
46.7
71.1
68.9
69.3
69.3
cRT Kahg 21
20199
58.8
44.0
26.
47.3
73.2
68.8
66.1
LWS
2019
57.1
45.2
29.3
47.7
71.0
69.8
68.8
69.5
PCL
Li A
34.7
26.1
12.3
27.5
48.5
45.9
41.7
44.5
SwAV
2020
37.5
28.3
15.6
30.1
51.9
48.4
43.7
47.0
BYOL
Grniee 20200
37.7
28.9
16.3
30.6
52.3
48.6
44.1
47.2
SCL
Khosla
61.4
28.2
KCL
King al 2020
62.4
49.0
29.5
51.5
68.6
TSC
63.5
72.6
SBCL.
63.8-0.3
51.3:0.3
3122+0.4
53.4=0.3
73.3:0.2
7194003
68.6=0.3
70,8=0.3
The results on ImageNet-LT and iNaturalist 2018 are in Table We can see that SBCL outperforms the baselines with a large margin over the two datasets. In addition, on iNaturalist 2018 dataset, SBCL outperforms the previous SOTA method by 0.7% Many, 1.3% on Medium, 0.8% Few and 1.1% on All, which shows the effectiveness of the proposed method solving real-world long- tailed recognition problems such natural species classification. Besides, SBCL is also better than existing contrastive learning method like KCL and TSC for all class splits, which demonstrates the effectiveness of the design principal of pursuing both instances- and
in contrastive

---

Under review as a conference paper at ICLR 2023
Table Performance comparison on CIFAR-I00-LT. Top-1 accuracy of the ResNet-32 (ee et 2016) under different imbalance ratios is reported. We also report the accuracy of our re-implemented important baselines (f) in same setting on LIFAR-100-LL. The columns of "Statistic (IR 100)" are results of different disjoint subsets on CIFAR-100-LT with imbalance ratio being 100.
Imbalance Ratio
Statistic (IR 100)
Method
100
50
10
Many
Medium
Few
CE
38.3
43.9
55.7
65.2
37.1
9.1
CB-CE CCu
2019
38.6
44.6
57.1
Focal Loss
2017
38.4
44.3
55.8
65.3
38.4
8.1
CB-Focal
Cur al. 2019
38.7
58.0
65.0
37.6
10.3
CE-DRW Cao al.
2019
41.4
58.1
CE-DRS Cao
2019)
41.6
45.5
58.1
LDAM (Cao et al. 2019
39.6
45.0
56.9
LDA.M-DW (Cao of
2019
42,0
46.6
58.7
61.5
41.7
20.2
M2m-ERM Kim et al.
2020
42.9
58.2
M2m-LDAM (Kim
2020
43.5
57.6
cRT
Kang el 2019)
43.3
46.8
58.1
64.0
44.8
18.1
LWS
Kang
2019
43.1
46.4
58.1
SCL
Khosla al.
2020
42.1
54.8
62.8
42.0
18.4
KCL
Kang et al.
2020
42.8
46.3
57.6
KCL-
42,8
46.4
57.5
63.4
42.5
19.2
TSC at 2022
43.8
47.4
59.0
TSC
43.5
47.6
58.7
63.7
43,2
20.4
SBCL
44.9
48.710.2
57.910.2
6..41103
45.3:10.2
22.2:10.3
learning. Table 2 summarizes the results on CIFAR-100-LT dataset. For CIFAR-100-L1 dataset, SBCL outperforms previous SOTA methods except for imbalance ratio 10. We hypothesize that is because the tail class of CIFAR- with imbalance ratio 10 has multiple samples, which makes hard to distinguish the performance of methods on the long-tailed recognition.
4.3 PERFORMANCE ON OTHER VISUAL TASKS
There a recent trend of using the contrastive learning to pretrain feature extractor for downstream visual tasks other than image classification (He et al., 2020). We curious about two questions: (1)
when the pretraining dataset is glass-imbalanced, how the downstream performance is affected? (2)
In such case, can our SBCL improve the learned feature extractor over existing contrastive learning baselines? To answer these questions, we use the object detection task of PASCAL VOC dataset as the evaluation suite and use ImageNe//ImageNet-.11 datasets as wess--alanneddiimaaan datasets. Following Kang et al. y: He et al. (2020), we first pretrain feature extractor on ImageNet/ImaaeeNer LT then further finetune for the downstream object detaction tasks using Faster R-CNN (Ren al. 2015) with R50-C4 backbone.
pretraining
The experiment results are shown Tables 3 From the results, we can see that pretraining on class- balanced data (ImgeNet) leads to consistently better results than that on class-imbalanced dataset (ImageNet-LT) pretraining the model on the ImageNet and ImageNet-LT datasets by the SBCL can perform slightly better than other baselines. In addition, the proposed SBCL significantly outperforms baselines on blass-imbalanced pretraining dataset, while achieve comparable performance on class- balanced ones. For the representation which trained the full ImageNet dataset, the performance advantage is not obvious. In Appendix A. we show additional experimental results of object detection and instance segmentation COCO (Lin al. 2014) dataset and SBCL also outperforms other baseline methods. Thus, we conclude that the proposed SBCL is not only helpful for image classification, but also other visual tasks.
4.4 ANALYSIS OF FEATURE DISTRIBUTION
To analyze the representation learned by SBCL, we firstly define the euclidean distance between given sample and other samples from the same/different classes as intra/inter-class distance. Concretely, the euclidean distance between a sample 2; and a set S is defined as D(2, S) = Enes || 27112- Then, the intra- and inter-class distance of sample can be defined as
7

---

Under review as a conference paper at ICLR 2023
D(2g, P.) and (2, D/P.) separately; and the intra- and inter-subelass distance of sample 2 be defined as (2,, M,) and ( P./M) separately.
Table 3: Object detection Results on PASCAL VOC.
Table 4: Ablation study on different components of SBCL.
Trag Net
Wan Magtiie chaser Desamii ienperutuee
CIEAR-IOD-LT
011
111
111
Infolaree Kein
TWI
97
01.3
15:
00.3
11.3
437216-2
5200102
1001
11 1:9008
963000
623601
58.600
5336602
44915-3
48.7362
$7.9102
Table 5: Average intra-classiiter-class distance of features learned with different contrastive learning methods.
Distance
Method
Many
Medium
Few
All
KCL
0.44
0.53
0.70
0.46
Intra-class
TSC
0.60
0.62
0.77
0.61
SBCL
1.00-10.01
0.902
0.88-10.02
0.99-10.01
KCL
1,27
1.26
1.24
1,26
Inter-class
TSC
1.32
1.31
1.29
1.32
SBCL
1.39:+0.01
1.38
1.37++0.01
1.39::00.01
Intra-class/Inter-laas distance. SBCL aims at learning compact representation space, in which representations from different classes are far from each other and the feature space spanned by representations of each class is invariant the long-tailed distribution. Though the intra/inter-class distance on the feature space, we compare SBCL on the previous methods (KCL, ,TSC) on CIFAR- 100-LT with imbalance ratio 100. The results the average distance are summarized in Table 5 and the distances of different groups are reported separately. The results show several good properties of SBCL over previous methods: (i) the inter-class distances of SBCL is larger than the previous methods, which implies that SBCL can push different classes far away from each other, and thus help the downstream tasks; (ii) the intra-class distance of SBCL is relatively more equal in different disjoint subsets than previous methods, which indicates that that head/tail classes have similar volume of the learned space and thus help balance different classes.
1.4
1.5
1.3
Intereittion ditunce
1.4
ta
chees
Intra
distance
(a) Feature distance in subclasses.
(b) Feature distance in classes.
Figure 2: Feature distance of subclasses and classes CIFAR-100-EL with imbalance ratio 100. We randomly sample instances from many-shot and medium-shot classes so that the size of each equals that of few-shot classes.
Feature distribution of SBCL. As shown in Figure 2a the distance between samples from the same subclass less than those from the same class but different subclasses. Meanwhile, in Figure 2b the inter-class distance is higher than the intra-class distance with stable value, which denotes features from the different classes are uniformly distributed on a hypersphere. The results in all indicate that the two-layer class hierarchy successfully captured and feature distribution achieves the core idea of SBCL.
4.5 ABLATION STUDIES
we train the feature extractor for several epochs using ordinary SCL KCL warm-up stage. As shown in Table such warm-up stage is beneficial since the performance drops when we remove the warm-up stage. This is likely because at the early stage of training, the extracted feature is not well-trained and the cluster assignment could be noisy and ineffective, hindering the efficacy of SBCL.
Warm-up. As mentioned in Section 3.3
8

---

Under review as a conference paper at ICLR 2023
Adaptive clustering. We are also curious about the efficacy of adaptive clustering and thus present the performance of SBCL with clustering only once and fixing cluster assignments during training. As shown in Table 4 without adaptive clustering, the performance decreases in all cases. The reason could be fixed cluster assignment is prone to noise when the model is not well-trained, while adaptive clustering would dynamically adjust the cluster assignments based on the current learned model, which is supposed to become better as the training proceeds.
Dynamic temperature. In Table 4 also study the effectiveness of dynamic temperature (Sec- tion/3.2 We remove the dynamic temperature and simply set 72 > 71 following (Hoffmann et al. 2022p. With the fixed temperature Tg. the performance of SBCL is significantly worse than that with dynamic temperature. We speculate that this is because dynamic temperature could help prevent the instances of a class to form overly large or small cluster in the feature space and therefore lead to better learned representations. Additionally, to evaluate the impact of dynamic temperature on other baselines, we apply the dynamic temperature on TSC, as reported in Appendix A.
5 RELATED WORK
Traditional methods for handling toiget recognition problem includes re-sampling and re- weighting. There are roughly two types of re-sampling techniques: over-sampling the minority classes (Shen et al. 2016 Zhong et al., al.] 2016)] Buda et al.] 2018: Byrd & Lipton 2019) and under- sampling the frequent classes (He & Gareial 2009 Japkowicz Stephen| [002] Buda et al. 2018). The re-weighting techniques assign adaptive weights for different classes or even different samples. The vanilla scheme re- re-weights classes proportionally to the inverse of their frequency (Huang et al.
2016: 2019 Wang et al. 2017). For class-level re-weighting methods, many loss functions including CB loss (Cui et al. 2019). LDAM loss (Cao et al. 2019) and Balanced softmax loss (Ren et al. 2020) were recently proposed, while level re- weighting methods include Focal loss (Lin et al. 2017) and Inffuence-balancee loss (Park et al. 2021). Recently, two-stage algorithms have achieved remarkable performance for long-tailed recognition, such classifier re-training (cRT) (Kang ct al.19)2011 learnable weight scaling (I (LWS) (Kang et 2019), and Mixup Shifted I Label-Aware Smoothing model (MiSLAS) (Zhong et al. (021). Mcanwhile, bilateral branch network (BBN) (Zhou cct al. 2020) uses an additional network branch for re-balancing. RIDE (Wang et al., 202 use multiple branches named experts, each learning to specialize in the entire classes. LADE (Hong et al.,
2021) assumes the prior of test class distributions is available and accordingly post-adjust model predictions. PaCo (Cui et al., 202T applies parametric lass-wise learnable centers to rebalance in contrastive learning. BCL (Zhu et al. (022) proposes multi-branch framework to achieve alass-averaging and complemett in the training process.
To boost the performance of the two-stage algorithms, researchers have introduced supervised contrastive learning (Khosla et al. 2020) to the first feature-learning stage and proposed k-positive contrastive loss (KCL) (Kang et al. (020) and targeted supervised contrastive learning (TSC) (Li eet al., 2022). While achieving the state state-of-the the- performance, these methods inject class-balance in the contrastive learning objective, inevitably leading to instance-imbalane during training. In this work, we instead propose to achieve both subclass- and instance-balance in the contrastive learning object. Our method is also related recent studies of clustering-based deep unsupervised learning (Dosovitskiy et al., 2014: Xie et al., 2016 Liao et al./ [2016] Yang et al. [0166] Caron et al. 2018 2020), especially those that leverage contrastive learning (Li al. 2021b a. Wang et al. 202 Guo
et al. 2022). However, they target at general unsupervised representation learning scenario, while our method is tailored for long-tailes recognition where the training data is immensely glass-mmbalanced.
CONCLUSION
In this paper, we introduced Subelass-balancing Contrastive Learning (SBCL) for long-tailed recog- nition. It breaks down the head classes into multiple semantically-cohert subclasses via subclass- balancing adaptive clustering and incorporates bi granularitty contrastive loss that encourages both subclass- and mustance-balanee Extensive experiments on multiple datasets demonstrate that SBCL achieves state-of-the art single-model performance on benchmark datasets for long-tailed recognition.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897-6907, 2022.
Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 770-785. Springer, 2017.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Nerworks, 206:249-259, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In
International Conference on Machine Learning, pp. 872-881 PMLR, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distributin-awaaa margin loss. Advances Neural Information Processing Systems, 32, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In European Conference on Computer Vision, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiay Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5008-5017, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0 04297, 2020b.
Collobert, Ronan, Weston, and Jason. unified architecture for natural language processing: deep neural networks with multitask learning. In International Conference Machine Learning, 2008.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0868-9277,2 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1430, 2015.
Qi Dong, Shaogang Gong, and Xiatian Zhu, Imbalanced deep learning by minority class incremental rectification. IEEE transactions pattern analysis and machine intelligence, 1166)1167-1381, 2018.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Advances in Neural Information Processing Systems, 27, 2014.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07728, 2018.
10

---

Under review as a conference paper at ICLR 2023
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In Proceedings of the ieee/cvf International Conference computer vision, pp. 6391-6400, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hese: Hierarchical contrastive selective coding. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 21(9):1263-122 1284, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9733 2020.
David Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. the Association for the Advance of Artificial Intelligence, 2022.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang
Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6626-6633 2021.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5375-5384, 2016.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions pattern analysis and machine intelligence, 42(1 1):2781-2794, 2019.
Ahmet Iscen, André Araujo, Boqing Gong, and Cordelia Schmid. Class-balanced distillation for long-tailed visual recognition. Proceedings of the IEEE international conference on computer vision, 2021.
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429 449, 2002.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long- snilla recognition. International Conference on Learning Representations, 2019.
Bingyi Kang, Yu Li, Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference Learning Representations, 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 13:18611-18777 2020.

---

Under review as a conference paper at ICLR 2023
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18896-13905 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2)833-97, 1955.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference Learning Representations, 2021a.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang. Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering.
AAAI Conference on Artificial Intelligence, 2021b.
Renjie Liao, Alex Schwing, Richard Zemel, and Raquel Urtasun. Learning deep parsimonious representations. Advances Neural Information Processing Systems, 29, 2016.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin. Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In European conference on computer vision, pp. 181-196, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pp. 69-84. Springer, 2016.
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 735-744, 2021.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Xie Saining, Girshick Ross, Dollar Piotr, Tu Zhuowen, and He Kaiming. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1492-1500, 2017.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pp. 467-482, Springer, 2016.
12

---

Under review as a conference paper at ICLR 2023
Tony Shen, Ariel Lee, Carol Shen, and C Jimmy Lin. The long tail and rare disease research: the impact of next-generation sequencing for rare mendelian disorders. Genetics research, 2015.
Aditya Singh and Alessandro Bay. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 2019.
Merrielle Spain and Pietro Perona. Measuring and predicting importance of objects in visual world. 2007.
AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, al. Saliencymix: A saliency guided data augmentation strategy for better regularization. International Conference on Learning Representations, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 8769-8778, 2018.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 2495-2504, 2021.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-awen experts. International Conference on Learning Representations, 2021a.
Xudong Wang. Ziwei Liu, and Stella Yu. Unsupervised feature learning by cross-level instance- group discrimination. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12586-12595, 2021
Yu-X Xu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning model the tail. Advances in Neural Information Processing Systems, 30, 2017.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. pp. 487. PMLR, 2016.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2016.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in Neural Information Processing Systems, 33:19290-19301, 2020.
Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu. Me-net: Towards effective adversarial robustness with matrix estimation. International Conference on Machine Learning, 2019.
Han-Jia Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001.01388, 2020.
Qiaoyong Zhong, Chao Li, Yingying Zhang. H Sun, S Yang, D Xie, and S Pu. Towards good practices for recognition & detection. In CVPR workshops, volume I, pp. 2, 2016.
Yaoyao Zhong, Weihong Deng, Mei Wang. Jiani Hu, Jianteng Peng, Xunqiang Tao, and Yaohai Huang. Unequal-training for deep face recognition with long-tailed noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7812-7821, 2019.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
Boyan Zhou Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for loig-tailed visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9719-9728. 2020.
13

---

Under review as a conference paper at ICLR 2023
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattemn Recognition, pp. 6908-6918 2022.
George Kingsley Zipf. The psycho-biology of language: An introduction to dynamic philology. Routledge, 2013.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL EXPERIMENT RESULTS
Accuracy of each class CIFAR-100-LT. We visualize the accuracy of each class of both SCL and SBCL on CIFAR-I00-LT with imbalance ratio 100 (Figureß) D. From the results, we can see that SBCL improves performance on tail classes over SCL without the expense the perforamnce the head classes.
SCL (Acc: 42.1%) SBCL (Acc: 44.9%)
Image Number
Accuracy (%))
Class index
Figure 3: Accuracy of each classes CIFARI00-LE The black line is the class distribution, and the classes in the left part are head classes while those in the right part are tail classes.
The per-class weight norm of a linear classifier trained on top of features learned by SBCL in different training stages. Figure shows the per-class weight norm of linear classifier trained top of features learned by SBCL in different training stages on CIFAR- SEAR-100-LE 100-LT. From the figure, we can see that as the training proceeds, the per-class weight norm becomes model balanced even when training the linear classifier, the original cross-entropy loss is used.
Weight epoch Weight norm opoch 600 Weight opoch 1000
MM M Wo M Way
10
Per-class weight
Class index
Figure 4: Change in weight norm the linear classifier based on the representations trained by SBCL on CIFAR-100-LT with imbalance ratio 100 during training.
Table 6: Different selection of negative instances in SBCL on CIFAR-100-LT with different imbalance ratio. The 'Class label' row the first term of loss function is constructed by class label and the "Subclass label' row are subclass label.
Imbalance Ratio
Negative samples
100
50
10
Class label
43.7
47.6
56.8
Subclass label
44.9
48.7
57.9
Selection of negative instances in SBCL. Our proposed loss in Eq. Eq.A.oom consisss of two supervised contrastive losses with subclass and class labels respectively. The first term regards instances in
15

---

Under review as a conference paper at ICLR 2023
different subclasses negative instead of these in different classes; In Table we show that such design choice leads to better performance than using instances in different classes as negative, which illustrates the effectiveness of exploiting the rich semantic in head classses.
distance. To leverage instance semantic coherence balance the feature space, we expect instances of high semantic coherence to form a more concentrated cluster than other instances in the same class. So, we embed the subclass-balanee balaneinnnnn adaptive clustering strategy on previous methods to illustrate this on CIFAR- 1000TT with imbalance ratio 100. In Table 7 we report the putra-subclass//nee-ssuby TSC, the results show that SBCL achieves to concentrate instances from the same subclass and pulls instances from different subelasses away on all splits. We also note that the inter-subelass distance of SBCL is invariant with the decreasing of group split. This means the head class could be split into many subclasses separately, which constructs balance feature space for the long-tailes recognition.
intra-subcclas/Intee-abbee
distance on different splits. Compare with KCL and
Table 7: Intrassubeeass/iieerssullla distance features learned with different contrastive learning methods.
Metric
Method
Many
Medium
Few
All
KCL
0.27
0.40
0.69
0.30
Intra-subelass distance
TSC
0.40
0.48
0.76
0.42
SBCL
0.68++0.02
0.76+0.02
0.87=0.03
0.70=t0.02
KCL
0.46
0.59
0.70
0.48
Inter-subelass distance
0.61
0.69
0.77
0.63
SBCL
1.02:10.01
0.99-10.01
0.89-10.02
1.01
Secciass iater ong adactive durletng aggncchm
42.1
(30
100
1556
2909
2014
2003
106
(a)
300
1206 136
Taining
LAWringea
(c)
Figure 5: Analysis of SBCL as a loss function of diffferett hyperparameters on CIFAR-100-LT
with imbalance ratio 100. (a): Sample number in clusters with different cluster algorithm. (b): Top-1 accuracy of SBCL/TSC as function of different batch size. (c): Top- 1 accuracy of SBCL/TSC as function of different pretraining epochs. (d): Top-1 accuracy SBCL/TSC as a function different learning rates.
Hyperparameter analysis CIFAR-100-LT. Figure 5a and Table 8 show the distribute situation of sample number in subclasses obtained by different cluster algorithms on CIFAR-100-L5 with imbalance ratio 100. For Kmean cluster algorithm, the imbalance phenomenon of subclasses is obvious. When using our proposed cluster algorithm, the imbalance ratio of sample number in subclasses deceases from 40 9.5. And the standard deviation of sample number CIFAR-100-HE
16

---

Under review as a conference paper at ICLR 2023
is relatively small, which denotes the number of samples in most subclasses keeps stable in certain range.
Figure 5b shows the impact of batch size SBCL/TSC. We find that larger batch sizes have significant advantage over the smaller oncs. This is because larger batch sizes provide more negative examples facilitate convergence. However, the over-large batch size hurts the model performance. And SBCL and TSC are equally sensitive to batch size on CIFAR-IO0-LT.
Figure 5c shows the curve of the accuracy of SBCL/TSC the number of training epochs. From the curve, we can see that the performance of SBCL and TSC both converge after 800 epochs. When the model is trained with SBCL over 600 epochs, its performance already exceeds TSC.
In Figure 5d we display the performance of SBCL with different learning rates on CIFAR-100-EE with imbalance ratio 100. As shown in the figure, the learning rate has significant impact on the performance, and we set the learning rate as 0.5 for CIFAR-100-LT.
Table 8: Distribution of sample number in subclass on CIFAR-100-L3 with imbalance ratio 100.
Dataset
Max
Min
Average
Std
Imbulance ratio Max/Mii)
clustering algorithm
10.34
6.27
babelass-balancinn adaptive clustering algorithm
19
10.34
1.60
9.5
Combining TSC with dynamic temperature. According to Table 4 dynamic temperature ef- fectively contributes the improvement of accuracy. We also add the dynamic temperature the second term of TSC (Li et al. 2022) and the experiment results are shown in Table 9 However, the improvement of the dynamic temperature on TSC less significant than that on our method, which is reasonable because we introduce dynamic temperature for the loss to distinguish between class and subclass, while TSC does not have subclass and therefore the dynamic temperature is less effective.
Table Combination of TSC and SBCL with dynamic temperature.
CIFAR 100-IT
Dynamic temperature
TSC
SBCL
Imbalance Ratio
100
30
10
100
50
10
43.5
47.6
58.7
43.7
47.8
57.0
43.9(40.4)
48.0(+0.4)
59.2(+0.5)
44.9(+1.2)
48.7(40.9)
57.9++09))
Warm-up on ImageNet-LT. Instead of using the SCL at the warm-up stage for CIFAR datasets, KCL is adopted for ImageNet-LT and iNaturalist 2018 datasets to warm up the feature extractor. As Table TO shows, warm-up phase makes feature extractor improve accuracy on all splits of ImageNet- LT. This is because it prevents cluster assignment from feature random distribution at the begining and avoids using the SCL to make the feature space dominated by the head class the warm-up stage.
Table 10: SBCL with and without warm-up stage on ImageNet-L.T
Method
Many
Medium
Few
All
SBCL without warm-up
62.9
49.6
29.3
52.0
SBCL with warm-up
63.8
51.3
31.2
53.4
Advantages of cluster validity. Actually, previous studies (Kang et al.) 2020 Li et al. 2022) have proven that randomly sampling balanced instances as positive pairs (such as KCL, TSC) is better than sampling all instances of the same class positive pairs (such as SCL). However, this strategy may destruct instance semantic coherence. Table II we replace the first team (regard subclasses as positive pairs) with the balanced positive sampling strategy (KCL) to prove this on ImageNet-LT. As the results show, subclass- subelass-balancing adaptive clustering strategy brings more improvement to SBCL than balanced positive sampling strategy.
17

---

Under review as a conference paper at ICLR 2023
Table 11: Subclass-balancing adaptive clustering strategy improves more than balanced positive sampling strategy on ImageNet-LT.
Methad
Many
Medium
Few
All
FCL
61.4
47.0
28.2
49.8
KCL
62.4
49.0
29.5
51.5
TSC
63.5
49.7
30.4
SBCL (KCL)
63.3
49.5
30.6
52,2
SBCL
63.8
51.3
31.2
53.4
COCO object detection and instance segmentation. In this section, following the experiment Mask R-CNN (He al.] (017)) conduct the object detection and instance segmentation experiments on COCO dataset. The schedule is the default 2x in (He) al. 2020). Table [2] shows the pretrained model trained by SBCL outperforms it learned with other contrastive learning for the downstream tasks.
setting in (He al., 2020),
Table 12: Object detection and instance segmentation results COCO dataset. The representa- tion model is trained on ImageNet and ImageNet-LT. We report results in bounding-bow AP (Apb)) and mask AP (Apm)).
ImageNet
ImageNet-LL
Method
AP
APPo
AP
AP
AP10
AP
CE
40.1
59.8
43.3
38.1
57.4
41.2
APM
C1. eH al. 2020
40.4
60.1
44.1
39.7
59.4
42.7
KCI KKnng al 2020
40.8
60.6
44.0
39.4
59.1
42.6
SBCL
41.1
60.811022
44,23101
40.01102
59.610.2
43,0110.
random init.
51.5
33.5
31.4
51.5
33.5
CE
34.9
56.6
37.0
33.3
54.2
35.4
Apma
He
KCL KKnng at at. 2020
35.5
57.4
37.8
34.4
55.8
36.4
37.320.1
Combining SBCL with ensemble-based methods. Another line of research to address the long- tailed problem the ensemble-based methods, such RIDE (Wang et al.) 2021ap, which incorporates multiple models in a multi-expert framework. Here we show that SBCL can also be leveraged to boost the performance RIDE (Wang al.] [2021a). state-of-the-arr ensemble-based method. To implement SBCL with RIDE, we follow (I.i et al. (022) to simply replace the feature extractor on stage-1 training in RIDE with that trained with SBCL and keep the stage-2 routing training unchange. As shown in Table 13 applying SBCL to RIDE improves its performance with a significant gap, outperforms the combination of TSC and RIDE on all different number experts.
Table 13: Performance of the combination of SBCL and state-of-the-arr ensemble-based method
RIDE Wang et al. 2021a) with ResNet-50 (He al. 2016) on ImageNet-LT.
Method
Many
Medium
Few
All
(2 experts)
65.8
$1.0
34.6
54.4
RIDE (3 experts)
66.2
51.7
34.9
54.9
RIDE (4 experts)
66.2
52.3
36.5
55.4
TSC+ RIDE experts)
68.4
51.3
36.4
55.9
TSC+ RIDE (3 experts)
69.1
51.7
36.7
56.3
TSC+ RIDE (4 experts)
69.2
52.4
37.9
56.9
SBCL RIDE (2 experts)
68.6
51.9
36.5
56.2
SBCL RIDE (3 experts)
69.2
52.4
36.9
56.8
SBCL. RIDE (4 experts)
52.6
38.1
57.1
Combining SBCL with PaCo CCui et al. 2021 and BCL (Zhu et al., 2022). PaCo CCui et al. 2021) and BCL (Zhu et al.; 2022) proposed variants of supervised contrastive loss and jointly train both the proposed loss and classification loss to improve long- maigtail recognition, while we focus on the two stage pipeline, especially the first stage of representation learning. In this experiment,
18

---

Under review as a conference paper at ICLR 2023
show that using models pretrained with both TSC and SBCL initialization could improve the performance both PaCo and BCL. The results can be found in Table 14 and we can see that SBCL renders larger performance gain than TSC. The improvement over PaCo and BCL demonstrates the effectiveness of SBCL. in long-tail recognition, and shed lights on potential future work to evaluate the combination of multiple techniques of long-tail recognition achieve new SOTA results.
Both PaCo and BCL adopt a single-stage stage pipeline where a classifier is trained with both classification loss and supervised contrastive loss. To further test the effectiveness of SBCL, we replace the supervised contrastive loss they used with SBCL and retrain all other techniques they proposed for fair comparison. The results can be found in Table 15 over PaCo and BCL.
We can see that using SBCL could improve
Table 14: Performance of the combination of SBCL and extended contrastive methods (PaCo (Cui et al. 202T) and BCL (Zhu et al. )) with ResNext-50 (Saining et al. 2017 on ImageNet-LT.
Method
Many
Medium
Few
All
PaCo
64.4
55.7
33.7
56.0
BCL
67.9
54.2
36.6
57.1
TSC+ PaCo
66.4
53.8
35.7
57.1
TSC+ BCL
69.0
56.3
37.8
58.7
SBCL PaCo
66.9
56.1
38.4
57.9
SBCL BCL
69.5
56.7
39.0
59.2
Table 15: Top-1 accuracy on CIFAR-100-LT with ResNet-32 (He et 2016
Method
Many
Medium
Few
All
CE
65.2
37.1
9.1
38.3
PaCo
52.0
BCL
69.7
53.8
35.5
53.9
SBCL PaCo
68.3
54.8
30.8
$2.3
SBCL BCL
69.4
54.5
37.6
54.5
Combining SBCL with two-stage methods. MisLAS (Zhong al. 2021 provides two major technical contributions to improve the two-stage pipeline of the longtail recognition, i.e., label-aware smoothing and shifted batch normalization. Both techniques are designed for improving the second stage (the classifier learning), while our method is to improve the first stage of representation learning. Thus, MisLAS could be combined with SBCL and TSC. In Table 16 combing MisLAS with both SBCL and TSC. The results show that both SBCL and TSC improve the performance of MisLAS and SBCL renders more performance boost than TSC.
report the results of
Table 16: Performance of the combination of SBCL and MisLAS (Zhong et al. 2021 with ResNet-
50 (He et al. 2016) on ImageNet-LT.
Method
Many
Medium
Few
All
MISLAS
52.7
TSC+ MISLAS
63.7
50.5
36.0
53.6
SBCL MiSLAS
64.1
52.0
36.4
54.5
Freezing the pretrained model for object detection. We assess the representation trained ImagNe//ImagNett-l for the downstream detection task. We freeze the pretrained backbone to train the Faster R-CNN (Ren reports the average mAp on PASCAL VOC dataset. The representation trained by SBCL obtains better performance for object detection.
2015). and use the same training schedule in Section 4.3 Table 17
Hyperparameter studies. Here, we study the effect of hyperparameters 3 and 6. Note that controls the balance of two loss terms in Eq. 4 and determines the lower bound the cluster size in Eq. 3 Specifically, CIFAR-100-L3 with imbalance ratio 100, vary the values of from (0.1,
19

---

Under review as a conference paper at ICLR 2023
Table 17: Object detection on PASCAL VOC when the pretrained model is frozen.
Method
Pretrain dataset
mAP
ImageNet
79.3
Supervised
ImageNct-LT
70.9
ImageNet
80.5
KCL Kang et al.
2020
ImageNet-LT
73.4
ImageNet
80.7
SBCL(ours)
ImageNet-LT
74.5
0.2, 0.5, 0.8, 1.0, 2.0) with 8 10 and the value of 5 from (5, 10, 20, 30, 50, 100) with 3 = 0.2. The results are summarized in Table We observe that the smaller 3 values (between 0.1 and 0.5) can achieve relatively good performance, with the best being 0.2. This observation aligns with our intuition of emphasizing the subclass-level contrastive loss, because smaller 3 is equivalent to putting more weights on the first term of Eq. 4 which corresponds to the subclass-level contrastive, For the values between 5 and 30 yield high accuracy, with the best being 10. We can see that large values (6 50, 100) lead significant drop in performance. We argue that this because large value would result in subclasses that contain more instance than tail classes and therefore affect the bablass--bllane, leading to suboptimal performance. In addition, smaller ó value (6 = 5) also causes performance drop; the reason could be small cluster size may let similar instance being assigned to different clusters and therefore affect the learned representations. Therefore, we fix 8 0.2 and 5 = 10 for all experiments.
Table 18: Hyperparameter study of 3 and å CIFAR-100-LT with imbalance ratio 100.
8
0.1
0.2
0.5
0,8
1.0
2.0
ACC(%)
44.6
44.9
44.5
44.1
43.9
42.1
30
50
100
ACC(%)
44.3
44.9
44.6
44.3
42.9
42.3
Visualization of generated clusters. In Figure show the clustering results of ImageNet-LL training images generated by subclass subclass-balancing adaptive clustering algorithm. From the results, we that the algorithm is able to find the subclasses with similar patterns, helping the model learn semantic coherent representations. For example, the two subclasses in the bottom-left are telephone with/without human.
(a) Shark
(b) Monkey
(c) Telephone
(d) Airplane
Figure 6: Visualization of subclasses generated by SBCL. Images with green and orange boarder are randomly drawn from different subclasses within the same classes. We can see that SBCL could produce semantically coherent subclasses.
20

---

Under review as a conference paper at ICLR 2023
A.2 ADDITIONAL INFORMATION
Benchmark datasets statistical information and Implementation Details. We summarize the statistical information the three benchmark datasets in Table 19 Following (Kang et al.) 2019 2020 Li al. 2022), apply SBCL on the long-tailed recognition by using two-stage training strategy: (i) train the representation with SBCL; (ii) learn a linear classifier on top of the fixed representation. The training process is the same as TSC (Li al.; (022). Thus, we use TSC default hyperparameters and implementation details for the representation learning. For CIFAR-100-LT dataset, all experiments are performed on NVIDIA RTX 3090 GPUs. For ImageNet-LT and iNaturalist 2018 datasets, we perform the experiments on 8 NVIDIA RTX 3090 GPUs. The detailed hyperparameters of TSC and SBCL are given in Table 20
For the classify learning, training the linear classifier strategy is the same with TSC (Li ct al. 2022 so, we use TSC default hyperparameters and implementation details for the classifier learning. For the detect model learning, we follow MoCo (He et al.) (220)) to adopot the same setting, hyperparameters and evolution metrics with R50-C4 backbone. For Pascal VOC dataset, we train Faster R-CNN (Ren) al., 2015) on VOC07+12 and evaluate on the test set of VOC07. For COCO dataset, we train Mask R-CNN (He et al. 2017) on train2017 set and evaluate on val2017 set.
Table 19: Statistics of datasets. The imbalance ratio p = no /TC.
Dataset
classes
training data
test data
imbalance ratio
50,000
10,000
{100, 50, 10}
ImageNet-LT
1,000
115,846
50,000
256
iNaturalist 2018
8,142
437.513
24,426
500
Table 20: Hyperparameters used by different loss functions for benchmark datasets. The detailed hyperparameters of iNaturalist 2018 are the same as the ImageNet-LT.
Iet-eNet-LT
CIFARI00-LT
Hyperparameters
TSC
SBCL
TSC
SBCL
module
MoCo
MoCo
SimCLR
SimCLR
warm-up epoch
200
200
0
10
epoch
400
400
1000
1000
batch size
256
256
1024
1024
learning rate
0.1
0.5
0.5
learning rate schedule
cosine
cosine
cosine
cosine
memory size encoder momentum
65536
65536
0.999
0.999
feature dimension
128
128
128
128
softmax temperature
0.07
0.07
0.1
0.1
k-positive number
6
4
hyperparameter of / hyperparameter of
0.2
0.2
0.2
0.2
20
10
hyperparameter of a
Limitations. SBCL has some limitations. First, clustering the head class in SBCL takes a long time on the training phase, especially for ImageNet-LTT and iNaturalist 2018. Second, SBCL requires knowing the number of samples in each class to decide the cluster number; so, it is not applicable to problems where the number of samples is unknown.
Social impacts. This work aims to propose novel representation learning to help people resolve the bias in the real world data recognition, which might has positive social impact. We do not foresee any form of negative social impact induced by our work.
Privacy information in data. All datesets we used in the experiment are public. The datasets only include the pictures, which most are animals and plants. No private information is included.
21

---

Under review as a conference paper at ICLR 2023
Baseline information. We report the accuracy of KCL and TSC different benchmark datasets from (Li et al. 2022). For SwAV (Caron et al. 2020), PCL (Li et al. 2021a and B BYOL
Grill
al. 2020 their official open-souree implementations.
SWAV offical implementation: itp.ememtaaionnttttss://///////a
2PCL offical implementation:
BBYOL offical implementation:
cree/master/byal
22


Reviews:
[{"review_id": "o5_ZNPjEMd5", "reviewer": "Reviewer_8khH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for contrastive learning on a long-tailed training dataset. The proposed method is called Subclass Balancing Contrastive Learner (SBCL), which uses clustering within the head classes to determine fine-grained subclasses of size similar to the tail clusters. The new subclass labels for the head instances are used w/ tail classes to perform supervised contrastive learning. The paper contains extensive experiments over several vision benchmarks, including classification, object recognition and instance segmentation tasks. In almost all cases, SBCL performs better than several other baselines.\n", "summary_of_the_review": "(Please see the strengths and weaknesses above.) I would appreciate if the authors respond to some of my questions and concerns. ", "strength_and_weaknesses": "**Strengths**\n\nS1: Extensive set of experiments on several vision tasks show SBCL does better than other representation learning methods for learning on long-tail distribution. Many ablations were also included to show several design choices made. \n\nS2: I think the idea of clustering the head class(es) into sub-classes and using the cluster ids as labels is novel.\n\n**Weaknesses**\n\nW1: While there are extensive sets of experiments, the quantitative results (Table 1-5) do not come with standard errors/deviations. The experiments were done w/ 5 random seeds, then I wonder why have the standard errors not been included in the table? Understanding how reliable these improvements seems like an important thing to report.\n\nW2: There is no discussion on the convergence of the model, and the number of training steps/epochs, loss trajectory or learning rate used. Since SBCL uses warm-up and adaptive clustering which may lead to a jump in the loss fn, studying these properties seems highly relevant. It is unclear if the number of training steps/epoch used for the training the feature extractor is the same for all the baselines and the proposed SBCL.\n\nOther weakness/questions:\n- I wonder why the disjoint subset analysis was not shown for the CIFAR-100 LT dataset?\n- The conclusion from the analysis in section 4.4 lacks clarity. What is the conclusion from this analysis? What is the significance of the finding: “the intra-class distance of SBCL is almost invariant with the decreasing of group split”. Why the increase in the intra-class distance for KCL & TSC is attributed to the head class? The discussion in this section needs some work.\n- Was the negative sampling bias corrected when computing the loss function? While SBCL balances the label distribution (via cluster labels for head instances), it is unclear that this would mitigate the sampling bias in selecting negatives. Typically, models use log(P(x)) correction [1] when using in-batch softmax loss to prevent the head instances to be overly treated as negatives than the tail instances.Having a baselines that uses this logit adjustment would be a definite plus.\n\n[1] Menon et al. (2021), Long-tail learning via logit adjustment, (ICLR 2021).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "abiTdSRnVLW", "reviewer": "Reviewer_QXy7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method called subclass balancing contrastive learning for long-tailed problems. The key idea is to divide head classes into subclasses to balance the number of instances in each subclass. The proposed method is evaluated on widely used image classification datasets as well as some additional vision tasks such as object detection and semantic segmentation. ", "summary_of_the_review": "I think the motivation of the paper is sound and the proposed method is somehow reasonable. However, I have doubt on the effectiveness of the proposed method since I do not think the current experiment is not sufficient for supporting it. As such I do not think this paper is ready for publication. I would like to hear the feedback from the authors on the points I listed in the weakness. \n\n### after rebuttal\nI became more positive about the paper through the rebuttal and I increased my score accordingly, but I still have some concerns on the effectiveness of the proposed method.", "strength_and_weaknesses": "## Strength\n1. The proposed method is evaluated on wide variety of datasets. Especially the evaluation on other tasks than image classification, i.e., object detection and semantic segmentation, is given.\n1. The effectiveness of the proposed method is shown in the experiment at least to some extent.\n1. The paper generally reads well\n\n## Weakness\n1. The comparison with stronger baselines such as [A1-3] is missing. The reported accuracies in these works are actually higher than the proposed method.\n1. According to Table 4, the performance of the proposed method on CIFAR100-LT becomes 43.7 if the dynamic temperature is not applied. This performance is lower than TSC. I am afraid this suggests that it is the dynamic temperature that makes the paper perform better than other contrastive learning methods. To show the effectiveness of the proposed idea, it is necessary to show the results of other contrastive learning methods such as SCL, KCL, and TSC with the dynamic parameters. \n1. I am afraid the performance of the proposed method is strongly affected by the batch size since the distribution in each mini batch is important in calculating the loss in Eq. (4). It would be nice if the authors could provide the analysis on this point.\n1. Table 5: “(i) the inter-class distances of SBCL is larger than the other methods, which implies that SBCL can push different classes far away from each other, and thus clear the decision boundaries between classes; ” It is true that the inter-class distances of SBCL is larger than those of the other methods, but the intra-class distance of SBCL is also larger. Actually, the intra-class distances of other methods are much smaller than those of the proposed method, which means that the other methods have favorable characteristics.\n1. P8 “We attribute that to feature space dominated by the head class.” Please rephrase and elaborate more so that the claim of this sentence becomes clearer.\n\n[A1] Zhong+, “Improving Calibration for Long-Tailed Recognition”, CVPR 2021    \n[A2] Cui+, “Parametric Contrastive Learning”, ICCV 2021  \n[A3] Zhu+, “Balanced Contrastive Learning for Long-Tailed Visual Recognition”, CVPR 2022\n\nMinor points.\n1. P2, (c) “… and instant segmentation” -> “instance segmentation”\n1. Eq (2): $\\tilde{V}_i^k$ -> $\\tilde{P}_i^k$?\n1. The second line in Section 3 $|V^+_{i, k}|$ does not appear in equation (2)\n1. P8 “successful captured” -> ““successfully captured””\n1. P9 “well-haved” What does this mean?\n1. P9: “the table 4” -> “Table 4”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XRHBUuMzKfL", "reviewer": "Reviewer_aZoT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes subclass-balancing contrastive learning (SBCL) for long-tailed recognition. It divides head classes into several fine-grained subclasses via adaptive clustering, to enforce equal probability of head/tail classes to be engaged. After clustering, a bi-granularity contrastive loss is proposed. In addition, dynamic temperature and warm-up are also used for SBCL. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "The motivation is clear. The proposed method is reasonable though some related works have similar ideas.\nThere are some weak aspects in empirical study, as pointed out above.  ", "strength_and_weaknesses": "Strength:\n\n(1) The motivation of this work, i.e., dividing head classes to sub-classes to deal with class imbalance, is clear. The basic idea of enforcing both instance- and subclass-balance by leveraging supervised contrastive learning is reasonable. \n\n(2) The paper is well organized and clearly presented. The method seems easy to reproduce as well. \n\n(3) Experiments on LT datasets and downstream tasks are adequate. \n\nWeaknesses:\n\n(1) Though the motivation and proposed method are reasonable, the comparison with similar long-tailed contrastive learning approaches is not complete. In addition to KCL and TSC, there are some other related solutions for long-tailed recognition problem, like PaCo [1] and BCL [2]. What is the advantage of sub-class clustering over these solutions? \n\n(2) There are quite a few hyper-parameters, like temperature, $\\beta$, $\\delta$ and $\\alpha$ in Eq. 5. Though the authors provide some hyper-parameter study on CIFAR-100-LT, I still wonder how the hyper-parameters change across datasets? Do I need to re-adjust these hyper-parameters with lots of efforts for a new dataset to achieve good performance? \n\n(3) In Table 1, it seems these contrastive methods are implemented on different code base. To verify the effectiveness of SBCL over the SCL baseline, it is better to use your own implementation on SCL. \n\n(4) There are quite a few works trying to re-balance the supervised contrastive loss for long-tailed recognition. What if we instead use non-contrastive methods like BYOL? For non-contrastive methods, the subclass-balancing problem naturally disappears. \n\n\n[1] Parametric contrastive learning, ICCV 2021.\n\n[2] Balanced Contrastive Learning for Long-Tailed Visual Recognition, CVPR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VX7iLAnTdbo", "reviewer": "Reviewer_S8iv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on improving supervised contrastive learning to learn better features over class-imbalanced dataset and to achieve better long-tailed recognition (LTR). The paper argues that, in the literature of LTR, typical class-balancing techniques such a data reweighting, resampling and supervised contrastive learning (SupCon) introduce biases as data instances are imbalanced although classes are balanced in training. Motivated by this, the paper proposes a method called \"subclass-balancing contrastive learning (SBCL)\", which clusters training data in balanced subgroups within (head) classes such that subgroups and tail classes are balanced in size. SBCL samples positive pairs of data within subgroups or tail classes in SupCon. It reports improved LTR performance in experiments.", "summary_of_the_review": "In the context of long-tailed recognition, the paper proposes a method to cluster head classes into small subgroups such that subgroups and tail classes have roughly equal number of training data, allowing for more balanced learning of feature representations using Supervised Contrastive Learning. However, the paper lacks justifications for some design choices and results, and misses citations and lacks analyses. Therefore, the paper is rated as \"marginally below the acceptance threshold\".", "strength_and_weaknesses": "Strength:\n- Clustering head classes into subgroups for group-level balanced learning is interesting in the context of long-tailed recognition.\n- The study of long-tailed pretraining for downstream tasks is interesting.\n\n\nBelow are weaknesses.\n\nFor LTR, it is misleading to state that \"the major challenge (of LTR) is the discrepancy between the imbalanced training data distribution and the balanced test set.\" In the real world, testing data should also follow the same long-tailed destruction, but the requirement or evaluation metric is \"class balanced\", i.e., it requires recognizing all imbalanced classes equally well.\n\nWhile the paper states that \" it might be wise to break down the head classes into multiple semantically coherent subclasses,\" it should discuss whether the clustering groups correlate to the \"semantically coherent subclasses\". In other words, how to guarantee the clusters are aware of subclasses?\n\nAlgorithm 1 describes a heuristic method about clustering data of a head class into equal-size subgroups. However, would subgroups produced later be noisy (as they will have large distances to their mean features) compared to those generated in early steps? Can authors provide more analyses or visualizations?\n\nTable 2: While SBCL achieves better performance than the compared methods, [R1] performs much better by tuning weight decay in training. [R1] finds that regularizing network weights (towards more balanced norms in network filters) is crucial for LTR, and doing so leads to state-of-the-art. Therefore, it is unclear why SBCL improves LTR (slightly): whether it is due to balanced subgroups of head classes or whether it helps learn balanced weights in networks. Authors should provide more analyses.\n\n[R1] Alshammari, et al., \"Long-Tailed Recognition via Weight Balancing\", CVPR 2022\n\n\nSection 4.3 and Table 3: Can authors list results by \"training from scratch\" (i.e., without pretraining), and \"freezing backbone\" (i.e., not finetuning the whole pretrained network). The two baselines are important as comparisons can clearly show how much imbalanced/balanced pretraining helps the downstream detection task.\n\nTable 5: It is confusing to compare the absolute distances of intra- and inter-class distances. Instead, a ratio between intra- and inter-class distances might better show how well the per-class examples are separated in the feature space. If computing such a ratio, SBCL seems to perform worse than the other methods. Can authors discuss this point?\n\nThe paper does not show how many epochs/iterations needed for convergence by SupCon and SBCL, and for SBCL's warm-up learning. Can authors provide a learning curve, e.g., loss vs. iterations/epochs, accuracy vs. iterations/epochs?\n\n\ntypos:\n- \"a instance\" in Section 3\n- \"a overly-loose/dense cluster\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject