Global Contrastive Learning for Long-Tailed Classification
Authors: Thong Bach
Anh Tong
Truong Son Hy
Vu Nguyen
Thanh Nguyen-Tang
Venue: Accepted by TMLR ()

Abstract: We consider the long-tailed classification problem in which a few classes in the training data dominate the majority of the other classes. For concreteness, we focus on the visual domain in this paper. Most current methods employ contrastive learning to learn a representation for long-tailed data. In this paper, first, we investigate $k$-positive sampling, a popular baseline method widely used to build contrastive learning models for imbalanced data. Previous works show that $k$-positive learning, which only chooses $k$ positive samples (instead of all positive images) for each query image, suffers from inferior performance in long-tailed data. In this work, we further point out that k-positive learning limits the learning capability of both head and tail classes.  Based on this perspective, we propose a novel contrastive learning framework that improves the limitation in k-positive learning by enlarging its positive selection space, so it can help the model learn more semantic discrimination features. Second, we analyze how the temperature (the hyperparameter used for tuning a concentration of samples on feature space) affects the gradients of each class in long-tailed learning, and propose a new method that can mitigate inadequate gradients between classes, which can help model learning easier. We name this framework as CoGloAT. Finally, we go on to introduce a new prototype learning framework namely ProCo based on coreset selection, which creates a global prototype for each cluster while keeping the computation cost within a reasonable time and show that combining CoGloAT with ProCo can further enhance the model learning ability on long-tailed data.

Published in Transactions on Machine Learning Research (09/2023)
Global Contrastive Learning for Long- Tailed Classification
Thong Bach
thongbtqu@gmail.com
Anh Tong
Korea Advanced Institute of Science & Technology anhtong0kaist.aco
Truong Son Hy
Indiana State University
TruongSon.
HySindstate. Huinngoo..Hyinnsstatae
Vu Nguyen
Amazon vu@ieee. org
Thanh Nguyen-Taang
Johns Hopkins University
aguyentêcs, edu
Reviewed on OpenReview:
Abstract
We consider the long-tailed classification problem in which few classes in the training data dominate the majority of the other classes. For concreteness, we foeus on the visual domain in this paper. Most current methods employ contrastive learning to learn representation for long-tailed data. In this paper, first, we investigate k-positive sampling, popular base- line method widely used to build contrastive learning models for imbalanced data. Previous works show that k-positive lcarning, which only chooses positive samples (instead all positive images) for each query image, suffers from inferior performance in long-tailed data. In this work, we further point out that k- k-positive learning limits the learning capability of both head and tail classes. Based on this perspective, we propose a novel contrastive learn- ing framework that improves the limitation in k-positive learning by enlarging its positive selection space, it can help the model learn more semantic discrimination features. Sec- ond, we analyze how the temperature (the hyperparamcter used for tuning concentration of samples on feature space) affects the gradients of each class in long-tailed learning, and propose a new method that can mitigate inadequate gradients between classes, which can help model learning casier. We name this framework CoGloAT. Finally, we go on to introduce new protolype learning framework namely ProCo based on coreset selection, which creates global prototype for each cluster while keeping the computation cost within reasonable time and show that combining CoGloAT with ProCo can further enhance the model learning ability on long-tailed data. Our code is available at CoGloAT ProCo
1 Introduction
Real-world data usually follows long-tail distribution, where only few classes dominate the dataset (namely, head classos). In contrast, most other classes have a small number of samples (namely, tail classos). This long-tailed data poses a major inferential challenge to traditional deep learning models whose training is biased by the head classes and whose performance quickly deteriorates when the data is imbalanced (Wang

---

Published in Transactions on Machine Learning Research (09/202))
Pull
Pull
Push
Push
Push
Push
Push
Pull
Pull
Pull
Figure 1: Overview of how CoGloAT and ProCo learn during training. CoGloAT learns to pull closer images of the same class while pushing apart images of different classes across-batches. Meanwhile, ProCo builds protolype (center) for each class, by which points of the same class are pulled loward the class protolype whereas points of the other classes are pushed away.
et al. 2020 Cao et al. 2019 Zhang et al. 2021c). Various approaches have been proposed to address such
issues, with typical mothods including dlass-balance re-sampling (Shen et al. 2016 More 2016 Zhang
et al., 2021c). class-level re- re-weighting (Alshammari et al. 2022 Lin et al. 2017 Zhang et al. 2021c and
ensemble learning (Zhou et al. 2020 Wang al., 2020 Zhang et al. 2021b). However, the prior approaches
rely on the classical cross-entropy losses which are sensitive to imbalanced data (Wang et al. 2020 Zhang
ct al. 2021c).
Many studies have shown that contrastive learning is more robust to noisy or unbalanced data (Khosla et al. Kang et al.) p020 therefore this algorithm is also widely applied to solve imbalanced data and achieves
2020
impressive results (Chen & Hc 2021 Li al.) 2022b Zhu et al. 2022 Liu et al.] 2021 Yang & Xu
2020
Its success is based on the contrastive loss function and large number of negative samples (He et al.
2020
which will help the model Iearn more robust and semantic discrimination feature (Liu et al.) 2021
Yang &
Xu 2020) thereby helping the model generalize well on the training data, and subsequently easily transfer to the test data or other data domains (Chen et al. 2020a Liu et al. 2021). Moreover, unsupervised contrastive learning creates more balance feature space, even when the data is highly skewed (Kang et al., 2020), which is the main factor that results in the failure of the previous methods. However, the model trained in an unsupervised manner often does not perform well since it does not use label information; thus it fails to learn rich semantic discrimination features due to lack of label information. Different from unsupervised contrastive learning, supervised contrastive learning can learn more semantic fcature space (Kang et all 2020p. Nonetheless, using the label during training makes features more imbalanced as compared with their unsupervised counterpart. To overcome this problem in supervised contrastive learning, instead of using all positive samples in the batch for training, Kang et al. (2020) proposes k-positive selection, a supervised contrastive learning method in which we limit the number of positive samples to k. therefore, it can reduce the imbalance effect. However, when the data extremely long-tailed, using k-positive alone is not enough to tackle the imbalance problem (Li ct al. 2022b). Besides contrastive learning, prototype learning (Li al. 2022b; Cui et al. 2021 Zhu al.j 2022) is also widely used to tackle long-tailed problems. This method constructs a prototype (center) for each class and pulls samples of the same class closer to the center while pushing samples of other classes away. But it remains clusive as to how to build good prototype and efficient prototype loss function.
Motivated by the limitations of the prior approaches, in this paper, we propose a training method namely Contrastive learning with i-Global positive selection and Adaptive Temperature (CoGloAT), which con- structs a more balanced feature space with better semantic discrimination features. In CoGloAT, we first use label information to select k-positive samples across batches for the query. This so-called k-global positive
2

---

Published in Transactions on Machine Learning Research (09/2023)
selection is in stark contrast with k-positive selection (Kang et al. 2020) which selects positive samples only within batch. Expanding the selection space across batches helps the query connect to more positive sam- ples; therefore the model can learn richer semantic discrimination features. Besides, we analyze the gradient of our model when training with imbalanced data, pointing out that the temperature value has an important role during the training. Based on the analysis, we propose a new method namely Adaplive lemperature. In this method, we view temperature as a re-balanced parameter, which can help us reduce the imbalance effect between head and tail classes during the training stage by balancing their gradients. Moreover, we propose prototype learning approach namely ProCo, which uses data distillation to efficiently construct prototypes. We empirically show that jointly training ProCo with CoGloAT can further improve the model performance.
In summary, our main contributions are a series of simple yet effective improvements that take a "global" perspective into contrastive learning to address the long-tailed (visual) classification (thus the name global contrastive lcarning):
Contrastive learning with k-global selection and adaplive temperature (CoGloAT): We introduce a new training strategy that can adapt to any contrastive framework to improve its performance in long-tailed learning namely k-global positive selection. After that, we analyze the model gradient and propose re- re-balanced techniques using adaptive temperature to further tackle the imbalance problem. We empirically evaluate with different settings and show that our model outperforms the state-of-the-art model with large gap.
Prototype learning via corcsct selection (ProCo): We propose a new prototype Icarning method named ProCo. Compared to standard prototype methods, ProCo efficiently computes protolypes from distilled coreset. Based this coreset, calculated prototypes that are used in ProCo can represent global features for the original dataset without using all the training data.
2
Related work
Long-tailed recognition one of the challenging problems in computer vision. There are many studies
focusing on this problem with different solutions (Zhang et al. 2021c Zhu et al.] 2022 Shen al. 2016 More 2016 Alshammari et al. 2022 Lin al. 2017 Zhu et al. 2022 Kang et al. 2020 Li et al. 2022b)
The general idea is to try to re-balance the effect between head and tail classes during training from data perspective or model perspective.
Data perspective The most obvious way to solve long-tailed problems is to create balanced dataset, where the samples of head and tail classes need to change to become more balanced. There are two popular approaches including over-sampling (Shen et al. 2016 Sarafianos et al. 2018). and under-sapling (More
2016 Drummond et al. 2003; Buda et al. 2018 While the former simply creates more data for tail classes by duplicating samples, average samples, or learning its distribution then genorating more data. The latter needs to reduce the samples of head classes by random removal procedures. However, the over-sampling approaches easily lead overfitting (Bunkhumporapat et al.) 2012 More 2016); sampl-g loss data information (More 2016). To overcome the limitation of the two approaches,
Pacnceess,Prrrkr Park al. (2022) introduces a over-sampling method that creates larger tail samples by utilizing rich context fcatures in head classes. Besides, we can use other data augmentation techniques (Li et. al. 2021 Zhang et al. 2021d) which can give us a larger size for tail class, or data distillation techniques (Wang et al. 2018 Killamsetty et al. 2021 Ghadikolaci et al., 2019 Zhang et al. 2021a) to extract a smaller set for head classes while ensuring it can represent the distribution of the whole data.
Model perspective There are many techniques utilizing models to lackle long-tailed learning including re- re-weighting (Alshammari et al.] 2022 Lin et al.] 2017 Khan et al. 2017), ensemble learning (hou et al. 2020 Wang et al. 2020 Zhang et al. 2021b), or two stages training (Kang al.] 2019 Zhou al. (0220). In re-weighting methods, we set weights for each class, usually reducing head class dominating by setting it small weight or enlarging the weight of tail classes to emphasize its contribution. The simplest way is to set weights
3

---

Published in Transactions on Machine Learning Research (09/2023)
inverse to number of samples in class. Besides, normalizing the weight of network (Alshammari et al. 2022)
also shows an impressive result, or extends focal loss to Equalized Focal Loss (Li et al. 2022a] for efficient one-stage object detection. Along this direction, decoupling techniques use two stages of training, where we need to learn a good representation in the first stage and fine-tune it in the second stage. This training strategy shows that. without carefully designing a model, we can also achieve good performance (Kang el al. 2019). Another typical method usually used in long-tailed learning is ensemble (Zhou et al.) 2020) Wang ee al.) 2020 Zhang et al., 2021b), where we can joint-trainin a multiple branches network, each branch will contribute and learn specific information during training, then we combine the results of these branches together to predict the final result for long-tailed data.
Contrastive learning iiee learning has got altetiion recently because of its success in training for large unlabeled data (Chen et al. 2020a He et al. 2020 Chen et al. 2020c Doersch et al. 2015 Grill ct al. 2020 Caron et al.] 2018 Chen et al., 2020b 2021 Tian et al.) 2020), and labeled data (Khosla et al. 2020 In the decoupling method that we have mentioned above, the key problem is how can we train a strong backbone model at the first stage then use it as pre-trained and fine-tune the whole top layer of the model in the second stage. This is similar to the behavior of training contrastive model (He et al. 20200 Chen et al. 2020c a. Chen & He. 2021), where we use a large amount of unlabeled data to learn invariant features from input data, then use those features to initiate for downstream tasks. Therefore, castrastive learning becomes a popular method in the first stage of training of decoupling. Recently, there have been an increasing number of studies trying to understand the behavior of contrastive learning on long-tailed data, For example, Kang et al.] (2020) explore that self-supervised learning creates balance but lacks semantic feature space, and they propose k-positive learning which simply limits the number of positive samples in batch not greater than k to ensure the balance between head and tail class. Liu et al. (2021) hypothesize that contrastive learning works better than supervised learning because il can transfer useful features from head classes to tail classes. Then they analyze the gradient and propose a method following theoretically- principled labelddisribbttionn-aaar margin loss (LDAM) (Cao et al. 2019) to re-balance feature space. [Zhu et al. 2022) show that supervised contrastive learning is not the optimal solution for long-tailed data and introduce class averaging and class complement in the loss function to make SupCon possible when training with long-tailed data. Zhoo et al. (2022) Jiang et al. (2021) utilize the memorization effect of deep neural networks to recognize tail sample and enhance it in both data and model perspective.
Prototypical contrastive learning Training on self-supervised learning with long-tailed data generates more balanced feature space than the supervised counterpart. However, when the dataset is extremely skewed, it is still dominated by head class. Several works use prototype learning to re-balance feature space et al., 2021 Li et al.] 2022b Zhu et al. 2022) to overcome this issuc. The key idea is to generate a set of prototypes (centers) for each class, then pull all samples of the same class closer to its prototypes and push samples from other classes far away. However, how to efficiently construct such prototypes is an active research area. Previous works (Li et al.) 2020 2022b) construct these prototypes by utilizing the large queue size in MoCo to save computation times. Nonetheless, it can not adapt to another contrastive framework, and when the dataset size is too large, data in the queue size can not represent the whole dalaset distribution. In this paper, we propose another approach that can generate prototypes to cover the training data distribution without using all samples within an acceptable time.
3 Method
The training samples S = {(z, wل)رار are i.i.d. drawn from a distribution over instance space X and label space )'= = (1....,CC C} where C' is the number of classes. Our goal is to learn function f that maps the input
data space X to the label space D which minimizes the misclassification error err( { (x [1f (z) # v}l. Inspired by previous works (Kang et al. 2020 Li ct al. 2022b Khosla et al. 2020 we proposc a general framework that can apply to existing contrastive learning models e.g., MoCo (He el al. 2020), SimCLR (Chen) et al. 2020a) to address the imbalance issue in long-tailed learning. Our model can create rich semantic features while reducing the dominating of head class during training. This framework includes two modules namely CoGloAT and ProCo. In the first section, we introduce Contrastive learning with k-global selection and adaptive temperature (CoGloAT), method that can help contrastive learning models learn more easily
4

---

Published in Transactions on Machine Learning Research (09/2023)
in the long-tailed setting. Then, further improve Global contrastive learning by prototypical learning, where we propose another efficient method for prototype learning via. coreset selection (ProCo).
3.1
Contrastive Learning with k- Global Positive Selection and Adaptive Temperature
3.2
K-global positive selection
SupCon (Koslla et al. 2020 and its improvement version for taiged learning: k-positive learning (Kang ctt al. 2020 Li al. 2022b) select positive sample from the same batch for each query sample. However, this selection strategy can limit the learning ability of the model, especially in tail classes, where the class size is small. More specifically, for arbitrary class i, the probability that & samples of a class i appear in the same batch will be proportional to the number of samples in i. Hence with the tail class where the class size is usually small, it is more difficult to have enough k positive samples in the same batch, and this makes the learning on these classes harder. To overcome this problem, we propose k-global positive selection. Our method expands the positive selection space for each query sample from within the batch to across batches. To implement the proposed k-global selection algorithm, we have redefined the DataLoader. Specifically, our DataLoader will include an additional variable containing information about the index of samples with the same labels in the training data. During training, for each sample in batch B, we will randomly select k positive samples with the same label in training data (global selection) instead of choosing k positive samples with the same label in batch (batch selection). This simple modilication helps the query sample connect with all other positive samples of that class, therefore enlarging its selection space, and helping the model learn casier. Our formula for across- across-batched batches positive selection is defined as:
N
Leontrastive
1
Σ
Σ
((i,j),
(1)
N(k + 1) t=1
of Elet JuPt
where
exp (v: /T/T)) /T)
{(i,j) = =log
exp (ws uz /T) Eejev. exp (wa 0z/T)
Here of is the augmented view of query vr p is the k posilive samples selected by k-global positive, and Vi are examples in the current batch excluding
This selection strategy is simple yet extremely helpful for tail classes because, in long-tailed data, the total number of tail classes is usually small. Therefore we can gather all the samples in tail classes together, each sample can utilize information from all other samples to learn, then our approach can efficiently learn the feature space of tail classes.
3.3 Adaptive Temperature
In this soction, we discuss the gradient of the contrastive learning model, which has played an important role in classical machine learning to mitigate the inbalance effects (Tang et al. 2020: Ren et al. 2020). However, this result has not been explored in the contrastive setting. Therefore, based on the previous work of Waang & Liu (2021), we investigate its behavior in the context of the contrastive model and introduce technique called adaptive temperature based temperature value, which can effectively reduce the magnitude difference of gradients between head and tail classes.
Gradient analysis Normalizing the gradient between head and tail classes has received much attention in cross-entropy learning (aan al. 2021). However, in contrastive learning, this problem has not been exploited much. As introduced in Khosla et al. (2020) with input 2; and embedding function f0, the
5

---

Published in Transactions on Machine Learning Research (09/2023)
gradient of embedding vector 2; f(z)) is defined:
OL
dz,
Σ
Zp
Pip
PP(i))
+ Σ
unPmn
(2)
PEP(()
nEN(1)
Here, L is the out in
Khosla et al. 2020 which is defined as:
exp Zp/T
sup out
log
PP(1) PEP()
Z(EA(1) exp Fi Ra/
where т is temperature parameter, P(i) is set of all positive samples of 2, (set of all samples in batch that have the same label with z;), (i)) is set of all negative samples (set of samples have different class label with 2)), and P.p = exp (z; /p/T)/ EaeA( exp (2; Za/T) where A(i) is set of all samples in the batch excluding (which means: A(i) 2())UZUN())
In cach batch, the gradient of sample will be equal to the gradient of a positive sample (P(i)) plus the gradient of negative samples (N())) and then divided by lemperature т. For easier analysis, set:
OL
Σ 2p PEP(()
Pip
(3)
O2
P(1))
OL sup
sup
Σ
Zn Pin
(4)
Oz
nEN(()
Now we rewrite the Eq. 2 as the combination of positive gradient and negative gradient
aL;
[aL;
aL, 0z,
0z:
O21
Applying the above formulas to GloCo, we will have IP(i)] k, which means that the number of positive samples for each query 2; is fixed. The gradient of positive samples in Eq. will be recalculated as follows:
OL
exp 2p/T) Σ 3p ElEA exp (2) Za/T)
dzh
PEP(i.)
where Pi, the set of & positive samples across batches of query x,, Thus, choosing only h positive samples during training will make the positive gradient of samples between different classes more balanced. This another explanation for why KCL. (Kang et al., 2020)) works well on imbalanced data. However, as mentioned in Eq. the gradient of embedding z; is combination of positive and negative gradients. Choosing positive samples helps to balance the positive gradient but does not balance the negative gradient. Expanding Eq. 4 for the negative gradient we have:
aL 0z
exp (2: 2n/T)
Σ 2n EnEA(t) exp (2s Za/T)
(6)
REN()
From this formula, when the negative sample similar to the query (hard negative). its gradient becomes larger. Therefore the negative gradient will be contributed mainly from the hard negative samples. Besides, we can divide hard negative samples into two types: false negative (samples have the same class as æ; but are considered negative) and true negative (similar samples of other classes). We have:
OL
@hard false negative + dhur
negative
(7)

---

Published in Transactions on Machine Learning Research (09/202))
95.0
95.0
96.0
96.0
97.0
97.0
99.0
99.0
Figure 2: Feature space of four tail classes when using constant temperature (left) and adaptive temperature (right). Adaptive temperature helps the model learn more separable feature space.
In Eq. while the true negative does not depend the imbalance of the data, the false negative will be proportional to the imbalance factor, since classes with more samples (head class) will have more false negatives because we just choose positive samples for training, then the remained samples are considered as negative samples), resulting in the negative gradient of the head class being much larger than that of the tail classes, making training more difficult. Therefore, we can remove the influence false hard negatives, it will help the learning bocome more stable and balanced.
Adaptive temperature formulation Work by Wang & Isola (020); Wang & Liu 2021 has shown that when the value of is small, the model will focus only on points near H; (hard negative), and as I increases, the model will focus more on further points. Especially, when is large enough, the model considers all points (normal negative and hard negative) to contribute equally. Therefore in head classes, to should increase the value of and vice versa, for tail
reduce the inffuence of false-hard negative samples, classes, because the number of false-hard negalives is not large, the reduction of will not affect the negative gradient too much. Besides, 7 can be viewed as a re-weighted parameter in Eqq 2 where tail classes with small J will enlarge their gradient and vice versa. Based on these observations, instcad of using temperature as a constant parameter, we can assign different temperature values to samples of different classes with the property that head classes will have large r values and tail classes will have smaller т values. To ensure the above property, propose formula where the temperature of each class will be proportional to the number of samples of that class:
7, = + (1 (1-)) frequent (class))
(classs,x))))
where 7; is the temperature of class i, 7 is the minimum value of temperature, frequent(class)) is the number of samples in class i, and (requent(class@aan is the number of samples of largest class.

---

Published in Transactions on Machine Learning Research (09/2023)
Centering
Prototype learning
Figure 3: Prototype learning via coreset selection pulls images closer to their centers. This will encourage the samples in the same class to be closer together, thus increasing semantic discrimination feature space.
3.4 Prototype Learning via Coreset (ProCo)
Previous works (Li et al. 2020 Zhu et al. 2022) determine prototype as a mean representation of feature space for each class. During the training, these prototypes are viewed as positive samples in the contrastive loss. Therefore, prototype training is similar to clustering (Li et al.) 2020), where it pulls samples in the same class closer to its prototype (the role of a prototype here is like centroid of class) while pushing samples from another class far away. Recently, few applications using prototype learning have been proposed to mitigate imbalanced data problems. TSC (Li et al. 2022b) re-balance feature space by pre-defining balanced largets, then moving samples in class close to their assigned balanced prototypes based on the matching algorithm. BCL (Zhu et al. 2022) analyzes the behavior of supervised contrastive learning, then proposes two new concepts named: class-averaging, and class-complement based on prototypes to mitigate the imbalance effect supervised contrastive learning. In this paper, we propose another approach to protolype learning. Firstly, we tackle the trade-off between global prototype representation and computation cost (the total running time for calculating prototype), our method achieves global mcan reprosentation for cach class while saving significant compulation cost due to the estimation ion mean being conducted on the coreset (the set contains important samples of the training data, where its size is significantly smaller than the training set). This makes our prototypes globally represent the feature in data without using all training samples. Secondly, we propose a new prototype loss function and empirically show our loss function can work well on imbalanced data and improve the contrastive learning model significantly. The detailed effect of this module is shown in Table 5.2
Coresct selection for calculating prototype The main challenge in prototype learning is how we efficiently calculate protolypes, both in computation time and representation ability. Previous works determine prototypes from the momentum queue feature in MoCo [Li ct al. 2020 2022b), which can yield consistent protolypes with low computation costs. However, this method can not adapt to another contrastive learning model that does not use large queue size (Chen et al.) 2020a). Besides, when the dataset size is extremely large (Krizhevsky
al. 2017), calculating the prototypes based on samples the queue can yield local prototypes. To solve the above limitations, we distill the original dataset to extract its most important samples, then we will calculate protolypes according to this set. Our algorithm is as follows: we first use coreset generation algorithm (Killamsetty et al. 2021) coreset. During the training, we update our prototypes based on the coresct at the beginning of cach epoch. Calculating protolypes from the coreset encourages the centers to represent their class globally, while our method can easily adapt to any framework and dataset size.
data distillation method to extract the
Prototype loss function After having a coresct, we augment two views of it and calculate two prototype sets for every two views. Then we define symmetric prototype loss function, based on contrastive learning loss. This loss function uses the calculated prototypes the positive sample for query having the same
8

---

Published in Transactions on Machine Learning Research (09/2023)
class label. Assume each query e augment two views ". o822 We define loss as the contrastive loss between each pair of the query sample and its prototypes:
X
Σ Σ
log
4N
24 XeLn
( (E(1.2)
+
CEC2
where Gigg is the prototype (center) of query in corcsct 9, i is calculated from coresct and C are the list protolypes from coreset g excluding G.gg Our proposed protolype formulation different from others [Li al. 2022b Zhu et al. 2022] in its symmetric loss between different views of query and coreset, which can help the model learning achieve more stability.
Joint training CoGloAT and ProCo The final loss function will be the weighted combination between CoGloAT and Proco, which is defined as:
fitnall = aLCoGIAT (1 (x) LrroCo
where is the hyperparameter that weights the CoGloAT and ProCo loss during training, which will be tuned during the training.
4 Analysis
To further understand the benefit of our model, extensively evaluate several aspects of the contrastive learning model based two important characteristics: Alignment and Uniformity, Besides, discuss the limitations of the current formula for calculating Intra Alignment and Inter Uniformity, followed by our improved version that can be calculated more efficiently. Finally, we evaluate these characteristics ics in our models to further understand their behavior when learning on long-tailed data.
Intra Alignment This metric the average distance between all samples in the same class on feature space (Li et al. 2022b). It is similar to a measure of dispersion: variance, meaning when the alignment value is small, samples in the same class will lie close each other, and vice versa. The alignment formula is defined as:
Σ Intra C 2m1 DF.
lof
1/
(9)
where F; is the feature space of samples in class i, (uf, off) are the two arbitrary feature vectors in feature space F, This formula is good choice to estimate the spread out of samples in a class. However, it takes a lot of time to calculate. For example, with a class has n samples, the time complexity of Intra Alignment is O(n"), so when the number of samples a class is large, is computationally expensive to calculate Intra Alignment. To overcome this issue, we first need to construct centers for each class, then calculate the variance by:
Variance
1 1 C
Σ
variance;
(10)
In this formula, variance, the variance of class and can be calculated by:
variance,
Σ 10 - call
(11)
where Cj is the center of class i. This reduces the computational cost from O(n²) to O(n).

---

Published in Transactions on Machine Learning Research (09/2023)
CIFAR-1O-LT
CIFAR-100-IT
Metric
Imbalance factor
100
50
100
50
KCL
0.43
0.42
0.42
0.55
0.57
0.45
Intra
Ours
0.41
0.41
0.39
0.44
0.43
0.41
KCL
0.65
0.62
0.61
0.84
0.85
0.63
Variance
Ours
0.61
0.60
0.57
0.66
0.65
0.60
KCL
0.70
0.75
0.83
0.90
0.95
0.93
Inter
Ours
0.73
0.79
0.85
0.98
1.00
0.98
KCL
1.10
1.25
2.03
1.91
2.00
2.31
Improved Inter
Ours
1.20
1.29
2.20
2.23
2.36
2.41
Table 1: We measure two common characteristics of feature space, which have been introduced in section Intra Alignment and Inter Uniformity. These measurements are conducted on the learned features using our model (Ours) and the baseline KCL (KCL) on two different datasets: CIFAR-IO-LT and CIFAR- 100-LT with different imbalance factors (the ratio between largest and smallest class in terms of frequent) range in {10, 50, 100}. From these results, we can highlight, two insights: (i) Our proposed measurements: Variance and Inter yield better results (in terms of computation cost for alignment and accuracy for uniformity) compared with the original formulas, (ii) Training with CoGloAT + ProCo creates a more robust feature space (it. returns smaller Intra Aligmment and larger Inter Uniformity values compared with KCL).
Inter Uniformity Different from Intra Alignment, Inter Uniformity is used to measure the distance be- tween classes (Li et al., 2022b). It is the total distance between the centers of all classes. When Inter Uniformity is large, it means feature space is more separable, and vice versa. The original formula of Inter Uniformity is defined as:
C Σ
|c: c112 Inter C(C 1) i=1 111
Σ
(12)
where represents as a mean feature vector of class based on its feature space F,, and we can achieve it via: 6; = E, [. oL j is the index of feature vector in F. This formula however has a caveat: does not use the alignment information, and consequently does not always reflect the intended meaning of "inter uniformity". concrete example is when we have two pairs of classes, that have the same center distance, but in the first pair each class has a high variance, and the second pair each class has a low variance. The Inter Uniformity of the second pair should be larger because the boundary between classes is larger, but with the above formula, it will return the same result. Therefore, we modify the original uniformity formula by adding variance information to capture the above situation. The proposed formula is follows:
2
C
|G - cj
Interimprove
Σ C(C - i=1 1
(13)
variance; + variances
where variance; is defined in Eq.
The above formula cstimates the uniformity of data better when combined with alignment information. Table shows the effectiveness of this formula, while the original uniformity formula indicates that KCL has a similar uniformity value with CoGloAT + ProCo in some settings, its improved version: Inter smproved Inpooee yields a more consistent result, which show by CoGloAT + ProCo gives higher uniformity in all settings. The results from the improved uniformity formula are more reasonable because prototype learning helps the model learn more separable feature space, therefore increasing its uniformity.
10

---

Published in Transactions on Machine Learning Research (09/2023)
5 Experiments
5.1 Dataset and implementation details
CIFAR-10-LT and CIFAR-100-LT For small-scale data, we conduct experiments on CIFAR-1O-LT and CIFAR-100-LT The original CIFAR-10 and CTFAR-100 contain 50,000 32x32 images for the training and 10,000 32x32 images for the testing. The label of both datasets follows a uniform distribution, where each class has the same number of samples. There are 10 classes in CIFAR-10 and 100 classes in CIFAR-100, respectively. Based on this data, its imbalance versions: CIFAR-IO-LT and C1FAR-100-LT are created following (Cao et al. 2019), where we create the imbalance training data while keeping the test data unchanged. The imbalance training data is defined through the imbalance ratio = mraa(n:)/min(n) between the most frequent and least frequent size classes. Long-tailed imbalance follows the exponential decay in sample size across classes. In this paper, we run the experiment under E {10, 50, 1003
ImageNet-LT This is long-tailed version of ImageNet dataset (Deng et al. 2009,, where the long- tailed data is constructed by selecting the subset following the Pareto distribution with power value a 6. Basically, this dataset contains 115,800 of 224x224 images from 1000 different classes, where the largest class has 1280 images and the smallest contains 5 images. And similar to the CIFAR data, the test set of ImageNet-LT is the same as its original test version.
Implementation details Similar to TSC [Li al. 2022b use Mocov2 (a common self-supervised learning algorithm (Chen et al., 2020c) here with the same configurations in TSC. For CIFAR data, we use batch size 256, initial learning rate 0.1, SGD optimizer with momentum 0.9, and we train the ResNet32 as the backbone models for 1,000 epochs. After pre-training on our framework, we fine-tuned the top-classifier layer of the pre-trained model with LDAM (Cao et al., 2019) loss (motivated by minimizing the margin-basod generalization bound) and Reweight (Wang et al. 2017) (the weight of each class is proportionally inverse to its frequent) for CIFAR-1O-LT, and Gross-Entroyy
loss combine Reweight for CIFAR-100-LT. Both of them use the same learning rate equal to 0.1 and batch size equal to 256. We conduct the experiments on three different imbalance factors (10, 50, 100}, where a larger imbalance factor means the data is challenging to learn. The result in Table 5.2 shows that our model can oulperform the current stale-of-the-arl model on all datasets across a range of imbalance factors by a large margin. For ImageNet-LT, its training strategy is similar to CIFAR data, we just change the backbone from ResNet-32 to ResNet-50, this model is trained on 200 epochs before fine-tuning the classifier layer with an additional 100 epochs before making the final prediction. The performance of our model with the ImageNet-L.T dataset can be found in Table 5.2
5.2 Result
CIFAR-10-LT and CIFAR-100-LT Table 5.2 shows that our model outperforms the current state- of-the-art model on both datasets with different imbalance factors. In particular, on CIFAR100-100-11 the model incrcasod by approximately 3%, compared to the best model: TSC (Li ct al. 2022b) (50.4% vs 47.4%). This improvement comes from the efficient extraction of positive samples from the k- klglobal positive selection and unsupervised loss function, helping the model learn balanced and semantic features. Besides, CoGloAT alone already outperforms TSC (Li et al.) 2022b) (k-positive + prototype learning). This suggests that CoGloAT is stronger baseline in contrastive learning when solving long-tailed problems.
ImageNeet-LT In table we compare the performance of our model with other baselines on the ImageNet-IT dataset. Our model achieves a slight. improvement of 0.3% over the baseline (KCL). This modest gain is likely due to the fact that we have not performed any hyperparameter optimization/selecde sslecctio for the ImageNet-LT dataset; instead, use the same hyperparameters as the CIFAR-100-LT dataset. We expect that fine-tuning our model on ImageNet-I.T will lead to a more significant improvement.
I For each setting, we evaluate it with 3 seeds and report its accuracy and standard deviation. The results of other methods are extracted from TSC (Li et al. 2022b) which do not have a standard deviation. 2Due our resource constraints.
11

---

Published in Transactions on Machine Learning Research (09/2023)
COFAR-10-LT
CIPAR-100-LT
Method
Imbalance factor
100
50
10
100
50
10
CE
70.4
74.8
86.4
38.3
43.9
55.7
CB-CE Cui et al.
2019
72.4
78.1
86.8
38.6
44.6
57.1
Focal
Lin et al.
2017
70.4
76.7
86.7
38.4
44.3
55.8
CE-DRW
CCao ct a. 2019)
75.1
78.9
86.4
40.5
44.7
56.2
LDAM
Cao al. 2019
73.4
76.8
87.0
39.6
45.0
56.9
LDAM-DRW
Cao et al.
2019
77.0
80.9
88.2
42.0
46.2
58.7
M2m-LDAM
Kim et a.l
2020
79.1
87.5
43.5
57.6
PCL Cui al.
202
52.0
56.0
64.2
BCT.
Zhu et al.
2022
84.3
87.2
91.1
51.9
56.6
64.9
KCL
Kang et a..
2020
77.6
81.7
88.0
42.8
46.3
57.6
TSC
et al.
2022b)
79.7
82.9
88.7
43.8
47.4
59.0
CoGloAT CoGloAT + ProCo
81.420.1
84.5+0.4
88.910.11
45.4::..1
49.5=t0.1
60.7
81.6++0.3
85.5+0.1
89.4=0.2
46.0+0.2
50.4++02
60.8+001
Table 2: The top-1% acc run CIFAR-10-LT and CIFAR-100-T.T with Resnet-32 backbone. In this table, the accuracy of CoGloAT + ProCo outperforms the previous state-of-the-art model by a large margin in all experiments, especially on CIFAR-100-LT, which increases 3% (on imbalance factor 50). This result suggests that our model can be viewed as a new baseline for contrastive learning in long-tailed recognition.
Method
Many
Medium
Few
OLTR
[Liu et al.
2019
35.8
32.3
21.5
32.2
LWS
Kang et al.
2019
57.1
45.2
29.3
47.7
PCL
Cui et al.
2021
57.0
BCL
Zhu et al.
2022
56.0
KCL
Kang et al.
2020
61.8
49.4
30.9
51.5
TSC
[Li et. al.
2022bl
63.5
49.7
30.4
52.4
KCL
59.5
50.3
38.5
49.5
Ours
59.0
49.5
40.9
49.8
Table The top-1% ace run on ImageNet-LI) Here KCL} means the baseline model that we have reproduced.
12

---

Published in Transactions on Machine Learning Research (09/2023)
CIFAR-10-LT
CIFAR-100-LT
Model
Imbalance factor
100
50
100
10
KCL
12.0
14.0
21.0
14.5
15.5
22.5
Global selection
28.5
34.0
50.5
34.0
37.5
57.5
Adaptive temperature
12.5
14.5
22.0
15.5
16.5
22.5
ProCo
22.0
25.0
35.0
26.0
27.5
36.0
Table 4: The running time (in seconds) different settings for one training epoch. Global selection increases the time proportionally to the number of k (we choose k=3 in our model) positives, while ProCo adds extra time to update the class centers after each epoch and compute the prototype loss.
CIFAR-100-LT
Loss type
Imbalance factor
100
50
10
Adaptive
43.7 #0.1
48.6
#0.2
57.9
#0.1
Proco
44.3 +0.1
47.8
+0.1
58.7
+0.2
Table 5: The accuracy of Adaptive temperature and ProCo alone on CIFAR-100-LT dataset. Each of these modules independently improves the accuracy the model. Therefore, is possible combine various model architectures with one of these techniques: k-global selection, adaptive temperature, or ProCo improve their performance.
5.3 Ablation study
Rumning time of different modules Contrastive learning models benefit from adding k-global selection ProCo, but these modules also increase the rumning time. To better understand the efficiency of our model when being applied to different scenarios, measure the running time of the baseline model when it is trained jointly with k-global positive selection, adaptive temperature, and ProCo, individually. The detailed result is shown in Table 4
How does each module contribute to the baseline? We evaluate the contribution of each module by combining it with the baseline model KCL (Kang et al. 2020): adaptive temperature, and prototype lcarning. Training with these modules individually improves learning accuracy. Table 5.3 shows the detailed results.
How do different loss functions contribute to the model performance? In the k-positive selection, sampling positive samples in the queue of MoCo leads to trade-off between equalizing positive samples across classes, and semantic discrimination of the learned features. Reducing the value & makes the model learn more balanced features but reduces the quality of the model, and vice versa. In CoGloAT, avoid this trade-off, we construct two independent modules: while k-global positive selection has the role of learning semantic discrimination features, unsupervised contrastive loss is responsible for balancing learned space between classes. This learning strategy helps the model learn more efficiently. In Table G we show the behavior of CoGloAT when using three different loss functions: unsupervised, supervised, and supervised contrastive losses with k-positive selection. While the model learns with unsupervised loss (the default loss function in CoGloAT) has the best accuracy, combining it with k k-positive loss yields an accuracy higher than the supervised sotting. Besides, CoGloAT + k-positive loss outperforms the original k-positive version by a large gap (1.3% 1.5%). This suggests CoGloAT to be strong bascline for other methods in long-tailed learning.
13

---

Published in Transactions on Machine Learning Research (09/2023)
CIFAR-100-IT
Loss type
Imbalance factor
100
50
10
Supervised
43.7 #0.1
48.6 +0.2
57.9
#0.1
44.3 #0.1
47.8
#0.1
58.7 #0.2
Unsupervised
45.4 +0.1
50.4 +0.2
60.8 =0.1
Table 6: The accuracy of CoGloAT with different loss types. Unsupervised contrasttive loss outperformance its supervised counterparts
6 Conclusion and Future work
Conclusion In this paper, we gave an overview of the current works on long-tailed data with contrastive learning, both its performance and limitations. After that, we introduced two new methods named CoGloAT and ProCo which can improve the training of contrastive learning models for long-tailed data in different ways, For each method, we created a stronger model built upon leveraging the advantages of the previous method while improving its limitations. Then we conducted variety of experiments to highlight the contribution of each method. Finally, reviewed the previous evaluation metric in contrastive learning, discussed its remaining problems, and proposed our improved version.
Future work Our model solely tackles data imbalance based on the general characteristics of contrastive learning. It is advantageous to build a robust baseline model like CoGloAT. However, combining contrastive learning with other methods to mitigate data imbalance such as class sampling, class re- re-weighting, logit adjustment, etc. has not been explored thoroughly our work. Therefore, is an interesting direction include the above methods in CoGloAT to have both theoretical and experimental observations. Another avenue for future work is to develop rigorous theoretical framework to explain the benefits of our contrastive learning framework for general long-tailed classification problems.
Acknowledgments
We would like to thank Nhat Khang Ngo for his help in running some of our large-scale experiments.
References
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0897-6907, 2022.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-200 259, 2018.
Chumphol Bunkhumpornpat, Krung Sinapiromsaran, and Chidchanok Lursinsap. Dbsmote: density-based synthetic minority over-sampling technique. Applied Intelligence, 36(3):664 684, 2012.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distrbbttion-wwaee margin loss. Advances in neural information processing systems, 32, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp.
132-149, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural informalion processing systems. 33:22243 22255, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 15750 15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arx preprint arXiv:2003.0420 2020c.
Xinlci Chon, Saining Xic, and Kaiming He. An empirical study training self-supervised vision transformers. In Proceedings of the HEEE/CVI International Conference on Computer Vision, pp. 9640 9649, 2021.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Paramctric contrastive learning. In Proceedings of the IEEE/CVE international conference on compuler vision, pp. 715 724, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. live number of samples. In Proceedings of the IEEE/CVP conference. computer vision and pallern recognition, pp. 9268 9277, 2019.
loss based on effec-
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 255. leee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In Workshop on learning from imbalanced datasets II, volume 11, pp. 1-8. Citeseer, 2003.
Hossein Shokri Ghadikolaei, Hadi Ghauch, Carlo Fischione, and Mikael Skoglund. Learning and data selec- tion in big datasots. In International Conference on Machine Learning, pp. 2191-2200. PMLR, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pp. 4927 4939. PMLR, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 10100.90000111 2019.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for represen- tation learning. In International Conference on Learning Representations, 2020.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost- sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-35) 3587, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 18673, 2020.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Gen- eralization based data subset selection for efficient and robust learning. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 8110 8118, 2021.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 13896-13905, 2020.
Alex Krizhevsky, lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Bo Li, Yonggiang Yao, Jingru Tan, Gang Zhang, Fengwei Yu, Jianwei Lu, and Ye Luo. Equalized focal loss for dense long-tailed object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6990-6990999 6999, 2022a.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.c arXi::2055.44660, 2020.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng, Metasaug: Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/C /CF conference on compuler vision and pallern recoguilion, pp. 5212 5221, 2021.
Tianhong Li, Peng Cao, Yuan Yuan, Lijic Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 6918 6928, 2022b.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980 2988, 2017.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXic preprint orXiv: 10. 10.05025, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long- tailed recognition in an open world. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2537-25466, 2019.
Ajinkya More. Survey of resampling techniques for improving classification performance in unbalanced datasets. Xrxiv preprint arxiv:1608.0 06048, 2016.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6887 6896, 2022.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tai-tailed visual recognition. Advances in neural information processing systems, 33:4175 4186, 2020.
Nikolaos Sarafianos, Xiang Xu, and loannis A Kakadiaris. Deep imbalanced attribute classification using visual attention aggregation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 680-697, 2018
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convo- lutional neural networks. In European conference computer vision, pp. 467-488 Springer, 2016.
Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1694, 2021.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33: 1513 1524, 2020.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In European conference computer vision, pp. 776-794. Springer, 2020.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint
arXi0:1811.10959, 2018.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprint ar arX::200000018 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances neural
information processing systems, 30, 2017.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 13:19290-19331, 2020.
Shaoyu Zhang, Chen Chen, Xiyuan Hu, and Silong Peng. Balanced knowledge distillation for long-tailed lcarning. arXiv preprint arXiv:2104.106 10510, 2021a.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.098: 2021b.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. Xie preprint urXiv:2110.00 04596, 2021c
Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 4447-3455, 2021d.
Boyan Zhou, Quan Cui, She-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CV /CF conference on computer vision and pattern recognition, pp. 9719-9728, 2020.
Zhihan Zhou, Jiangchao Yao, Yan- Wang, Bo Han, and Ya Zhang, Contrastive learning with boosted memorization. In International Conference Machine Learning, pp. 27367-27377. PMLR, 2022.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu- Lu-Gang Jiang, Balanced contrastive learning for long-tailed visual recognition. Tn Proceedings of the TEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 6908 6917, 2022.
17


Reviews:
[{"review_id": "kkyBY0HvgX", "reviewer": "Reviewer_5hQe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The authors used a binary classification problem to theoretically analyze the motivation of the proposed method. GloCo can lead to a smaller imbalance factor, thus gaining improved performance.\n2. The authors leveraged the existing coreset selection method for generating the effective prototype, which is reasonable. \n3. The empirical results seem to show that the proposed method can lead to significant improvement. \n\nWeaknesses:\n\n1. The global selection needs to select the positive samples globally instead of a minibatch, which could consume more running time than baselines. It would be better for the authors to analyze the running time as well.\n2. The authors only provide the performance of GloCo and GloCo with ProCo. However, it is still unknown how effective is the proposed ProCo. It would be better for the authors to provide the performance of ProCo. \n3. The experimental results are not comprehensive. The authors only used ResNet-32 as the backbone model and CIFAR-10/100 as the training set. I suggest the authors provide the results of various backbone models such as ResNet-50 and WideResNet trained on some large-scale datasets such as ImageNet to support their claim. \n4. In Table 1, I suggest the authors provide the results of the baseline. Without the results of baseline, it is hard for me to judge the effectiveness of the proposed method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes two techniques to improve supervised contrastive learning on long-tailed datasets. The first proposed technique called GloCo selects the positive samples globally instead of the minibatch and dynamically schedules the temperature hyperparameter to re-balance the training loss. The second technique called ProCo generates the prototype based the selected coreset and then uses the prototype training to further improve the performance. This paper provides the theoretical analyses of the proposed techniques and the empirical results to support their claim.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the comments in the section Weaknesses and make corresponding changes. ", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDzhCiBRO2", "reviewer": "Reviewer_aM4e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper aims at a specific problem in long-tailed recognition using contrastive learning. The proposed methods are simple and effective.\n2. The intuitions of the methods are mathematically proved, which is appreciated. The math derivations also provided insight into the problem.\t\n3. The paper’s ideas are well written and clarified.\n\nWeaknesses:\nThe main concern is about the experiments.\n1. Empirical analysis are insufficient. The paper lacks experiments on larger datasets, such as ImageNet-LT or iNaturalist.\n2. While the authors claim their method can outperform the SOTA methods, many other methods, especially contrastive learning ones, yield better performance. Refer to paperswithcode.com for the baselines.\n3. Table 1 lacks comparison with other baseline methods. Comparance between the proposed two methods cannot verify anything.\n4. Ablation studies on Adaptive temperature and ProCo are missing. \n5. Writing could be further improved, such as “balanced feature space” instead of “balance feature space”, “inspired by” instead of “inspiration from” etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles long-tailed recognition using contrastive learning and prototype learning. The authors propose GloCo using k-global positive selection and adaptive temperature. Another method, ProCo, is also introduced to improve prototype learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above suggestions in weak part.\n\nAdditional questions:\n1. What is align_i in the Inter_improve formulation?\n2. How is GloCo combined with ProCo in training?", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LIpYQQBxOY", "reviewer": "Reviewer_VF6P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The motivations are clearly described;\n2. The performance improvement is overall significant. \n\nWeaknesses:\n1. Some core descriptions of the proposed techniques are missing, e.g., how to perform the $k$-global positive selection, and the definition of $L_i^{sup}$.\n2. It seems that Prop. 1 is insufficient to explain the proposed $k$-global positive selection. What is the relationship between the so-called batch/global balance ratio and the long-tailed classification performance?\n3. Although the proposed adaptive temperature (Eq. (13)) sounds reasonable, it is unclear how it balances the gradients. \n4. Experiments are conducted on only small-scale datasets, CIFAR-10 and CIFAR-100. It will be more convencing if experiments on large-scale datasets like imagenet and Inaturalist are provided.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to solve the long-tailed visual classification problem via contrastive learning. Specifically, the authors propose to sample $k$ positive samples for each query from the whole training set instead of a batch and adaptively balance the gradients by adjusting the temperature. Moreover, they propose a prototype learning method based on the coreset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses part.", "broader_impact_concerns": "NAN", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The current submission worked on long-tailed visual recognition based on contrastive learning. It has three contributions: global selection of k positive data, adaptive temperature, and coreset selection (which is the prototype learning part). We may regard all of the three contributions as major contributions, but they are relatively independent and the final proposal is the combination of them, making the submission look more like engineering than science. I think the authors did a good job for their rebuttal, as one reviewer said that
> After reading the response to all the reviews, I lean towards accepting the paper for the updated and added experiment results.

On the other hand, two reviewers argued for rejection (one changed his/her mind later after I clarified the acceptance criteria):
> *Official recommendation:* I still think that the proposed theory is insufficient to support the proposed method. Hence, I weakly recommend rejecting this paper.
*Internal discussion:* My major concern is that it is still unclear if Prop. 1 has anything to do with the final performance. The problem comes from the fact that the imbalanced ratio is defined as the ratio of two combinatorial numbers (how many ways to pick out (fixed) k positive instances ) instead of the actual sample frequency. Also the notion of selection space is also quite confusing, why does calculating an imbalance ratio require an extra space?

> *Official recommendation:* Thanks for the authors' responses. Most of my concerns have been solved. However, the improvement on the large-scale datasets is really insignificant and the proposed method is even less efficient since it needs to consumes more running time to do the global positive selection (ref. response 1). Therefore, I lean to rejection.
*Internal discussion:* Thanks for AE sharing the Acceptance Criteria of TMLR. I think the empirical results can support the claims. Therefore, I lean toward acceptance.

According to the evaluation criteria, the novelty/significance is not the main point, but it is whether or not the claims made in the submission are supported by accurate, convincing and clear evidence. I personally think the theory is not a major issue even if it has no business with the final performance (because this is an algorithm paper rather than a theory paper). I also agree with a reviewer that the empirical results can support the claims. Therefore, we should accept the submission for publication.
Decision: Accept (poster)