Temperature Schedules for self-supervised contrastive methods on long-tail data
Authors: Anna Kukleva
Moritz Böhle
Bernt Schiele
Hilde Kuehne
Christian Rupprecht
Venue: ICLR 2023 poster (2023)
Keywords: contrastive learning
long-tail data
self-supervised learning
temperature
analysis

Abstract: Most approaches for self-supervised learning (SSL) are optimised on curated balanced datasets, e.g. ImageNet, despite the fact that natural data usually exhibits long-tail distributions. In this paper, we analyse the behaviour of one of the most popular variants of SSL, i.e. contrastive methods, on imbalanced data. In particular, we investigate the role of the temperature parameter $\tau$ in the contrastive loss, by analysing the loss through the lens of average distance maximisation, and find that a large $\tau$ emphasises group-wise discrimination, whereas a small $\tau$ leads to a higher degree of instance discrimination. While $\tau$ has thus far been treated exclusively as a constant hyperparameter, in this work, we propose to employ a dynamic $\tau$ and show that a simple cosine schedule can yield significant improvements in the learnt representations. Such a schedule results in a constant `task switching' between an emphasis on instance discrimination and group-wise discrimination and thereby ensures that the model learns both group-wise features, as well as instance-specific details. Since frequent classes benefit from the former, while infrequent classes require the latter, we find this method to consistently improve separation between the classes in long-tail data without any additional computational cost. 

Published as a conference paper at ICLR 2023
TEMPERATURE SCHEDULES FOR SELF-SUPERVISED CONTRASTIVE METHODS ON LONG-TAIL DATA
Anna Kukleva"), Moritz Böhle"1, Bernt Schiele', Hilde Kuehne Christian Rupprecht
MPI for Informatics, Saarland Informatics Campus, 2 Goethe University Frankfurt,
MIT-IBM Watson AT Lab, University of Oxford {akukleva, mboehl ee) @mpi inn -mpg. de
ABSTRACT
Most approaches for self-supervised learning (SSL) are optimised on curated balanced datasets, e.g. ImageNet, despite the fact that natural data usually exhibits long-tail distributions. In this paper, we analyse the behaviour of one of the most popular variants of SSL, i.e. contrastive methods, on long-tail data. In particular, we investigate the role of the temperature parameter 7 in the contrastive loss, by analysing the loss through the lens of average distance maximisation, and find that a large T emphasises group- wiiep discrimination, whereas a small leads to higher degree of instance discrimination. While has thus far been treated exclusively as constant hy hyperparameter, in this work. propose to employ a dynamic and show that simple cosine schedule can yield significant improvements in the learnt representations. Such schedule results in a constant 'task switching" between emphasis on instance discrimination and sooupwwiee discrimination and thereby ensures that the model learns both group-wise features, as well details. Since frequent classes benefit from the former, while infrequent classes require the latter, we find this method consistently improve separation between the classes in long-tail data without any additional computational cost.
sestance-speciffe
INTRODUCTION
Deep Neural Networks have shown remarkable capabilities at learning representations of their inputs that are useful for variety of tasks. Especially since the advent of recent self supervised learning (SSL) techniques, rapid progress towards learning universally useful representations has been made.
Currently, however, SSL on images is mainly carried out benchmark datasets that have been con- structed and curated for supervised learning (e.g. ImageNet (Deng et al., 2009), CIFAR (Krizhevsky et al., 2009). etc.). Although the labels of curated datasets are not explicitly used in SSL. the structure of the data still follows the predefined set of classes. In particular, the blassebalanced nature of curated datasets could result in a learning signal for unsupervised methods. As such, these methods are often not evaluated in the settings they were designed for, i.e. learning from truly unlabelled data. Moreover, some methods (e.g. (Asano et al., 2019; Caron al., 2020)) even explicitly enforce uniform prior over the embedding or label space, which cannot be expected to hold for uncurated datasets.
In particular, uncurated, real-world data tends to follow long-tail distributions (Reed, 2001), in this paper, we analyse SSL methods on long-tailed data. Specifically, we analyse the behaviour of contrastive learning (CL) methods, which are among the most popular learning paradigms for SSL.
In CL, the models are trained such that embeddings of different samples repelled, while embed- dings of different 'views" (i.e. augmentations) of the same sample are attracted. The strength of those attractive and repelling forces between samples is controlled by temperature parameter т, which has been shown to play a crucial role in learning good representations (Chen et al., 2020c;a). To the best of our knowledge, has thus far almost exclusively been treated as a constant parameear
In contrast, we employ a dynamic T during training and show that this has a strong effect on the learned embedding space for long-tail distributions. In particular, by introducing simple schedule for r we consistently improve the representation quality across a wide range of settings. Crucially, these gains are obtained without additional costs and only require oscillating 7 with a cosine schedule.
"qqual contribution. Code available at: guthuu..com/mmmussscemnnnuuuu schedules

---

Published as a conference paper at ICLR 2023
This mechanism is grounded in our novel understanding of the effect of temperature on the contrastive loss. In particular, we analyse the contrastive loss from an average distance maximisation perspective, which gives intuitive insights to why a large temperature emphasises group-wise discrimination, whereas a small temperature leads to a higher degree of instance discrimination and more uniform distributions over the embedding space. Varying during training ensures that the model learns both group-wise and instance-specific features, resulting in better separation between head and tail classes.
we earry out an extensive analysis of the effect of r on imbalanced data; we analyse the contrastive loss from an average distance perspective to understand the emergence of semantic structure; we propose simple yet effective temperature schedule that improves the performance across different settings; we show that the proposed scheduling is robust and consistently improves the performance for different hyperparameter choices.
Overall, our contributions are summarised as follows:
2 RELATED WORK
representation learning (SSL) from visual data is a quickly evolving field. Recent methods are based on various forms of comparing embeddings between transformations of input images. We divide current methods into two categories: contrastive learning (He et al., 2020; Chen et al., 2020c;a; Oord et al., 2018), and non-contrastive learning (Grill et al., 2020; Zbontar et al., 2021; Chen & He, 2021; Bardes et al., 2022: Wei et al., 2022; Gidaris et al., 2021; Asano et al., 2019; Caron et al., 2020; He et al., 2022). Our analysis concerns the structure and the properties of the embedding space of contrastive methods when training on imbalanced data. Consequently, this section focuses on contrastive learning methods, their analysis and application to imbalanced training datasets.
Contrastive Learning employs instance discrimination (Wu et al., 2018) to learn representations by forming positive pairs of images through augmentations and a loss formulation that maximises their similarity while simultancously minimising the similarity to other samples. Methods such as MoCo (He et al., 2020; Chen et al., 2020c), SimCLR (Chen al., 2020a;b), SWAV (Caron et al., 2020), CPC (Oord et al., 2018), CMC Tian et al. (2020a), and Whitening (Ermolov et al., 2021) have shown impressive representation quality and down-stream performance using this learning paradigm. CL has also found applications beyond SSL pre-training, such as multi-modal learning (Shvetsova et al., 2022), domain generalisation (Yao et al., 2022), semantic segmentation (Van Gansbeke et al., 2021), 3D point cloud understanding (Afham et al., 2022), and 3D face generation (Deng et al., 2020).
Negatives. The importance of negatives for contrastive learning is remarkable and noticed in many prior works (Wang et al., 2021; Yeh et al., 2021; Zhang et al., 2022; Iscen et al., 2018; 8; Kalantidis et al., 2020; Robinson al., 2020; Khaertdinov et al., 2022). Yeh et al. (2021) propose decoupled learning by removing the positive term from the denominator, Robinson et al. (2020) develop an unsupervised hard-negative sampling technique, Wang et al. (2021) propose to employ a triplet loss, and Zhang et al. (2022); Khaertdinov et al. (2022) propose to improve negative mining with the help of different temperatures for positive and negative samples that can be defined as input-independent or input-dependent functions, respectively. In contrast to explicitly choosing a specific subset of negatives, we discuss the Info-NCE loss (Oord et al., 2018) through the lens of an average distance perspective with respect to all negatives and show that the temperature parameter can be used to implicitly control the effective number of negatives.
Imbalanced Self-Supervised Learning. Learning on imbalanced data instead of curated bal- anced datasets is an important application since natural data commonly follows long-tailed distribu- tions (Reed, 2001; Liu et al., 2019; Wang et al., 2017). In recent work, Kang et al. (2020), Yang & Xu (2020), Liu al. (2021), Zhong et al. (2022), Gwilliam & Shrivastava (2022) discover that self-supervised learning generally allows to learn a more robust embedding space than a supervised counterpart. Tian et al. (2021) explore the down-stream performance of contrastive learning on stan- dard benchmarks based on large-scale uncurated pre-training and propose multi-stage distillation framework to overcome the shift in the distribution of image classes. Jiang et al. (2021); Zhou et al. (2022) propose to address the data imbalance by identifying and then emphasising tail samples during training in an unsupervised manner. For this, Jiang et al. (2021) compare the outputs of the trained model before and after pruning, assuming that tail samples are more easily 'forgotten' by the pruned model and can thus be identified. Zhou et al. (2022), use the loss value for each input to identify tail samples and then use stronger augmentations for those. Instead of modifying the architecture
2

---

Published as a conference paper at ICLR 2023
or the training data of the underlying frameworks, show that simple approach -ie. oscillating the temperature of the Info-NCE loss (Oord et al.. 2018) to alternate between instance and group discrimination can achieve similar performance improvements at low cost.
Analysis of Contrastive Learning (CL). Given the success of CL in representation learning, is essential to understand its properties. While some work analyses the interpretability of embedding spaces (Bau al., 2017: Fong & Vedaldi, 2018; Laina et al., 2020; 2021), here the focus lies on understanding the structure and learning dynamics of the objective function such as in Saunshi et al. (2019); Tsai et al. (2020); Chen et al. (2021). F.g. Chen et al. (2021) study the role of the projection head, the impact of multi-object images, and a feature suppression phenomenon. Wen & Li (2021) analyse the feature learning process to understand the role of augmentations in CL. Robinson et al. (2021) find that emphasis on instance discrimination can improve representation of some features at the cost of suppressing otherwise well-learned features. Wang & Isola (2020); Wang & Liu (2021)
analyse the uniformity of the representations learned with CL. In particular, Wang & Liu (2021) focus on the impact of individual negatives and describe a uniformity-tolerr tolerance dilemma when choosing the temperature parameter. In this work, we rely the previous findings, expand them to long-tailed data distributions and complement them with an understanding of the emergence of semantic structure.
3 METHOD
In the following, we describe our approach and analysis of contrastive learning on long-tailed data. For this, we will first review the core principles of contrastive learning for the case of uniform data (Sec. 3.1). In Sec. 3.2, we then place a particular focus on the temperature parameter 7 in the contrastive loss and its impact the learnt representations, Based on our analysis, in Sec. 3.3 discuss how the choice of F might negatively affect the learnt representation of rare classes in the case of long-tailed distributions. Following this, we describe a simple proof-of-concept based on additional coarse supervision to test our hypothesis. We then further develop temperature schedules (TS) that yield significant gains with respect the separability of the learnt representations in Sec.
3.1 CONTRASTIVE LEARNING
The Info-NCE loss is a popular objective for contrastive learning (CL) and has lead to impressive results for learning useful representations from unlabelled data (Oord et al., 2018; Wu et al., 2018; He et al., 2020; Chen et al., 2020a). Given set of inputs {z1..... NN }. and the cosine similarities st between learnt representations u. ((Ax,)) and VU =((((z))) of the inputs, the loss is defined by:
N
exp (SH/T)
Le
log exp (Sa/T) + > jzy; exp (sig/T)
Here, A(:) applies a random augmentation to its input and f and g are deep neural networks. For given F,, we will refer to u, as the anchor and to By as a positive sample if i=j and as a negative i.j. Last, denotes the lemperature of the Info-NCE loss and has been found crucially impact the learnt representations of the model (Wang & Isola, 2020; Wang & Liu, 2021; Robinson et al., 2021).
Uniformity. Specifically, small 7 has been tied to more uniformly distributed representations, see Fig. For example, Wang & Liu (2021) show that the loss is "hardness-aware i.e. negative samples closest to the anchor receive the highest gradient. In particular, for a given anchor, the gradient with respect to the negative sample uy scaled by its relative contribution to the denominator Eq. (1):
Oc. ahe Osij
x |softmax x Du,
(2)
Oug Osig
Oug
As result, for sufficiently small r, the model minimises the cosine similarity to the nearest negatives in the embedding space, as softmax approaches an indicator function that selects the largest gradient. The optimum of this objective, in turn, is to distribute the embeddings as uniformly as possible over the sphere, as this reduces the average similarity between nearest neighbours, see also Figs. and 3.
Semantic structure. In contrast, large has been observed to induce more semantic structure in the representation space. However, while the effect of small r has an intuitive explanation, the phenomenon that larger r induce semantic structure is much more poorly understood and has mostly
3

---

Published as a conference paper at ICLR 2023
Epoch
1000 a)
T=0.1
b)
T=0.5
c)
7=1.0
1000
800
800
#samples
600
600
400
400
200
0
100
200
300
400
500
100
200
300
400
500 0
100
200 300
400
500
bins
bins
bins
embedding space during training. To measure coverage we uniformly sample 500 bins on the unit sphere. Each training sample is assigned to the closest bin and we plot a histogram of the assignments. X-axis: bins. Y-axis: number of training samples in bin. Colors denotes epochs: light is the 1st epoch of training, dark is the last. For small 7 (a) the representations are more uniformly distributed (ef. Sec. 3).
Figure 1: Coverage
been described empirically (Wang & Liu, 2021; Robinson et al., 2021). Specifically, note that for any given positive sample, all negatives are repelled from the anchor, with close-by samples receiving exponentially higher gradients. Nonetheless, for large т, tightly packed semantic clusters emerge. However, if close-by negatives heavily repelled, how this be? Should the loss not be dominated by the hard-negative samples and thus break the semantic structure?
To better understand both phenomena, propose view the contrastive loss through the lens of average distance maximisation, which we describe in the following section.
3.2 CONTRASTIVE LEARNING AS AVERAGE DISTANCE MAXIMISATION
As discussed in the previous section, the parameter T plays crucial role in shaping the learning dynamics of contrastive learning. To understand this role better, in this section, we present novel viewpoint on the mechanics of the contrastive loss that explain the observed model behaviour. In particular, and in contrast to Wang & Liu (2021) who focused on the impact of individual negatives, for this we discuss the cumulative impact that all negative samples have on the loss.
To do so, express the summands L. of the loss in terms of distances dij instead of similarities sty:
1- 8i
2
S di
x T
and Cts = exp(ds)).
(3)
This allows us to rewrite the loss Li as
expl (di)) exp (-du) + Eefes exp (-dis)
C. - log
= log
1+ ci Σ exp (-dis)
(4)
J
As the effect ou of the positive sample for given anchor the same for all negatives, in the following we place a particular focus on the negatives and their relative influence on the loss in Eq. (4); for discussion of the influence of positive samples, please see appendix A.4.
To understand the impact of the temperature т, first note that the loss monotonically increases with the sum S; = Ejjs exp(-dis) of exponential distances in Eq. (4). As log is continuous, monotonic function, base the following discussion the impact of on the sum S,.
For small 7, the nearest neighbours of the anchor point dominate S., as differences in similarity amplified. As a result, the contrastive objective maximises the average distance to nearest neighbours, leading uniform distribution over the hypersphere, see Fig. 3. Since individual negatives dominate the loss, this argument is consistent with existing interpretations, e.g. Wang & Liu (2021), as described the previous section.
For large 5, (e.g. v 2 1), on the other hand, the contributions to the loss from a given negative are on the same order of magnitude for wide range of cosine similarities. Hence, the constrastive objective can be thought of as maximising the average distance over a wider range of neighbours. Interestingly, since distant negatives will typically outnumber close negatives, the strongest cumulative contribution
4

---

Published as a conference paper at ICLR 2023
a)
b)
romalised magnitude
1.0-
0.1
0.1
==02
== 0.2
0=0.3
r= 0.3
0.5
T 0.5
T 0.5
0.2
1.0
1.0
0.0
-1.0
-0.6
-0.2 0.2 0.6 1.0
-1.0
-0.6
-0.2
0.2
0.6
1.0
-1.0
-0.6
-0.2 0.2 0.6 1.
Similarity
similanty
similarity
Figure 2: Loss contribution by similarity. X-axis: cosine similarity between anchor and negative, All curves are normalised such that their max y-valuc is 1. a): influence of an individual negative sample to the loss depending its similarity to anchor for different b): average histogram of distribution of negatives over the hypersphere with respect to their similarity to the anchor; e): cumulative impact that negative samples have on the loss. cumulative contribution of negatives shifts left, towards less similar samples, in contrast to individual contributions of negatives. As T + oc, the cumulative distribution coincides with the histogram b).
to the contrastive loss will come from more distant samples, despite the fact that individually the strongest contributions will come from the closest samples. To visualise this, in Fig. 2a, we plot the contributions of individual samples depending on their distance, as well as the distribution of similarities Sij to negatives over the entire dataset in Fig. 2b. Since the number of negatives at larger distances (e.g. Sij R 0.1) significantly outnumber close negatives (st > 0.9), the peak the cumulative contributions shifts towards lower similarities for larger т, as can be seen in Fig. 2c; in fact, for 7 00, the distribution of cumulative contributions approaches the distribution of negatives.
Hence, the model can significantly decrease the loss by increasing the distance to relatively 'casy negatives" for much longer during training, i.e. to samples that are easily distinguishable from the anchor by simple patterns. Instead of learning 'hard' features that allow for better instance discrimination between hard negatives, the model will be biased to learn easy patterns that allow for group-wise discrimination and thereby increase the margin between clusters of samples. Note that since the clusters whole mutually repel each other, the model is optimised between the expanding forces between hard negatives (i.e. within a cluster) and the compressing forces that arise due the margin maximisation between easy negatives (i.e. between clusters).
find trade-off
Importantly, such bias towards easy features can prevent the models from learning hard features- i.e. by focusing on group-wise discrimination, the model becomes agnostic to instance-specific features that would allow for a better instance discrimination (cf. Robinson et al. (2021)). In the following, we discuss how this might negatively impact rare classes long-tail tailed distributions.
3.3 TEMPERATURE SCHEDULES FOR CONTRASTIVE LEARNING ON LONG-TAIL DATA
As discussed in Sec. 1, naturally occurring data typically exhibit long-tail distributions, with some classes occurring much more frequently than others; across the dataset, head classes appear frequently. whereas tail classes contain fewest number of samples. Since self-supervised learning methods are designed to learn representations from unlabelled data, it is important to investigate their performance on imbalanced datasets.
Claim: Tail classes benefit from instance discrimination. As discussed in Sec. 3.2, sufficiently large 7 required for semantic groups emerge during contrastive learning this emphasises group-wise discrimination. However, as shown by Robinson et al. (2021), l), this can come at the cost of encoding instance-specifie features and thus hurt the models' instance discrimination capabilities.
We hypothesise that this disproportionately affects tail classes, as tail classes consist of only relatively few instances to begin with. Their representations should thus remain distinguishable from most of their neighbours and not be grouped with other instances, which are likely of different class. In contrast, since head classes are represented by many samples, grouping those will be advantageous.
To test this hypothesis, we propose to explicitly train head and tail classes with different т, to emphasise group discrimination for the former while ensuring instance discrimination for the latter.
'To ohtain the cumulative contributions, group the negatives into 100 non-overlapping bins of size 0.02 depending their distance the anchor and report the sum of contributions of given bin.
5

---

Published as a conference paper at ICLR 2023
Thead = 1.0 Ttail =.1
T=0.1
T= 0.5
T= 1.0
KNN: 49.71 89.93 20.08
KNN: 49.24
KNN: 45.76
KNN: 70.91 93.68 68.27
93.20 10.03
92.71 13.01
Figure 3: Representations of a head and tail class. Visualisation of the influence of representations of two semantically close classes (trained with all 10 classes). Red: single head class and blue: single tail class from CIFARIO-LT Small 7=0.1 promotes uniformity, while large 7= 7= creates dense clusters. With [head/ttait) we refer to coarse supervision described in Sec. 3.3 which separates tail from head classes. In black / /red/ blue, we respectively show the average kNN accuracy over all classes / the head class/ / the tail class.
Controlling r with coarse supervision. We experiment on CIFARIO-LT long-tail variant of CIFARIO see Sec. 4.1) in which we select different depending whether the anchor U, is from head or a tail class, i.e. the 5 most or least common classes. We chose a relatively large (Thend =1. 1.0) for the 5 head classes emphasise group-wise discrimination and relatively small
Experiment:
(Taii (..1) for the 5 tail classes to encourage the model to learn instance-diseriminaaiig features.
As can be seen in Fig. 3, this simple manipulation of the contrastive loss indeed provides significant benefit with respect to the semantic structure of the embedding space, despite only weakly supervising the learning by adjusting T according
coarse (frequent/infrequen)) measure of class frequency.
In particular, in Fig. 3, we show the projections of a single head class and a single tail class onto the three leading PCA dimensions and the corresponding kNN accuracies. We would like highlight the following results. First, without any supervision, we indeed find that the head class consistently performs better for larger values of T (e.g. 1.0), whereas the tail class consistently benefits from smaller values for (e.g. 0.1). Second, when training the model according to the coarse 7 supervision as described above, we are not only able to maintain the benefits of large 7 values for the head class, but significantly outperform all constant versions for the tail class, which improves the overall model performance on all classes; detailed results for classes are provided in the appendix.
Temperature Schedules (TS) without supervision. Such supervision with respect to the class frequency is, of course, generally not available when training unlabelled data and these experiments are only designed to test the above claim and provide an intuition about the learning dynamics on long-tail data. However, we would like point out that the supervision in these experiments is very coarse and only separates the unlabelled data into frequen! and infrequent classes. Nonetheless, while the results are encouraging, they are, of course, based on additional, albeit coarse, labels. Therefore. in what follows, we present an unsupervised method that yields similar benefits.
In detail, we propose modify according a cosine schedule, such that it alternates between an upper (T4) and lower (r_) bound at a fixed period length T:
Teoa (t) (r_ - --) (1 + cos
(5)
here, denotes training epochs. This method motivated by the observation that 7 controls the trade-off between learning easily separable features and learning instance-specifer features.
Arguably, however, the models should learn both types of features: i.e. the representation space should be structured according to easily separable features that (optimally) represent semantically meaningful group-wise patterns, whilst still allowing for instance discrimination within those groups.
Therefore, we propose to alternate between both objectives as in Eq. (5), to ensure that throughout training the model learns to encode instance-specifie patterns, whilst also structuring the representation space along semantically meaningful features. Note that while we find a cosine schedule work best and to be robust with respect to the choice for T (Sec. 4.3), we also evaluate alternatives. Even randomly sampling from the interval [+] improves the model performance. This indicates that the task switching between group-wiss wise discrimination (large 7) and instance discrimination (small 7) is indeed the driving factor behind the performance improvements observe.

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTAL RESULTS
In this section, we validate our hypothesis that simple manipulations of the temperature parameter in Eq. (1) lead to better performance for long-tailed data. First, we introduce our experimental setup in Sec. 4.1, then in Sec. 4.2 we discuss the results across three imbalanced datasets and, finally, we analyse different design choices of the framework through extensive ablation studies in Sec. 4.3.
4.1 IMPLEMENTATION DETAILS
Datasets. We consider long-tailed (LT) versions of the following three popular datasets for the experiments: CIFARIO-LT, CIFARI and ImageNer100-LTT. For most of the experiments, we follow the setting from SDCLR (Jiang et al., 2021). In case of CIFARIO-LT/CIFA R10-LIT, 11000/ the
original datasets (Krizhevsky al., 2009) consist of 60000 32x32 images sampled uniformly from 10 and 100 semantic classes, respectively, where 50000 images correspond to the training set and 10000 to a test set. Long-tail versions of the datasets are introduced by Cui et al. (2019) and consist of subset of the original datasets with an exponential decay in the number of images per class. The imbalance ratio controls the uniformity of the dataset and calculated as the ratio of the sizes of the biggest and the smallest classes. By default, we use an imbalance ratio 100 if not stated otherwise. Experiments in Tab. 1, Tab. are the average over three runs with different permutations of classes. ImageNet100-LLT is a subset of the original ImageNet-100 (Tian al., 2020a) consisting of 100 classes for a total of 12.21k 256x256 images. The number of images per class varies from 1280 to 25.
Training. We use an SGD optimizer for all experiments with a weight decay of le-4. As for the learning rate, we utilize linear warm-up for 10 epochs that is followed by a cosine annealing schedule starting from 0.5. We train for 2000 epochs for CIFAR 10-LT and CIFAR 100-LT and 800 epochs for ImageNet100-LT, For CIFAR L0-LT and L00-LT we use 8 (He et al., 2016) backbone. For ImageNet100-L1 we use ResNet50 (He et al., 2016) backbone. For both the MoCo (He et al., 2020) and the SimCLR (Chen al., 2020a) experiments, we follow Jiang et al. (2021) and use the following augmentations: resized crop, color jitters, grey scale and horizontal flip. MoCo details: we use a dictionary of size 10000, a projection dimensionality of 128 and projection head with one linear layer. SimCLR details: we train with batch size of 512 and projection head that has two layers with an output size of 128. For evaluation, we discard the projection head and apply 12-normalisation Regarding the proposed temperature schedules (TS), we use period length of 4=400 with 1.0 and 7_ 0.1 if not stated otherwise; for more details, see appendix A.2.
Evaluation We use k nearest neighbours (kNN) and linear classifiers to assess the learned features. For kNN, we compute 12-normalised distances between LT samples from the train set and the class- balanced test set. For each test image, we assign it to the majority class among the top-k closest train images. We report accuracy for kNN with 1=1 (kNN @ 1) and with x= 10 (kNN @ 10). Compared to fine-tuning or linear probing, kNN directly evaluates the learned embedding since it relies on the learned metric and local structure of the space. We also evaluate the linear separability and generalisation of the space with linear classifier that we train on the top of frozen backbone. For this, we consider two setups: balanced few-shot linear probing (FS LP) and long-tailed linear probing (LT LP). For FS LP, the fhw-shot train set is direct subset of the original long-tailes train set with the shot number equal to the minimum class size in the original LT train set. For LT LP, we use the original LT training set. For extended tables, see appendix A.3.
4.2 EFFECTIVENESS OF TEMPERATURE SCHEDULES
Contrastive learning with TS. In Tab. I we present the efficacy of temperature schedules (TS) for two well-known contrastive learning frameworks MoCo (He et al., 2020) and SimCLR (Chen et al., 2020a). We find that both frameworks benefit from varying the temperature and we observe consistent improvements over all evaluation metrics for CIFARIO-EE IO-LT and CIFARI00-LT, i.e. the local structure of the embedding space (kNN) and the global structure (linear probe) are both improved. Moreover, we show in Tab. 3 that our finding also transfers to ImageNet100-LT.. Furthermore, in Tab. 2 we evaluate the performance of the proposed method on the CIFAR 10 and CIFAR100 datasets with different imbalance ratios. An imbalance ratio of 50 (imb50) reflects less pronounced imbalance, and imb 150 corresponds to the datasets with only 30 (CIFAR10) and 3 (CIFAR 100) samples for the
7

---

Published as a conference paper at ICLR 2023
smallest class. Varying during training improves the performance for different long-taile tailed data; for a discussion on the dependence of the improvement on the imbalance ratio, please see the appendix.
CIFARIO-LT
CIFARI00-LT
method
KNN@1
KNN@ 10
FS LP
LT LP
KNN@
kNN@ 10
FS LP
LT LP
MoCo
63.54
64.56
69.31 72.87
65.11
28.69
28.75
26.86
30.41
MoCo + TS
64.99
65.01
66.86
30.31
29.75
28.97
32,05
SimCLR 59.84
60.19
68.29
61.86
28.81
28.12
25.70
31.20
SimCLR TS
63.09
62.91
71.86
65.03
31.06
30.06
28.89
33.28
Table Effect of temperature scheduling. Comparison of MaCo MoCo+TS and SimCL. SimCLR+TS
on COFARIO-LT and CIFAR 100-IT with KNN, few-shot and long-tail linear probe (FS LP and L.T LP).
CIFAR-IO-LT
CIFAR-100-LT
imb 50
imb 150
imb S0
imb 150
method
kNN@
FS LP
kNN@
FS LP
kNN@
FS
KNN@ 10
MoCo
69.12
74.16 76.37
59.13 60.83
65.76
32.22
33.53
25.36
22.73
MoCo TS
71.49
68.59
33.24
35.03
26.75
22.78
Table 2: Effect of imbalance ratio. MoCo vs MoCo+TS on CIFARIO-LT and CIFARI00-LT for imbalance ratio 50 (imb50) and 150 (imb150). Evaluation metries: kNN classifier and few-shot linear probe (FS LP).
CIFAR-IO-LY
CIFAR-100-HE kNN@
ImageNet 100-LT
method
KNN@ 10
FS LP
LS
FS LP
LT
LP
kNN@ 10
FS LP
LT LP
SimCLR
60.19
68.29
61.68
28.12
25.70
31.20
38.00
42.64
44.82
SDCLR
60.74
71.03
64.99
29.22
27.28
34.23
37.36
42.74
46.40
SimCLR TS
62.91
71.86
65.03
30.06
28.89
33.28
38.86
45.18
47.26
Table Comparison with SDCLR. SimCLR SDCLR vs SimCI.R+TS CIFARTO-LT, CIFAR and ImageNet100-LT. Evaluation: kNN classifier, few-shot (FS LP) and long-tail linear probe (LT LP).
TS vs SDCLR. Further, we compare our method with SDCLR (Jiang et al., 2021). ). In SDCLR, SimCLR is modified s.t. the embeddings the online model are contrasted with those of a pruned version of the same model, which updated after every epoch. Since the pruning is done by simply masking the pruned weights of the original model, SDCLR requires twice as much memory compared to the original SimCLR and extra computational time prune the model every epoch. In contrast, our method does not require any changes the architecture or training. In Tab. we show that this simple approach improves not only over the original SimCLR, but also over SDCLR in most metrics.
4.3 ABLATIONS
In this section, we evaluate how the hyperparameters in Eq. (5) influence the model behaviour.
Cosine Boundaries. First, we vary the lower T- and upper 74 bounds of for the cosine schedule. In CIFAR10 with FS LP. We
Tab. 4 we assess the performance of MoCo+TS with different 7_ and observe a clear trend that with wider range of J values the performance increases. We attribute this to the ability of the model to learn better 'hard' features with low 7 and improve semantic structure for high r. Note that 0.07 the value for 7 in many current contrastive learning methods.
TS
FS LP
0.2
0.3
0.4
0.5
1.0
fixed
68.89
step
70.18
0.07
69.46
68.86
71.29
71.83
73.26
rand 70.26
0.1
68.17
70.34
71.25
72.31
72.87
oscil
71.50
0.2
68.89
69.37
70.12
69.65
71.42
cos 72.31
Table 4: Influence of cosine boundaries. Best performance with the largest difference between r_ and 74., CIFARI0 with MoCo+TS, evaluating few-shot linear probes (FS LP).
Table 5: Alternative Schedules. Constant, step function, and random sampling. All
functions are bounded by 0.1 and 0.5.
8

---

Published as a conference paper at ICLR 2023
Cosine Period. Further, we investigate if the length of the period T in Eq. (5) impacts the performance of the model. In Tab. 6, we show that modifying the temperature T based on the cosine schedule is beneficial during training independently of the period T. The performance varies insignificantly depending on 7 and consistently improves over standard fixed r 002.2 whereas the best performance achieve with 70000. Even though the performance is stable with respect the length of the period, it changes within one period as we show in Fig. 4. Here, we average the accuracy of one last full period over different models trained with different I and find that the models reach the best performance around 0.7 T. Based on this observation, we recommend to stop training after (n- 0.3) T epochs, where is the number of full periods.
74
T
T/ #epochs
FS LP
72-
no
fixed
68.89
200
0.1
71.86
400
0.2
72.87
68
1000
0,5
72.47
2000
1.0
72.22
67-
4000
2.0
72.10
0.0
0.2
0.4
period T
0.6
0.8 1.0
Table 6: Influence the period length T. Few-shot linear probe accuracy (FS
Figure Dependence on relative time one period. Blue: Average F$ LP of last period of the models trained with = 200, 100, 1000, 2000. Light blue: variance. Yellow: Relative cosine value over relative time. CIFARIO-LT trained with MoCo+TS.
LP) of MoCo+TS on CIFARIO-LA,
Alternatives to Cosine Schedule. Additionally, we test different methods of varying the temperature parameter and report the results in Tab. 5: we examine linearly oscillating (oscil) function, step function, and random sampling. For the linear oscillations, we follow the same schedule as for the cosine version, as shown on the right of Tab. 5. For the step function, we change from a low (0.1) to a high (0.5) value and back every 200 epochs. For random, we uniformly sample values for from the range [0.1, 0.5]. In Tab. 5 we observe that both those methods for varying the r value also improve the performance over the fixed temperature, while with the cosine schedule the model achieves the best performance. These results indicate that it is indeed the task switching between group-wise and instance- discrimination during training which is the driving factor for the observed improvements for unsupervised long-tail representation learning. We assume the reason why slow oscillation of the temperature performs better than fast (i.e. random) temperature changes grounded in learning dynamics and the slow evolution of the embedding space during training.
CONCLUSION
In this work, we discover the surprising effectiveness of temperature schedules for self-supervised contrastive representation learning on imbalanced datasets. In particular, we find that simple cosine schedule for 7 consistently improves two state-of ofte--of-hh-arrr contrastive methods over several datasets and different imbalance ratios, without introducing any additional cost.
Importantly, our approach is based on a novel perspective on the contrastive loss, in which the average distance maximisation aspect is emphasised. This perspective sheds light on which samples dominate the contrastive loss and explains why large values for 7 can lead to the emergence of tight clusters in the embedding space, despite the fact that individual instance always repel each other.
Specifically, we find that while large thus necessary to induce semantic structure, the con- comitant focus group-wise discrimination biases the model to encode easily separable features rather than Instance-specifi details. However, in long-tailed distributions, this can be particularly harmful to the most infrequent classes, as those require higher degree of instance discrimination to remain distinguishable from the prevalent semantic categories. The proposed cosine schedule for T overcomes this tension, by alternating between emphasis on instance discrimination (small and group-wist wise discrimination (large T). As a result of this constant 'task switching', , the model is trained to both structure the embedding space according semantically meaningful features, whilst also encoding instance-specific details such that rare classes remain distinguishable from dominant ones.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The paper proposes an analysis and a method to improve the performance of self-supervised rep- resentation learning methods based the contrastive loss. The method and investigation in this paper do not introduce any ethical issues to the field of representation learning, as it is decoupled from the training data. Nonetheless, we would like point out that representation learning does not automatically prevent models from learning harmful biases from the training data and should not be used outside of research applications without thorough evaluation for fairness and bias.
ACKNOWLEDGEMENTS
C. is supported by VisualAI EP/T028572/1 and 1RC-UNION--OG--1000012..
REFERENCES
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thi- lakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In CVPR, 2022.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varrance-ivaarane--cvvaaaaee regularization for self-supervised learning. In ICLR, 2022.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, pp. 6541-6549, 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong supervieee learners. In NeurIPS, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. NeurIPS, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020c.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2009.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In CVPR, 2020.
Aleksandr Ermolov, Aliaksandr Siarohin. Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, pp. 8730-8738, 2018.
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online bag-of-visual-word. generation for self-supervised learning. In CVPR, 2021.
10

---

Published as a conference paper at ICLR 2023
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. NeurIPS, 2020.
Matthew Gwilliam and Abhinav Shrivastava. Beyond supervised vs. unsupervised: Representative benchmarking and analysis of image representation learning. In CVPR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondfej Chum. Mining on manifolds: Metric learning without labels. In CVPR, 2018.
Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In ICML, 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. NeurIPS, 2020.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. ICLR, 2020.
Bulat Khaertdinov, Stylianos Asteriadis, and Esam Ghaleb. Dynamic temperature scaling in con- trastive self-super learning for sensor-based human activity recognition. IEEE Transactions on Biometrics, Behavior, and ldentity Science, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tro Laina, Ruth Fong, and Andrea Vedaldi. Quantifying learnability and describability of visual concepts emerging in representation learning. NeurlPS, 33:13112-13122 2020.
Iro Laina, Yuki M Asano, and Andrea Vedaldi. Measuring the interpretability of unsupervised representations via quantized reversed prohing. In ICI.R, 2021.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In NeurlPS 2021 Workshop Distribution Shifts: Connecting Methods and Applications, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiiv1807..3777 2018.
William Reed. The pareto, zipf and other power laws. Economics Letters, 2001.
Joshua Robinson, Ching-Yoe Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In ICLR, 2020.
Joshua David Robinson, Li Sun, Ke Yu, kayhan Batmanghelich, Stefanic Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? In NeurlPS, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Nina Shvetsova. Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at once reee-mmlti-moone fusion transformer for video retrieval. In CVPR, 2022.

---

Published as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? NeurIPS, 2020b.
Yonglong Tian, Olivier Henaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uneurated data. In ICCV, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Self-supervised learning from multi-view perspective. In ICLR, 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In ICCV, 2021.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In ICCV, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML, 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning model the tail. NeurIPS, 2017.
Chen Wei, Haoqi Fan, Saining Xie, Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self self-supervised visual pre-training, In CVPR, 2022.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In ICML, 2021.
Zhirong Wu, Yuanjun Xiong, Yu Stella, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbllanced learning. In NeurlPS, 2020.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pcl: Proxy-based contrastive learning for domain generalization. In CVPR, 2022.
Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. arXiv preprint arXiv:2/ 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Sulf-super learning via redundancy reduction. In ICML, 2021.
Chaoning Zhang, Kang Zhang, Trung X Pham, Axi Niu, Zhinan Qiao, Chang D Yoo, and In So Kweon. Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying moco. In CVPR, 2022.
Oliver Zhang, Mike Wu, Jasmine Bayrooti, and Noah Goodman. Temperature uncertainty in contrastive learning. arXiv preprint arXiv:2 10.04403, 2021.
Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. Is self-supervised learning more robust than supervised learning? arXiv preprint arXiv:2206.052: 05259, 2022.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang Contrastive learning with boosted memorization. In International Conference Machine Learning, pp. 27367-27377. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PSEUDO-CODE FOR REPRODUCIBILITY OF COSINE SCHEDULE
Algorithm 1 Cosine Schedule
Require:
period T 2 0, T- = 0.1, = 1.0
ep - current epoch
lau +- (T+ T-) x (1=np.cos22xpppp xep/T)) +
Insert algorithm into your favourite contrastive learning framework check out!
A.2 IMPLEMENTATION DETAILS
Evaluation details. Following Jiang et al. (2021), we separate 5000 images for CIFAR10/100-11 validation set for each split. As discussed in the main paper, the performance of the model depends on the relative position within a period T. Therefore we utilise the validation split to choose checkpoint for further testing on the standard test splits for CIFARIO/ 10/10LLL.. Preciscly, for each dataset, we select the evaluation epoch for the checkpoint based only on the validation set of the first random split; the other splits of the same dataset evaluated using the same number of epochs. Note that for ImageNet1004 (00-LT there is no validation split and we select the last checkpoint as in Jiang al. (2021). For fair comparison, we also reproduce the numbers from Jiang al. (2021) in the same way.
Division into head, mid, and tail classes. Following Jiang et al. (2021), we divide all the classes into three categories: head classes are with the most number of samples, tail classes are with the least number of samples and mid are the rest. In particular, for CIFAR 10-LT for each split there are head classes, 3 mid classes, and tail classes; for CIFAR100-LT there are 34 head classes, 33 mid classes, 33 tail classes; for ImageNet100-15 head classes classes with more than 100 instances, tail classes have less than 20 instances per class, and mid are the rest.
A.3 EXTENDED RESULTS
Extension of Fig. 3 In Fig. 5 we provide full results of kNN accuracy on CIFAR 10 when the model trained with different fixed 7 values and with coarse binary supervision. Especially tail classes are improved by instance discrimination (small Tril).
KNN
1.5-
1.0
[0.1.1.0]
Accurasy
0.6
0.4
0.2
2025
6mer
dng
mrgg
truck
Class
Figure 5: kNN accuracy for CIFARIO-LT trained with MoCo. Comparison between 7 = 0.1, 0.5, 1.0.
10.1, 1.0] denotes coarse binary supervision with Thead 1.0 and Ttall 0.1. MAA: mean average accuracy over classes.
13

---

Published as a conference paper at ICLR 2023
Head-mid-tail classes evaluation. In the following, present detailed comparison of SimCLR and SimCLR+TS on head, mid, and tail classes on CIFARIO-LT in Tab. on CIFARI00-1E in Tab. and ImageNet100-LT in Tab. 9. We observe consistent improvement for all evaluation metrics for all types of classes over the three datasets.
CIFAR-IO4 -IFAR-IO-LT
kNN@1 Mid
kNN@
method
Head
Tail
Head
Mid
Tail
SimCLR
84.93
3.44
54.08
4.24
32.14
# 7.44
88.03
+ 3.32
53.76
= 4,80
29.52
9.44
SimCLR TS
87.24
+ 3.05
58.96
= 5.21
35.02
= 8.27
89.92
+ 2.97
59.31
4.69
30.51
12.38
FS LP
LT LP
method
Head
Mid
Tail
Head
Mid
Tail
SimCLR
76.38
+ 5.24
63.20
+ 2.95
62.60
= 3.63
89.52
+ 3.15
56.98
4.74
29.88
8.11
SimCLR TS
80.54
5.02
66.50
+ 4.38
65.67
+ 4.07
91.73
2.49
62.09
4.21
32.38
+ 9.23
Table Detailed evaluation on CIFARIO-LT. Evaluation metries: kNN@ 1010, FS LP states for few-shot linear probe, and states long-tail linear probe. We report the average performance with the standard deviation over three different random splits for different sets of classes: head, mid, and tail.
CIFAR-I00-LT
kNN@ @1
KNN@ 10
method
Head
Mid
Tail
Head
Mid
Tail
SimCLR
53.87 57.14
2.12
24.56
1.51
7.26
0.39
58.46
46
1.79
22.15
=
2.83 3.05
SimCLR TS
1 1.95
26.00
1 1.20
8.31
1. 0.57
61.93
1.88
24.22
4 2.23
1. 0.54
FS LP
LT LP
method
Head
Mid
Tail
Head
Mid
Tail
SimCLR
33.48
1.24
24.25
2.12
19.12
1.35
62.19
1.80
26.56
= 1.46
3,92
+ 0.46
SimCLR + TS
37.5
+ 1.33
27.64
- 199
21.26
+ 0.66
65.24
+ 2.04
29.20
1.48
4.42
+ 0.26
Table 8: Detailed evaluation on CIFARI00-LE Evaluation metries: kNN@ 1,10, FS LP states for few-shot linear probe, and LT LP states for long-tail linear probe. We report the average performance with the standard deviation over three different random splits different sets of classes: head, mid, tail.
ImageNet100-LT
kNN@1 Mid
kNN @ I0 Mid
method
Head
Tail
Head
Tail
SimCLR SimCLR TS
55.13
30.00
10.71
58.51
29.70 29.53
8.71
57.23
30.26
13.14
60.41
10.14
FS LP Mid
LT LP Mid
method
Head
Tail
Head
Tail
SimCLR
51.79 60.41
36.77
30.29
67.59 70.67
36.47 38.85
9.43
SimCLR TS
40.38
33.57
10.29
Table 9: Detailed evaluation on ImageNet100-LTT Evaluation metrics: kNN@ 1,10, FS LP states for few-shot linear prohe, and LT LP states for long-tail linear probe. We report the average performance for different sets of classes: head, mid, and tail.
Influence of TS Uniform Long-Tailed Distributions. To further corroborate that TS par- ticularly helpful for imbalanced data, we apply TS for the uniformly distributed data. In Tab. 10, we can observe that the cosine schedule yields significant and consistent gains for the long-tailed We assume
version of CIFAR IO CIFARIO-LTT but not for the uniform one CCIFARI-Unnifon that both head classes and tail classes for long-tail distribution should be expected to benefit from better separation between the two: on the one hand, the tail classes form better clusters and are thus the other hand, the clusters of the head classes
easier to classify based on their neighbours, "purified', which should similarly improve performance. Weather, for the uniform distribution, we do not observe such influence of TS and the performance changes only marginally.
14

---

Published as a conference paper at ICLR 2023
CIFARIO-Uniform kNN@10
CIFARIO-LT
method
kNN@1
LP
LT LP
kNN@1
kNN@10
FS LP
LT
MoCo
83,47
84.87
90.19
87.70
63,00
64.10
68.89
63.99
MoCo TS
83.78
85.85
90.02
87.40
65.68
65.91
72.31
66.64
Table 10: Influence of 'S on uniform vs long-tailed distribution. Comparison of MoCo vs MoCo+TS CIFARIO- Rn-UUoom and CIFAR-LT-imb10 onc split. Evaluation metrics: kNN classifier, FS LP denotes fow-shot linear probe, LT LP denotes long-tail lincar probe.
A.4 INFLUENCE OF THE POSITIVE SAMPLES ON CONTRASTIVE LEARNING
In Sec. 3.2, we particularly focused on the impact the negative samples on the learning dynamics under the contrastive objective, they likely are the driving factor with respect to the semantic structure. fact, we find that the positive samples should have an inverse relation with the temperature and thus cannot explain the observed learning dynamics, as we discuss in the following.
To understand the impact of the positive samples, first note their role in the loss (same as Eq. (4)):
C log (1 + c....)
(6)
In particular, ca scales the entire sum S, Σ بهز exp(-d.j) dıy)- As such, encoding two augmentations of the same instance at large distance is much more "costly' for the model than encoding different samples close to each other, as each and every summand S; amplified by the corresponding Cy. As
result, the model will be biased "err the safe side' and become invariant to the augmentations, which has been one of the main motivations for introducing augmentations in contrastive learning in the first place, cf. Tian al. (2020b); Chen et al. (2020a); Caron al. (2020).
Consequently, the positive samples, of course, also influence the forming of clusters in the embedding space as they induce invariance with respect augmentations. Note, however, that this does not contradict our analysis regarding the impact of negative samples, but rather corroborates it.
In particular, eæ biases the model become invariant the applied augmentations for all values of т; in fact, for small r, this invariance is even emphasised as increases for small t and the influence of the negatives is diminished. Hence, if the augmentations were the main factor in inducing semantic structure in the embedding space, should have the opposite effect of the one we and many others (Wang& & Liu, 2021; Zhang al., 2022; 2021) observe.
Thus, instead of inducing semantic structure on their own, we believe the positive samples to rather play critical role in influencing which features the model can rely on for grouping samples in the embedding space; for detailed discussion of this phenomenon, see also Chen et al. (2021).
15


Reviews:
[{"review_id": "yU9odsdmVN", "reviewer": "Reviewer_bPFj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores self-supervised contrastive learning for long-tail data. It presents an empirical study of the impact of the temperature parameter on the head and tail categories. Based on the study, the paper proposes a simple yet effective solution to improve the imbalance pretraining which involves a dynamic temperature schedule. Compared to the vanilla contrastive learning and the SDCLR baseline, the proposed method demonstrates better performance on three long-tail benchmarks: CIFAR10-LT, CIFAR100-LT, ImageNet100-LT.", "summary_of_the_review": "The paper provides an empirical analysis of the impact of the temperature parameters on self-supervised contrastive long-tail pretraining. A simple yet effective temperature scheduling solution is also proposed and demonstrates superior performance on three long-tail benchmarks. In general, I'm happy to learn about the good performance of this simple solution but would prefer some clarifications on the evaluation ([W2]). The paper could be stronger if it includes further investigations to interpret the results of TS ([W1]) and analysis of non-contrastive self-supervised methods ([W3]).\n\n***\nAfter rebuttal:\n\nMy concerns are addressed in the rebuttal. I thereby raise the score to 8.", "strength_and_weaknesses": "Strength\n\n**[S1]** Topic: the paper investigates self-supervised learning for long-tail data. This is one of the first steps toward applying self-supervised approaches to uncurated data, which is of great interest to the community;\n\n**[S2]** Insight: the paper provides an empirical analysis of the impact of the temperature parameter on the head and tail categories;\n\n**[S3]** Technology: the proposed dynamic temperature scheduling is technically sound;\n\n**[S4]** Performance: compared to the vanilla contrastive learning and the SDCLR baseline, the proposed method demonstrates better performance on three long-tail benchmarks: CIFAR10-LT, CIFAR100-LT, ImageNet100-LT.\n\n**[S5]** Presentation: the writing is clear.\n\n---\n\nWeaknesses\n\n**[W1]** Interpretation:\n\ni) I find it difficult to interpret the results of TS. The study in Sec.3.3 and Fig. 3 seems to indicate that TS achieves a trade-off between optimizing the performances of the head or tail classes: as shown in Fig.3, TS slightly reduces the performance of the head classes but significantly improves the performance of the tail classes. On the other hand, results in table 7/8/9 show that TS consistently improves the performance of both the head and tail classes. The improvements on the head and tail classes are also comparable: +2.1%/+2.4%  kNN@1 for the head/tail classes of ImageNet100-LT (table 9). It looks like TS is more than a simple trade-off?\n\nii) it is also unclear to me why decreasing the temperature during the training is the optimal scheduling solution (table 4 and 5). It looks like TS learns the head classes better at the early stage of the training then progressively prioritizes the tail classes. I wonder if TS may hurt the performance of the head classes at the later stage of the training. I would also be curious why alternative schedules perform worse than TS.\n\nOverall, I’m happy to learn about the good performance of TS but the interpretation of TS deserves more investigation than what is presented in the current paper.\n\n**[W2]** Evaluation. The results of SDCLR in table 3 look different from what were reported in the [SDCLR](https://arxiv.org/abs/2106.02990) paper. I wonder if these results are reproduced? If this is the case, I wonder if there is any difference in the experimental setting compared to the SDCLR paper. For example, the SDCLR paper reported results with 500 epochs pretraining while TS is trained with 800 epochs (Sec. 4.1), I wonder if different methods shown in table 3 share the same experimental configurations\n\n**[W3]** The analysis in Sec. 3.2 and 3.3 is limited to contrastive learning. It is not clear to me if TS could be beneficial to non-contrastive SSL methods that use a temperature parameter, e.g. SwAV, DINO, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "o2frdgmrFkv", "reviewer": "Reviewer_RFLZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the long-tailed learning problem by varying softmax temperature during the course of training. The authors argued that different magnitudes of the temperature value actually induce different learning preferences, which makes it possible to gradually switch the learning objective by carefully setting up a temperature scheduler. The proposed method has been evaluted on serveral long-tailed learning benchmarks, demonstrated its effectiveness.", "summary_of_the_review": "An interesting method, yet with limited empirical improvments.", "strength_and_weaknesses": "Strength:\n- The proposed method is interesting and is of a unique view.\n- The proposed method requires no computation overhead, which is a tempting merit.\n\nWeaknesses:\n- The evaluation is not comprehensive. Only contrastive learning based methods are compared against.\n- The improvement is quite limited compared with other long-tailed learning methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E3-mXt8KQf", "reviewer": "Reviewer_Fp8W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Contrastive learning objectives are a popular and effective method for self-supervised learning. Most variations of the contrastive objective have a static temperature hyperparameter $\\tau$. Previous work suggests that large values of $\\tau$ lead to improved group-wise discrimination while small values lead to improved instance-wise discrimination. The authors study the effect of varying this temperature parameter in 'long-tail' settings with imbalanced data. They posit that long-tail class performance benefits from instance-wise discrimination ability and run initial 'coarse supervision' experiments to test this claim. Since this method requires (weak) labels, the authors also propose a dynamic temperature schedule that can be used in unsupervised settings. In a series of ablation experiments, they find that a cosine schedule that gradually switches between the two phases leads to the best performance. ", "summary_of_the_review": "I think the work is interesting and important, but have some concerns detailed in the Strengths and Weaknesses section. \nI would be likely to raise my score and recommend acceptance if these concerns were addressed. \n\n***Update after author response***\nThe authors have addressed these weaknesses to my satisfaction. I have raised my score accordingly.", "strength_and_weaknesses": "Strengths:\n- The topic is interesting and offers a promising direction for deepening our theoretical understanding of contrastive learning.\n- The work is clear and easy to follow with the main theoretical claim explicitly laid out and experiments set-up to test it directly.\n- The experiments are conducted with a variety of models and datasets. \n\n\nWeaknesses:\n- It is unclear to me whether the cosine schedule is actually the optimal one. While it clearly outperforms the rapidly changing schedules like the step function, how does it compare, for example, to a linear (oscillating) schedule?\n- The marginal performance gain from MoCo to MoCo+TS does not seem significantly higher in the imb 150 vs imb 50 case. It isn't entirely clear whether the results support the hypothesis posited initially by the authors, otherwise one might expect the marginal gain to be higher in the more imbalanced case. It would be useful to compare marginal performance gain on CIFAR10-LT and on CIFAR10 to confirm that the improvement from the temperature schedule is actually related to the data imbalance. \n- There have been a few recent papers studying temperature in contrastive learning. In particular, how does your dynamic temperature scaling compare to the method in 'Dynamic Temperature Scaling in Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition' (Khaertdinov et al., 2022)?\n\n***Update after author response***\nThe authors have addressed these weaknesses to my satisfaction.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster