On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learning.
Authors: Jianhong Bai
Zuozhu Liu
Hualiang Wang
Jin Hao
YANG FENG
Huanpeng Chu
Haoji Hu
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised learning
long-tail learning
out-of-distribution data

Abstract: Though Self-supervised learning (SSL) has been widely studied as a promising technique for representation learning, it doesn't generalize well on long-tailed datasets due to the majority classes dominating the feature space. Recent work shows that the long-tailed learning performance could be boosted by sampling extra in-domain (ID) data for self-supervised training, however, large-scale ID data which can rebalance the minority classes are expensive to collect. In this paper, we propose an alternative but easy-to-use and effective solution, \textbf{C}ontrastive with \textbf{O}ut-of-distribution (OOD) data for \textbf{L}ong-\textbf{T}ail learning (COLT), which can effectively exploit OOD data to dynamically re-balance the feature space. We empirically identify the counter-intuitive usefulness of OOD samples in SSL long-tailed learning and principally design a novel SSL method. Concretely, we first localize the `\emph{head}' and `\emph{tail}' samples by assigning a tailness score to each OOD sample based on its neighborhoods in the feature space. Then, we propose an online OOD sampling strategy to dynamically re-balance the feature space. Finally, we enforce the model to be capable of distinguishing ID and OOD samples by a distribution-level supervised contrastive loss. Extensive experiments are conducted on various datasets and several state-of-the-art SSL frameworks to verify the effectiveness of the proposed method. The results show that our method significantly improves the performance of SSL on long-tailed datasets by a large margin, and even outperforms previous work which uses external ID data. Our code is available at \url{https://github.com/JianhongBai/COLT}.

Published as a conference paper at ICLR 2023
ON THE EFFECTIVENESS OF OUT-OF-DISTRIBUTION DATA IN SELF-SUPERVIE
LONG-TAIL LEARNING
Jianhong Bai'-, Zuozhu Liu Hualiang Wang', Jin Hao?, Yang Feng',
Huanpeng Chu', Haoji Hu
'Zhejiang University, "The Hong Kong University of Science and Technology. Harvard University, 'Angelalign Technology
ABSTRACT
Though Self-supervised learning (SSL) has been widely studied as a promising technique for representation learning, it doesn't generalize well on long-tailed datasets due to the majority classes dominating the feature space. Recent work shows that the long-tailed learning performance could be boosted by sampling ex- tra in-domain (ID) data for self-supervised training, however, large-scale ID data which can rebalance the minority classes are expensive to collect. In this paper, we propose an alternative but easy-to-use and effective solution, Contrastive with intto--distribution (OOD) data for Long-Tail learning (COLT), which can effec- tively exploit OOD data to dynamically re-balance the feature space. We empiri- cally identify the counter-intuitive usefulness of OOD samples in SSL long-tailed learning and principally design novel SSL method. Concretely, we first localize the 'head" and 'tail" samples by assigning a tailness score to each OOD sample based on neighhorhoods in the feature space. Then, we propose an online OOD sampling strategy to dynamically re-balance the feature space. Finally, we enforce the model to capable of distinguishing II) and OOD samples by distribution- level supervised contrastive loss. Extensive experiments are conducted on various datasets and several state-of-the-art SSL frameworks to verify the effectiveness the proposed method. The results show that our method significantly improves the performance of SSL on long-tailed datasets by a large margin, and even out- performs previous work which uses external ID data. Our code is available https: / htt//gtttuueete com / Ji Jiaaaaaaaoonnn BraaannooaaaaCO
INTRODUCTION
Self-supervised learning (SSL) methods (Chen et al. 2020 et 2020 Grill et al. 2020) provide distinctive and transferable representations in an unsupervised manner. However, most SSL meth- ods are performed on well-curated and balanced datasets (c.g., ImageNet), while many real-world datasets in practical applications, such as medical imaging and self-driving cars, usually follow a long-tailed distribution (Spain & isting SSL methods exhibit severe performance degradation when exposed imbalanced datasets.
Recent research (Liu et al. (02T) indicates that ex-
To enhance the robustness of SSL methods under long-tailed data, several pioneering methods (Jiang jet al. 2021b; Zhou et al. 2022) are proposed for a feasible migration of cost-sensitive learning, which is widely studied in supervised long-tail learning (Elkan 2001; [Sun et al.) 2007) Cui et al.) 2019b, Wang et al. 2022). The high-level intuition of these methods is to re-balance classes by adjusting loss values for diffferent classes, i.c., forcing the model to pay more attention to tail sam- ples. Another promising line of work explores the probability of improving the SSL methods with external data. (Jiang al. (021a) suggests re-balancing the class distributions by sampling external in-distribution (ID) tail instances in the wild. Nevertheless, they still require available ID samples in the sampling pool, which is hard to collect in many real-world scenarios, e.g., medical image diagnosis (Ju et al. 2021) or species classification (Miao et al. 2021)
"qqual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2023
SmiLLA
Ta SimCLR Head
3.00
SmELA. COz
TallCOO
3.75
Head-COLT
Uniformity
3.70
Aligmment
-0.42
3.00
-0.48
3.50
.0.5
3.45
300
150
250
150
205 Epoch
350
430
450
500
200
350
450 Epoch
500
660
(a) Uniformity
(b) Alignment
Feature space uniformity of different SSL frameworks. [b): Visualization of the alignment property of samples in minority classes and majority classes w/ or w/o COLT. The exper- iment is conducted with ResNet-18 on CIFAR-100-LT.
Figure 1:
la
The aforementioned findings and challenges motivate challenging setting: when the ID data not available, can we levenage the out-of-distrbbttonn (OOD) data improve the performance of SSL in long-tailed learning? Compare to MAK Jiang et al. aa that assumes external ID samples are available, while we consider a more practical scenario where we only have access to OOD data that can be easily collected (e.g., downloaded from the internet). A very recent work (Wei et al. 2022) proposes to re-balance the class priors by assigning labels to OOD images following a pre-defined distribution. However, performed in a supervised manner while not directly applicable to the SSL frameworks.
investigate another practical and
In this paper, we proposed novel and principal method to exploit the unlabeled OOD data to im- prove SSL performance long-tailed learning. As suggested in previous research, the standard contrastive learning would naturally put more weight on the loss of majority classes and less weight on that of minority classes, resulting in imbalanced feature spaces and poor linear separability on
tail samples(Kang et al.) 2020 Li et al. 2022). However, rebalancing minorities with ID samples, no matter labeled or unlabeled, is quite expensive. To alleviate these issues, we devise a framework, Contrastive Learning with OOD data for Long-Tailed learning (COLT), to dynamically augment the minorities with unlabeled OOD samples which close tail classes in the feature space. As illustrated in Fig. T our COLT can significantly improve SSL baselines in terms of the Align- ment and Uniformity Wang & Isola 2020). two widely-used metrics evaluate the performance of contrastive learning methods, demonstrating the effectiveness of our method.
The pipeline of our method is illustrated in Fig. 2 To augment the long-tail ID dataset, we define a tailness score to localize the head and tail samples in an unsupervised manner. Afterward, we design an online sampling strategy dynamically re-balance the long-tail distribution by selecting OOD samples close (with large cosine similarity in the feature space) to the head or tail classes based on predefined budget allocation function. We follow the intuition to allocate more OOD samples to the tail classes for rebalancing. Those selected OOD samples are augmented with the ID dataset for contrastive training, where an additional distribution-leww level supervised contrastive loss makes the model aware of the samples from different distributions. Experimental results on four long-tail datasets demonstrate that COLT can greatly improve the performance of various SSL methods and even surpass the state-of-the-art baselines with auxiliary ID data. We also conduct comprehensive analyses to understand the effectiveness of COLT. Our contributions can be summarized as:
We raise the question of whether we can and how to improve SSL on long-tailed datasets effectively with external unlabeled OOD data, which is better aligned with the practical scenarios but counter-intuitive to most existing work and rarely investigated before.
We design novel yet easy-to-use SSL method, which is composed of tailness score estima- tion, dynamic sampling strategies, and additional contrastive losses for long-tail learning with external OOD samples, alleviate the imbalance issues during contrastive learning.
We conducted extensive experiments on various datasets and SSL frameworks to verify and understand the effectiveness of the proposed method. Our method consistently out- performs baselines by a large margin with the consistent agreement between the superior performance and various feature quality evaluation metrics of contrastive learning.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Supervised learning with imbalanced datasets Early attempts aim to highlight the minority sam- ples by e-balancing strategy. These methods fall into two categories: re-sampling at the data level (Shen et al., 2016 Zou et al.) 2018 Geifman & El- Yaniv, 2017), reiweighting at the loss (gradi- ent) level (Cao et al. 2019 Jamal et al.] 2020p. Due to the usage of label-related information, the above methods can be generalized the unsupervised field. (Kang et al. 2019) suggests that the scheme of decoupling learning representations and classifiers benefits long-tail learning. The feasibility of two-stage training promotes the exploration in unsupervised scenarios.
Self-supervised long tail learning (Yang & Xu 2020) is, to our best, the first to analyze the per- formance of SSL methods in long-tail learning and verify the effectiveness of self-supervised pre- training theoretically and experimentally. However, (Liu et al. (021)) shows that SSL methods although more robust than the supervised methods are not immune to the imbalanced datasets. Follow-up studies improve the ability of SSL. methods on long-tailed datasets. Motivated by the observation that deep neural networks would easily forget hard samples after pruning (Hooker et al.) 2019), (Jiang et al. 202 proposed a self-competitor to pay more attention to the hard (tail) sam- plcs. BCL (Zhou et al. 2022) involved the memorization effect deep neural networks (Zhang al. 202 into contrastive learning, i.e., they emphasize samples from tail by assigning more powerful augmentation based on the memorization clue. We show that our method is non-conflict with existing methods and can further improve the balancedness and accuracy (Section 4.2).
Learning with auxiliary data Auxiliary data is widely used in the field of deep learning for different purposes, e.g., improving model robustness (Lee et al., 2020), y combating label noise (Wei et al., 2021 OOD detection (Liang et al., 2018 Hendryyck et 2018a),, domain generalization (Li et al,,
2021 Liu et al. [2020] Long et al. 2015). neural network compression (Fang et al. (022). training large models (Alayrac et al. 2022; Brown et al.) 2020). In long-tail learning, MAK (Jiang et al.
2021ap suggests tackling the dataset imbalance problem by sampling in-distribution tail classes' data from an open- wpendworld sampling pool. On the contrary, we explore the probability of helping long-tail leamning with OOD samples, i.e., none of the ID samples are included in the sampling pool. Open- Sampling (Wei et al. (2022) utilizes the OOD samples by assigning label to each sample following a pre-defined label distribution. Their work performed under supervised scenarios, and the OOD data is not filtered, which results in massive computation overhead.
3 METHOD
3.1 PRELIMINARIES
Unsupervised visual representation learning methods aim to find an optimal embedding function ʃ.
which projects input image X E RRCHW to the feature space Z Bd with = [xx), such that retains the discriminative semantic information of the input image. SimCLR (Chen et al., (2020) is one of the state-of-the-art unsupervised learning frameworks, and its training objective is defined as:
N
explzi 21/77
LCL.
log N
exp(z, 2t7r) + Ex, ez 20p(z;-2,1
where (z,, =,t) is the positive pair of instance i, श indicates the negative samples from the negative practice, a batch of images is augmented twice
set Z- and r is the temperature typer-parameter in different augmentations, the positive pair is formulated as the two views of the same image, and the negative samples are the views of other images.
3.2 LOCALIZE TAIL SAMPLES IN
TRAINING
Due to the label-agnostic assumption in the pre-training state, the first step of the proposed method is to localize tail samples. As mentioned earlier, the majority classes dominate the feature space, and tail instances turn out to be outliers. Moreover, the minority classes have lower intra-class consistency (Li et al., (022). Hence, a sparse neighborhood could be a reliable proxy to identify the tail samples (More analysis can be found in Section 4.4). Specifically,
top-k% (k:
3

---

Published as a conference paper at ICLR 2023
Pipeline:
Projected feature space
External DOD data
Unsupervised clustering
sampling
A
Sompling
Contrastive Backbone
Sompied 000 dota
Re fe-sampling every epochs
Contrastive backbone
Oistribution awareness
Augmented train set
Projection head
000 dota
Encoder
Samole
Tailness score
data
view 2
Projection head
Ditributionn supervised astirastive loss
22
Notations:
many
Sampling area
X
X
Sampled/Unsamples
Sample budget
OOD
Lower
Toriness score
Higher
median
means cluster
Push apart
*
Cluster conter
Figure 2: Overview of Contrastive with Out-of-distribution data for Long-Tail Tail learning (COLT). COLT can be easily plugged into most SSL frameworks. Proposed components are denoted as red.
in practice) largest negative logits of each sample to depict the feature space neighborhood during training. Given training sample æ;, its negative logits P; is the following:
exp( Zi 2, /T)
PT
(2)
exp z; 2//T) Ex, ez exp(z; 2, /T)
Considering implementing SimCI. (Chen al. 2020) with batch size B, each image has 2( negative samples. Then, we define s Enp- Pi as the tailness score for each ID instance
4.0 st si a;, During training, perform a momentum update to the tailness score, i.e., 8t mss:n=1 + (1 m)s;" in where
[0,1) is the momentum coefficient. The momentum update makes the tailness score more robust and discriminative to the tail samples. A higher value of sf indicates sample æ; has more sparse neighborhood in the feature space and implies that belongs to the tail classes with larger probability. Experiments in Fig @ejepirrcal demonstrated that tail samples could be effectively discovered by our proposed tailness score.
3.3 DYNAMICALLY BE-BALANCE THE FEATURE SPACE WITH ONLINE SAMPLING
The core of our approach is sample OOD images from the sampling pool Sood and further balance the original long-tail tail ID dataset and the feature space. First, we obtain C' feature prototypes ze, from ID training set Sia via K-means clustering. Note that jection layer since the contrastive process performed on this layer. The cluster- wise tailness score of is defined as the mean of tailness score in cluster C, i.e., ct = Ezzec, s//cal, here |ci| is the number of instances in cluster c,. Then, we obtain each cluster's sampling budget K' as follows:
the features the last pro-
8f mean(ss) 86
K" K sofomaaa(s//aa
std((s))
where K refers to the total sampling budget, K' C is the sampling budget assigned to each cluster, sf is the normalized cluster tailness score. Empirically, assign sampling budget to the tailness clusters to be consistent with the idea of re- balancinn the feature space. We sample OOD images whose feature is close (higher cosine similarity) the ID prototypes Zc,-
4

---

Published as a conference paper at ICLR 2023
To fully exploit the OOD data, we re-sample from the Sood every T epoch. The motivation behind this is: i) the sampled OOD data can be well-separated from Sid after a few epochs, therefore becoming less effective to re-balance the feature space; ii) over-fitting to the OOD data can be toxic to the ID performance (Wei al. (022). From another perspective, this online sampling strategy lets the ID training set (especially the tail samples) continuously be exposed to the more newly sampled effective negative samples, forcing the model gives more distinctive embeddings and better fitting the ID distribution. The online sampling process is summarized in Algorithm
Algorithm 1 The overall pipeline of COLT.
Algorithm 2 our online sampling strategy.
ID train set Sid. OOD dataset Sood. model 0, sample budget K, cluster number C. similarity cient m, warm-up epochs w, sample interval r, metric sim(-). hyper-parameter Te-
Input: ID train set Sta. OOD dataset Sood. sam-
Input:
ple budget K, train epoch T, momentum coeffi-
cluster number C, peperrparameter k, Te-
Output: new train set Strain-
Output: pre-trained model parameter 0r. Initialize: model parameter Oo. the original train set Strain Sid.
Calculate both ID features 2nd and OOD fea- tures 2 aod through model 0;
Obtain C' ID prototypes ze, via K-means clus- tering in the projected feature space; Calculate cluster-wise tailness score by sf
if epoch then Train model 8, with Eq. 1 and compute sf;
end for epoch 1,... T do
Ex,ec, st/|cal;
Assign each cluster a sample budget Ky with Eq. 3
if' epoch 2 w then
if (epoch- - w) %r then
Initialize the sample set Ssample = 0; for O,... C do Initialize subset S sampie = 0; while |S samplel < Ky do u arg max æ, ESoo sim(zjj Za;):
Sample by performing Algorithm 2
end if
Calculate the supervised contrastive loss
with Eq. 5
Use Strain to train 0 epoch with Eq. 6
else
S Semmpl = S sumple Ufu}: end while
Use Strain to train O eepoch with Eq. I
end if
S Seample Seample S end for
Saamllei
Compute si. and update s epocc end for
Strain Strain U Ssample-
3.4 AWARENESS OF THE OUT-OF-DISTRIEE
DATA
Section 3.2 and Section 3,3 introduce our sampling strategy toward OOD images. To involve the sampled OOD subset Sample in training, a feasible way is directly using the augmented training set (containing both ID) and OOD samples) to train the model with Eq. [ However, we would argue that giving equal treatment to all samples may not be the optimal choice (details in Section 4). One natural idea is to let the model be aware of that there are two kinds of samples from different domains. Henee, we define an indicator & to provide weakly supervised (distribution only) information:
(1, Φ; E Sid -1, xc Sood-
o(z,)
(4)
Afterward, we add a supervised contrastive loss (Khosla et al. 2020 to both ID and OOD samples:
N
exp(z; xp/T)
LSCL N
- log PEP(3)
(5)
exp(z; zp/T) + EneN(a) exp(z, En/T)'
where P(i) = ((xp) ((x))) is the set of indices of the same domain within the mini-batch, [P()) is its cardinality and the negative index set (() = (n. Φ(x,,) * ((x;;) contains index from different distribution. Fig 3e illustrates that the proposed distribution-awarren only the overall performance but also facilitates more balanced feature space. It's worth noting that the proposed loss only utilizes the distribution information as the supervised term, while the labels for both ID and OOD samples are unavailable during the self-supervised training stage. Finally, we scale the supervised loss with a and add it to the contrastive loss in Eq I
loss improves not
LCOIT Leu + aLscI.
5

---

Published as a conference paper at ICLR 2023
Table Test accuracy (%) and balancedness (StdL) on CIFAR-IO-LT and CIFAR-I00-LT.
Method
CIFAR-IO-LT
CIFAR-100-LT
Metric
Many f
Median f
Few
Std +
All f
Many 1
Median
Few f Std 4
All
SimCLR +COLT
82,40 87.50
73.91
70.19
5.11
75.34
51.50
45.58
45.96
2.71
47.65
81.65
80.80
2.98
83.15
57.94
56.74
57.72
0.52
57.46
SDCLR
86.69
82.15
76.23
4.28
81.74
58,54
55.70
52.10
2.64
55,48
+COLT
90.87
84.28
81.45
3.95
85.41
63.28
60.85
59.42
1.59
61.18
BCL-I 86.97 89.03
82,40
76.45
4.31
81.99
58.92
54.63
53.58
2.31
55.70
+COLT
85.10
80.36
3.55
84.86
61.12
57.03
55.82
2.27
57.98
Table 2: Test accuracy (%) and balancedness (Stdl) on ImageNet-100-LT and Places-LT.
Method
ImageNet-100-1E
Places-LT
Metric
Many t
Median f
Few
Std 4 All f
Many 1
Median
Few f Std 4
All 1
SimCLR +COLT
70.96
65.33
61.89
3.74
67.08
40.02
46.61
49.38
3.93
44.78
75.13
71.38
66.62
3.48
72.22
41,55
48.40
50.54
3.83
46.36
SDCLR
71,13 75.13
66.04
62.31
3.61
67.54
40.13
46.61
48.90
3.71
44.73
+COLT
70.25
67.69
3.08
71.82
41.72
48.42
50.78
3.84
46.47
4
EXPERIMENTS
In this section, first introduce the datasets and experimental settings (Section[4] and evaluate the proposed COLT in three aspects: accuracy and balancednesss(Sectionff (Section 4.3) Then, verify whether method can 1), localize tail samples, 2), re-bal re-balance the feature space. Finally, we provide comprehensive analysis of COLT (Section 4.4.
versatility and complexity
4.1 DATASETS AND SETTINGS
We conduct experiments on four popular datasets. subsets sampled from the original
are long-tail
(Cui et al., 2019a). We set the imbalance ratio to 100 in default. Following (Wei al. 2022), we 300K Random Images (Hendrycks) ct al. 2018b) as the OOD dataset. ImageNet-100% 100-LT is proposed by (Jiang et al. 202 with 12K images sampled from ImageNet-100 (Tian al. 2020 with Pareto distribution. We use ImageNet- (Hendrycks et al. 2021) as the OOD dataset. Places-LT (Liu et al., 2019) contains about 62.5K images sampled from the large-scale scene-centric Places dataset (Zhou al. 2017) with Pareto distribution. Places-Extra69 (Zhou et al. 2017) is utilized as the OOD dataset.
Evaluation protocols To verify the balancedness and separability of the feature space, we report performance under two widely-used evaluation protocols in SSL: linear-probing and few-shot. For both protocols, we first perform self-supervised training on the encoder model to get the optimized visual representation. Then, we fine-tune a linear classifier on top of the fixed encoder. The only difference between finear-probing and few-shot learning is that we use the full dataset for linear probing and 1% samples of the full dataset for few-shot learning during fine-tuning.
Measurement metrics As common practice long tail learning, we divide each dataset into three disjoint groups in terms of the instance number of each class: (Many, Median, Few). By calculating the standard deviation of the accuracy of the three groups, we can quantitatively analyze the balancedness of a feature space (Jiang et al. 2021b). The linear separability of the feature space evaluated by the overall accuracy.
Training settings We evaluate our method with SimCLR (Chen et al. 2020) framework in default. We also conduct experiments on several state-of-ther methods in self-supervised long tail learn- ing (Jiang al. 2021b; Zhou et al.) 2022). We adopt Resnet-18 (He et al. 2016) for small datasets (CIFAR-IO-LT/CAE 10-LT/CIFAR IT0-LT), and Resnet-50 for large datasets (ImageNet M00-LT/Places-DD re- spectively. More details can be found in Appendix.

---

Published as a conference paper at ICLR 2023
Table Compare the proposed COLT with random sample and MAK under the same sampling pool and sampling budget. The best performance under each setting is marked as bold.
ID
Sampling pool
Budget
Method
Protocol
Many
Median t
Few 1 Std
All f
dataset
few-shot
51.76
41,45
38.12
5.81
45.04
random
linear-probing
72.22
66.49
63.45
3.64
68.33
few-shot
52.49
43.54
39.00
5.65
46.48
5K
MAK
linear-probing
73.24
67.62
64.14
3.75
69.36
few-shot 54.10
46.01
42.32
4.92
48.69
COLT
Image
Image Net-R
lincar-probing
74.52
69.67
67.53
2.92
71.28
Net-100
few-shot
52.82
42.88
40.27
5.41
46.42
random
linear-probing
74.33
68.52
62.65
4.77
70.02
few-shot
54.33
45.01
40.12
5.90
48.01
10K
MAK
linear-probing
75.57
68.20
66.29
4.07
70.83
few-shot 54.26
46.54
43.38
4.57
49.14
COLT
linear-probing
75.13
71.38
66.62
3.48
72.22
few-shot 30.61
33.94
37.55
2.83
33.45
random
linear-probing
40.21
47.59
50.51
4.33
45.51
Places 365
Places 69
few-shot 30.41
34.47
37.59
2.94
33.62
10K
MAK
linear-probing 40.83
47.78
50.72
4.15
45.86
few-shot 31.04
34.65
37.49
2.64
33.91
COLT
linear-probing
41.55
48.40
50.54
3.83
46.36
Table 4: Compare the test accuracy (%) on PingeeNet-100-L11 of the proposed COLT with MAK which use ID data. The best performance marked as bold.
Method Extra type
Sample set
Many t
Median t
Few
Std 4
All
ID
IN-900
75.7
70.4
66.9100.
3.0:0.4
7..0
MAK
ID & OOD
IPM
74.7=10.2
66.6.10.7 69.2:10.7
3.3
71.1
OOD
ImageNet-R
75.6:10.4
68.2:1..8
66.3+0.8
4.140.6
70.8=0.5
COLT
OOD
ImageNet-R
75.3++0.3
70.9++0.8
69.5+
2.4
72.4 0.3
4.2
COLT'S ACCURACY, BALANCEDNESS AND VERSATILITY
The main results of the proposed approach in various datasets and settings are presented in Table
I and Table 2 We sample K = 10, 000 OOD images on every r = 25 epoch for CIFAR-IO- for 1000-N COLT significantly outperforms
LT/CIFAR-I00-LE Places-LT, and the baseline (vanilla SimCLR) by large margin (about 10% for long-tail tail CIFAR, 5% for ImageNet- 100-LT, 1.6% Places-LT), Besides, the performance gain of the minority classes (Median & Few) is more notable (e.g., about 12% for long-tailed CIFAR-100) 100). Meanwhile, COLT yields balanced feature space. Following previous works (Jiang et al.) 202Tb) (Zhou et al. 2022). we measure the balancedness of feature space through the accuracy standard deviation from Many, Median and Few. COLT significantly narrows the performance gap between the three groups (much lower Std), which indicates we learn more balanced feature space.
To evaluate the versatility COLT, we carry out experiments on top several improved SSL frameworks for long-tail learning, i.e., SDCLR (Jiang et al., 2021b) and BCL (Zhou et al.) 2022). Table [ and Table 2 also summarized COLT performance on these two methods. We can observe that incorporating our method into existing stafe-of-the-an methods consistently improve their performance, which indicates that our method is robust to the underlying SSL frameworks.
4.3 COLT VS BASELINES WITH AUXILIARY DATA
We also compare COLT with methods that make use of external data. MAK (Jiang et al. 2021a) is the fate-of-the-are method that proposes sampling strategy re-balance the training set by sample

---

Published as a conference paper at ICLR 2023
Table 5: Comparison semi-supervised and self-supervised methods when leveraging OOD data.
SimCLR
Method
FixMatch
FlexMatch
ABC
DARP
SimCLR
Supervised
+COLT(IOK)
Accuracy
44.13
47.38
50.40
51.22
50.94
47.65
57.46
Daodd ea
Inogenee.130
muges
Haces
Acaraac
Imbelanie
Sample budge:
(a) External dataset
(b) Sample budget
lac
Accuroic
Imbo ance rado
interval
(c) Distribution loss
(d) Sampling interval
Epach
typperparramtterr
(c) Tail discovery.
(f) Ablation on hyper-paramcter kt.
Figure 3: Analytical experiments COLT on CIFAR- (Ba): accuracy when changing the
external OOD dataset. (3b): accuracy when sampling different numbers of OOD samples 300K Random Images. (3c): Top-1 accuracy and standard derivation (Std) of COLT with or without the
proposed distribution loss. (3d): accuracy with various sampling intervals (3ep: A higher drait
and lower Phead implies mining tail samples more precisely. 3f accuracy with various k.
in-distribution tail class instances from an open-world sampling pool. We compare the proposed COLT with MAK in Table 3 noting that "random" refers random sampling from the external dataset according to the budget. We observe both higher accuracy and balancedness under different sample budgets ImageNet-100 and Places. It indicates COLT leverages OOD data in a more efficient way. Furthermore, we ask the question that whether OOD samples can replace ID samples to help long-tail learning. We obtain a positive answer from empirical results Table 4 We compare the result of COLT and MAK on auxiliary data which involve ID samples. COLT achieves better performance on most of the metrics, even compared with sampling an entirely IDD dataset.
On the other, COLT has less computational overhead. MAK applies three-stage pipeline: pre-train the model with ID samples, sample from the sampling pool, and re-train a model from the begin- ning. In contrast, COLT samples during the training process, resulted in single-stage pipeline. The online sampling strategy not only fully utilized the external datasets but also reduced the computa- tion overhead significantly" learning. Different from ours, they use an large data budget (300K for CIFAR), while COLT im- proves the baselines with much smaller budget (10K for CIFAR).
ppen-igm (Wei et al. 2022) also OOD data to help long-tail
8

---

Published as a conference paper at ICLR 2023
4.4 ANALYSIS AND ABLATION STUDY
The choice of OOD dataset We conduct experiments on 1IFAR-100-LL and replace the OOD dataset while maintaining other settings unchanged. As shown in Fig 3a our method improves the ID accuracy when using 300K Random Images (Hendrycks et al. 2018b), STL (Coates et al.) 201T) ImageNet-100, Places with 9.81% %, 5.19%, 9.43%, 5.74% respectively. Besides, sampling on Gaussian Noise provides limited help (less than 1%) or degradation to ID accuracy.
loss COLT introduces a supervised contrastive loss to explicitly separate samples from different distributions. We conduct an ablation study on the proposed loss Fig and the results show that the proposed loss not only improves the overall accuracy but also significantly alleviates the imbalance (i.e., much lower std).
The effectiveness of distribution-awareen
The effect of sampling budget In Fig 3b we compare the performance gains of COLT under dif- ferent budgets. COLT consistently outperforms the random sampling strategy, i.e., leveraging OOD samples more effectively. Moreover, though a larger budget will give better performance, the per- formance gain almost plateaus with a budget of 10-15k in COLT, indicating better data efficiency.
Comparison with semi-supervised methods Performing semi-supervised learning also a natural choice for utilizing external unlabeled data. We implement FixMaich (Sohn et al. 2020), FlexMatch (Zhang et al. 2021a), ABC (Lee et al. 202T), and DARP (Kim et al. 2020) on long-tailed CIFAR- 100, the first two are general semi- supervisd methods, and the last two elaborately designed for long-tail learning. In Table [S] we compare the results in such semi- supervise learning scenar- ios (labeled: CIFAR-100-LT, unlabeled: 300K Random Images) to supervised, self-supervised, and COLT. It can be observed that 1), external unlabeled OOD data can also be helpful when perform- ing semi-supervised learning 2), the performance gains of COLT (about 10%) are more significant than incorporating OOD data via semi-supervised training. This could be attributed to most semi- supervised methods considering unlabeled data is also ID. may need some special design for unlabeled OOD data, e.g., resist some "toxic" samples or redesign the pseudo labels for OOD data.
Changing of pyper-parameters We also show the effect of syper-parameters involved in COLT. Fig 3d shows the empirical results of changing resample interval r. We can observe that reasonably small intervals lead to higher accuracy. Fig 31 shows the classification accuracy when changing k. The limited fluctuations in performance prove that COLT is robust to hyper-parameters.
The ability of tail sample mining Recall in Section 3.2 we localize tail samples by assigning a predefined "tailness score" to each sample. In order to verify the effectiveness of tailness score, we select the top 10% samples with the highest tailness score as a subset and calculate the ratio of the
Toss percentage of [Major/ / Minor] samples in this subset to the percentage of whole dataset:ó TrSsia where T denotes the target group. Sta is the whole in-distribution dataset, S is the subset of samples which have top 7% highest tailness score, 7 is set 10. reflects the ability to identify tail samples: when the target group is Minor/Major, higher/lower & indicates method localizes tail samples well. As illustrated in Fig 3e COLT discovers more samples from the tail than BCL.
5 CONCLUSION AND LIMITATIONS
In this paper, we propose novel SSL pipeline COLT, which is, our best, the first attempt to extend additional training samples from OOD datasets for improved SSL long-tailed learning. COLT includes three steps, unsupervised localizing head/tail samples, re-balancing the feature space by online sampling, and SSL with additional distribution-levy supervised contrastive loss. Extensive experiments show that our method significantly and consistently improves the performance of SSL on various long-tailed datasets. There are nevertheless some limitations. First, more theoretical analyses are needed to better understand the effectiveness of OOD samples. Besides, for a given long-tail ID dataset, how to specify the best OOD dataset that gives the largest improvements is also worth exploring. We hope our work can promote the exploration of OOD data in long-tal scenarios.
A Ithough we sample for multiple times (every epochs) in COLT, the time of performing sampling once is less than training for one epoch; therefore be ignored compared to 1.7x training epochs brought by MAK.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (Grant No. U21 62106222), the Zhejiang Provincial key RD Program of China (Grant No. 2021C01119), the Natural Science Foundation of Zhejiang Province, China(Grant No. LZ23F020008), the Core Technology Research Project of Foshan, Guangdong Province, China (Grant No. 0200001004990 and the Zhe- jiang Sniversity-Angllaiiee Inc. R&D Center for Intelligent Healthcare. Jianhong Bai would also like to thank Huan Wang from Northeastern University (Boston, USA), Denny Wu from University of Toronto, and Hangxiang Fang from Zhejiang University for their guidance and help.
REFERENCES
Jean-Bay Ban-itptitt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14100. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distrbuutoon-aaree margin loss. Advances in neural information processing systems, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 115-223..2 JMLR Workshop and Conference Proceedings, 2011.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class- balanced loss based on effective number samples. In Proceedings of the IÉEE/CVF conference on computer vision and patlern recognition, pp. 9268-9277, 2019a.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. blasn-bllanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268 08879,977 2019b.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
Gongfan Fang, Yifan Bao, Jie Song, Xinchao Wang, Donglin Xie, Chengchao Shen, and Mingli Song. Mosaicking to distill: Knowledge distillation from out-of-domain data. Advances in Neural Information Processing Systems, 34:11920- 11932, 2021.
Yonatan Geifman and Ran EI- Yania..v. Deep active learning over the long tail. arXiv preprint 17Xiv:1711.00941 2017.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738, 2020.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2018a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2018b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International 2021.
Conference Computer Vision, pp.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint axXiv:991.005244 2019.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang. and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 7610-7619, 2020.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data open-world sampling. Advances Neural Information Processing Systems, 34:5997-6009, 2021a.
Ziyu Jinng, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, pp. 4927-4939. PMLR, 2021b.
Lie Ju, Xin Wang, Lin Wang, Tongliang Liu, Xin Zhao, Tom Drummond, Dwarikanath Mahapa- tra, and Zongyuan Ge. Relational subsets knowledge distillation for long-tailed retinal diseases recognition. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 3-12. Springer, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Bingyi Kang, Yu Li, Sa Xie, 7ehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. International Conference on Learning Representations, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673 2020.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of lseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems, 33:14567-14579, 2020.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-EI-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhi- nay Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classifi- cation. Dataset available from bamps//gghhua..compppnmmggg com/openimages, 2017.
Ananya Kumar, Aditi Raghumathan, Robbie Matthew Jones, Tengyu Ma, and Perey Liang. Fine- tuning can distort pretrained features and underperform duttof-disttibutton. In International Con- ference on Learning Representations, 2021.
Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abe: Auxiliary balanced classifier for class- imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 34: 7082-7094, 2021.
Sachyung Lee, Changhwa Park, Hyungyu Lee, Jihun Yi, Jonghyun Lee, and Sungroh Yoon. Remov- ing undesirable feature contributions using -ustr--diitribution data. In International Conference Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2023
Bo Li, Yezhen Wang. Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for semi supervisee domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1104-1113, 2021.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8918-6928, 2022.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detec- tion in neural networks. In International Conference on Learning Representations, 2018.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-25515551 2546, 2019.
Ziwei Liu, Zhongqi Miao, Xingang Pan, Xiaohang Zhan, Dahua Lin, Stella X Yu, and Boqing Gong. Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406-12415 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Zhongqi Miao, Ziwei Liu, Kaitlyn M Gaynor, Meredith S Palmer, Stella Yu, and Wayne M Getz. Iterative human and automated identification of wildlife images. Nature Machine Intelligence, (10):885-895, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pp. 467-482. Springer, 2016.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying supi-supe learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Merrielle Spain and Pietro Perona. Measuring and predicting importance of objects in visual world. 2007.
Yanmin Sun, Mohamed S Kamel, Andrew KC Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern recognition, 40(12):3358-38) 3378, 2007.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
conference on computer vision, pp. 776-794. Springer, 2020.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE transactions on pattern analysis and machine intelligence, 30(1 1911)198-19991 2008
Hualiang Wang, Siming Fu, Xiaoxuan He, Hangxiang Fang, Zuozhu Liu, and Haoji Hu. Towards calibrated hyper-sphere representation via distribution overlap coefficient for long- tailed learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel. October 23-27, 2022, Proceedings, Part XXIV, pp. 179-196. Springer, 2022.
12

---

Published as a conference paper at ICLR 2023
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939 PMLR. 2020.
Yifei Wang. Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2021.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34:7978-7992, 2021.
Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling: Exploring out-of- distribution data for re-balancing long-tailed datasets. In International Conference Machine Learning, pp. 23615-23630. PMLR, 2022.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving iasasimmalanced learning. Advances in neural information processing systems, 33:19290-19301 2020.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling, Advances in Neural Information Processing Systems, 14:18408-18419 2021a.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021b.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4066; 2017.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pp. 27367-27377. PMLR, 2022.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for mantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
13

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
We uploaded the code to ensure the reproducibility of the proposed method, which can be found at:
APPENDIX A DATASETS AND TRAINING DETAILS
are first introduced by (Cui et al. 2019a), which are long-tail sub- sets sampled from the original CIFAR 1O//IFOROOO 100. The imbalance ratio is defined as the instance number of the largest class divided by the smallest class. To better reflect the performance dif- ference, we set the imbalance ratio to 100 in default. Following (Wei et al., 2022). we use 300K Random Images (Hendrycks al. 2018b) as the OOD dataset. In addition, we also conduct ex- periments with OOD datasets as S-L-10 10 CCoates et al., 2011). which contains 5, 5,000 labeled images and 100,000 unlabeled images in 10 classes with resolution of 96x96. We use all the unlabeled images as external OOD data.
CIFAR-10-LT/CIEAR-III
ImageNNet-10--1 is proposed by (Jiang et al. 202 contains about 12K images sampled from ImageNet- 100 (Tian et al., (2020) with Pareto distribution. The instance number of each class ranges from 1,280 to 5. We use ImageNet-R (Hendrycks et al. 202I)) as the OOD dataset. The dataset contains 30K images with several renditions (e.g., art, cartoons, deviantart) of ImageNet classes.
Places-LT The original Places (Zhou et al. 2017) is a large-scale scene-centric dataset. Places- LT (Liu et al.) 2019) contains about 62.5K images sampled from Places with Pareto distribution. The instance number of each class ranges from 4,980 to 5. Places-Extra69 (Zhou et al. 2017) is utilized as the OOD dataset. It includes 98,721 images for 69 scene categories besides the 365 scene categories in Places.
Training details We implement all our techniques using PyTorch (Paszke al. 2017) and conduct the experiments using RTX3090 GPUs. We evaluate our method with SimCLR (Chen et al.) (020) framework with batch size 512 for small datasets CCIFAR-IO-LTCCIFAA (IVCIFAR-IOO-LTT) and 256 for large datasets EmmgeNNe-100011/PPaeees.
in default. We adopt Resnet-18 (He et al.] 2016) for small datasets and Resnet-50 for large datasets, respectively. In our paper, we evaluate COLT's perfor- mance under two evaluation protocols in self-supervised learning: linear-probing and few-shot. For both protocols, we first perform self-supervised training on the encoder model to get the optimized visual representation. Then, we fine-tune linear classifier on top of the encoder (fixed during train- ing the classifier). The only difference between linear probing and few-shot learning we use the full dataset for linear probing and 1% samples of the full dataset for few-shot learning during fine- tuning. We keep all settings the fine-tuning stage (e.g., optimizer, learning rate, batch size) the same (Jiang et al.) 2021b).
Mainly Following (Jiang et al. 2021b Zhou et al. 2022 Jiang et al. 2021a), we pre-train all the baselines and COLT with 2000 epochs CIFAR 10/100, 1000 epochs ImageNet- 100,10000000000000 500 epochs on Places. As for the fine-tuning stage, the "linear probing"" and "few-shot" results are produced by fine-tuning the classifier for 30 epochs and 100 epochs, respectively. To make fair comparison, we implement COLT and all baselines with the same data augmentation strategies. We sample K 10, 000 OOD images on every 25 epoch for CIFAR--0-LLTCIITAR--OO0A Places-LT, and r 50 for ImageNet-100--2
APPENDIX B MORE EMPIRICAL RESULTS
We present the experiment results on ImageNet- 100 with ImageNet-R or Places69 as the external OOD dataset in Table 6 We compare COLT with methods that make use of external data. MAK Jiang et al., 2021a) is the state-of-the-art method that proposes a sampling strategy to re-balance the training set by sample in-distribution tail class instances. Note that "random" refers to randomly sampling from the external dataset according to the sampling budget. We observe both higher accu- racy and balancedness under different sample budgets different OOD datasets. It indicates COLT
2 A debiased subset of 80 Million Tiny Images (Torralba et al. 2008 with 300K images. 80 Million Tiny Images constructed by the corresponding images of 53,464 nouns from internet search engines.
14

---

Published as a conference paper at ICLR 2023
leverages OOD data in more efficient way. Besides, the performance gain of the minority classes (Median Few) is more notable. Meanwhile, COLT yields balancedness feature space. Following previous works (Jiang et al. 2021b) (Zhou et al. 2022). measure the balancedness of fea- ture space through the accuracy's 's standard deviation (Std) from Many, Median and Few. To further demonstrate that proposed COLT is also effective the non-curated open-world datasets, conduct experiments on ImageNet- 100-LT with 50K subset of Open Images (Krasin et al.) (0117) dataset of about million images belonging over 6000 categories.) the OOD dataset. We can also observe a significant improvement in both the accuracy (especially for (Median, Few) and the balancedness of the feature space.
Table 6: Accuracy (%) and balancedness (Std) on ImageNet- with different OOD datasets.
Sampling
Budget Method
Protocol
Many
Median
Few
Std 4
All 1
pool
few-shot
48.92
39.54
34.15
6.10
42.51
None
linear-probing
69.53
63.74
59.88
3.97
65.50
few-shot 52.97
44.03
39.98
5.43
46.99
random
linear-probing
73.49
67.57
63.05
4.28
69.29
few-shot 54.16
45.17
41,43
5.34
48.19
Places69
10K
MAK
linear-probing 74.69
68.60
64.63
4.14
70.46
few-shot 55.82
48.32
43.49
4.83
49.24
COLT
linear-probing
75.23
71.42
66.46
3.59
72.26
few-shot 51.76
41.45
38.12
5.81
45.04
random
linear-probing
72.22
66.49
63.45
3.64
68.33
few-shot 52.49
43.54
39.00
5.65
46.48
5K
MAK
linear-probing
73.24
67.62
64.14
3.75
69.36
few-shot 54.10
46.01
42.32
4.92
48.69
COLT
Image
linear-probing
74.52
69.67
67.53
2.92
71.28
Net-R
few-shot 52.82
42.88
40.27
5,41
46.42
random
linear-probing
74.33
68.52
62.65
4.77
70.02
few-shot 54.33
45.01
40.12
5.90
48.01
10K
MAK
linear-probing 75.57
68.20
66.29
4.07
70.83
few-shot 54.26
46.54
43.38
4.57
49.14
COLT
linear-probing 75.13
71.38
66.62
3.48
72.22
Table 7: Accuracy (%) and balancedness (Std) on ImageNet-100 with external 50K Open-Images Krasin et al. 2017) OOD dataset.
Method
Budget
Protocol
Many
Median t
Few f
Std 4
All +
few-shot
49.74
41.00
36.62
5.45
43.84
SimCLR
linear-probing
70.82
65.33
62.31
3.52
67.08
few-shot 54.46
47.12
43.23
4.66
49.48
5K
linear-prohing 75.62
70.87
67.29
3.41
72.26
SimCLR+COLT
few-shot 57.54
48.12
44.00
5,67
51.26
10K
linear-probing
76.87
71.00
69.23
3.27
73.06
APPENDIX C How DO OOD DATA CONTRIBUTE TO LONG-TAIL LEARNING?
OOD DATA RE-BALANCE THE FEATURE SPACE
As suggested in previous research, the standard contrastive learning would naturally put more weight on the loss of majority classes and less weight that of minority classes, resulting in imbalanced
15

---

Published as a conference paper at ICLR 2023
feature spaces and poor linear separability tail samples Kang al. 2020 Li et al. 2022). i.e., the majority classes dominate the feature space and tail instances turn out to be outliers.
To quantitatively analyze the imbalance of the feature space, define metric called Normalized Misclassification Matrix (NMM):
may/
Σ
mak
N
k=1
ITg
TR
where Tk is the k-th split of the long-tailed train set, satisfying Sid = UR-1 and T. T, = Vi * j. In this paper, we follow the common practice long-tail learning that split the dataset to Sia = T (Trew: TMediam: Manyy according to the instance number in each class, and ITyl denote the class number in split Tj. maj represents the number of (misclassified) instances belonging to split T; but are classified split Tý. Note that ma indicates both the ground truth label and the (wrong) prediction fall into split Te. Intuitively, if the feature space perfectly balanced (i.e., equal margin between different classes), each element in NMM is approximately 1.0, i.c., the misclassified samples nearly randomly fall into each split. On the contrary, higher (lower) mean margins between classes in T; and T, result in lower (higher) maj-
The results are shown Fig Fig 4a indicates when the train set is balanced, the mean margin of different classes is approximately equal (NMM;j 1), note that the split (Trew, TMedian TMany in Fig 4a is consistent with Fig 4b and Fig 4c Fig reflects that majority classes have a higher margin other classes than minority classes.
other words, the model is more likely to confuse samples from a minority class with other minority classes, implying an imbalanced feature space. Fig4ccexhibil the result of COLT based on SimCLR with 10K OOD data. It's observed that COLT alleviates the imbalance issue since COLT augments minority classes with more instances which could be interpreted as an implicit loss re-weighting strategy.
OOD DATA BRIDGE INSTANCES FROM MINORITY CLASSES
In this section, demonstrate the effect of OOD data the perspective of contrastive learning. A recent work (Wang et al. 2021) gives a thcoretical understanding of contrastive learning based on augmentation overlap. Concretely, they suggest that the samples of the same class could be very alike after aggressive data augmentations. Thus, the pretext task aligning positive samples can facilitate the model to learn class-separable representations. They define the augmentation graph G = (V, 8) as: N samples are the vertices of the graph, and there exists an edge eig when sample i and sample j has overlapped views. According their theory, intra-class augmentation overlap a sufficient condition for gathering features from the same class. In this case, we compute the ratio of connected nodes (degree class k, denoted as score s Ideally, each class should have s = 1, which indicates all samples from the same class will be clustered together during the contrastive training process. A smaller k indicates lower intra-class consistency and vice versa.
zero) to measure the extent of intra-class augmentation overlap in
0:91
1:41
1.05
1:21
1.31
1.05
0.6
33
13
13
1.00
1.08
0.20
1.27
1:33
0.70
1:7
1.77
0.5
C3
88
1.01
0.30
1.26
1:1
1.20
(a) SimCLR(BL)
(b) SimCLR(LT)
(c) SimCLR+COLT(LT)
Figure 4: Normalized Misclassification Matrix (NMM) on the test set of CIFAR- with different
frameworks and train sets. 44)): SimCLR trained on balanced CIFAR-100 (4b): SimCLR trained on
loigetailed CIFAR-100, (4c): implement COLT on top SimCLR trained on long-tailed CIFAR-
100.
16

---

Published as a conference paper at ICLR 2023
(a) SimCLR (majority classes' mean connectivity: SimCLR+COLT (majority classes' mean 0.73 minority classes' mean connectivity: 0.46)
tivity: 0.76 minority classes' mean connectivity: 0.67)
The augmentation graph of CIFAR-10. Similar to (Wang et al. 2021 We choose random subset test images and randomly augment them 20 times. Then, we calculate the instance distance in the representation space and draw edges for image pairs whose smallest view distance is below a small threshold. We visualize the samples with t-SNE and denote edges between ID instances in black and edges between ID and OOD samples, forming new connections in red.
Figure 5:
R² 0.928
R² 0.958
Cbsstr Talleess Score
0.22
Cluster tailness
0.29
0.10
0.05
3.50
0.0
54
3.8
5.0
0.2
64
Minarity Propartion
Minonty Proportion
(a) CIFAR-10-LT
(b) CIFAR-100-LT
3.30
R² 0.915
0.926
Clister Tal 186655 Scone
0.20-
2.20
Tailness
0.10
0.00
5005-
3.004
0.00
Minority Proportion
Minority Proportion
ImageNet- 1maggeNet1000-LT 100-LT
(d) Places-365-I.T
Figure 6: Linear regression results between the minority proportion in cluster and the cluster's tailness score on long-taile CIFAR, ImageNet-100 100, and Places. We set cluster number 10.
We visualize the augmentation graph of CIFAR- 10 in Fig 5 and compute the connectivity score for each class. We observe smaller 5. for minority classes than for majority classes, which is also con- sistent with the theoretical analysis in (Wang al. 2021 Nevertheless, since we dynamically add OOD samples around ID (especially minority classes) samples, more instances from the ID minority class be reachable on the augmentation graph via OOD samples (majority/minority connectiv- ity improvement: 0.03/0.31). The red edges in Fig (5b) illustrate that OOD samples help long-tail learning by bridging samples from minority classes, further improving intra-class consistency.
17

---

Published as a conference paper at ICLR 2023
APPENDIX D MORE ANALYSIS ABOUT COLT
Different tail estimation strategies In our paper, we defined a tailness score based on top-k% (A = 2 in practice) largest negative logits to localize the head and tail samples in an unsupervised manner. We also make a comparison to other alternative strategies. In Tab. g we provide results on locating tail samples by a radius-based definition (with radius as the sum of negative logits and select different radius thresholds) of tailness score or simply using the method in BCL (Zhou et al.) 2022). The sampling budget is set to 5K, and the ID and OOD dataset is CIFAR-100- Random Images respectively. We can notice that there is no significant difference between COLT with Top-k % and radius-based methods, while both of them surpass BCL.
and 300K
Is the unsupervised clustering reliable? In the paper, we propose to perform a K-means clustering to the samples from Std, then calculate the cluster-wise sampling budget via the tailness score. Since our ultimate goal is to sample more (less) OOD data similar to the minority (majority) samples, it's natural to ask how well does this clustering work, whether we assigned more budger to the tail samples. To this end, we propose to measure the clustering and sampling quality by the ratio of minority samples in a cluster and its corresponding cluster-wise tailness score. Results are visualized in Fig 6 It's observed that the minority proportion of some clusters is close to 1, while others are composed of samples from majority classes. Besides, the cluster- wise tailness score shows a linear correlation of the minority proportion, implying we do allocate more sampling budget to tail classes according to Eq. 3
Impact of the cluster number Recall in Sec 3.3 we first perform K-means clustering to the ID samples, then select OOD samples close (with a large cosine similarity in the feature space) to the head or tail classes based on the budget allocation function. To verify 1), whether the cluster number will dramatically affect the performance 2), how the performance is when clustering based on the ground-truth label information rather than self-supervised manner. We conduct experiments with various cluster number C' in Tab. 9 the OOD dataset is 300K Random Images (Hendrycks et al. 2018b), and the sampling budget set to 5K. Note that the supervised clustering method is referred to as Oracle. The results show the performance of COLT will not be significantly affected by the number of clusters. Furthermore, cluster according to the label information (Oracle) achieves a similar performance compared to the unsupervised clustering, implying the K-means clustering can be used as a feasible alternative to roughly separates the minority from the majority classes in self- supervised scenarios.
The scale / balancedness of the OOD dataset In our work, we evaluate the proposed COLT on both balanced datasets, which used as OOD datasets in previous works (Kumar al. 202 I Wei ct al. 2022) and non-curated woendworld datasets such as 300K Random Images (Hendrycks et al. 2018b). Although COLT performs well the aforementioned datasets, we intend to further ask will COLT perform well when the OOD dataset is also long-tailed? Tab. [[]shows the performance of COLT on CIFAR- 100 when the OOD dataset (ImageNet-100) has various imbalance ratios. The similar accuracy suggests COLT is robust to the imbalancedness of the OOD dataset to some extent. This could be attributed to the dynamic sampling procedure filtering out most of the unhelpful OOD samples so that the performance will not be dramatically affected by changes in the external OOD dataset.
Another observation can be found from Tab. I where we measured how the OOD dataset's scale influences COLT, We form the external OOD dataset of different scales by gradually increasing the number of samples in Random 300K Images (Hendrycks et al. (0088)) and implementing COLT on those subsets of Random 300K Images. Conform to intuition, the scale of the OOD dataset is positively correlated with the performance of COLT since we may select more desired samples in a larger candidate set. However, the problem can be tricky when the dataset is also changed. For example, we observe a similar performance gain on 1000-Net-100-LT when utilizing Places-69 (about 98K images) and ImageNet-R (30K images) as the auxiliary OOD dataset, which implics the scale of the OOD dataset is not the only factor affecting the performance.
18

---

Published as a conference paper at ICLR 2023
Table 8: Comparison on different tail estimation strategies.
Mothod
COLT
Radius-based
BCL
Threshold
0,08
0.09
0.10
0.11
0.12
Accuracy
54.20 0.35
53.19 0.47
53.33
0.28
34.00
0.21
53.87
0.22
53.0
0.34
52.97 0.30
Table Ablation study on the cluster number C' on CIFAR-IO0-LT.
Cluster Number C
20
100
100 (Oracle)
Accuracy
54.20 + 0.35
53.61 + 0.18 54.06 + 0.22 53.81 + 0.29
56.16 0.10
Table 10: COLT's accuracy when OOD dataset is also imbalanced.
Imbalance ratio
10
50
100
Accuracy
52.98 + 0.33
52.63 + 0.19
52.74 + 0.25
52.66 = 0.41
Table 11: COLT's accuracy with different scale OOD dataset.
OOD Images Number (K)
100
200
300
Accuracy
53.43 + 0.26
53.99 + 0.18
54.20 + 0.35
19


Reviews:
[{"review_id": "pbNpBD6_bE", "reviewer": "Reviewer_9FQm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method where out-of-domain (OOD) data are used to learning better representations via self-supervised learning (SSL) in a setting where we have unlabeled in-domain (ID) data that follow a long-tail distribution. The paper proposed to define an unsupervised \"tailness\" score, while an online sampling strategy is proposed for sampling the OOD data mostly closer to the tail classes. Experiments are conducted on 4 long-tail datasets", "summary_of_the_review": "The paper deals with an interesting task and offers a solution that to my knowledge is novel, works better than recent works on the datasets it is tested on. Adding more realistic evaluations on bigger and real long-tail datasets, as well as expanding on aspects like the use of k-means clustering or different, more robust criteria for the tailness loss would make me more positive towards acceptance. Looking forward to the authors' responses.", "strength_and_weaknesses": "Strengths:\n\n- Self-supervised Learning (SSL) on long-tail data is a very important and interesting problem; training on balanced datasets like imagenet is highly artificial in that regard.\n\n- Improving SSL on long-tailed datasets effectively with external unlabeled OOD data is a realistic scenario that was shown to work for supervised learning by Wei et al (2022).\n\n- The method seems to give strong gains for SimCLR for long-tail SSL.\n\nWeaknesses:\n\n- The authors say that they \"obtain C feature prototypes from ID training set S_id via K-means clustering.\". How does one expect to capture a long-tail distribution with kmeans? How well does this clustering work, also when looking at clustering accuracy for the head and tail classes? Also, how do you set K? Is there a further assumption that we know how many classes exist? \n\n- Defining the \"tailness\" score as simply the sum of the \" top-k%\"  largest negative logits seems like an unstable heuristic. Does k transfer across datasets? Did you consider other alternatives, eg radius-based? \n\n- Although tackling a realistic problem in theory, the datasets used are mostly toy-level. Results on a real long-tail dataset like iNaturalist would be more convincing.\n\nNotes:\n- It is hard to undestand which line is which in Fig 3e, please fix the typo of two methods having the same line color and style", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m1K-gcIcK9", "reviewer": "Reviewer_EuL2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors aim to employ OOD data to improve self-supervised learning in the long-tailed setting.  To achieve that, they use tailness score estimation, dynamic sampling strategies, and additional contrastive losses for long-tailed learning with additional OOD samples.  The authors conduct experiments on various datasets and several state-of-the-art SSL frameworks to evaluate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed method is novel and has shown significant improvement from the reported results. However, this work does not provide any insight or understanding about why it can work. Since this work can be treated as an SSL version of the recent work [1], the contribution of this work is not sufficient for this conference. So, I recommend a weak reject for this work.\n\n-----\n\nThe response from the authors has addressed my concerns so I decide to improve my score to 6.", "strength_and_weaknesses": "Strength:\n- The idea in this work is novel. Before this work, It has not been shown how to exploit the benefits of OOD data for SSL long-tailed setting. \n- The improvements shown in the reported results are significant. From the reported results, the proposed method can effectively utilize the benefits of OOD examples in the SSL long-tailed settings.\n\n\nWeakness:\n- The contribution of this work is limited. This work can be seen as an SSL version of the recent work [1], which has shown that OOD can be beneficial to long-tailed learning. Although this work provides some technical contributions by adapting OOD data for SSL methods, this work cannot provide new insight for this problem.\n- This work does not provide any in-depth understanding about why OOD examples can improve SSL in long-tailed setting. In this paper, the authors simply propose a method with three components, but do not explain why we could use OOD examples there.\n\n\n[1] Wei, H., Tao, L., Xie, R., Feng, L., & An, B. (2022, June). Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets. In International Conference on Machine Learning (pp. 23615-23630). PMLR.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lpaTUFNCWE", "reviewer": "Reviewer_QM2L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies self-supervised learning (SSL) for long-tailed datasets. In particular, it proposes to leverage out-of-distribution (OOD) data to improve model performance when facing imbalanced data, with a new framework proposed called Contrastive with Out-of-distribution (OOD) data for Long-Tail learning (COLT). It first proposes to localize tail samples by assigning a tailness score to each OOD sample, and then uses an online OOD sampling strategy to dynamically re-balance the feature space. Finally, a distribution-level supervised contrastive loss is proposed. The paper performs experiments on several long-tailed datasets and verifies that COLT improves the performance over other baselines.", "summary_of_the_review": "The paper studies an interesting yet important problem, utilizing OOD samples for long-tailed learning. The paper proposed several interesting components to form a self-supervised learning framework called COLT, which is later demonstrated to deliver better results.\n\nThe overall idea of leveraging OOD data seems to be novel and not explored before in this field, which is plausible.\n\nHowever, currently there are several drawbacks and weaknesses, in terms of Methods and Exepriments. The writing quality also needs to be significantly improved.\n\nIn summary, in my view, the current elaboration and presentation do not meet the standards of ICLR. I am open to increase my evaluation if my principal concerns in the Weaknesses section are addressed in a convincing manner.", "strength_and_weaknesses": "# Strength \n+ The investigation of OOD data for improving long-tailed recognition is under explored. The problem itself is important and practical, and the technique introduced is well motivated and clearly stated.\n+ The experiments are thorough and cover large-scale benchmark datasets in the field. The performance is good -- compared to other CL baselines, the improvements on certain datasets are even larger than 10%.\n+ The ablation studies are thorough. They confirm that each of the components should be useful for long-tailed data.\n+ The analysis and visualization of the feature balanceness / ability for tail discovery are effective and clear.\n\n---\n# Weaknesses\n\nUnfortunately, there are several major weaknesses that exist in the current paper.\n\n### Methods & Experiments\n\n- One drawback is that the paper fails to compare with semi-supervised learning methods for long-tailed recognition.\n1. The main contribution of the paper is to consider OOD dataset in long-tailed learning, and design SSL algorithm to incorporate the OOD samples into the training process. However, one natural choice when given more unlabeled data is to use *semi-supervised* learning. For example, how does COLT compare to SOTA semi-supervised learning methods [1-3] when leveraging OOD data?\n\n[1] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.\n\n[2] FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling.\n\n[3] SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning\n\n2. Moreover, in the long-tailed literature, methods have already been proposed to use unlabeled data for combating data imbalance [4-6]. Although they might use ID samples, it is still encourgaed to compare with these standard methods in the field to demonstrate the performance is convincing.\n\n[4] CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning. CVPR 2021.\n\n[5] ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning\n\n[6] Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning\n\n- Another part I find confusing is the choice of OOD dataset.\n1. How does OOD defined in the first place? Does it refer to covariate shift from the original data? Is the OOD strength controlled across different datasets? Overall, the authors did not give a principled way to define / contruct the OOD dataset.\n2. Is the OOD dataset used also imbalanced / long-tailed? Will the imbalance of OOD dataset influence the results of COLT?  If so, how does it influence the performance? Unfortunately the analysis is missing here.\n3. What is the amount (i.e., # of samples) of the OOD dataset? Will the number of OOD samples influence the performance?\n\n- It is unclear how many epochs needed for training with COLT. In fact, no descriptions are provided for the actual training details. Does COLT need more training epochs, similar to contrastive learning methods like SimCLR / MoCo? What about training details for other baselines? \n\n- Similar to above, contrastive learning-based methods usually use strong data augmentation tricks during training. However, many current methods do not use them, making the comparison unfair. Does COLT involve strong augmentations during training? This again goes to the comment that no detailed training setups are provided. If so, the authors should comprehensively compare COLT with current non-contrastive learning-based methods (such as RIDE) by adding such tricks to them. \n\n- The proposed method can obtain enhancement on all many / medium / few-shot classes (e.g., on ImageNet-100-LT), which is perhaps surprising and a great benefit, but also leads to many questions. Many current methods sacrifice the head performance. Can the author explain where the benefit comes from, and why the current methods cannot achieve this goal?\n\n### Writing\n- The overall writing quality is not good. The writing needs to be significantly improved.\n- The references in the text are somehow mixed with the main text without clear separation, making it really hard to read, especially for the related work. Please fix accordingly.\n- There are many typos in the text, I suggest a careful rereading. Here are some examples:\n  - page 7, \"about 12% for longtail CIFAR-100\" -> \"about 12% for long-tailed CIFAR-100\"\n  - page 7, \"COLT yields balancedness feature space\" -> \"COLT yields a balanced feature space\"\n  - page 8, \"but also significantly alleviates the imbalancedness\" -> \"but also significantly alleviates the imbalance\"\n\n---\n# Other questions\n- The authors showed that using a distribution-level supervised contrastive loss to separate ID / OOD samples can lead to better performance. I do not fully understand the intuition here. The OOD samples used in the experiments are still largely overlapped with the ID samples (e.g., Places-LT uses still \n\"places\" samples as OOD dataset; the contents do not change). That being said, the high-level semantic information are still there. Directly separating them may not lead to better feature space as many of the features can be shared across ID / OOD samples. Can you explain the rationale behind using this SCL loss?\n- The distribution-level supervised contrastive loss is essentially try to discriminate between two sets of samples (ID or OOD). I wonder why not directly using a discriminator loss? What would be the differences?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U2p7LLoGeB7", "reviewer": "Reviewer_SHmZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies self-supervised long-tailed learning with the auxiliary of out-of-distribution data by designing a sampling strategy together with a distribution-level supervised contrastive loss. Empirical validation demonstrates the effectiveness of the proposed method on a range of datasets compared with previous methods. ", "summary_of_the_review": "It is an interesting and promising work, and the technical novelty and some claims should be improved. I can consider to raise the score if above concerns can be well addressed.", "strength_and_weaknesses": "Strength\n\n(1) The experiments on a range of datasets demonstrate that it is promising to improve the performance of self-supervised long-tailed learning via external data. \n\n(2) The proposed method is orthogonal to a range of existing methods and can be consistently improve the current self-supervised long-tailed learning methods.\n\nThere are some concerns that need to be improved.\n\n(1) The first contribution in the Introduction is not very meaningful, since MAK has given such a setting and a rough answer to this problem. It is advised to refine this point, better claiming the challenges in this direction.\n\n(2) The proposed method combines dynamic sampling guided by tailed area estimation and an additional supervised contrastive learning, which slightly makes the novelty incremental. Although it shows the additional supervised contrastive learning helpful in Figure 3(c), it also confuses the key challenge of this topic. \n\n(3) The external datasets are also relatively controversial, since it seems that the authors also choose the datasets tightly in the same domain, e.g., ImageNet-100 with ImageNet-R, instead of the non-curated open-world datasets. Here, some details are not clear. What is 300K random Images in Hendrycks 2018b. for CIFAR-LT. \n\n(4) It is not proper to roughly compare the computational overhead between COLT and MAK, since the periodic sampling on the basis of the clustering is also expensive. It might be more reasonable to compare the time cost relative to the plain backbone without the sampling augmentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster