Improving Contrastive Learning on Imbalanced Data via Open-World Sampling
Authors: Ziyu Jiang
Tianlong Chen
Ting Chen
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
sampling
balancedness

Abstract: Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes. That implies a tantalizing possibility of scaling them up beyond a curated “seed" benchmark, to incorporating more unlabeled images from the internet-scale external sources to enhance its performance. However, in practice, larger amount of unlabeled data will require more computing resources due to the bigger model size and longer training needed. Moreover, open-world unlabeled data usually follows an implicit long-tail class or attribute distribution, many of which also do not belong to the target classes. Blindly leveraging all unlabeled data hence can lead to the data imbalance as well as distraction issues. This motivates us to seek a principled approach to strategically select unlabeled data from an external source, in order to learn generalizable, balanced and diverse representations for relevant classes. In this work, we present an open-world unlabeled data sampling framework called Model-Aware K-center (MAK), which follows three simple principles: (1) tailness, which encourages sampling of examples from tail classes, by sorting the empirical contrastive loss expectation (ECLE) of samples over random data augmentations; (2) proximity, which rejects the out-of-distribution outliers that may distract training; and (3) diversity, which ensures diversity in the set of sampled examples. Empirically, using ImageNet-100-LT (without labels) as the seed dataset and two “noisy” external data sources, we demonstrate that MAK can consistently improve both the overall representation quality and the class balancedness of the learned features, as evaluated via linear classifier evaluation on full-shot and few-shot settings. The
code is available at: https://github.com/VITA-Group/MAK.

Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling
Ziyu Jiang', Tianlong Chen', Ting Chen', Zhangyang Wang²
'eexas A&M University. PUniversity of Texas at Austin, 'Google Research, Brain Team jianzziyustamu. edu, {tianlong. chen, Suttsaaaaa iamtingchen&google.o com
Abstract
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes. That implies tantalizing possibility of scaling them up beyond curated "seed" benchmark, to incorporating more unlabeled images from the internet-scale external sources enhance its performance. However, in practice, larger amount of unlabeled data will require more computing resources due to the bigger model size and longer training needed. Moreover, open-world unlabeled data usually follows an implicit long-tail class or attribute distribution, many of which also do not belong to the target classes. Blindly leveraging all unlabeled data hence can lead to the data imbalance as well as distraction issues. This motivates to seek a principled approach strategi- cally select unlabeled data from an external source, in order to learn generalizable, balanced and diverse representations for relevant classes. In this work, we present open-world unlabeled data sampling framework called Model-Aware K -ccenter (MAK). which follows three simple principles: (1) tailness, which encourages sampling of examples from tail classes, by sorting the empirical contrastive loss expectation (ECLE) of samples over random data augmentations; (2) proximity, which rejects the -iitr---iitriittion outliers that may distract training: and (3) diversity, which ensures diversity in the set of sampled examples. Empirically, using ImageNet- 100-LT (without labels) the seed dataset and two "noisy" ex- ternal data sources, we demonstrate that MAK can consistently improve both the overall representation quality and the class balancedness of the learned features, evaluated via linear classifier evaluation on full-shot and few-shot settings. The code is available https: htapa://githann /ggbtttt...... www../........e...e.. GrTTTAGGrr
Introduction
Contrastive learning has been successfully applied to learning strong visual representations in an unsupervised manner [155]. The promise of label-free learning makes it appealing to scale up contrastive learning with massive unannotated data, e.g., from internet-scale sources [6]. However, training with a larger amount of unlabeled data is not cheap. As shown in [ 2] contrastive learning needs bigger model and longer training compared to supervised counterparts. With larger amount of data, also requires more compute resources (for bigger model and longer training). With a limited computing budget, it is likely that the out-of-distribution data in the wild would suppress the learning of relevant features.
Moreover, different from standard benchmarks that are carefully curated and well balanced across classes (c.g., ImageNet), the data distribution in the open world are extremely diverse and always exhibits long tails 7 [). A few very recent works [9I1 have studied whether contrastive learning can still generalize well in those loig-tail scenarios. And they found that while contrastive learning learns more balanced feature space than its supervised counterpart, it still exhibits certain vulnerability
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

to the Long-tailed data ... Therefore, blindly gathering unlabeled data in the wild may exacerbate the imbalancedness issue of the training data.
Here we consider the problem of sampling ppen-world unlabeled data for improving the representation learning, not just for the head classes but also for the tailed classes. To make it clear, we describe the problem setting in more detail.
Assume that we start from relatively small ("seed") set of unlabeled training data, where the data distribution could be highly skewed yet unspecified. We aim to retrieve an extra set, with a given sampling budget, of freely available images from some external sources (e.g., the Web), to enhance self self-supervised representation learning for targeted distribution (of seed set). is worth noting that in setup, do not use labels for data sampling or training (e.g. [22), but only use labels as a way to evaluate the representation. By training on the retrieved unlabeled examples, our goal is to learn "stronger and fairer" visual representations that improve not only the overall quality but also the balancedness across various class attributes.
Problem Setting:
It is worth pointing out some unique challenges in our problem setup. 1) We may start from skewed set, and the actual class imbalancedness is unknown to us due to label unavailability, making the most approaches handling imbalance in the (semi-)superrisc) setting like pseudo-labeling, re-sampling, or loss re- seeiiiitingg [13 inapplicable. 2) Adopting a pre-trained backbone trained on imbalanced seed data with tail classes under-learned may amplify unfairness, as it is harder to retrieve images for poorly learned tail classes. 3) Widely existing irrelevant outlier samples in the open world are harder to detect given the lack of label information.
In this work, we conduct the first systematic study on leveraging additional unlabeled data from external sources when performing contrastive learning on a long-tail seed training set. We observe that, while randomly sampling more unlabeled data can effectively help overall accuracy, shows only limited and unstable benefits to the balancedness. Thus we aim to seek principled open-world unlabeled data sampling strategy, and follows three principles outlined below.
The first principle we follow is tailness. Intuitively, we would like to sample more examples that correspond to tail classes. Since neither labels nor class predictions are are unavailable, we seek another proxy for tailness, using each sample's training loss to identify "hard samples". However, compared to similar ideas adopted in supervised learning 19, our proxy signal is weaker (only contrastive loss, and not directly compared to the class label) and much noisier due to the strong random augmentations in contrastive learning. In view of that, we propose to instead use an empirical expectation of sample loss over multiple random augmentations the proxy. Our experiments verify that this new proxy of empirical contrastive loss expectation (ECLE) can reliably and stably spotlight more samples from tail classes.
We further complement the tailness principle with two other principles: proximity and diversity. Without proximity to examples in the seed set, the large-loss samples could simply be outliers, and training with those may hamper the model generalization on the target distribution. Thus we incorporate feature distance regularizer between new external samples and seed training samples to reject too "far-away" samples from the former. Without diversity in the retrieved images, the benefits of the additional data can be significantly lower. We hence include another divverity-promotiie term in sample selection. Eventually, our ideas could be mathematically unified into one framework called Model-Aware K-center (MAK) (as shown in Figure I] which could be viewed as a principled extension the core-set active learning 200, from supervised to unsupervised representation learning.
Our contributions are summarized below.
New problem and insight: leveraging external unlabeled data could help contrastive learning on imbalanced data, but needs strategic sampling to avoid backfiring on balancedness.
New tail sample mining for contrastive learning: an empirical contrastive loss expectation (ECLE) is proposed to adaptively mine under-learned samples (usually on the tail classes), while alleviating the substantial impact of random data augmentations.
New principled sampling framework: our unified formulation of Model-Aware K-center (MAK) seamlessly integrates ECLE, Out-of cuttr--Distribution (OoD) rejection and diversity promotion. It also extends the ideas of core-set active learning 20 to unsupervised representation learning.
2

---

Re-balancing via sampling
Sampling pool
Default training dataset
Head
Head class
Taliness: ignore head class samples
class
Tail class
Diversity: drop redundent data
OoD data
Long tail distribution
Proximity: Reject OoD outliers
Figure 1:
The overview of the proposed MAK sampling framework. MAK re-balances the long tail distribution via sampling additional data from a sampling pool. MAK are composed by three components: tailness, proximity and diversity.
Compelling empirical results: When training on ImageNet-100-L1 the MAK can consistently yield accuracy and balancedness improvement over the competitive random sampling method across different sampling datasets and budgets.
2
Our Method
2.1
Overview
contrastive learning, the representation is learned enforcing an anchor sample to be similar to another positive sample while being different from negative samples. Among various popular contrastive learning frameworks, SimCLR [I] is easy-to- toi-tlmmemee while yielding near state-of-the-an performance. utilizes two augmented views of the same data the positive pairs while all other augmented samples in the same batch are treated as negative samples. Assuming random data augmentation function A(- 8) where 0 is the hyperparameter to be sampled from some empirical distribution θ each time. With 0,, 0,,2 ~ θ, vi. The SimCLR loss associated with the i-th sample in the batch could be represented as:
Preliminary:
S" (A(u;, 02,1), A(u;, 02,2)) LCL.1 log S" (A(u;, 111), A(u,, 0,,2)) + Σnjev (A(1;, 0,,1)
(1)
where v; denotes the negative samples sampled from distribution V-. ST represents the similarity function with a temperature term of r defined as 8" (a, b) exp (a b/T). The overall term measures the entropy for sampling A(e,, 0,,2) among all A(e, 2,2)) and v,-
Hereinafter, we focus on SimCLR the backbone to develop our framework. However, our idea rather plug-and-play and can be applied with other contrastive learning frameworks adopting the two-branch design with data augmentation views 3 4 which we will explore as our future work.
Principles: As demonstrated in Figure the principles behind MAK ensure tailness, proximity and diversity. While tailness is for sampling more tail class data, proximity ensures the sampled data in-distribution. Besides, Diversity further enforce the diversity of the sampled data, preventing sampling redundant similar samples.
2.2 Spotting Hard Samples from Tail Classes:A New Proxy for Tailness
In supervised learning, one can spot "hard examples" which yield the largest loss values and assign
higher weights for accelerated training [19, and for handling data imbalance [21]. However, such benefit does not extend to contrastive learning straightforwardly, Importantly, the contrastive loss (T)
largely depends on the random augmentations A(, and thus display high randomness. observed
in experiments, the loss variations caused by selecting two similar/dissimilar augmentation views

---

could outweigh the loss difference caused by sample memorization or learning difficulty. Hence naively picking samples with the largest contrastive loss will only produce highly noisy selections.
To eliminate the randomness caused by augmentations, we turn to the following new proxy value for the i-th sample, that is designed to "smooth out" random augmentations by integrating over them:
c ct. = 2:.1100.-~ (Oc,,i(O,,,, 02227, Us, V)
In practice, the expectation is approximated by the sample mean, e.g., drawing 10,,,, 02,23 for M times and then averaging corresponding LCL,i values, We denote the new loss ( as the empirical contrastive loss expectation (ECLE) for the i-th sample. Note a similar "randomized smoothing" idea was utilized before to learn robust classifiers [22] [2], though in completely different context from ours. We then sort and choose those with the largest ECLE values 2 as hard samples. Section 3.4 experimentally verifies that ECLE eliminates the randomness within reasonable computational budgets, and larger ECLE values display correlation with tailness.
2.3 Proximity and Diversity
The new ECLE proxy lays the foundation for sampling hard examples. While they are found correlate with Tailness, the alignment remains be weak and noisy in the open-world setting, since the external sources can have shifted or non -vverlalddpped class distribution with the target ones, e.g., containing "distractor" classes. Those outliers behave like zero-shot minorities and are likely to incur large losses, since they are from completely unseen and unwanted classes. Adopting only the ECLE proxy might easily pick those outliers, hurting feature learning and generalization on the underlying distribution.
Proximity:
We hence construct a regularization term that promotes proximity via rejecting OoD outliers. Let s be the new additional set that we have sampled from the external sources, and s be the seed training set. We let A (t)) denote the feature distance between two samples. We define D(s , s²) as the average feature distance between each sample in s and its nearest sample in s", formalized as:
D(s 8)) = 8I
Σ JEst
min A(z,, xz)
(3)
In practice, to compute A(r;, R, we the normalized cosine distance, by passing Iy through the currently trained backbone and using their extracted features before feeding projection heads. For further efficiency, instead of calculating ( over the whole 5 we pre-compute the set of feature prototypes from s" using K K-means clustering, denoted as sy. and then compute (ss st).
Oversampling too many external images both would add to the training overhead, and might not necessarily help since we might get redundant samples. Assume that we have a sampling budget: s' |s'| < K, the challenge is how to attain the informative samples within the size limit. We introduce the following regularization term:
Diversity:
(s' U Sau) max min 2z)
iES jEs
Here Saul denotes all samples from the seed training set and the external sources (s s E Sat). Conceptually, minimizing the term (4) boils down to choosing |s"| center points on top of the given |s8) points, such that the largest distance between any data point from Sall and its nearest center point from s U 8 is minimized. It equals to finding a diverse subset cover for the dataset Sau using minimax facility location formulation 224, also known as the name of K K-center (00
2.4 Model-Aware K-Center: A Unified Framework
Taking together, we end up with solving the following constrained optimization:
max s1: SK
D (s* U.s., ,,
(5)
The goal of (5) is to find sample set 5 from the external source, such that it could simultaneously: (i) mine more data for tail classes while overcoming augmentation randomness, by sorting external samples by their ECLE values (tailness); (ii) reject the out-of-distribution outliers that might distract

---

Algorithm 1: A greedy heuristic to efficiently solve MAK.
Require:seed training set s external data Sau Sau)s", sampling budget K, candidate set size C, feature distance function A (cosine distance in practice), coefficient (0, 1).
Train feature extractor f on s with self-supervised method;
Calculate ECLE L and average feature distance D (s", with f as in Equation 2 and 3
respectively;
a))NND(D (s9, (4)) where Construct set S": from Sau',5' 555, find samples whose score are top C' largest among all;
Summarize C(,, and D (6")))) with score q = aa=
[(v) -=meante) is the normalization function: std(e)
// tailness proximity
Construct set Initialize s
while |s\ < K do // Apply K-center greedy algorithm for diversity u arg m minjes (
s=sUp {u}
end
return 8 = si
training. by constraining feature distances from the seed set (proximity): and (iii) control the sample volume under K while ensuring sample diversity, by K-center sample selection (diversity).
We name our optimization 5 Model-Aware K-center (MAK), since it inherits the vanilla "data- level" K-center objective while injecting "model-level" sample selection criteria, including largest (smoothed) model losses as well as closer feature distances. Those mnodel-aware" terms are necessary in our unsupervised re-balancing goal and our open-world setting, for automatically re-balancing the additional samples in favor the seed set's minority classes, and for rejecting outliers from open-world external data, respectively.
Exactly solving (5) would be NP-hard (25), and we instead refer to an empirical greedy routine with moordinate-deseent heuristic. As indicated in Algorithm [ we start by training feature extractor on the seed training set so, which is then used for calculating ELCE loss L and average feature distance D (s", x²). Afterwards, we sample preliminary pool S' with top C largest tailness and proximity samples (C > K is candidate set size). We employ a score to combine these two metrics, defined as:
aN - - (1-
where (C)) y-mean(u) is normalization term and (0. 1) is coefficient. From S', we then std(o)
sample a diverse subset s of size K, leveraging the K-center greedy algorithm.
Experiment
3.1 Settings
Seed Training Datasets We employ ImageNet-1000LLT as the seed training dataset, which intro- duced in [TI] It is a long-tail version ImageNet-100 100 [26]. The sample number of each class is determined by a down-sampled Pareto distribution used for ImageNet-LT [27]. It includes 12.21K images, with the sample number per class ranging from 1280 to 5.
Sampling Datasets We consider two datasets for sampling: ImageNet-900 and ImageNet--laces--mixs i) ImageNet-900 ImageNet-909 900 is composed by the rest part of ImageNet (28) excluding ImageNet- 100 26). It total contains 1.14 Million images belonging to 900 classes. ii) ImageNett-Plaes--ir We build a sampling dataset including both in-distribution and OoD data. While ImageNet-900 employed as source of in-distribution data, the OoD data is sampled from Places, a large scale scene-centric database. 200 random sampled classes from ImageNet-900 and Places are mixed together. The resultant dataset is called MmageNet-Places-Me
and contains 1.24 million images in total, where 0.25 million and 0.99 million are from ImageNet-900 and Places, respectively. While the
5

---

Table 1: Compare the proposed MAK with random sampling and K-center (0 in terms of the linear separability performance and sho- performance when fixing the sampling budget to be 10K images. The seed training dataset is ImageNet- MeageNet-100-LT. 100-LT. The sampling dataset is ImageNet- t means the metric is the higher the better, and y means the metric is the lower the better. We run each experiment three times and report the error bar. The best performance under each setting is marked as bold.
and Paageeee-Places-Mix(tee
Sampling Budget dataset
method
Protocol
Many
Medium
Few 1
Std (imbal- ancedness)
All
linear separability
71.2:10.8
65.3:10.7
62.7:10.9
3.610.5
67.311.77
None
few-shot
52.640.3
40.5+11.5
32,5#111
8.340.4
44.22:100
linear separability
74.610.3
69.7:10.4
66.1+:1.2
3.5:::.5
71.2:10.2
random
few-shot
56.6+:1.2
48.6++0.4
43.7+117
5.3+1.2
51.1+0.1
10K
linear separability
73.640.3
68.611.88
64.5=0099
3.820.3
70.0:10.4
K-center
few-shot
55.0+:0.4
45.840.3
39.1+1.1
6.540.6
48,5+0.2
IN900
linear separability
76.1+:0.6
70,8++055
69.3+
3.0-0.11
72.7++0.4
MAK
few-shot
57,4+0.6
48.9+0,2
1..3++1.5
4.8+0.2
51,900,4
linear separability
75.7+0..
71.8+0.1
69.6+1.1
2.6+0.4
03.0=t.11
random
Sew-shot
57.41:0.7
49.911.33
45.9:10.4
4.8:0.22
52.3:1..5
20K
linear separability
78.0110.8
73.410..6
7..410.3
2.4
75.1
MAK
few-shot
59.0:10.9
52.910.5
50.0
3.850.5
5..910.4
linear separability
73.8+0.7
67.9
65.1
3.610.2
69,8
random
Sew-shot
55,5100.5
45.8
38.910.7
6.910.3
48.7:10.4
IPM
10K
linear separability
73.010.6
67.7
65.411.5
3.210.4
69.5100.4
K-center
few-shot
54.240.11
45.6::0.4
38.4-0099
6.520.3
48.0
linear separability
74.7:10.2
69.2.10.7
66.610.77
3.3
71.1:0.5
MAK
few-shot
56.8+0077
45.1+00.9
42.6+
6.2+0.1
49.3+00.7
ImageNet-900 a setting with a lot of in-distribution data, ImageNet-Places-- is built for testing a less friendly case where there fewer in-distribution data in the sampling dataset.
Training settings We conduct the experiments with the SimCLR framework. We follow the settings of SimCLR (1 including augmentations, projection head setting, optimizer, temperature, and learning rate. We use Resnet-50 29 for all experiments (including feature extractor f).
Sampling settings When calculating loss enforcing term. Augmentation repeat number M is set as 5 (see explanation 3.44. For K-mean clustering of proximity, the number of centers K is set as 10. For the optimization process, the coefficient is set as 0.3, and the candidate set size C' is set as 1.5 K. As we will show Section 3.3, the hyper-parameters choosing not sensitive.
Evaluation protocol To verify the balancedness of a feature space, we adopt two evaluation protocols: linear separability performance and few-shot performance following (II). 1) linear separability performance: The balancedness of feature space can be reflected with the linear separability w.r.t all classes [1] Testing involves three steps [0] : Pre-train a model f with contrastive learning on the imbalanced ImageNNet dataset ii) Fine-tune a linear classifier with visual representation produced with a balanced dataset (by default, the full dataset where the subset is sampled from) iii) Testing the accuracy on testing dataset for the linear classifier 2) few-shot performance: Few-shot learning an important application for Contrastive Learning (2] Testing it can better reflect the influence of feature embedding for down-stream tasks. The main difference on testing few-shot performance compared to linear separability performance lies in step ii): the whole model are fine-tuned on 1 %% samples of the full dataset from where the long tail dataset is sampled. Note that we choose to tune the whole model instead of employing linear evaluation as in [T since it can yield higher accuracy and can be more practical setting.
Evaluation metrics For measuring the balancedness, we divide dataset into three disjoint groups Many, Medium, Few according to the size of each class following OLTR [27]. Specifically. Many includes classes each with over 100 training samples, Medium includes classes each with 20- 100 training samples and Few includes classes under 20 training samples. For both linear separability performance and few-shot performance, we report the average accuracy of classes for each group. The standard deviation (Std) of three groups' accuracy is used for measuring imbalancedness I.].

---

Table 2: Ablation of each component of the proposed MAK when optimizing different terms with respect to linear separability performance and few-shot performance when fixing sampling budget to be 10K images. The seed training dataset is ImageNet- 100-LT The sampling dataset is ImageNet-9000 The first two rows denote the random sampling baseline. t means the metric is the higher the better, and + means the metric is the lower the better. We each experiment three times and report the error bar. The best performance under each setting is marked as bold.
Tailness Proximity
Diversity
Protocol
Many 1
Medium
Few
Stil (imbal- ancedness)
All 1
linear separability
74.6::0.3
69.7:10.44
66.1
3.5+50.5
71.2:10.2
few-shot
56.61:1.2
48.61:04
43.7
5.311.2
51.110.1
linear separability
74.5:10.6
69.2.10.6
66.311.1
3,440.6
70,910.44
few-shot
55.7:10.4
46.5+0.5
40,2+:.66
6,410.4
49.3:10.4
linear separability
74.0:1009
68.4:10.7
6..5
3.610.3
70.2:10.4
few-shot
55.0++0.2
46,6+0.33
1..8+1.4
5,8+0.5
49.1+0.3
linear separability
73.6:10.3
68.620.8
64.520.9
3,840.3
70.0110.4
few-shot
55.0+10.4
45,8+0.3
39.1
6,540.6
48.5+0.2
linear separability
75.8++0.44
69.9+0.33
69.8+113
2,9+0.5
72.2+0.2
few-shot
57.7++0.7
48.0+1.0
46,4+0.7
5.0+0.77
51,5++0.3
linear separability
76.1=10.6
70.8=10.5
69.3+0.88
3.0+0.1
72.7+10.4
few-shot
57.4:1006
48.9110.2
1.5
4.8
51.910.4
3.2 MAK Yield Higher Accuracy and Balancedness Improvement
We compare three sampling methods: random, K-center, and MAK at Table When the sampling dataset ImageNet-900,, and the sampling budget is set 10K, all three sampling methods could improve accuracy compared to the baseline without sampling additional data. Among them, the proposed MAK yielded the highest performance in terms of balancedness and accuracy. Specifically, compared to the closest competitor, it increases the (Std(imbalancedness)) Accuracy] by [-0.5% 1.5%], respectively, in terms of the linear separability performance. Also, for the few-shot perfor- mance, it improves (Sid(imbalancednes)) Accuracy] by [-0.5%. 0.8%], respectively. It is worth noting that the proposed MAK improves the performance Few groups by at least 2.6% compared to other sampling methods, indicating that MAK is beneficial for tail classes. The improvement is consistent as the sampling budget increases yp to 20K, showing the robustness of MAK towards the sampling budget. An intriguing finding is that the naive K-center algorithm fails to improve the performance compared to the random sampling baseline. The intuition behind this is that too large diversity can compromise the difficulty of negative samples, overfitting the model.
Besides, for the sampling dataset with a large amount of OoD data, the proposed MAK can still yield consistent performance and balancedness improvement. Compared to random sampling baseline, it improves (Sdd(imbalanceedness) Accuracy] by [-0.3%, 0.3%] for linear separability performance and [-0.7%, 0.6%] for few-shot performance, respectively. Compared to the K-center, while the Std of MAK is marginally lower than K-center sampling by 0.1%, it yields higher performance in terms of all other three metrics.
3.3 Ablation Study
In this section, we ablation study the performance with different components in MAK. As shown in Table 2, when employing any single component of the MAK (tailness, proximity or diversity) the accuracy of linear separability performance and few-shot performance would decrease compared to random sampling baseline. The intuition behind is: i) Only tailness: many OoD outliers would be sampled. ii) Only proximity: would lead large redundancy since samples sampled are all similar to the seed training dataset. iii) Only diversity: too weak negative samples (Same as sampling with K-means). By combine tailness and proximity, the accuracy and bulancedness of both protocols could achieve an improvement over the random sampling baseline. By combining the diversity term, the accuracy can be further improved while the balancedness stays in the same level.
Components ablation:
7

---

parrme- robustness: We further look into the sensitivity of the hyper-parameters in the proposed MAK. As demonstrated in table 3 compared to default settings, when us- ing bigger Ks, the accuracy linear separability marginally drops while the balancedness fluctuates (which may be caused by randomness). For few-shot performance, the accuracy keeps at the same level while the bal- ancedness marginally drops. When it comes a, there is only marginally change when a increases to 0.5 and 0.7 for all metrics except balanced- ness of few-shot performance, which marginally decreases. For linear separability performance, employing larger could lead marginally drop in accuracy while improving bal- ancedness. Meanwhile, for few-shot performance, the accuracy improves while the balancedness fluctuates.
Table 3: Study the sensitivity of hyperpaarameterss of MAK in terms of accuracy and balancedness for linear separability few shot when sampling 10K samples on ImageNet 900. Table (a), (b), (c) study the K-means center number K, the coefficient a and the candidate set size C, respectively.
(a)
Protocol
linear separability
few shot
K
Std I (imbal- ancedness)
All
Std + (imbal- ancedness)
All
20
3.3 2.5
72.3
5.2
51.9 51.7
50
72.1
4.9
(b)
Protocol
linear separability
few shot
Std (imbal- ancedness)
All
Std (imbal- ancedness)
All
3.0
72.6
5.4
51.8
0.7
2.8
72.7
5.1
51.4
(c)
Protocol
linear separability
few shot
| (imbal- ancedness)
All
Std 4 (imbal- ancedness)
All
3.4 Further Analysis
ECLE proxy correlates with data distribution We start by verifying if the ELCE proxy reflect the tail- ness. To this end, in the test set of MatgeNet-100-L33 we measure the likelihood of sampling tail classes when sampling samples with large ECLE. For fair comparing,
2.0 x K
2.0
72.3
5.3
52.2
2.5 x
2.7
71.8
4.1
52.8
employ the metric of to mitigate the influence of different group size As demonstrated Figure 2] as the repeat number M increases, of few and medium would increase while & of many decrease. When M 10, é would become to be steady with respect to the repeat number, where @ of the minority is almost twice larger than the majority, which proves that ECLE can be used to spotlight the tail classes with a small M. According to the empirical result, we adopt M = 10 for MAK in practice. Note that M = 10 corresponds times of forward pass for SimCLR, which is reasonable computational cost.
loss target group's percentage in samples with 10% highest loss
data percentage the target group
Computational overhead analysis. The com- putational overhead for the proposed MAK mainly lies in two aspects: 1) The training on the seed dataset. 2) Computing ECLE. The overall computation time overhead is equivalent 700
many
medium
minor
[Selected
1.2
training epochs on ImageNet- Netgeo0-t-0 100-LT with 10K addition data. For a fair comparison, we train the closest competitor (random sampling) by 700 more epochs (1700 epochs in total), the resul- tant [accuracy, std (imbalancedness)) becomes 17..240.1, 3.55 and [5.. 5.8 for linear separability performance and few-shot performance, respectively. Compared to train- ing by 1000 epochs, the imbalancedness for few- shot performance increase while other metrics stay the same, which maybe because of the over- fitting. The proposed MAK can still surpass it in terms of performance.
0.6
repeat number
Figure 2: The correlation between ECLE and tail- ness on ImageNet- 100-LT dataset with respect to different repeat number M. The correlation is reflected with é. practice, we select M 10 as it can reflect the tailness.
Imbalanced downstream task analysis. To evaluate the proposed method can address the im- balancedness for the downstream imbalance task, we linear evaluate the proposed MAK sampling

---

must thm proneneer
with stsareryy promettio
-30
(a)
(b)
(c)
Figure 3: Compare diversity sampled dataset among different methods with t-SNE, (a), (b) and (c)
denotes the features sampled with random, MAK without diversity promotion, and MAK, respectively. Three figures are on the same scale.
method on ImageNet-1002) 100-TT. When sampling from ImageNet-900 with 10K budget, the [accuracy, (imbalancedness)) the proposed MAK is 152.9-0.2. 24.8+ which improves the closest competitor (random sampling) of [52.1 25.4 via improving accuracy by 0.8 and reducing std (imbalancedness) by 0.6, demonstrating the effectiveness of our proposed method.
Visualizing diversity. To further verify the function of the diversity component, we visualize the features of the sampled dataset with different sampling method in Figure Compared to the random sampling method, MAK without diversity promotion would collapse to small area of the feature space, indicating that there are many appearance similar images. By employing diversity promotion, the features expand to large space and avoid redundancy sampling.
Related works
Supervised Learning with Class Imbalance. Most literature tackles the imbalance problem in supervised learning. They can be divided to two classes: class re-sampling [13 and loss or gradient re-weighting [8 21 30. However, neither can extend to unsupervised setting directly, as they require either label information or label-related prediction scores. Recent works also found that training the backbone and the classifier could be decoupled 331 (32, which motivated the exploration of pre-training imbalanced data, followed by downstream fine-tuning.
Self-supervised Learning on Imbalanced Data. The line of applying self-supervised learning on inherently imbalanced data stems from [9] The authors find that simply pre-training with pre-text tasks like rotation (33) and contrastive learning (c.g.. Moco (3) could bring consistent improvements, compared to end-to-end learning without pre-training, implying its regularization effect for more balanced feature space. Follow-up works confirm the superiority of contrastive learning [00 of alleviating imbalance, but also reveal that that benefit does not equal full immunity. In this work, we are the first to explore novel direction for improving the feature balancing contrastive learning further, via sampling additional data from the open world.
Active learning. Our approach is related to active learning. has been well-studied in traditional ML. literature for sampling-efficient learning, such as information theoretical methods 34 ensemble
approaches 35| 36). uncertainty-baat
methods 37 and Bayesian methods (40)
In general, active learning is known to be less popular for deep learning, partially because the most effective theoretically justifiable active learning algorithms rely finite capacity assumptions about the model class, which deep learning disobeys. Examples of recent success include core-set based (20 and loss-based training (41), both developed for supervised learning. We show that in our special setting where balancedness of concern, sampling strategically and actively also becomes relevant.
Additionally, many have combined unsupervised learning for active learning, such as the classical pre-clustering [42] and one/few-shot learning 43 In deep learning, pre-trained embeddings have been found to enhance active learning performances due to calibrating prediction scores better

---

Recent work [47] in turn employed active learning for efficient pre-training. None of the above concerns imbalanced issues. The closet work related ours is [8 in (semi-Jsupervised deep active learning for long-tail visual recognition. The authors present feature density matching using self-suuprvision as well as novel pseudo-label estimators, assisted by rotation- pre-training [49] Differently from that, our method does not need to bootstrap from a labeled training set.
44 46].
5
Conclusion and Discussion of Broader Impact
The data sampled from open- wornd always show long tail distribution, further hurting the balanced- ness of contrastive learning. We tackle this important problem by proposing a unified sampling framework called MAK. It significantly boosts the balancedness and accuracy of contrastive learning via strategically sampling additional data. We believe our techniques are beneficial for improving balancedness on long-tailed data in realistic applications.
On the other hand, the proposed methods are mainly evaluated on academic datasets. When applying on real applications like autonomous driving and medical diagnosis, besides the imbalance problem, there are also problems like fair or private. This reminds us to carefully check our method has risk of producing unfair or biased outputs in the future.
References
[1] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[2] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong supervise leamners. arXiv preprint arXiv:2006. 10029, 2020.
[3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9729-9738. 2020.
[4] Jean- conn-Bstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent: A new approach to self-supervised learning, arXiv preprint 2rX1v:2006.07333, 2020.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0 09882, 2020.
[6] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux. Min Xu, Pengchao Wang. Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2 103.0/988... 2021.
[7] Xiangxin Zhu, Dragomir Anguclov, and Deva Ramanan. Capturing long-tail distributions of object subcatcgories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 915-922, 2014.
[8] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
[9] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving dass-imbalanced learning. arXiv preprint arXiv:2006 07529, 2020.
[10] Bingyi Kang, Yu Li, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for rep-resentation lcarning. In International Conference on Learning Representations, 2021.
[11] Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning, 2021.
[12] Qizhe Xie, Minh Sinh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (0687-10696 2020.
10

---

[13] Li Shen. Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference computer vision, pages 467-482. Springer, 2016.
[14] Nitesh Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[15] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289-305, 2018.
[16] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class- baass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268-9277, 2019.
[17] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking dass-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7610- 7619, 2020.
[18] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aahel-distribution-avaee aware margin loss. arXiv preprint 1906v:1906.0744 2019.
[19] Angela H Jiang. Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger. Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv: 1910.00762, 2019.
[20] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
[21] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE internutional conference computer vision, pages 2980-2988, 2017.
[22] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[23] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 20/9 IEEE Symposium on Security and Privacy (SP), pages 656-672. IEBE, 2019.
[24] Gert W Wolf. Facility location: concepts, models, algorithms and case studies, series: Contributions to management science. International Journal of Geographical Information Science, 25(2):331-333, 201
[25] William J Cook, WH Cunningham, WR Pulleyblank, and Schrijver. Combinatorial optimization. Oberwolfach Reports, 5(4):2875-2942, 2009.
[26] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: (et:.999000555 05849, 2019.
[27] Ziwei Liu, Zhongqi Miao, Xiaohang 7han, Jiayun Wang. Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 2537-2546, 2019.
[28] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. Ieee, 2009.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[30] Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8577-8584, 2019.
[31] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis, Decoupling representation and classifier for long-tain tailed recognition. arXiv preprint arXiv: arXiv:1910.0000 191000977.. 2019.
11

---

[32] Junjie Zhang, Linggiao Liu, Peng Wang, and Chunhua Shen. To balance or not to balance: A simple-yet- effective approach for learning with long-tailed distributions. arXiv preprint arXiv:1912.04496, 2019.
[33] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 10Xiv:1803.07722 2018.
[34] David IC MacKay. information-basee objective functions for active data selection. Neural computation, 4(4):590-604, 1992.
[35] Andrew Kachites McCallumzy and Kamal Nigamy. Employing em and pool-based active learning for text classification. In Proc. International Conference Machine Learning (ICML), pages 359-367. Citeseer, 1998.
[36] Yoav Freund, H. Sehastian Seung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning, 28(2):133-16) 1997.
[37] Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classifi- cation. Journal of machine learning research, 24Nov): 2001.
[38] Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos, Multi-class active learning for image classifi- cation. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2372-2379, IEEE, 2009.
[39] Xin and Yuhong Guo. Adaptive active learning image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 859-866, 2013.
[40] Ashish Kapoor, Kristen Grauman, Raquel Urtasun. and Trevor Darrell, Active learning with gaussian processes for object categorization. In 2007 IEEE 11th International Conference on Computer Vision, pages 1-8. IEEE, 2007.
[41] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 93- 2019.
[42] Hieu Nguyen and Amoll Smeulders. Active learning using pre-clustering. In Proceedings of the twenty-first international conference on Machine learning, page 79, 2004.
[43] Mark Woodward and Chelsea Finn. Active one-shot learning. arXiv preprint arXiv: /70kv://00..00 700.00659, 2017.
[44] Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar, Deep active learning for named entity recognition. arXiv preprint /rXiv:1707.05928 2017.
[45] Michelle Yuan, Hsuan-Tiees Lin, and Jordan Boyd-Graber. Cold start active learning through self- supf-supe language modeling. arXiv preprint 20X::20010.99535, 2020.
[46] Yuan-Hong Liao, Amlan Kar, and Sanja Fidler. Towards gond practices for efficiiennyy annotating large-scale image classification datasets. arXiv preprint arXiv:2104. /2690, 2021.
[47] Shuvam Chakraborty, Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Shechan, and Stefano Ermon. Efficient conditional pre-training for transfer learning. arXiv preprint 1rXiv:2011.102:. 2020.
[48] Denis Gudovskiy, Alee Hodgkinson, Takuya Yamaguchi, and Sotaro Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9041-9049, 2020.
[49] Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning hy predicting image rotations.
In International Conference Learning Representations (ICLR), 2018.
[50] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[51] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint ar 2001 2020.
12

---

Appendix:
Improving Contrastive Learning on Imbalanced Seed Data via Open-Worke Sampling
This appendix contains the following details that we could not include in the main paper due to space restrictions.
(Sec.
A Details of the computing infrastructure. (Sec.
Details of the employed hyperparameters.
A Details of computing infrastructure
Our codes are based on Pytorch [5). and all models are trained with NVIDIA A100 Tensor Core GPU.
B Details of hyper-parameter settings
B.1 Pre-training
We identically follow 11 for pre-training settings except the epochs number: we pre-train for 1000 epochs for all our experiments following [[] (Including the feature extractor).
B.2 Fine-tuning
When fine-tuning for linear separability performance, the optimizer is set as SGD with momentum of 0.9 and initial learning rate 30 following rate by 10 times at epochs 10 and 20. When fine-tuning for few-shol performance, we follow [2] fine-tuning from the first MLP projection layer. We train for 100 epochs with batch size 64. The initial Ir is set as 0.02 and cosine learning rate decay without warm up is employed [2]
[5T]. we train for 30 epochs and decrease the learning
13


Reviews:
[{"review_id": "vEuehBNiNOf", "reviewer": "Reviewer_DGts", "summary": "-This paper aims to improve contrastive learning by incorporating more open-world unlabeled data. However, such data may introduce a class imbalance problem. To solve the problem, the authors propose a sampling strategy to select a subset of the imbalanced unlabeled data following three principles: tailness, proximity, and diversity, which encourages sampling tail-class examples, rejecting outliers, and ensures diversity in the selected set. They also provide extensive experiments which demonstrate the effectiveness of the proposed sampling method.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-The writing of the paper is clear and easy to follow. The treatment is thorough, proceeding from introducing the SimCLR backbone, designing the sampling objective with the tailness, proximity, and diversity principles, providing a heuristic algorithm for it, to implementing experiments for the proposed method.\n\n-The ablation study in the paper decomposes the three components of the proposed method and it is interesting to see how each of them contributes. And the effort made on demonstrating the hyperparameter robustness is also a positive point to apply the proposed method for general purposes.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-In the introduction, the authors mentioned using each sample’s training loss to identify “hard samples”...It is unclear to me what “hardness” means in the context of contrastive learning. Is it more accurate to say under-learned samples, or rare samples? I think the word “hard sample” is more often used in supervised learning, as shown in line 113, they yield large loss values. And it is often used in label-noise learning literature, where they differentiate noisy (hard) and non-noisy data by large and small loss respectively.\n\n-In line 103, it seems that the right-hand side of s^\\tau(a,b) does not include b?\n\n-What is the \\epsilon in Eq. 2?\n\n-I wonder if the authors have considered other distances for Eq 3, e.g., other norms? Is this an important factor in the experiment?\n\n-How was the threshold t in Algorithm 1 selected in the experiments?\n\n-In the experiment section, it makes sense that the balancedness may be reflected by the linear separability of all classes in the feature space, so the linear separability and the few-shot performance (accuracy) may be useful for evaluating the balancedness. But solely saying that the standard deviation (std) of the accuracy measures the imbalancedness may be a bit misleading.\n\n-In Table 1, if we increase the sampling budget, we can see that the std of linear separability performance of the proposed method gets better. But it seems that the std of the few-shot performance gets a bit worse. Could the authors elaborate more on this inconsistency? I guess it may be some randomness. But from the current table, we may conclude that the imbalancedness does not get better in such cases, which may not be true. Because the accuracy clearly improves. I hope the authors add more explanations on this point.\n\n--Minor typo\n\n-Line 67: “an empirical expectation of loss” is a bit weird to me, maybe using “an empirical version of the expected loss” is better\n\n-Line 84: adaptive-->adaptively\n\n-Line 86: the abbreviation MAK is already introduced in Line 77\n\n-Line 87: OoD firstly appears without the full word\n\n-Line 100: .-->,\n\n-Line 102: .-->,\n\n-Line 127: varies-->verifies\n\n-Line 187: togather-->together\n\n-Line 210: OLTR firstly appears without any explanation\n\n-Line 213: We-->we\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10hours"}, "extra_text": {}}, {"review_id": "sZYPIScHjdW", "reviewer": "Reviewer_JDg1", "summary": "\nThe paper aims to seek a principled approach of selecting a subset of unlabeled data from an external source that are relevant for learning better and diverse representations. In particular, it proposes an open-world unlabeled data sampling strategy called **Model-Aware K-center** (MAK) for long-tail learning. It follows three simple principles, **tailness** to encourage sampling of examples from tail classes; **proximity** to reject the out-of-distribution outliers; **diversity** to ensure diversity in the set of sampled examples. They use ImageNet-100-LT as the target dataset and two external data sources. The experiments  demonstrate that MAK can improve the representation quality. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The paper is easy to follow. The organization is good.\n2. The motivation of this work looks clear and reasonable. For open-world long-tailed unlabeled data, it is reasonable to sample more tail data, reject out-of-distribution data, and ensure diversity in the sampled set. \n3. The authors demonstrate that their method improves on random selection on ImageNet-LT, and conduct ablation studies to show the separate effects of the proposed three constraints (tailness, proximity, and diversity).\n\n**Weaknesses**\n1. The proposed method lacks novelty. Firstly, for *tailness*, using the largest loss to select the hard examples is common, and using multiple random augmentations to eliminate the randomness is more like an implementation trick. For *proximity*, considering the most dissimilar samples as outliers is also common. And the *diversity* is similar to [1]. Therefore, I think the paper just applies some existing methods but lacks methodological improvements.\n2. The performance gains are marginal, and the methods introduce extra computations cost compared to random selection. As shown in Table1, compared to random selection, MAK only brings about 0.9% gain on *linear separability performance* and about 0.2% gain on *few-shot performance*. In addition, MAK uses multiple random augmentations for SimCLR and a greedy heuristic algorithm, which introduces computation costs. The computation costs and training time of MAK and baseline should be added and compared.  \n3. From Table 2, it does look like *Diversity* does not help much. It brings no performance gain on *linear separability performance* and only 0.2% gain on *few-shot performance*. \n4. The description of the Evaluation Protocol is not very clear. For *few-shot performance*, the authors claim that they use 1% samples of the full dataset from where the long tail dataset is sampled. I still can not understand how to select the 1% samples. Why not use the ImageNet-LT training set to train a classifier?\n\n[1] Active learning for convolutional neural networks: A core-set approach, in ICLR 2018. \n\n\n*****************************************************\nThe authors have addressed the limitation raised in the review. So I raise my rate to accept.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have claimed the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "gwHtWXJBVsd", "reviewer": "Reviewer_HcS6", "summary": "In this paper, the author improves the accuracy and balancedness of the contrastive learning via proposing a \nunified sampling framework called Model-Aware K-center, which follows the principle of tailness, proximity and diversity. The extensive experiments verified the consistent improvement compared to other sampling methods. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the author improves the accuracy and balancedness of the contrastive learning via proposing a \nunified sampling framework called Model-Aware K-center, which follows the principle of tailness, proximity and diversity. The extensive experiments verified the consistent improvement compared to other sampling methods. \n\nPro:\n1. the idea sampling open-world unlabeled data for improving the representation learning is interesting.\n2. New problem and insight to contrastive learning.\n3.The performance of experiments is  good.\n4. Good writing.\n\n\nSome issues:\n1. the idea ( sampling open-world unlabeled data) is similar (same) to paper Learning bounds for open-set learning (ICML 2021).  Could you discusses the relationship in your paper ?\n\n2. Contrastive learning with the idea  sampling open-world unlabeled data can be used to solve open-set learning ?\n\n3. The  curse of dimensionality will happen ? sampling open-world unlabeled data may face the curse of dimensionality, becausw you do not know the manifold dimension of datasets. If it happens, how can you solve it ?\n\nPlease answer above issues.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "NDgdSe4J0J", "reviewer": "Reviewer_j3dN", "summary": "This paper improves the accuracy and balancedness of contrastive learning by proposing a unified sampling framework called Model-Aware K-center. It follows three principles: tailness, proximity, and diversity. As a new problem setting, the authors reported extensive experiments to show their method's consistent improvement over other sampling methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ This paper studies an important new problem. The data sampled from the open world always show a long tail distribution, thus potentially hurting the balancedness of contrastive learning and leading to “unfair” downstream class transferability. This is a refreshing yet timely angle to study contrastive pre-training. \n+ The authors also clearly lay out the unique challenges in this new setting, which is helpful to understand. The three principles and the optimization formulation look novel to me. The finding of ELCE (the big influence of augmentation randomness) is intuitive but sensible. \n+ Since this is a new problem, I especially like that the authors perform due diligence in designing benchmark experiments. They created two mixed unlabeled sets for sampling that purposely include OoD samples at different levels. ImageNet-LT is adopted as the main testbed. The reported numbers and ablation studies are thorough and convincing. \n+ The writing quality is very high. The paper is clearly structured, and the logic is coherent. The mathematics is concise and correct to my best knowledge.\n\nCons:\n-  While I understand this paper's main point is on improving balancedness, I have to point out the accuracy gain looks underwhelming.  Especially, while linear separability has an okay 0.9% margin over the closest competitor, the few-shot gain is much less impressive (only 0.2%). Since few-shot is often the main application scenario, this brings up a concern for MAK's practicality.\n- I’m not sure if K-center is necessary. Why not more computationally convenient alternatives, such as K-means? Also, correct me if I’m wrong -- the actual Algorithm 1 does not involve solving K-center, but instead used a check-and-go heuristic to inject diversity. Am I understanding that right?\n- Computing ECLE could be expensive due to multiple feedforward passes.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern was found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper worked on contrastive learning on class-imbalanced data and proposed *model-aware K-center* for the purpose. It makes use of additional unlabeled dataset which is bigger than the targeted unlabeled dataset for contrastive learning in terms of the number of instances and the number of classes. According to three principles, namely *tailness*, *proximity*, and *diversity*, certain unlabeled data are sampled from the pool dataset to re-balance the targeted dataset. The writing is clear, the motivation is strong, the idea is novel, and the results are significant. Thus, it should be accepted for publication.

It is quite interesting that the proposed method can satisfy the tailness and the proximity at the same time as shown in Figure 1, because the former sounds like accepting long-tail **classes** and the latter sounds like rejecting long-tail **instances**. Since contrastive learning should be regarded as **pre-training**, it may not be very natural if the label space for the downstream tasks is fixed at the time of contrastive learning (it is unsupervised and there is no label at all). Moreover, since the targeted dataset is smaller, is it possible that a class exist in both datasets but **no data has been drawn from this class in the targeted dataset** and then all the data of this class in the bigger pool dataset become out-of-distribution data by the proposed method (i.e, some data should be accepted but will be rejected since this class is missing in the targeted dataset)? Perhaps I have some misunderstanding because I didn't carefully go through the full paper by myself, but I believe clarifying my questions (not concerns, just questions) are very helpful and can maximize the impact of your work.

BTW, the following paper should be related to your work, which can sample in-distribution data and reject out-of-distribution data though it considered to enlarge but not re-balance/enrich the targeted dataset (i.e., quantity vs quality):

Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive-Unlabeled Compression on the Cloud. NeurIPS 2019.
Decision: Accept (Poster)