Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper improves the accuracy and balancedness of contrastive learning by proposing a unified sampling framework called Model-Aware K-center. It follows three principles: tailness, proximity, and diversity. As a new problem setting, the authors reported extensive experiments to show their method's consistent improvement over other sampling methods.
main_review: Pros:
+ This paper studies an important new problem. The data sampled from the open world always show a long tail distribution, thus potentially hurting the balancedness of contrastive learning and leading to “unfair” downstream class transferability. This is a refreshing yet timely angle to study contrastive pre-training. 
+ The authors also clearly lay out the unique challenges in this new setting, which is helpful to understand. The three principles and the optimization formulation look novel to me. The finding of ELCE (the big influence of augmentation randomness) is intuitive but sensible. 
+ Since this is a new problem, I especially like that the authors perform due diligence in designing benchmark experiments. They created two mixed unlabeled sets for sampling that purposely include OoD samples at different levels. ImageNet-LT is adopted as the main testbed. The reported numbers and ablation studies are thorough and convincing. 
+ The writing quality is very high. The paper is clearly structured, and the logic is coherent. The mathematics is concise and correct to my best knowledge.

Cons:
-  While I understand this paper's main point is on improving balancedness, I have to point out the accuracy gain looks underwhelming.  Especially, while linear separability has an okay 0.9% margin over the closest competitor, the few-shot gain is much less impressive (only 0.2%). Since few-shot is often the main application scenario, this brings up a concern for MAK's practicality.
- I’m not sure if K-center is necessary. Why not more computationally convenient alternatives, such as K-means? Also, correct me if I’m wrong -- the actual Algorithm 1 does not involve solving K-center, but instead used a check-and-go heuristic to inject diversity. Am I understanding that right?
- Computing ECLE could be expensive due to multiple feedforward passes.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: -This paper aims to improve contrastive learning by incorporating more open-world unlabeled data. However, such data may introduce a class imbalance problem. To solve the problem, the authors propose a sampling strategy to select a subset of the imbalanced unlabeled data following three principles: tailness, proximity, and diversity, which encourages sampling tail-class examples, rejecting outliers, and ensures diversity in the selected set. They also provide extensive experiments which demonstrate the effectiveness of the proposed sampling method.
main_review: -The writing of the paper is clear and easy to follow. The treatment is thorough, proceeding from introducing the SimCLR backbone, designing the sampling objective with the tailness, proximity, and diversity principles, providing a heuristic algorithm for it, to implementing experiments for the proposed method.

-The ablation study in the paper decomposes the three components of the proposed method and it is interesting to see how each of them contributes. And the effort made on demonstrating the hyperparameter robustness is also a positive point to apply the proposed method for general purposes.

limitations_and_societal_impact: -In the introduction, the authors mentioned using each sample’s training loss to identify “hard samples”...It is unclear to me what “hardness” means in the context of contrastive learning. Is it more accurate to say under-learned samples, or rare samples? I think the word “hard sample” is more often used in supervised learning, as shown in line 113, they yield large loss values. And it is often used in label-noise learning literature, where they differentiate noisy (hard) and non-noisy data by large and small loss respectively.

-In line 103, it seems that the right-hand side of s^\tau(a,b) does not include b?

-What is the \epsilon in Eq. 2?

-I wonder if the authors have considered other distances for Eq 3, e.g., other norms? Is this an important factor in the experiment?

-How was the threshold t in Algorithm 1 selected in the experiments?

-In the experiment section, it makes sense that the balancedness may be reflected by the linear separability of all classes in the feature space, so the linear separability and the few-shot performance (accuracy) may be useful for evaluating the balancedness. But solely saying that the standard deviation (std) of the accuracy measures the imbalancedness may be a bit misleading.

-In Table 1, if we increase the sampling budget, we can see that the std of linear separability performance of the proposed method gets better. But it seems that the std of the few-shot performance gets a bit worse. Could the authors elaborate more on this inconsistency? I guess it may be some randomness. But from the current table, we may conclude that the imbalancedness does not get better in such cases, which may not be true. Because the accuracy clearly improves. I hope the authors add more explanations on this point.

--Minor typo

-Line 67: “an empirical expectation of loss” is a bit weird to me, maybe using “an empirical version of the expected loss” is better

-Line 84: adaptive-->adaptively

-Line 86: the abbreviation MAK is already introduced in Line 77

-Line 87: OoD firstly appears without the full word

-Line 100: .-->,

-Line 102: .-->,

-Line 127: varies-->verifies

-Line 187: togather-->together

-Line 210: OLTR firstly appears without any explanation

-Line 213: We-->we
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the author improves the accuracy and balancedness of the contrastive learning via proposing a 
unified sampling framework called Model-Aware K-center, which follows the principle of tailness, proximity and diversity. The extensive experiments verified the consistent improvement compared to other sampling methods. 

main_review: In this paper, the author improves the accuracy and balancedness of the contrastive learning via proposing a 
unified sampling framework called Model-Aware K-center, which follows the principle of tailness, proximity and diversity. The extensive experiments verified the consistent improvement compared to other sampling methods. 

Pro:
1. the idea sampling open-world unlabeled data for improving the representation learning is interesting.
2. New problem and insight to contrastive learning.
3.The performance of experiments is  good.
4. Good writing.


Some issues:
1. the idea ( sampling open-world unlabeled data) is similar (same) to paper Learning bounds for open-set learning (ICML 2021).  Could you discusses the relationship in your paper ?

2. Contrastive learning with the idea  sampling open-world unlabeled data can be used to solve open-set learning ?

3. The  curse of dimensionality will happen ? sampling open-world unlabeled data may face the curse of dimensionality, becausw you do not know the manifold dimension of datasets. If it happens, how can you solve it ?

Please answer above issues.

limitations_and_societal_impact: No potential negative societal impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
The paper aims to seek a principled approach of selecting a subset of unlabeled data from an external source that are relevant for learning better and diverse representations. In particular, it proposes an open-world unlabeled data sampling strategy called **Model-Aware K-center** (MAK) for long-tail learning. It follows three simple principles, **tailness** to encourage sampling of examples from tail classes; **proximity** to reject the out-of-distribution outliers; **diversity** to ensure diversity in the set of sampled examples. They use ImageNet-100-LT as the target dataset and two external data sources. The experiments  demonstrate that MAK can improve the representation quality. 

main_review: **Strengths**
1. The paper is easy to follow. The organization is good.
2. The motivation of this work looks clear and reasonable. For open-world long-tailed unlabeled data, it is reasonable to sample more tail data, reject out-of-distribution data, and ensure diversity in the sampled set. 
3. The authors demonstrate that their method improves on random selection on ImageNet-LT, and conduct ablation studies to show the separate effects of the proposed three constraints (tailness, proximity, and diversity).

**Weaknesses**
1. The proposed method lacks novelty. Firstly, for *tailness*, using the largest loss to select the hard examples is common, and using multiple random augmentations to eliminate the randomness is more like an implementation trick. For *proximity*, considering the most dissimilar samples as outliers is also common. And the *diversity* is similar to [1]. Therefore, I think the paper just applies some existing methods but lacks methodological improvements.
2. The performance gains are marginal, and the methods introduce extra computations cost compared to random selection. As shown in Table1, compared to random selection, MAK only brings about 0.9% gain on *linear separability performance* and about 0.2% gain on *few-shot performance*. In addition, MAK uses multiple random augmentations for SimCLR and a greedy heuristic algorithm, which introduces computation costs. The computation costs and training time of MAK and baseline should be added and compared.  
3. From Table 2, it does look like *Diversity* does not help much. It brings no performance gain on *linear separability performance* and only 0.2% gain on *few-shot performance*. 
4. The description of the Evaluation Protocol is not very clear. For *few-shot performance*, the authors claim that they use 1% samples of the full dataset from where the long tail dataset is sampled. I still can not understand how to select the 1% samples. Why not use the ImageNet-LT training set to train a classifier?

[1] Active learning for convolutional neural networks: A core-set approach, in ICLR 2018. 


*****************************************************
The authors have addressed the limitation raised in the review. So I raise my rate to accept.

limitations_and_societal_impact: Yes, the authors have claimed the limitations of their work.
review_title: null

Meta-review: The paper worked on contrastive learning on class-imbalanced data and proposed *model-aware K-center* for the purpose. It makes use of additional unlabeled dataset which is bigger than the targeted unlabeled dataset for contrastive learning in terms of the number of instances and the number of classes. According to three principles, namely *tailness*, *proximity*, and *diversity*, certain unlabeled data are sampled from the pool dataset to re-balance the targeted dataset. The writing is clear, the motivation is strong, the idea is novel, and the results are significant. Thus, it should be accepted for publication.

It is quite interesting that the proposed method can satisfy the tailness and the proximity at the same time as shown in Figure 1, because the former sounds like accepting long-tail **classes** and the latter sounds like rejecting long-tail **instances**. Since contrastive learning should be regarded as **pre-training**, it may not be very natural if the label space for the downstream tasks is fixed at the time of contrastive learning (it is unsupervised and there is no label at all). Moreover, since the targeted dataset is smaller, is it possible that a class exist in both datasets but **no data has been drawn from this class in the targeted dataset** and then all the data of this class in the bigger pool dataset become out-of-distribution data by the proposed method (i.e, some data should be accepted but will be rejected since this class is missing in the targeted dataset)? Perhaps I have some misunderstanding because I didn't carefully go through the full paper by myself, but I believe clarifying my questions (not concerns, just questions) are very helpful and can maximize the impact of your work.

BTW, the following paper should be related to your work, which can sample in-distribution data and reject out-of-distribution data though it considered to enlarge but not re-balance/enrich the targeted dataset (i.e., quantity vs quality):

Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive-Unlabeled Compression on the Cloud. NeurIPS 2019.
Decision: Accept (Poster)