Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper mainly studied how the negative samples can affect the model performance in supervised learning CIO works. Through the experiments, this work has a few interesting findings, including the majority of negative samples are not important for the model learning, only a small subset of hard samples determine the model importance. These hard examples are also closely related with positive samples (more semantically similar).  We can see from experiments that it's very important  to fairly treat negative samples in supervised learning tasks. However, there is no frameworks proposed to help improve the learning representation or speed up the training task.  In general, the readers are more interested in the solutions after realizing the importance of negative samples treatment during the experiments.  It would be necessary to include the corresponding solutions by automatically setup these negatives samples in CID related task.
review_title: The exploration is useful, and would like to see a corresponding framework to be used in the community. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
This paper argues that in contrastive self-supervised learning, different negative instances have different importance. This importance is relevant to the ``difficulty" of negative instances. On ImageNet and MoCo2, the authors show that using the most difficult 5% negative instances can achieve similar performance compared with using all negative instances. However, the most difficult 0.1% of negative instances yield bad performance.

I recommend to reject this paper due to the following major concerns: 1) study is performed on a single dataset, which is not convincing; 2) study is performed on a single method, which casts doubts on whether the conclusions hold for other methods; 3) this study does not seem to have practical value.

While this study is interesting, it lacks rigor, in the following aspects.
1. The study is only performed on a single contrastive self-supervised learning method: MoCo2. It is unclear whether the conclusions hold for other contrastive SSL methods, such as BYOL and many others.
2. The study is conducted on a single dataset: ImageNet. It is unclear whether the conclusions hold for other datasets.
3. Another concern is this study does not seem to have practical value. In each iteration during training, finding the hardest examples for a query needs to calculate the inner-product between this query and all other training examples, which is computationally very heavy. 
4. In the author's measure of difficulty, the difficulty is a function of network weights. In early stage of the training, the network weights are random, which implies that the calculated difficulty may be meaningless. Can the authors comment on this?

However, the paper does have a few strong points.
1. The paper is well-written. The organization is clear and the paper is easy to follow.
2. The studied problem is interesting and novel. 


Other comments.
1. Figure 5a is difficult to interpret. The author may consider to reorganize it.
2. In Figure 3, only three temperature values were considered, which may not be very convincing.

-----------------------------------------------------------------------------------------------------------------------------------
Update: I read the authors' rebuttal. The authors didn't address my concern "The study is conducted on a single dataset: ImageNet. It is unclear whether the conclusions hold for other datasets."  sufficiently. I would like to keep my original rating. 

review_title: Interesting study, but lacking rigor

final_score: rating: 2: Strong rejection
review_content: 
The findings of this work are that for contrastive learning, most of the negatives deemed easily separable are unnecessary, the most important negatives are somewhere in the top 5% closest to the positive sample, and that some of the exceedingly hard examples are detrimental.

-In general, I felt the main findings of this work to be roughly in line with what we already know about contrastive learning. We can easily look at this work's findings with respect to the soft SVM margin, in that only the examples close to the decision boundary should matter (max margin), but some difficult examples  (the aforementioned exceedingly difficult ones) make the data inseparable, so we allow some violation (slack terms).  While I'm not suggesting that slapping a soft SVM here would solve the problem, there is a large body of SVM-based detection/classification literature that precedes the findings of this work.

-Validity of WordNet as a measure of semantic similarity: Section 4 uses WordNet distances to estimate the semantic similarities between classes by finding their shared subtree root. The deeper the subtree, the more semantically similar. While I do not dispute the claim of the hardest negatives being from semantically similar classes. Different parts of the WordNet synset tree have semantic hierarchies of varying levels of coarseness. A 2 hop distance in one subtree could easily be more of a semantic jump than a 3 hop distance in another. 

-The exist prior works dealing with the neglected semantic hierarchies in ImageNet by setting up hierarchical classifiers. An example is [1].

-I would further argue that there's some nuance in the correlation between semantic similarity and example hardness, in that it really depends on your choice of feature representation. Visual features will naturally correlate with closer semantic levels in visually-defined categories. However, this will not necessarily hold for semantic categories defined by function, in that two visually distinct items may fall under close semantic labels. 

-The related works section claims object detection works have not "explicitly involved negative examples as in CID." I have to imagine this statement is poorly phrased, as [2] (also cited in this paragraph) very explicitly mines for  face-like non-face patterns. There is a very long list of hard-negative mining works in object detection.

Overall, I value the empirical impact of this work, in that the rather detailed analysis may lead to improvements to future versions of the contrastive feature learning task. However, I do not find the findings of this work to be sufficiently novel for this conference, and therefore cannot recommend this work for acceptance in its current state.



[1] Yan et al. HD-CNN: Hierarchical Deep Convolutional Neural Networksfor Large Scale Visual Recognition. ICCV 2015

[2] Sung and Poggio. Example-Based Learning for View-Based Human Face Detection. TPAMI 1998
review_title: Good empirical analysis, but findings are not sufficiently novel

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors carried out a series of experiments to analyze the impact of negative samples in contrastive learning (instance discrimination - CID). In particular, they try to identify which difficulty range is important for representation learning. Of the many recent self-supervised learning approaches, they chose MOCO V2 as the testbed. They trained the MOCO model from an ImageNet pre-trained one. Various settings, which correspond to various ways of filtering our hard or easy negatives, were used. Hardness of samples are measured based on embedding distance to the query. I.e. ones with large distance are easy. Their main findings are, for negative samples, 1) Using the 5% hardest is enough for downstream tasks, 2) the easiest 95% of them were unnecessary and insufficient, 3) The hardest 0.1% is harmful and  4) hard negatives were more semantically similar to the query. 
In general, in my opinion, this is a paper in which the authors tried to answers many interesting practical questions. The author provided experiments and convincing evidences for a number of insights. My main reservations with this paper are: 
1) most of the points are not new and are elaborations of what were pointed out before elsewhere, for example, in semi-hard mining for distance metric learning.
2) the empirical results are only within the context of MOCO2 and for a linear classification task. It is not clear how such numbers as 0.1%, 5% or 95% would change when adopting other frameworks such as BYOL or SwAV… The reported gains seems a little bit sensitive to the temperature parameters of MOCO.
3) The sample hardness is measured based on embedding distance, which would be evolved during the training process itself.  It is not clear how accurate it is especially in the early stage of training.

My suggestion for improvements is that either to empirically show that their findings (numbers) are consistent across a number of frameworks and downstream tasks, or to provide some theoretical justification for their findings if only MOCO v2 is used.

review_title: Nice work, but needs more experiments or some theoretical justification

Meta-review: This paper empirically studies the impact of different types of negatives used in recent contrastive self-supervised learning methods. Results were initially shown on Mocov2, though after rebuttal simCLR was also added, and several interesting findings were found including that only hardest 5% of the negatives are necessary and sufficient. While the reviewers saw the benefit of rigorously studying this aspect of recent advances in self-supervised learning, a number of issues were raised including: 1) The limited scope of the conclusions, given that only two (after rebuttal) algorithms were used on one datasets, 2) Limited connections drawn to existing works on hard negative mining (which is very common across machine learning including metric learning and object detection), and 3) Limited discussion of some of the methodological issues such as use of measures that are intrinsically tied to the model's weights (hence being less reliable early in the training) and WordNet as a measure for semantic similarity. Though the authors provided lengthy rebuttals, the reviewers still felt some of these issues were not addressed. As a result, I recommend rejection in this cycle, and that the authors bolster some of these aspects for a submission to future venues. 

I would like to emphasize that this type of work, which provides rigorous empirical investigation of various phenomena in machine learning, is indeed important and worth doing. Hence, the lack of a new method (e.g. to address the selection of negatives) was not the basis of the decision. While the paper clearly does a thorough job at investigating these issues for a limited scope (e.g. in terms of datasets), a larger contribution is expected for empirical papers such that 1) we can ensure the generality of the conclusions (across methods and datasets), 2) we have a conceptual framework for understanding the empirical results especially with respect to what is already known in adjacent areas (e.g. metric learning and object detection), and 3) we understand some of the methodological choices that were made and why they are sufficiently justified. 
Decision: Reject