Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Briefly, this paper presents a simple yet effective method (i.e., IFND) for self-supervised contrastive learning by effectively handling the false-negative samples. Specifically, the authors propose to cluster samples in the embedding space and assign the cluster indices as pseudo labels in an incremental manner during training. The experimental results are extensive and demonstrate the effectiveness of the proposed method.
main_review: Strengths:
1) The paper is easy to follow with clear motivation. The discussion and analysis about “false negatives” on contrastive learning in Fig1 is quite valuable. 
2) The authors justify the effectiveness of the proposed method according to the extensive evaluations. Besides, the ablation study also validates the contribution of the main component of the proposed IFND.

Weaknesses:
1) The technical novelty of the proposed method is somewhat marginal since some of its main components (i.e., hierarchical semantic definition) have already been mentioned in PCL (Li et al., 2021). The authors should present more detailed analyses and discussions to highlight the difference between PCL and their proposed IFND. For instance, it will be better if the authors can conduct experiments on the object detection task to demonstrate the consistently superior performance of IFND over PCL. 
2) The acceptance rate of the pseudo label is determined without careful consideration. According to Appendix B, the authors empirically set the acceptance rate in a hyperparameter-free manner (i.e., Epoch / Training Epoch). This might make the learning process unstable due to the accumulated pseudo-labeling errors. It might be better if the authors can explore some robust learning techniques (e.g., self-paced learning) to improve learning efficiency.
3) It seems that the experiment results do not validate the effectiveness of the introduced attraction strategy according to Table 5.
summary_of_the_review: This paper introduces some interesting insights for contrastive learning. However, its presentation and experiment design could be improved. I will raise my rating if most of my concerns have been well addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes incremental false-negative detection (IFND), which has several improvements over prototypical contrastive learning (PCL). The key idea of PCL is to alternatively update (a) the pseudo-labels based on the current encoder and (b) update the encoder based on the current pseudo-labels. However, PCL often converges to the local minima since the bad encoder produces falsy pseudo-labels, further harming the encoder. To tackle the issue, INFD suggests including the confident pseudo-labels while uncertain ones incrementally. IFND consistently improves the PCL in all considered scenarios.
main_review: ### Pros

- Clear motivation over the prior work and a simple yet effective solution.
- Nice details on the method, e.g., linear scheduling of the acceptance rate, to be hyperparameter-free.
- Nice experimental setup, e.g., comparing with SimCLR and SupCon to provide lower and upper bounds.


### Cons/concerns

1. Limited novelty and inappropriate credit for PCL

The proposed method is somewhat incremental over the prior work, PCL. As PCL develops the core concept of pseudo-label-based contrastive learning, the paper should clearly illustrate the contribution of prior work and its novel innovations. While the paper briefly introduces PCL in the related work section, the paper should clearly state PCL as the preliminary in the main method section.

For example, the "hierarchical semantic definition" part of the paper (page 4) suggests using multiple clusters to consider the different hierarchy levels. However, this technique was already introduced in the original PCL paper (Eq. (11)).


2. Ablation study on the difference from PCL

Putting aside omitting uncertain pseudo-labels, IFND has several minor differences from the PCL. After clearly stating the differences, the paper could provide the ablation study for each modified component, e.g.,
- Confidence of pseudo-label: One may use Eq. (9) of PCL instead of the proposed Eq. (2), although the latter seems to be better as the former does not consider different classes.
- Number of clusters: The original PCL used {25000,50000,100000} but INFD uses {1000,3000,10000}. It can be unfair since the desired number of clusters are 1000 for ImageNet.


3. Comparison with other cluster-based (and other self-supervised) methods

Table 1 suggests that SwAV, another cluster-based method, performs the best. While the paper only verifies the merit over its predecessor, PCL, the empirical contribution could be limited if it underperforms than other self-supervised methods. Thus, the paper should compare IFND with SwAV for all experiments, under the same setup (e.g., 200 epochs).

While BYOL is not cluster-based method, it does not suffer from the suggested false negative issue, which weaken the motivation of this work. The paper could also verify the advantages over the BYOL to strengthen the desirability of proposed method.


Minor comments:
- While the paper introduces two losses: elimination and attraction, the paper concludes that elimination is better for considered scenarios. The paper could mainly introduce the single final method and leave the inferior one for the discussion.
- The definition of false negative $\mathcal{N}(i)$ could be introduced before it is used in Eq. (3).
summary_of_the_review: The paper proposes a solid yet somewhat incremental modification from the prior work, PCL. Overall, I think the paper is on the boarderline. I'm willing to raise my score if my concerns: (a) improper credit for PCL, (b) ablation study over PCL, amd (c) comparison with SwAV/BYOL are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper deals with the very important topic of “false negatives” in SSL and shows that “false negatives” have a huge impact on performance of SSL methods. One of the most interesting experiments they show is that as they increase the number of classes in the dataset the performance drops more and more. They propose a clustering based method which reduces the  “false negatives” and shows improved performances as compared to other state-of-the-art baselines.

main_review: Strengths: 
1) Well written and easy to follow paper with a very solid hypothesis and analysis of effects of “false negatives” on contrastive learning.
2) They have shown good results specially under classification settings in both supervised learning and semi-supervised learning setups.
3) Table4 shows really good analysis and shows how increasing the depth results in more and more poor performances. 
4) Better clustering results than SWaV: Table 3 results are very promising as they show that IFND achieves better NMI results than SWaV and MoCo-V2. This means that IFND is really able to form better clusters and show improved performances.

Weakness:
1) Results on BYOL and other non-contrastive methods: The method would be much more comprehensive if this method can be extended to other non-contrastive methods such BYOL as well. Recently BYOL and other non-contrastive methods are showing much more promise than contrastive methods. Hence changing the loss function using similar motivation from IFND would be really good to see. 
2) Object-Detection and Semantic Segmentation results on COCO/VOC: I couldn't find any results on object detection or segmentation. Adding those results would make the paper more complete.
3) Another interesting result would be to see the performance of IFND by pre-training on the COCO dataset. ImageNet in a way is already clustered and finding close “false negatives” would be easier on ImageNEt as compared to COCO. Results by pre-training on COCO would make the method much more convincing.

summary_of_the_review: I like the idea used in the paper and analysis shown in the paper. However lack of results on object detection and semantic segmentation coupled with lack of results using BYOL method, I'm leaning towards borderline rejection. Adding these results would make the paper stronger.
review_title: null

Meta-review: While the reviewers were somewhat split on this paper, they all found some strengths, and pointed out some weaknesses. Among these the main seems to be the somewhat incremental nature of the work, in particular with respect to PCL. As the authors point out, the differences w.r.t. PCL are meaningful and include the main thrust of the paper (removal of false negatives), and the results do indicate usefulness of the proposed approach. Given the wide interest in self-supervision I think the paper is above bar for acceptance.
Decision: Accept (Poster)