Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: To train a model with a noisy weakly supervised training set, this paper proposed a momentum prototypes method for label noise correction and OOD sample removal. Noise correction is done by a heuristic rule, that if the prediction is confident enough or the prediction on original label is higher than uniform probability, the label will be kept otherwise it is considered as OOD sample. For training the model, this paper jointly optimizes cross entropy loss on the corrected labels, as well as contrastive loss using prototypical examples and instances. 

Extensive experiments are performed on diverse tasks, including classification, transfer, robustness, detection and segmentation. 

My main concern about this paper is on the results. MoPro is a method proposed to handle the noisy training labels, while only the cross entropy baselines are compared. When I checked a few publications, and found that the reported results are significantly lower than the existing methods. For example, learning from noisy labels paper DivideMix [1] reports 77.32% top-1 accuracy  on WebVision and 75.20% top-1 accuracy on ImageNet. Based on the results, it's unclear to me why the proposed method is superior to the existing methods that learn from noisy labels. 
[1] DivideMix: Learning with Noisy Labels as Semi-supervised Learning.

In Table 3, comparison to self-supervised learning is OK but not very meaningful. It would be nice to include other weakly-supervised learning results [Mahajan 2018, Kolesnikov 2020] for fair comparison purposes.

=====================

Post Rebuttal: I would like to thank the authors for the new results on WebVision-mini and ImageNet-mini, this has partially addressed my concerns as Reviewer3 raised similar issues on the SoTA claim. Overall, I think this paper is well presented and the results are solid, thus updated my rating to reflect this.
review_title: A simple and clean method with weak experimental results

final_score: rating: 7: Good paper, accept
review_content: Caveat: I am not very familiar with the weak supervision literature.

Pros:
+ The method is fairly simple, a combination of three loss terms which have all been explored pretty comprehensively. The fact that a simple combination gives such solid performance improvements is a novel contribution.
+ The results are convincing and significant. I especially appreciate the low-shot transfer results: I think they are the true test of a trained feature representation.
+ The ablation and robustness studies are appreciated.

Cons:
- While experimentally the paper is on solid ground, there isn't much intuition presented as to preccisely why this combination of losses is the right thing. For example, the technique proposed in section 3.3 can also be seen as trying to use the consensus between two classifiers, one based on prototypes and the other parametric. Is this the right intuition? If so, does one need to use a prototype based classifier? Or would a nearest neighbor work fine?
- This paper inherits some of the mysteries of self supervised techniques: what function does the projector provide? what other architectural choices are important?
- I am not that familiar with the WebVision dataset, but this dataset has been collected by using Google Image Search ccirca 2017: Google might have been using internal convnets to rank images by that point, in which case it is possible that all we are doing here is replicating/slightly improving their internal network. A discussion of this would be good.


In general, I think this is a good paper, but would still like to see the authors' response to the points above.
review_title: Simple method with solid results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a weakly-supervised learning method based on the self-supervised contrastive loss to combine supervised learning and unsupervised contrastive learning. A label correction mechanism is proposed by utilizing distance in class prototypes. The overall combination is reasonable and convincing.

Prons: 

- As far as I know, this is the first work that combines weakly supervised learning and self-supervised learning.
- The performance on Webvision dataset is nice
- Writing it high-quality and easy to read.

Cons:

- Novelty: This method is greatly relying on an previous method, prototypical contrastive learning (PCL).  The key technical contribution is Eq (4) and (5), however, they are relatively naive and straightforward design choices. 
- Experiments are a bit distracting and insufficient. I am not convinced by the larger amount of experiment focusing on tasks not actually related to weakly supervised learning (including detection, instance segmentation). 
- Since it is a weakly-supervised learning method, only one table evaluates this task.
- Compared with unsupervised learning methods on downstreaming tasks are not that meaningful from my view, because the method accesses weak labels. Moreover, the noisy ratio of webvision dataset isn't that high, which can be inferred by the results in Table 6 (w/o L_pro & L_inst). 
- What is the motivation of validating the methods on so many downstream tasks?
- The method replies on a temperature T, which is tuned based on datasets. The paper uses different values for webvision versions. Can it be generalized in practice? 
- More weak labeled datasets more meaningful, such as Clothing1M, Food101, and real world datasets presented by this paper "Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels" (ICML2020)
- Distilling Effective Supervision from Severe Label Noise (CVPR2020) shows better Webvision results. So the SoTA claim in the abstract might not be less accurate. Proper discussions of related papers are encouraged.

review_title: Anonymous review

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to modify the recent proposed prototypical contrastive learning work (which is in turn based on improved momentum contrastive learning work) and apply it to noisy web images with detailed insights on how to combat noise. It is shown that the representation learned in this way not only achieves good results on classifying web images, but also shows great transfer performance.

+ I think it is a nice application of the recent unsupervised learning methods to settings that are more realistic (it is not practical to assume that there is absolutely no chance of getting a single label, but instead some label with noise). The proposed approach is reasonable, the motivation is strong, and the solution is (quite) simple and effective.
+ Quite a nice set of experiments are done, expanding not only the within the webvision benchmark, but also to other down-stream tasks like COCO object detection.
+ I like the robustness study showing that noisy images are good to train representations that generalize beyond normal ImageNet, but to other corrupted versions.

- In Table 1, it mentions that CE is already achieving really good results and beats most of the state-of-the-art. What might be the reason? Is it possible that stronger data augmentations are used? I am asking because most of the contrastive method these days are NOT using the same set of basic augmentations that ImageNet supervised methods are using. This makes the comparison not too fair to other methods. If possible, please consider report experiments that follow the same augmentation protocol of other compared methods.
- I noticed that different augmentations are used for the base encoder and the momentum encoder (Sec 4.2), why this is the case?
- Missing reference: (the first work that explores representation learning with web images) Chen, Xinlei, and Abhinav Gupta. "Webly supervised learning of convolutional networks." Proceedings of the IEEE International Conference on Computer Vision. 2015.  
(the work that invented the word "webly"): Divvala, Santosh K., Ali Farhadi, and Carlos Guestrin. "Learning everything about anything: Webly-supervised visual concept learning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014.
Both work actually explored some mechanism/design to de-noise web images.

review_title: Application of recent contrastive learning methods to web images and shows improvements

Meta-review: This paper provides an approach for weakly supervised learning by label noise correction and OOD sample removal. Overall, all reviewers agree paper is simple and approach makes sense. The experiments are solid with results on Webvision and ImageNet Mini (there were initial concerns but rebuttal handled some of those concerns). AC agrees with reviewers and recommends acceptance.

Decision: Accept (Poster)