Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new methodology for noisy-robust learning by augmenting the simlr methodology with a Mixup-style augmentation and noise-cleaning.

The paper proposes a new methodology for training models in the presence of label noise. The method is a combination of standard SimCLR: using standard image augmentations and enforcing a contrastive consistence loss between the emeddings of the weakly augmented and strongly augmented versions of the same image. In addition, to the standard augmentation, the method also uses a Mixup style augmentation for the strongly augmented images: that is the projection of a randomly selected image is combined (using a convex combination) with the original image and then another contrastive loss is enforced for the class prototyle.
The high dimensional embedding of the weakly augmented image is trained with the (noisy) label using a cross-entropy loss.
In addition, the high dimensional embedding is also reconstructed from the low-dimensional projection.

The final evaluation also contains a noise-cleaning step by generating pseudo-labels from the smooth-neighborhoods using the above embeddings.

The quality of the writing is high and the paper presents a plausible combination of several strong methods. The paper is nicely written, well motivated and can be followed easily.

The paper also presents significant improvement of artificially noisy versions of CIFAR-10 and CIFAR-100. Also it is also measured on versions on which the images are noised by using extra datasets (SVHN) to create more interesting augmentations of the datasets.

The weakness of the paper is that it is only measured on cifar-10/100 and these datasest are often not very representative of real noisy datasets with label noise of complicated structure.

Another weakness is that this paper presents a relatively complex approach composed of 4 different methods, each of them are well studied and with very limited ablation analyses. It is plausible that the combined approach should do well, still it has limited novelty as it is the straightforward combination of four known approaches also the paper does not give a disciplined overview and study of the contributions of the different components.

review_title: An interesting combination of Mixup, SimCLR,  denoising autoencoding and noise-cleaning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: #######################################################################

Summary:
 
The paper proposes noise-robust contrastive learning to combat label noise, out-of-distribution input and input corruption simultaneously. In particular, this paper embeds images into low-dimensional representations by training an autoencoder, and regularizes the geometric structure of the representations by contrastive learning. Furthermore, this paper introduces a new noise cleaning method based on the structure of the representations. Training samples with confident pseudo-labels are selected for supervised learning to clean both label noise and out-of-distribution noise. The effectiveness of the proposed method has been evaluated on multiple simulated and real-world noisy datasets. 

#######################################################################

Reasons for score: 
 
Overall, I vote for a weak acceptance. The proposed noise-robust contrastive learning introduces two contrastive losses: unsupervised consistency loss and supervised mixup prototypical loss. My major concern is about the clarity of the paper and some additional issues (see cons below). Hopefully the authors can address my concern in the rebuttal period. 
 
#######################################################################

Pros: 
 
1. The paper takes one import issue in deep learning: learning from noisy data.  
 
2. For me, the proposed supervised mixup prototypical contrastive loss is novel for learning with noisy data. Specifically, it injects structure knowledge of classes into embedding space by combining the mixup technique and prototypical contrastive loss. The design is reasonable and interesting. 
 
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. In particular, the proposed method outperforms several state-of-the-art robust learning methods in learning with label noise, out-of-distribution input and input corruption. 

#######################################################################

Cons: 
  
1. For the motivation, it would be better to provide more details about it, which seems not very clear to me. Particularly, it is unclear why the contrastive loss is duly used in the paper. Will the functionality of the unsupervised contrastive loss be achieved in the supervised prototypical loss? Additionally, the prototypical contrastive loss in equation (1) is an InfoNCE with normalized mean embeddings as the prototypes, which seems different from the ProtoNCE in the original paper [1]. It is better to clarify the the differences of the formulation and training strategy, and the reason of design of supervised prototypical loss in this paper. 

[1] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C.H. Hoi. Prototypical Contrastive Learning of Unsupervised Representations, In ICLR, 2020. 

2. As the key contribution of this paper, mixup prototypical contrastive loss combines mixup technique and prototypical contrastive loss. In the appendix, the authors have provided ablation study to show the effect of proposed losses, and it shows that the mixup prototypical loss is most crucial to the model’s performance. Here, the authors utilizes mixup in two ways: first is to create virtual training samples, and the other is to define the mixup version of prototypical contrastive loss as an weighted combination of two prototypical contrastive loss with respect to true class and virtual class. However, the effect of mixup augmentation, prototypical contrastive loss and mixup prototypical contrastive loss are unclear.  Since mixup has been shown to be an effective method against label noise, it would be more convincing if the authors can study the individual effect of each components of the proposed loss in the rebuttal period. 
 
3. The proposed method uses many data augmentations, e.g., standard crop and horizontal flip as weak augmentation and AugMix as strong augmentation in the unsupervised consistency contrastive loss, and mixup technique in the supervised prototypical contrastive loss. I am concerning about the fairness in the experimental comparison. It is unclear to me if the authors have applied the same data augmentation to all the compared methods. 
 
#######################################################################

Questions during rebuttal period: 
 
Please address and clarify the cons above. 
 

review_title: An extention of contrastive learning for training noise-robust deep networks. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors of the paper propose to use the contrastive loss, the mixup prototypical loss, and a reconstruction loss to regularize the learned representation in order to achieve robustness under various kinds of noise like label noise, out-of-distribution input, and input corruption. A noise-cleaning process based on the learned representation is also introduced to further enhance the results. Extensive experiments were conducted to demonstrate the effectiveness of the method. 

All in all, the paper is well written and easy to understand. The proposed method is well justified and seems to be technically sound. I also find the idea of using neighboring samples to perform noise-cleaning intuitive and interesting. A major complaint I have regarding this submission is the lack of novelty. While the empirical results are encouraging, the proposed method is merely a combination of various previously proposed methods. Moreover, the final objective used (Equation 8) also involves 3 hyper-parameters that can be hard to tune. How sensitive are the results when different weights are used? Why is the $w_{pc}$ changed across different datasets? Due to the limitation addressed above, I think the paper a very borderline submission.

Questions: 
1. It was demonstrated in the appendix that the prototypical loss is the most critical of all the losses used in this paper. However, no ablation study was done with respect to the effect of the mixup on the prototypical loss. How would regular prototypical loss perform without mixup in the context of these experiments?
2. It was stated that the labels for the pseudo-labeled samples were converted into hard labels. Does this lead to better performance than if soft labels are used? 
----------------------------------------------------------------------
After Rebuttal: 

I would like to thank the authors for answering my questions and addressing my concerns on hyperparameters. I also appreciate the authors' efforts in the additional ablation study conducted. While I still do think that the novelty of the paper is a little bit lacking, I think the experiments are carefully conducted and the empirical results seem to be encouraging. As such, I have raised my score to 6.
review_title: Review for "Noise-Robust Contrastive Learning"

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors have conducted a range of experiments to validate the performance of the proposed method. And according to the results, it is appealing that the proposed method achieves relative improvement compared to current SOTAs . However, there are some concerns as follows.

1) It is quite ad-hoc about the proposed method, which consists of multiple losses and two phases in the training progresses. The potential hyperparameter space is relative black-box to understand where is the bottleneck of learning with noisy labels in this method. It also requires careful hyperparameter-setting as shown in the implementation details.

2) Some contributions are over-claimed since many components have also used in previous works for learning with noisy labels, e.g., the idea of consistency loss and autoencoder in Bootstrapping [1].  Besides, the title of the paper is also misleading since it is about learning with noisy labels instead of contrastive learning.


[1] S. Reed et al. Training deep neural networks on noisy labels with bootstrapping. 
review_title: This paper introduces a noise-robust contrastive learning loss. The authors first learn robust representation to regularize the learning procedure with label noise. Then, a noise cleaning process is implemented to select some confident samples for partially finetuning. 

Meta-review: This paper proposes a framework to train a discriminative model robust against (i) label noise, (ii) out-of-distribution input, and (iii) input corruption. To tackle these problems, a complex model is proposed that combines several existing models including InfoNCE-style contrastive learning, prototypical contrastive loss, Mixup, and reconstruction loss. Noisy training labels are cleaned using a temporally consistent label smoothing mechanism, combined with a curriculum learning algorithm. 

Originally, the reviewers raised concerns regarding the limited ablation experiments and the lack of studies on real-world noisy labels. The additional experiments in the revised version addressed some of these concerns. Thus, the reviewers increased their rating slightly.

However, the reviewers in the discussion phase agree that the proposed method has a limited novelty, is complex, and involves many moving parts that require a careful design and hyperparameter tuning, and they do not recommend accepting the submission. I agree with the reviewers and recommend rejection. 
Decision: Reject