Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper performs a theoretical analysis on one kind of noisy label learning method. The paper also presents an empirical analysis.

main_review: Strength: (1) Theoretical analysis on one kind of noisy-label learning method.
Weakness: (1) The empirical analysis could be improved (see below).
summary_of_the_review: The experiment needs to be improved. The paper performs empirical analysis only on one setup. It could make the paper better if the authors could perform analysis on other settings (e.g., asymmetric noise, etc).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how self-supervised pre-training impacts the resistance of the neural network to noisy labels. The paper provides both theoretical analyses and numerical experiments of their study.  The main contribution of the provided study is (i) given a quality encoder pre-trained from SSL, a simple linear layer trained by the cross-entropy loss is theoretically robust to symmetric label noise (ii) providing insights for how knowledge distilled from SSL features can alleviate the over-fitting problem.
main_review: 
Pros: 
Providing a theoretical understanding of how self-supervised pre-trained features help to improve network resistance against noisy labels.
The paper conducted experiments that supported their theoretical claims

Cons:
The experiments were conducted under limited noisy ratio levels from 20 to 60 percent. No extreme noise levels were checked, i.e., >=80%. In addition, only symmetric synthetic noise scenario is consider, while more complicated asymmetric and real-life noise cases are not considered. 

summary_of_the_review: This paper aims to provide a theoretical understanding of why self-supervised pre-trained features help neural networks improve resistance under noisy labels. While the theoretical claims look solid, the important, more general settings outside of symmetric synthetic noise are missing (please see "Cons"). 
Hence, in the pre-rebuttal phase, my score is 5 and it can be re-evaluated once the authors will address my concerns 

Post rebuttal decision:
I am grateful to the authors for their responses to the reviewers' concerns during the rebuttal phase. Nevertheless, the paper presents an important study the experimental setup is not comprehensive enough to support the paper's concussions, Hence, I vote to reject the paper in its current form and maintain my current score, I encourage the authors to expand the experimental study by my and other reviews comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors illustrate why self-supervised learning can help learning in training with noisy labels.
The main contributions are:
1. The authors illustrate theoretically why learning good representation can help learning with noisy labels
2. The authors describe why fixed encoders are important.
3. The authors motivate a regularizer between the SL and SSL.
main_review: Strength:
1. Using SSL representation can help learning with noisy labels is an interesting discovery. Using theirs theories, the authors also motivate why fixed encoders are important
2. The authors motivate a regularizer from knowledge distillation to further improve the training performance.

Weakness:

Theoretically, (1-2); Experimentally (3-6); Clarity(7)
1. The novelty of the theorem 1 and 2 is limited. It is trivially true that when you have majority labels that are not noisy, you can learn an optimal classifier. 
2. The authors need to explain or cite when SSL give you representations that satisfy assumption 1.
3. It's not clear whether the authors use a validation set. Validation set is important for model selection in machine learning. The authors seem to choose the best test performance directly.
4. The motivation of downsampling is unclear. The authors mentions that downsampling can help satisfy the condition in Theorem 1. However, with downsampling, the target distribution may change and the Bayes classifier may be different. This may change the conclusion in theorem 1. Does downsampling also help for the baseline CORES?
5. Though the authors claim that they do not aim to get SOTA results, more datasets should be introduced. At least, I would like to see the results on cifar 100.
6. Somehow the authors mention the regularizer in section 4. but do not use it in Table 1. Why is that?
7. The paper can be re-organized. The authors describe the use of SSL first and then describe some experiments. After that, the authors describe the regularizer and some other experiments. They look like two separate parts. I suggest the authors put all the experiments and discussions in one section. 
summary_of_the_review: The authors propose an interesting idea that SSL can help learning with noisy-labels but due to the theoretical, experimental and clarity concerns above, I think the paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the usefulness of self-supervised features when encountering data with noisy labels. It presents an array of ideas on this topic: including the question of fine-tuning pre-trained representations, using ideas from distillation as regularisation to improve generalisation. Some theoretical and empirical results are relating to the authors arguments.  
main_review: Strengths:
- Understanding how self-supervised learning can be used, and particular with regards to the impact of fine tuning representations is an important question, as is dealing with label noise.
- The paper has quite a large scope of topics that it tries to connect.

Weaknesses:
- I find the paper somewhat fragmented, jumping between different ideas and topics without going too deep enough into any single one. It's not really clear to me that section 4 is related to section 2/3, they seem to studying quite distinct subjects?
- I find the lack of analysis regarding the specific type of SSL algorithm you use, and the effects that that may have, a weakness. The authors sidestep this by assumptions 1 and 4, but as seen in Figure 2c) on standard small vision datasets these assumptions do not seem to hold (never mind imagenet). What effect does using different SSL methods have on the results, even empirically?
- Many of the results do not seem particularly surprisingly to me and I'm not sure if the application to noisy labels is sufficiently novel. For example, it seems intuitive that one should not fine tune a pretrained NN if the fine tuning dataset is small, i.e. that there isn't much 'signal'. Likewise, it seems intuitive that one should do the same when faced with noisy data.
- It is not clear to me how you train h, and with what objective, in eq 3? If h is the regulariser surely it cant be trained towards g? Please can you explain this to me.
- I think the setting for fig 3 is a bit strange. Surely you should not stop training at the best test accuracy but rather for a fixed number of epochs to show the 'damage' of training with different noise levels for a fixed amount of training. It seems like for higher noise rates that the you are taking very few epochs by stopping based on test accuracy. The results would look more in your favour too I imagine and reflect well on fixed encoder, given fig 3d.

Minor comment:
- Is X_+ for label 1 or 0, it is defined as label 1 but in Theorem 3 it is 0?
summary_of_the_review: I do not recommend the paper for acceptance currently as I do not think it presents a coherent picture (of a very interesting topic) and also have some confusions about the presentation of the paper.
review_title: null

Meta-review: This paper presents an analysis of the robustness of self-supervised learning (SSL) features to noisy labels in downstream supervised learning, and provides empirical verification of the results (mostly in the symmetric noise setup); a SSL regularization scheme is also analyzed (section 4). While the paper contains plausible insights, the reviews share similar concerns that the analysis is mainly based on the noise being symmetric, and that the SSL features already have good class separation and Gaussian clusters, which are strong assumptions. Given that the assumptions are not theoretically verified, and that there is not sufficient empirical results in heavy non-symmetric noise scenario on large benchmark datasets, the reviewers think the paper does not provide practical guidance for noise label learning in its current form.
Decision: Reject