Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the behavior of contrastive learning (CL) and supervised learning (SL) frameworks in response to changes in data distribution. To achieve this, they develop a variety of data corruption strategies, including patch, pixel, and dataset-level corruption. For experiments, they conduct a comprehensive comparison with a list of recent SSL algorithms. The authors further conclude that CL is more robust to dataset-level corruption than SL, but much less so to pixel- and patch-level corruption. In addition, the authors note that uniformity is the key to the downstream robustness of CL. Overall, this work offers a number of intriguing insights that could inspire future research in contrastive self-supervised learning.
strength_and_weaknesses: Strength
- This paper proposed a systemic framework for evaluating the behaviors of SL and CL frameworks with various data corruption strageies.
- The authors presented a number of metrics for measuring and analyzing feature learning, including feature semantic fluctuation, figure unfiormlmity, and fetuar distance.
- The authors demonstrate that uniformity may be the key to the success of contrastive learning on downstream robustness.  
- Overall this paper is well-organized and easy to follow.

Weakness
- In section 3.3, the authors mention utilizing the same data augmentaions  for all methods. It is somewhat confusing. Did you apply any augmentation from the original CL algorithms?
- Some essential components of CL algorithms, such as batch size and temperature parameters, are not being discussed or studied. For instance, adjusting the batch size or temperature, for instance, could have a significant impact on the feature learning of negative samples for certain algorithms. Will this modification affect the current outcomes?
- It would be interesting to compare qualitatively how different corruption strategies impact the model's ability to learn semantic concepts/objects. Figure C.1 (attention maps) is a good illustration of how SL is superior to CL algorithms in capturing the main object of the image by applying patch shuffling. 
- This paper lacks a comprehensive analysis or discussion of the distinctions between CL algorithms. In section 4.1, for instance, the authors note that CL algorithms perform differently on pixel-level corruption. Besides the numerical difference, are there any additional explanations for why this happened?
- In addition, I would suggest adding an average number of all datasets for each algorithm in Table 2 or Table B.5. This would make it easier for readers to compare the performance of different algorithms across all datasets.
- The authors added a uniformity-promoting loss term to supervise learning algorithm in table 7 of section 5.1. How do the authors define the term uniformity-promoting loss, for instance, the definition in 2 feature uniformity?

clarity,_quality,_novelty_and_reproducibility: Overall, this paper is partially novel in that the authors design a systemic framework for evaluating the behaviors of SL and CL frameworks on different data corruption strategies. However, there is a lack of in-depth analysis/comparison on behaviors of various CL algorithms. I think the proposed method is reproducible.
summary_of_the_review: Please check my comments in Strength And Weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper discusses contrastive learning's robustness to data corruption compared with that of supervised learning. Two types of corruption, pertaining to corruption and downstream corruption are considered. A systematic study on multiple datasets is provided. 
strength_and_weaknesses: Pros:

1. The paper is well-written. I find it easy to follow and understand.

2. The paper does extensive experiments on multiple levels and two types of data corruption. 

3. The author provides a nice interpretation of why CL is more robust to downstream corruption by checking the feature uniformity. And leverage this fact to improve the SL's robustness to downstream corruption. This result is novel and interesting. 

Cons:

1. Unjustified metric: the paper proposes to use relative accuracy drop (RAP) as the robust metric. I am not convinced why the absolute accuracy drop (AAP) is not chosen here. 

2. Artificial Corruption: The author studies patch-level corruption. As the author admits in the paper "Note that patch shuffling is not commonly used in the standard augmentation pipeline". If patch-level corruption is not real and is usually not encountered in real practice, why should we care about the result of patch-level corruption? The author could argue that the study is done for the purpose of understanding CL's behavior. However, I do not see deeper and more intrinsic findings than CL is not robust to patch-level corruption.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and provides novel findings about CL's robustness. 
summary_of_the_review: Overall, I find the paper borderline. On one hand, it provides new findings of CL's robustness to data corruption. However, on the other hand, the paper includes many related studies without well motivating them. For example,  as I mentioned above, why should we study patch shuffling corruption? Why should we use related accuracy drop as the metric?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper design and conduct a series of robustness tests to quantify the behavioral differences between contrastive learning and supervised learning to downstream and pre-training data distribution changes, and conclude that contrastive learning is more robust than supervised learning under downstream corruptions. This paper also discover that diverging robustness behaviors between CL and SL, and even among different CL algorithms. After that, this paper give analyses and explanations, and provide a simple way to improve the downstream robustness of supervised learning.
strength_and_weaknesses: Strength: 

-This paper investigate an interesting problem, that is, how does changes in data distribution affect the robust behaviors of contrastive and supervised learning?

-The design of data corruption types ranging from micro-level(pixel-level, patch-level) to macro-level(dataset-level) can better evaluate pre-training algorithms’ behavior under various distortions.

Weaknesses: 

-This paper has mentioned many aspects of robustness such as: robustness to long-tail or noisy training data, Robustness of the model output, Robustness to input corruptions at test-time. However, in practice this paper only measure accuracy degradation on certain corruption, which seems trivial to some extent. It is better to say, the main phenomenon is CL benefits from some data augmentation more than SL in some cases.

-Contrastive learning methods shows more robustness compared with supervised learning seems to be a global known behavior in transfer learning field, as discussed in [A][B]. Demonstrating this experimental phenomenon is not novel.

-The analysis of contrastive learning from the perspective of uniformity and stable feature space have also been discussed by a widely known paper [C].

-The third contribution that “a simple way to improve the downstream robustness of supervised learning”, which adding an additional loss as contrastive learning to supervised learning is also not novel. For example, the third section of [A] also provide a MoCo-like loss to get a better supervised pretraining.

[A] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In ICLR, 2021.

[B] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In CVPR, 2021.

[C] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.

clarity,_quality,_novelty_and_reproducibility: This paper has clarified clearly. The sufficient experiment with analysis and visualization can fit for the discovered phenomenon.

However, this paper is not much novel for the analysis and discussion between contrastive learning and supervised learning. Details can be seen on above weakness.

summary_of_the_review: This paper has presented some meaningful intuitive experimental results. However, the main motivation that “Contrastive learning methods shows more robustness compared with supervised learning” and the main conclusion “CL’s robustness is related to a more uniform and stable feature space” and improvement of supervised learning by adding a loss seems less novel. From another perspective, some parts of this article are more like a patchwork of some existing theoretical verifications, and lacking their own verifications.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an empirical assessment of variations in behaviour of different representation learning paradigms when the underlying training sample is corrupted with noise. Authors focused on image classification tasks, and verified gap in performances post corruptions. They further evaluated against which kinds of corruptions each of supervised or self-supervised contrastive schemes observes the greater drop in prediction accuracy, and that analysis is carried out both when only classification heads are trained downstream, as well as when  training the complete embedding encoder. Authors further discussed metrics that can be measured from the representations across layers and that seem to correlate with performance gap.
strength_and_weaknesses: Pros:

+The empirical assessment covers a number of datasets and seems to consistently justify the posed conclusions.

+A diverse set of data corruption processes is considered.

Cons:

-Unclear motivation: It's unclear to me after reading through the manuscript what is the motivation for measuring this notion of robustness the authors posed. Most of the corruption approaches considered are unnatural, so we should not expect conclusions to hold similarly once we move to naturally occurring variations in the data. I would recommend including an explicit motivation statement in the text. To be explicit, the question I could not get an answer for from the text is: what do we get from determining whether supervision or self-supervision is less affected by unnatural corruptions?

-Missing related work: Robustness against distribution shifts has been largely studied in recent years and novel work should contextualize itself with respect to prior literature. It seems to me, for instance, that authors specifically consider the covariate shift cases where data marginals change (via perturbations), but class conditional distributions are unchanged. It would be then important to state that in the text, and further mention that conclusions hold for that setting only. There's also been prior work showing self-supervision to observe greater robustness to covariate shift than supervision, such as [1].

-Limited empirical assessment: The empirical evaluation has some limitations that should be mentioned. Conclusions do not control for task difficulty, and it's unclear whether varying levels of distortion impose varying levels of difficulty. I would suggest measuring some notion of distance between every considered dataset and corrupted versions, so one can get a sense of different behaviours against different levels of difficulty. The H-divergence [2] would be a natural candidate metric for such an analysis, but something else could be used as well. Moreover, the evaluation focuses on determining performance gaps, but lacks in explaining those. Authors hypothesize the main reason for differences across supervision and self-supervision are due to the effective dimension of the features, but that should be empirically verified. Finally, I would also suggest considering naturally occurring data perturbations.

-Clarity: I would say the write up could benefit of stating early on what is meant by terms that are usually overloaded in the literature. The main issue in my opinion is the usage of 'robustness'. There are several notions of robustness currently in use, so a definition should appear as early as possible. The same goes for somewhat generic terms such as stability and behaviour.

[1] Albuquerque I, Naik N, Li J, Keskar N, Socher R. Improving out-of-distribution generalization via multi-task self-supervised pretraining. arXiv preprint arXiv:2003.13525. 2020 Mar 30.

[2] Ben-David S, Blitzer J, Crammer K, Kulesza A, Pereira F, Vaughan JW. A theory of learning from different domains. Machine learning. 2010 May;79(1):151-75.
clarity,_quality,_novelty_and_reproducibility: Clarity: As discussed above, the write-up could improve in terms of defining upfront some terms that often overloaded in our community (e.g., robustness).

Quality: The empirical evaluation seems to have been well executed. Nonetheless, concerns remain in terms of the underlying motivations for such an assessment and the limitations in the conclusions themselves. In particular, it's hard to tell how the conclusions change depending on the "amount of shift", and there's no analysis indicating why the observed behaviours manifest as they do; i.e., what is it that causes the different behaviours?  Authors do hypothesize the nature of the training objective is the reason, since contrastive methods tend less to collapse to a subspace of the feature space, but that hypothesis should be verified empirically since there's recent work arguing contrastive losses yield embeddings of incomplete rank [3].

Novelty: The current version of the manuscript is limited in terms of novelty since it's already known that both considered learning schemes suffer from covariate shift, and that self-supervision seems to suffer less. I would say that extensions of the experiments showing why that is would be novel and informative to the overall community.

Reproducibility: The experiments setup are clearly described and should be easy to reproduce from the details in the text.

[3] Jing L, Vincent P, LeCun Y, Tian Y. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348. 2021 Oct 18.
summary_of_the_review: While the paper carefully carried out a somewhat large scale evaluation, I would say the current version still lacks some extra analysis in order for it to be conclusive (detailed above).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors investigate the differences in behavior of learned representations from supervised and contrastive learning. In particular, the authors focus on the robustness of the representations to downstream tasks in the presence of pixel-level, patch-level and dataset-level corruptions. The authors find that representations learned through contrastive learning are more robust to dataset-level corruptions than supervised learning; this robustness however is much less prominent with pixel-level and patch-level corruptions. The authors explore these questions on ResNet-50 and ViT architectures and test on ImageNet, CIFAR-10/100 and a litany of other small image recognition tasks. Additionally, the authors test a range of contrastive learning approaches including MOCO, BYOL, SimCLR, etc.

The reviewers commented positively on the systematic framework for evaluating the behaviors of supervised and contrastive learning, the organization and presentation of the material and numerous experiments conducted by the authors. The reviewers also expressed some concern about the lack of in-depth analysis of behaviors of various critical contrastive learning parameters (e.g. temperature, batch size, etc.); the lack of motivation about some of the corruption studies, particularly pertaining to pretraining, since many of the corruptions appear rather unnatural; questions about the overall implications of the study. With regard to the motivation, there was some discussion between the reviewers and authors but the reviewers were unconvinced particularly in the setting of corruptions applied to data during pretraining. As to the overall implications of the study, the results of this study were unclear. There were previous studies that have already explored how the performance of downstream tasks changes in the context of corruption, thus the novelty for similar results reported in this paper are limited. Likewise, given the concerns about motivation on corruptions applied during pretraining, it was difficult to state the new implications provided by this study. 

Given that none of the reviewers felt that their concerns were addressed, this paper will not be accepted at this conference. From my perspective, I would encourage the authors to revise the motivation for their study, particularly in light of the previous literature, and consider resubmitting to a future venue.

justification_for_why_not_higher_score: Motivation and implications unclear.
justification_for_why_not_lower_score: n/a
Decision: Reject