Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors explore the robustness of feature learning via self-supervised learning (SSL) and supervised learning (SL) with imbalance datasets.  Generally, SL can learn better features than SSL, and features are better from balanced than from imbalance datasets.  However, with imbalance data, SSL is more robust than SL in terms of the performance difference of features learned from balance and imbalance datasets.  That is, the performance does not drop as much for SSL with imbalance data from balanced data.   The robustness is observed from both in-domain (ID)  and out-of-domain (OOD) tasks (different downstream tasks).  They hypothesized that "SSL learns richer features from frequent data that are transferable to rare data."

Using a synthetic dataset with 2 frequent classes and 1 rare class, they observe that SL learns one feature to distinguish the two frequent classes and one feature to overfit the rare class.  However, SSL learns two features that can classify the 3 classes well. That is, SSL can learn label-irrelevant-but-transferable features from the frequent classes which can help classify the rare class.
 
To illustrate the observation from a synthetic dataset also exists in the real world, they construct a semi-synthetic dataset.  From CIFAR 10, they generate 5 fequent classes and 5 rare classes.  For the frequent classes, the right half of the image is replaced by a random half image, ie not relevant to the label.  For the rare classes, the left half is replaced by blank.  Using activation maps, they observe that SL learn features from only the left half (relevant to the label) but not from the right half (irrelevant to the label).   However, SSL learns features from both halves.

To further improve robustness, they adapt SAM to the imbalance scenario: rwSAM. SAM improves model generalization by penalizing loss sharpness. For SAM, the loss is uniformly low in the neighboring area.   To have a flatter region for rare examples, rwSAM reweights rare examples in the inner maximization step of SAM.  Since labels are not available, they estimate kernel density on feature vectors and the weight is inversely proportional to the density.  On two datasets, empirical results indicate rwSAM improves performance over SAM.


main_review: Strengths:

Empirical analysis, with theoretical justification, shows that SSL is more roboust than SL in feature learning from imbalance datasets.

Using a semi-synthetic dataset, they show that SSL learns label-irrelevant features that are useful for classifying rare classes. 

Adapting SAM by reweighting more on low density instances improves performance.

Weaknesses:

Theorem 1 could have more explanation.

Table 1:  Gap Freq and Gap Rare were not described

Table 2;  Which of the four classes are frequent (or rare)?
summary_of_the_review: The authors propose well-designed experiments to show SSL is more robust than SL in feature learning from imbalance data and SSL learns label-irrelevant features that could be useful for the rare classes.   The presentation can be improved as indicated above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the differences between representations pretrained with Self-Supervised Learning (SSL) and the standard Supervised Learning (SL) frameworks in the In-Domain (ID) and Out-Of-Domain (OOD) settings of the image classification task with an imbalanced dataset. 

The paper presents different results that are relevant to the computer vision and machine learning community: 
1) In the ID setting, representations pretrained with SL outperform those pretrained with SSL. However, in the OOD setting, representations pretrained with SSL outperform those pretrained with SL and are even robust to the imbalance factor.
2) An analysis in Section 3.1 on a toy dataset and semi-synthetic data (by cropping and fusing half images during pretraining) in Section 3.2 validate that SSL methods learn representations that do not overfit to labels and show improved accuracy at test time.
3) The paper also proposes a regularization framework (called rwSAM) that promotes flatter landscape for rare examples to improve performance at test time (see Table 1 and 2).
main_review: The paper is well written in general and its motivation is clear. 

Weaknesses: 

- The only task considered in the paper is image classification using the standard accuracy as the evaluation metric. Is the observed robustness also valid for other tasks? Although the paper presents a large range of experiments, the conclusions are less general than what is sold. The imbalanced datasets used in the paper are CIFAR-10-LT and ImageNet-LT, which are versions of CIFAR-10 and ImageNet that are originally balanced. Datasets that follow an intrinsic long-tail distribution (e.g., iNaturalist 2018) are not considered. Although the superior results of SSL in imbalanced settings are promising (e.g. the apparent robustness in Figure 2), it would be interesting to know if the conclusions in the paper generalize to datasets that are originally imbalanced (e.g., iNaturalist 2018).

- The improvement of the proposed regularization in Tables 1 and 2 does not seem statistically significant.

- The toy dataset setup in Section 3.1 is difficult to read.


Despite the weaknesses mentioned above, I think that the paper is worth publishing. It considers a problem that is relevant to the machine learning and computer vision communities. It shows that, in the OOD case or very imbalanced case, it is better to use SSL techniques (e.g. SimCLR or MoCo v2) since they do not overfit to the training samples and labels, and the imbalanced factor does not matter. 
The paper gives a sensible explanation of why this is the case, with detailed ablation studies. 
Although Table 2 uses diverse datasets such as CUB, Cars, Aircrafts and Pets, the improvement does not seem significant. It is then difficult to know whether or not the conclusions of the paper generalize to datasets other than CIFAR-10 and ImageNet.






minor typo in titles of Figure 1: "Imabalanced" => "Imbalanced"
summary_of_the_review: The paper considers the imbalanced dataset problem that is relevant to the machine learning and computer vision communities. The proposed analysis also makes sense and is interesting to the ML and CV communities.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on class-imbalanced learning and shows that self-supervised pre-training yields representations that are more robust to dataset imbalance.

main_review: This paper focuses on class-imbalanced learning and shows that self-supervised pre-training yields representations that are more robust to dataset imbalance.

Strength:
1) This paper is well-written and the conclusion is clear.

Weakness:
1) The conclusions are mainly observed from empirical results. There lack of theoretical insights. 
2) The generalization of the conclusions is not clear. Given an imbalanced dataset, how to decide which algorithm should I adopt? For example, on what conditions self-supervised learning is better than supervised learning, and on what conditions are not?
3) Some class-imbalanced learning methods are not compared. In section 4.1, the authors report the performance of the proposed rwSAM on imbalanced datasets. More related algorithms should be compared to show the effectiveness of the proposal.

summary_of_the_review: This paper focuses on class-imbalanced learning and shows that self-supervised pre-training yields representations that are more robust to dataset imbalance. The problem is interesting. But the results are mainly observed from empirical results, the generalization of the conclusions is not clear. The novelty and contribution of the paper are limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies and compares the performance of self-supervised representation learning methods against supervised representation learning when there are class imbalance. It shows empirically that self-supervised methods are more robust to class imbalance, i.e. the performance gap between models trained on balanced and imbalanced datasets is smaller. The paper also investigates the quality of the features learnt by SSL vs SL, and shows both theoretically in a limited setting and empirically on synthetic datasets that features learnt by SSL are diverse and capture characteristics that might be useful for rare classes. Finally, they provide a regularisation method for SSL to encourage model to learn better features for rare classes without having access to labels.
main_review: I enjoyed the paper. It reads well and has a robust set of experiments to investigate the effects of class imbalance on representation learning. Some suggestions and questions on how to improve the paper:

- The effect of class difficulty: while random sampling for rare classes is a good strategy to take, I wonder how the class difficulty may play a role. This may become important as usually rare classes are also harder classes (even for annotators). So, an experimental setup to test this would be to use the class performance of a model on a balanced dataset to deliberately more frequently choose challenging classes as rare.
- reSAM computational complexity: because of pairwise kernel within the regularisation term, I think this methods can't scale to very large datasets. So, what are your ideas on how to improve this? I thought of in-batch sampling, but that might be ineffective as it's possible samples in a batch are already far away from each other.

Minor comments:
- Abstract - first sentence: While the paper focuses on vision, SSL is not a vision specific. For example, NLP domain has been enjoying SSL since ELMo, BERT, ...
- NLP applications: following from above, would be interesting to study the effect of dataset balance on NLP applications where SSL is a dominant pre-training approach.
summary_of_the_review: I think this is an important paper with experiments and limited theory to show and explain why SSL methods are more robust to class imbalance problem which also occur frequently in practice.
review_title: null

Meta-review: This paper is proposed to investigate the robustness of self-supervised learning (SSL) and supervised learning (SL) in both balanced (in domain) and imbalanced (out of domain) settings. It can be concluded that SL can regularly learn better representations than SSL, and representations are better from balanced than from imbalanced datasets. The SSL is more robust than SL in the imbalanced settings, which is the crucial of this paper. Expect the experimental results, the authors also provided theoretical analysis to support their claims. The authors also extend a well-established method SAM into the Reweighted SAM as the technical contribution to better address the imbalanced setting. The paper is well written with clear logic to follow.
Decision: Accept (Spotlight)