Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the class-imbalance problem. In many practical cases, the data set has minority classes that have few samples. To understand the effect of highly imbalanced data sets, the authors perform extensive empirical studies for various ML models, including neural networks, gradient-boosted decision trees, and VMSs. The observations are as follows. Re-balancing and oversampling for the minor samples are not effective. They even degrade performance. Self-supervised pre-training is insensitive to imbalance. Bayesian is effective. Flatness-seeking regularization can provide more margin to the boundary of the minorities. 

strength_and_weaknesses: Strengths

- From experiments, the authors find interesting observations for class-imbalance cases

Weakness

- Since this paper considers imbalanced data sets, it is not appropriate to use accuracy as the unique performance metric. Other metrics like Precision, recall, bias, f1, CSI, ROC, should be considered.

- The authors should discuss and implement recent works addressing the class-imbalance problem. 

- Messages from this paper should be compared with recent works. For instance, one of the claims of this work is "minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization." However, there are some works that improve the performance by fitting the minority. 

clarity,_quality,_novelty_and_reproducibility: The novelty of this paper is not significant. 

The source code is not provided.

summary_of_the_review: This paper provides many experimental results and messages from the experiments for the class-imbalance problem. However, this work does not compare with recent algorithms and the performance metric is not proper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the learning of classifiers from class-imbalanced data. In the first part of the paper, the authors empirically investigate the effect of imbalanced classes on the performance of classification models. They find that i) oversampling minority class is ineffective, and that ii) adding too many examples of the majority class can be harmful as well. In the second part, they show that three existing methods can improve the performance of deep models in a class-imbalance scenario, namely i) using self-supervised pretraining followed by a supervised fine-tuning, ii) using Bayesian inference, and iii) using sharpness-aware minimization of the loss.
strength_and_weaknesses: Strength:

The paper addresses a topic of both practically relevant and theoretical interest. 

The authors identify three existing methods that show promising results when applied in class-imbalanced setting. 

Weaknesses:
 
The authors make a relatively strong conclusion about previously used methods, e.g., that the rebalancing of training data is generally not helpful. The authors' conclusions regarding representation learning are made on the basis of experiments, which are reasonable; however, they use a single NN architecture trained on a single dataset. For me, it is not clear that the observations would apply to different data distributions. A similar problem, i.e. using a single dataset, appears when evaluating the three proposed methods.

Quantities used to characterize data distribution and performance have flows that limit conclusions made and should be at least discussed. First, the class-imbalance ratio used as a way to characterize the entire class distribution completely neglects the intermediate classes, which certainly have a large impact on the results. Second, the classification error is not an appropriate metric in the cases where the test data are severally class-imbalanced, as the performance on the minority class has a negligible influence on the metric. 

For some unknown reason, the three proposed techniques are not compared against each other in terms of prediction performance nor in terms of computational demands.

clarity,_quality,_novelty_and_reproducibility: Most of the parts of the paper are clearly written. Only a description of some experiments lacks details. In particular, in the case of the experiment in sec 4.1., it is unclear what it is "the best test ratio". It is also not clear if the number of examples is fixed when varying the class-imbalance ratio. In Figure 6, "SSL Train Data" and "SSL Linear Evaluation" probably refer to the proposed methods, but the names are hard to match with the methods in the text.

Evaluation of benefits of Bayesian learning/inference and Sharpness-Aware Minimization for representation learning in a class-imbalanced setting is novel. The first method, SSL pretraining followed by fine-tuning on the same class-imbalanced dataset, seems novel but incremental change. 

For perfect reproducibility, the authors need to provide an exact description of the test protocol, e.g. what the number of the training and test examples was, or how the distribution in the intermediate classes was varied. 
summary_of_the_review: The paper comes with interesting observations regarding the usefulness of existing methods in a class-imbalanced setting. The main deficiency of this purely empirical paper is the limited number of datasets, which makes the conclusions rather speculative.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies representation learning under data imbalance. In particular, it performs extensive empirical investigation on the learning behaviors for different sets of models, and observe that  re-balancing class-imbalanced data is in general ineffective. Along this line, it studies self-supervised pre-training, Bayesian inference, and  flatness-seeking regularization for data imbalance, and verifies the advantages of these methods in the presence of imbalanced data across a variety of domains.
strength_and_weaknesses: # Strength 

+ The topic is interesting and timely. Data imbalance and few-shot learning is important and practical in real-world datasets, and in-depth analysis is plausible for understanding the behavior of representation learned under such data bias.

+ The writing and overall structure is good, and the paper is easy to follow.
---
# Weaknesses

Unfortunately, there are several major weaknesses that exist in the current paper.

### Unclear Contributions
The paper focuses on representation learning under class imbalance. However, many topics in this paper have already been extensively studied in the literature. I had a hard time figuring out the real contributions of this paper. Can you explain what's new in the paper, and position your work w.r.t the literature?

**Specifically**, the following "findings" in this paper have been well studied / known in the literature.
- _"re-balancing training data actually harms accuracy under imbalanced testing"_
1. This argument is actually trivial in its form. Without any label shift between training and testing, ERM should be the optimal solution [1]. This phenomenon actually motivates the logit adjustment method, where you only need to adjust the logit to compensate the training / testing label shift for a optimal classifier [1]. Going back to this paper, if the training / testing label distribution is the same, no matter how imbalanced they can be, you simply do not need to tune anything, as it is simply a in-domain generalization problem without any shifts.
2. Even under the above setup, the implicit assumption we have is $P_{train}(x|y) = P_{test}(x|y)$, where the only bias comes from the label shift between training and testing. Elaborating this assumption could potentially be a more interesting direction, which is however totally missing in the paper.
3. Perhaps the real interesting question here, is what if the test set is not balanced, but arbitrarily imbalanced (e.g., could be inversely long-tailed compared to the training label distribution). The observation here will easily break when extending to arbitrary testing label shifts. Such scenario is, however, also covered in the literature [2].

- _"oversampling which fit these samples do not improve generalization"_

1. Again, this is not even a proper takeway -- tons of papers in this field have already studied this phenomenon, which is why methods like logit adjustment, ensemble learning, self-supervised learning, contrastive learning, semi-supervised learning (with more unlabeled data) have been introduced. This observation is **out-dated** and do not provide anything new to the research field.

- _"A two-step process of (a) SSL pre-training on the imbalanced training set and ... accuracy across imbalance ratios."_

1. This takeaway is what has been exactly done in the paper two years ago [3]. Strictly speaking, nothing new is presented here, despite that less analysis is conducted in this paper. I suggest the authors doing a comprehensive literature review and correcly position the paper to the literature, and what new contribution/observation is made.

- _"Under class imbalance, models are particularly underspecified by the data"_

1. Similarly, past works (e.g., [4]) have studied this phenomenon. There are even new methods that involve uncertainty modeling for combating underspecification proposed [4] in imbalanced learning.

### Limited Experiments

- One big drawback in the experiments is that only small datasets are evaluated. CIFAR-10 has been standard but also too small for the data imbalance / long-tailed recognition field. The fields have advanced to larger and more practical datasets with higher resolution, such as ImageNet-LT or iNaturalist. Without validation on these large-scale datasets, the observations may not even be justifiable or convincing.

- No ablation studies across datasets / network architectures / optimization methods performed, which again makes the observation less convincing. If the author wanted to do an in-depth analysis, the first thing they need to make sure is that the observations **really persist** across a range of datasets / network architectures / etc.

- Despite that the title is "Representation learning ...", no real analysis on the learned feature space under class imbalance is performed. How exactly is the feature or representation balanced / imbalanced? More analysis and experiments are needed.

### Writing and typos
- page 4, "Takeaway" - the sentence is grammarly incorrect. You might want to change it to something as "Training with a train set exhibiting class imbalance similar to that of"

# References
[1] Long-tail learning via logit adjustment. ICLR 2021.

[2] Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition. 2021.

[3] Rethinking the Value of Labels for Improving Class-Imbalanced Learning. NeurIPS 2020.

[4] Striking the Right Balance With Uncertainty. CVPR 2020.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
Good.

Overall the writing is good. The structure is fine. Several typos exist.

## Quality
Fair.

Figures and tables are well presented. Several interesting visualizations.

## Novelty
Not good.

As I mentioned in the weakness part, almost all of the main observations in this paper have been extensively studied in the literature. I had a hard time figuring out what are the real contributions in the paper. Seems nothing new is really introduced here.

See the above weakness part for details.

## Reproducibility
N/A

No code is provided, though it seems straightforward to reproduce the results here, as many of them have been validated in the literature.
summary_of_the_review: The topic of this paper is interesting. However, as elaborated in the weakness part, none of the observations in the paper is actually new. The contribution is unclear and the novelty is low.

Currently there are several major drawbacks and weaknesses, in terms of contribution and exepriments. I view this work as a half-baked work and needs significant improvements.

In summary, I recommend rejection for the current paper. The paper needs to be significantly improved to meet the bar of top conferences like ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors explore and evaluate several techniques for class imbalance.   Instead of assuming the test set is balanced, they explore imbalance test ratio.   Based one one image dataset with NN and two tabular datasets with SVM and XGBoost, they made the following observations.

1.   training ratio similar to test ratio is desirable
2.   oversampling minority samples does not improve generalization
3.   self-supervised learning for pre-training before supervised learning improves accuracy
4.   using Bayesian inference to incorporate uncertainty increases accuracy


strength_and_weaknesses: Strengths:

1.  Instead of assuming the test set is balanced, they explore imbalance test ratio. 

2.  the paper is generally well written

Weaknesses:

1.  More datasets would be more convincing--one image dataset and two tabular datasets seem insufficient

2.  accuracy/error is one measurement of performance, however, for imbalanced datasets, other performance measurement such as F1 and AUC of ROC could be used.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written.  However, the contribution is low due to a small number of datasets and only one performance metric.    Reproducibility seems to be ok.
summary_of_the_review: The claims could be more convincing if more datasets and performance metrics (for imbalanced data) are used.  Currently, the claims seems to be stronger than the evidence can support.   The paper title stated representation learning, but some of the claims are not related to representation learning.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors consider learning under class imbalance, challenging the validity of widely used techniques via empirical validation and proposing solutions to address problems encountered. Based on experiments with the CIFAR dataset for neural networks and and {Adult, Forest Cover} tabular datasets with SVM and XGBoost, they make the observations itemized below:
- It is desirable to have the training label distribution be similar to the test label distribution (i.e., under/over sampling can lead to poor generalization performance under the assumptions in this paper)
- Minority samples are difficult to fit, but oversampling minority samples does not necessarily improve generalization on test data.
- Self-supervised learning for pre-training on the imbalanced dataset followed by supervised fine-tuning improves accuracy consistently across different imbalance ratios.
- Bayesian inference consistently improves performance on imbalanced data by mitigating uncertainty within the estimator. 
- Sharpness-aware minimization appears to improve performance by increasing the decision boundary margin relative to the minority examples.

The consensus strengths of this work include:
- Class imbalance is a widely-studied problem with significant practical implications in applied settings.
- The authors do make several interesting observations and provide methods to mitigate observed shortcomings; in particular, exploration of the train/test-imbalance ratios are interesting and seem a direction for needed further understanding.
- The paper is well-written and easy to understand.

The notable weaknesses of this work include:
- The exact contribution isn't clear; the 'controversial claims' with respect to existing methods aren't precisely stated enough (i.e., in mathematical form) to effectively evaluate. The reviewers pointed out several existing methods that deal with some interpretations of the the sampling-centric findings, which need to be addressed. Additionally from an empirical perspective, (1) claims that are counter to accepted understanding require evidence at the level of a systematic review and/or meta analysis or large-scale new experiments whereas this work has a limited number of datasets & experiments and (2) using accuracy as the metric for demonstrating this is clearly insufficient. While this second point was partially addressed during the rebuttal period, there wasn't sufficient interpretation/analysis to convincingly support these statements.
- There also isn't any theoretical analysis to support these claims (which would be needed given the lack of large-scale experiments). Additionally, there is a general absence of mathematical rigor to precisely understand what methodological shortcomings are being addressed in each case (as the improvements may not be due to the stated reasons). This could also be supported with ablation studies to some degree.
- Regarding the SSL, Bayesian, and SAM findings, there are existing works that require additional discussion and in some cases, direct comparison. The reviewers felts overall that there wasn't sufficient contextualization and contrastive evaluation wrt existing relevant work.

Overall, the consensus was that there are some potentially interesting findings. However, the work needs more contextualization/comparison wrt recent existing work, more experiments to adequately support the paper's claims, and more general rigor with respect to the core claims. Some interesting directions, but just needs more work to be a strong contribution to this subfield.

justification_for_why_not_higher_score: There was a general consensus that the findings of this paper are interesting, but there isn't sufficient analytical nor empirical support to entirely accept the claims or sufficiently evaluate the proposed solutions. Additionally more work is needed to make the paper more rigorous and contextualize with respect to recent work that also addresses some of these issues.
Decision: Reject